Hey, HearingandStudying,
when I was in Special Ed., it was as you describe. I didn't mind so much the idea of being away from the other kids, because I was attacked physically and verbally badly and often either wasn't let into other kids' playground games - or, I didn't understand the process or physical/spatial/mobility-related things in the game and messed up and was a target anyway. As a child and even into adolescence I was socially awkward/behind, cried easily and had a hard time putting things into words when I was upset. So - the part about being in a different setting temporarily, wasn't so difficult itself <though I had mixed feelings about speech therapy> but the "comings and goings" part was bad because Special Ed. was a separate building, with its own teachers, own lunch, buses, playground, doorways, etc. It even -smelled- different. Kids in there drooled and wore helmets. Some "mysterious person" <from a kid's perspective> would come each day to get me and take me down to that section. One time the speech path. suggested I try to come down by myself and said I should watch the clock and when the time came, just to leave the classroom . I was ashamed and embarrassed to admit I had no concept of anything on that old-style classroom clock. I didn't learn to use a clock with hands til high school and I still have problemsd with them. I have never worn a watch.
So the speech lady came to get me on the day I was supposed to go myself and asked me - why didn't you come down? I was waiting and waiting-
I said that I couldn't read the clock. Nothing more was investigated. I was probably about 10.
Other kids weren't supposed to be in that section at all. So it was the "weird" place <worse word but won't use it> and therefore I was - that word- and others like it, for going there.
I was also in a spontaneously-arranged "special ed" of sorts in 7th or 8th grade gym. Those years-middle school, here - were the worst of my school years because the attacks were worse and I needed glasses but put off telling my parents. I was scared all the time.
The gym teacher thought she was being kind by taking all the "misfits" who weren't performing <for a while I'd simply refused to play volleyball and sat and drew horses instead, because I didn't understand anything about the game and had physical issues with it. I just got in trouble for that> and sticking us into our own little group, in an adjacent, utility or storage type room off the gym. She would come and give us assignments and then check back in after she'd worked with the other kids.
This is again similar to what you're describing - closer actually, because it wasn't in any way official.
so far as CAPD or APD <Central Auditory Processing or Auditory Processing Disorder> here's a link for some info. on that:
http://www.capdsupport.org/Diagnosis/what-is-capd.html