yeah, its a tough balance. You dont want to come off as thin skinned but at the same time you dont want the boss to undermine your position within the company. Like giving other employees the idea that its pointless to talk to you cuz youll never hear them anyway.
Then other employees might start to ignore you and work around you. Giving you a bad reputation and affecting your usefullness to the company. Ruling out any chance of moving up in the company. I bet it happens pretty often
At my last job, a couple of co-workers made fun of my deaf accent. I did not like it at all. To me its the equivalent of calling a black person the n-word.