Okay I am late to the party but here it goes.
Definitely both cultural differences and attitudes are involved. Both ASL and English are learned in their own way, such as in speech, tone of voice tells one the mood of another while ASL relies on facial gestures, etc. How a deaf and hearing person approaches these differences comes down to how much they are willing to understand one another when it's not done in a "perfect" manner, whether a similar but inappropriate sign is used or somewhat slurred speech was misunderstood and so).
I agreed it is both cultural and attitude and sometime it didn't even get clear up through use of the interpreter. There are ASL signs and gestures means something different for hearie.
For instance...the wave down sign/gesture. When deaf uses that sign/gesture it can means "oh, you know what!" but to hearie, they thinks deaf person are dissing them. It still even happening with the interpreter present. With the interpreter hearie dont necessarily associates what they hear from interpreter with that particular sign/gesture, and may think interpreter left it out to "protect" deaf person from looking "bad". (Yes, seriously).
Likewise the use of gestures, and facial expressions, deaf person may come across as "hysterical" resulting in misunderstanding. There are other examples. i.e. talking loud, typing too fast or "hard" in face to face conservation.
I would say (at least from my personal experience) on it's often personal attitude that is brought on by ignorance which I think is created from a larger community ignorance(which one could argue would be "cultural", so I guess it could be a blend of the two).
Yep, it is the same with me. Sometime despite my best effort to educate them the differences they may not believe. So it comes down to personal attitude and open mindedness of the person.
I'll be honest here I am sure some of us, including myself, may not be fully familiar with English/hearie culture to catch ourselves doing something that may gave them the wrong impression until it is too late to prevent the misunderstanding.