To be honest, yeah. As horrible as it sounds, I get really tired of listening to people talk and have difficulty concentrating on the context of what they're saying. Pretty soon their voice becomes noise and I'd rather not bother with it.
Plus, some people have plain irritating voices...