I think minorities in general are often overlooked, and therefore feel less important or as though people don't think as highly of you. And then there's the fact that people are all too often uneducated about groups of which they're not a part, so they form stereotypes, which leads to prejudice. There's also fear of the unknown.
I know I've felt like people have looked down on me about various things. When I lived abroad I got a lot of crap about being an American. I've had people give me a hard time for being a vegetarian. Anything that makes you different can make you a target.
Of course, it is important to remember that just like we don't want to be looked down on, we should try not to look down on the majority. A lot of people are ignorant, but if we group them all together and start to stereotype, how does that make us better than them?