"First impression is everything."
Whether you like it or not, people do judge you based on what you look like because they think it's a tall-tale sign of what you like to do.
For instance, someone with beefy legs and broad shoulders is most likely to be a backpacker or a rugby player. Probably spend more time outside than he like to work.
Slender, well-toned? Probably someone who go to the pool a lot.
Scrawny? Probably read a lot, or spend too much time in front of video consoles.
Bodybuilder or strongman physique? Probably spend more time at the gym than he does at home.
Someone who looks bony? Probably too obsessive with dieting. Or is a drug addict.
Whether or not these are actually true indicators, stereotypes do exist, and they are constantly reinforced by the majority who choose those lifestyles.
So yes, it is a matter of perceptive, but you cannot deny the mainstream stereotypes.