That happened to me too. When I was a teenager I was with my friends at the mall and a "deaf" peddler came up to us with the fingerspelling chart and asked for money. When I started signing to him his face turned bright red and he walked away fast.
Don't those people realize that there are many people who know how to sign and they are probably going to get caught?