I sign it from mouth first to the ear.
Another variation that you may have seen is when a person stick the thumb in the ear (with the other four fingers closed) and then they open the fingers where the hand is open. This means "totally (profoundly) deaf" or "stone deaf."