Wow, I can totally relate to this! This is one of the reasons I have gone basically voice-off. It avoids this situation entirely. Yeah, relying on pen/paper or text messaging isn't easy, but it works for me.
Are you going to learn ASL? It really is a beautiful language, far more expressive than spoken english (IMO).