I am HOH and in most situations, I am the one having difficulties hearing in restaurants. But I still order for myself and of course, I still use my voice. I began losing my hearing later in life, in my 40s, so I use all my communication tools: my aids and my voice and my eyes. If I can't understand the server, I ask for a repeat, speak up, etc. I request a table in a quieter part of the place, not near speakers (if they have background muzak) or the kitchen or service stations, etc.
*However* in our ASL class which is for Late Deafened or Hard of Hearing beginners and a hearing family member, if applicable (like my husband), sponsored by the local Deaf Advocacy agency, we are required to be voice off. Because we use every tool available, most of us are used to using our voice, and straining to hear and not use ASL except in class. So, this makes us SEE and work on understanding signs others are using as well as signing ourselves, using body language (which we may not do in hearing world as much). So by voice off it forces us to fully concentrate on learning our new language.
So, I don't know if I would go voice off in "mixed company" of hearing and Deaf and Hard of Hearing, or not. But I would definitely explain to servers that they need to speak with the person directly and not look at me or someone else. Like you did.

Sometimes people working with the public don't have a lot of experience with people with hearing loss. They know what to do for wheel chairs, right? They can spot a blind person by their white cane or dog, right? But hearing loss is invisible until communication actually begins, so they are often just ignorant. I always ask for a repeat, and explain right away that I am HOH, the music is loud, loud people standing around us, so I couldn't hear, etc. Explain shouting no good, but being closer, speaking clearly, moving away from noise as much as possible, etc. When I do that, teach them what to do for me, I find people are grateful and ready to help as much as possible. Rather than look at me like I am either learning disabled or senile...
I am who I am. HOH. I spend 99% of my time in the hearing world with hearing people. I don't apologize for using all my tools in mixed company, either. And in that case, I would voice *and* sign whatever I knew how to sign... I'm not afraid to ask for help, etc. All of it, aids, voice, eyes, limited ASL, are my tools. I expect people to communicate according to what works best for them, and their choices and options, and the situation at hand (mixed or fully hearing). And be polite enough to work with me to communicate as I will try to do for them. Now, if I were at a Deaf event or gathering, and everyone else was voice off, then I would respect that and do so. And try hard to use what ASL I know, of course. So, depends on the situation and venue, and I would hope I would be sensitive enough to be observant and try to go with the flow. After all, the most important thing, is have a good time, be mutually respectful and inclusive as possible, and communicate, right?