I have a similar situation. I have a progressive hearing loss. Soon after I met my hearing wife, she tried to learn ASL. This has been a challenge because she has a brain injury from a car accident before we met which makes it difficult for her to remember things. She also gets overwhelmed with too much visual stimulation and it can send her brain into a freeze. We can have a conversation in the morning and maybe teach some new signs. By the afternoon, not only does she forget the signs but she might not remember we had a conversation at all.
We communicate by any means necessary - writing, lip reading, drawing, acting out, speech-to-text apps, text message and signing. After seven years she is roughly an intermediate signer but it fluctuates. A big help was to change from teaching ASL to Signed English. It was too much for her to both remember the signs AND remember ASL syntax.
This relationship requires a lot of patience and work on both sides. Both of you need to be very committed to make this work. Actually that is true for any marriage really.