I think the difference is having a disability vs being a disabled person. I think the inability to hear will always be a disability, but that doesn't necessarily make that person disabled. I absolutely see being hearing impaired as a disability (for myself) because of the difficulties it causes...
I think it can be offensive if it's used to refer to someone who doesn't identify with it. But on it's own, the term is not offensive. It's what I use for myself, because that's the most accurate description - my hearing is impaired.