Could an app that translates sign language to text be useful?

[Jay The Tech Guy]

Hello Everyone!

My name is Jay, I am currently completing a 4 year Bsc Computer Science degree.

I am making my Final Year Project where I am planning to make a Sign language translation application which translates sign language into typed text.

I want to develop an application that will actually help people and I really want this to be useful. So if anyone that knows anyone that is deaf or is deaf themselves please please please could you answer these questions to see if this could be of any help??

Please be as honest as possible if you think this would not help at all please let me know!

What are the main problems you face when signing, could an app that translates sign language to text be useful?

How/Where would an app that translates sign language to text be helpful? (if you had this where would you use it - if at all)

What is the current solution you use to translate sign language to text?

If you could have anything to help what would it be?

When teaching kids sign language would an app that translates sign language to text be helpful?

 

[DOD]

When you say text, I assume your objective is to transliterate from ASL to English. If it is even possible it will be no simple task. I am no expert by any means but here is my take based on what I think I understand. Anyone feel free to correct me where I am wrong.

First and foremost, ASL does not translate directly to English. Sign Language is visual and conceptual. English is spoken and written.

Other considerations;

• There are potentially up to 5 components of a sign including orientation, handshape, location, movement and facial-body expression.
• ASL syntax is not the same as English syntax
• Finger spelling
• Variations of signing while not considered "sign language" are used including Pidgin Signed English (PSE) and Signing Exact English (SEE). They are mutually exclusive.

There may be more considerations. It is an ambitious project and if you can pull it off I am sure it could be beneficial in many applications. I wish you luck.

There are some students that have made gloves to attempt this however they can only measure two components of a sign and it is intrusive because it requires the signer to wear the equipment. I also don't care for how they presented it on the youtube video I saw. While I am sure they are well intended, they are clearly ignorant to Deaf culture.

 

[Augusta86]

DOD- well said, I was thinking the same thing. I have heard of many different attempts (usually by students) to build translation technology for ASL and for the reasons you listed, the projects are usually successful for grades and media attention, but ultimately unsuccessful in function. Jay- I appreciate your idea, but I think an app that works more like a dictionary of English to ASL (text/clip entries of words and common phrases) would make more sense. Whenever someone tries to show me a video about the ASL gloves, or the tablet that "watches your ASL and speaks for you", or whichever new thing comes out, I always say I prefer a pen and paper. But, since you are asking, I will answer the questions for you
What are the main problems you face when signing, could an app that translates sign language to text be useful?
-Hearing people often become either frustrated or sympathetic (flip a coin!) when I sign. Obviously I wouldn't expect everyone to understand ASL, but when I try to use clear gestures with a hearing person, they can become nervous and confused just because they don't understand me. Rarely am I using and expecting ASL conversations with general public (which is why I would instead just write something down), but if I do happen to sign, that is the biggest problem.
How/Where would an app that translates sign language to text be helpful? (if you had this where would you use it - if at all)
-I guess this is a tricky question. Communication is needed most for me in airports, post office, maybe at a restaurant- but it would honestly be very off putting for both me and the hearing person if I was signing to my phone, having an app respond for me. I think it would add even more awkward layers to an already difficult situation. When I have a note to someone, it is the cleanest way to communicate without.
What is the current solution you use to translate sign language to text?
- I don't. Either I write (as in this moment typing), on a pad and paper, or the other person does. As far as needing text interpreted to ASL, that is never an issue for me (and I assume) most Deaf people, even when English isn't a first language. Reading this question makes me think "does he mean using an app to translate a sign or menu into ASL? That would be strange. Otherwise, interpreting someone's written response to me in ASL? Unnecessary- I think even first language signers would (and do everyday) be able to manage the written English.
If you could have anything to help what would it be?
-Genuine answer, without meaning to sound rude- the biggest help I can imagine is public awareness of Deafness, linguistic diversity, and if I can reach for the moon, rudimentary ABC fingerspelling from hearing people. The more practical answer, again, a notepad and a pen.
When teaching kids sign language would an app that translates sign language to text be helpful?
- I don't think so. I know they have DVD and computer-based ASL lessons (for example,Deaf Aotearoa, the New Zealand Deaf organization, has a great one called Sign Ninja, which is a kind of video game that helps people learn NZSL found here http://deaf.org.nz). Having video based interactive ASL learning is great, but only as a supplement to actually learning with a person one on one.

I don't think that was very helpful, but there it is. DOD made a great ending point, the good intentions people have for signing software are often a little bit disconnected from Deaf reality. I think if you are interested in helping Deaf people, you might be starting out wrong. Instead, take the opportunity to mingle here or in your community with Deaf people, make friends and go out in public with them to see what they experience, and then discuss it with them. Deaf culture should be your first step, just be sure that your project passes the "Deaf endorsement" test- engage with Deaf people and learn from them, rather than try to fix, teach or help them Best of luck!

 

[DeafNerdMommy]

I agree augusta, so many people come with designs yet nothing ever gets made. I think that is what I dislike the most.

 

[DeafDucky]

I think in some cases some components do get made like coding and all but an actual functioning (and not limited) product may not make it once the designer realizes that there's more to ASL than a straight translation- spoken Spanish to English text isn't easy either as every language has its own grammar and format.

Maybe someday so I think it's GOOD some are trying to think outside the box... there will always be failure before success.

*on that note if I manage to fund my coding bootcamp (at a standstill with the fundraising campaign...:/), I probably won't be doing anything connected to ASL even though I know it. (We'd be working on a project- I wouldn't know what to do mine on!).

 

[Tetracyclone]

Augusta 876-
It is interesting what you say about hearing people's response when you sign. My husband and I have been together 16 years with me going deaf gradually, when I use very simple, universally understood gestures, thinking it will encourage him to do so, he just stares at me blankly and continues to try and converse. There are people who cannot figure out how to shift the way they are communicating. We can't fix them.

 

[Jay The Tech Guy]

Thanks for your feedback guys + girls!! Really useful

 

[Jay The Tech Guy]

DOD- well said, I was thinking the same thing. I have heard of many different attempts (usually by students) to build translation technology for ASL and for the reasons you listed, the projects are usually successful for grades and media attention, but ultimately unsuccessful in function. Jay- I appreciate your idea, but I think an app that works more like a dictionary of English to ASL (text/clip entries of words and common phrases) would make more sense. Whenever someone tries to show me a video about the ASL gloves, or the tablet that "watches your ASL and speaks for you", or whichever new thing comes out, I always say I prefer a pen and paper. But, since you are asking, I will answer the questions for you
What are the main problems you face when signing, could an app that translates sign language to text be useful?
-Hearing people often become either frustrated or sympathetic (flip a coin!) when I sign. Obviously I wouldn't expect everyone to understand ASL, but when I try to use clear gestures with a hearing person, they can become nervous and confused just because they don't understand me. Rarely am I using and expecting ASL conversations with general public (which is why I would instead just write something down), but if I do happen to sign, that is the biggest problem.
How/Where would an app that translates sign language to text be helpful? (if you had this where would you use it - if at all)
-I guess this is a tricky question. Communication is needed most for me in airports, post office, maybe at a restaurant- but it would honestly be very off putting for both me and the hearing person if I was signing to my phone, having an app respond for me. I think it would add even more awkward layers to an already difficult situation. When I have a note to someone, it is the cleanest way to communicate without.
What is the current solution you use to translate sign language to text?
- I don't. Either I write (as in this moment typing), on a pad and paper, or the other person does. As far as needing text interpreted to ASL, that is never an issue for me (and I assume) most Deaf people, even when English isn't a first language. Reading this question makes me think "does he mean using an app to translate a sign or menu into ASL? That would be strange. Otherwise, interpreting someone's written response to me in ASL? Unnecessary- I think even first language signers would (and do everyday) be able to manage the written English.
If you could have anything to help what would it be?
-Genuine answer, without meaning to sound rude- the biggest help I can imagine is public awareness of Deafness, linguistic diversity, and if I can reach for the moon, rudimentary ABC fingerspelling from hearing people. The more practical answer, again, a notepad and a pen.
When teaching kids sign language would an app that translates sign language to text be helpful?
- I don't think so. I know they have DVD and computer-based ASL lessons (for example,Deaf Aotearoa, the New Zealand Deaf organization, has a great one called Sign Ninja, which is a kind of video game that helps people learn NZSL found here http://deaf.org.nz). Having video based interactive ASL learning is great, but only as a supplement to actually learning with a person one on one.

I don't think that was very helpful, but there it is. DOD made a great ending point, the good intentions people have for signing software are often a little bit disconnected from Deaf reality. I think if you are interested in helping Deaf people, you might be starting out wrong. Instead, take the opportunity to mingle here or in your community with Deaf people, make friends and go out in public with them to see what they experience, and then discuss it with them. Deaf culture should be your first step, just be sure that your project passes the "Deaf endorsement" test- engage with Deaf people and learn from them, rather than try to fix, teach or help them Best of luck!

Really Really helpful!!! Thank you so much! Do you have an email address where I could talk to you directly?

After reading that response I have learnt a great deal and although the idea seems good the practicality is not great.

I have thought about the points you have mentioned and I have thought about a possible solution.

Please could you give me your honest feedback about this idea:

If there was a camera connected to a pc say at the airport desk/restaurant, post office etc, and when you approached the desk you could then sign to that camera and it spoke out what you signed and/or displayed this on a screen for the person sitting on the desk.
And they could then type their reply and that can be show as text on a monitor which is facing yourself.

1. Would you feel comfortable signing to the camera which then displayed the gestures as text to the receptionist, and then text being shown on a monitor facing you with their replies?

2. Would you find this easier than writing on paper?

 

[Jay The Tech Guy]

When you say text, I assume your objective is to transliterate from ASL to English. If it is even possible it will be no simple task. I am no expert by any means but here is my take based on what I think I understand. Anyone feel free to correct me where I am wrong.

First and foremost, ASL does not translate directly to English. Sign Language is visual and conceptual. English is spoken and written.

Other considerations;

• There are potentially up to 5 components of a sign including orientation, handshape, location, movement and facial-body expression.
• ASL syntax is not the same as English syntax
• Finger spelling
• Variations of signing while not considered "sign language" are used including Pidgin Signed English (PSE) and Signing Exact English (SEE). They are mutually exclusive.

There may be more considerations. It is an ambitious project and if you can pull it off I am sure it could be beneficial in many applications. I wish you luck.

There are some students that have made gloves to attempt this however they can only measure two components of a sign and it is intrusive because it requires the signer to wear the equipment. I also don't care for how they presented it on the youtube video I saw. While I am sure they are well intended, they are clearly ignorant to Deaf culture.

Thank you so much for your reply!! Very useful food for thought, I'm doing some more research on existing implementation and I agree, this is challenging however this can be done via the use of AI. I am planning to use machine learning with a large data set extracted from existing videos showing various signs in various different situations which should tackle the points you mentioned.

I can understand how the gloves are impractical and it seems most students design without the input of possible users which could lead to a lot of "great" ideas which are not that useful in real life scenarios.

Once again thanks for your input! - And what are your thoughts about the question below?

If there was a camera connected to a pc say at the airport desk/restaurant, post office etc, and when you approached the desk you could then sign to that camera and it spoke out what you signed and/or displayed this on a screen for the person sitting on the desk.
And they could then type their reply and that can be show as text on a monitor which is facing yourself.

1. Would you feel comfortable signing to the camera which then displayed the gestures as text to the receptionist, and then text being shown on a monitor facing you with their replies?

2. Would you find this easier than writing on paper?

 

[DOD]

Thank you so much for your reply!! Very useful food for thought, I'm doing some more research on existing implementation and I agree, this is challenging however this can be done via the use of AI. I am planning to use machine learning with a large data set extracted from existing videos showing various signs in various different situations which should tackle the points you mentioned.

I can understand how the gloves are impractical and it seems most students design without the input of possible users which could lead to a lot of "great" ideas which are not that useful in real life scenarios.

Once again thanks for your input! - And what are your thoughts about the question below?

If there was a camera connected to a pc say at the airport desk/restaurant, post office etc, and when you approached the desk you could then sign to that camera and it spoke out what you signed and/or displayed this on a screen for the person sitting on the desk.
And they could then type their reply and that can be show as text on a monitor which is facing yourself.

1. Would you feel comfortable signing to the camera which then displayed the gestures as text to the receptionist, and then text being shown on a monitor facing you with their replies?

2. Would you find this easier than writing on paper?

The best solution would be for everyone to learn ASL.

Not to speak for anybody, but cameras are everywhere nowadays. Comfort level and ease of use are relative. It becomes comfortable if it is non intrusive. Nobody has to wear gloves or carry equipment. Nobody has to type or write anything. It becomes easy when we understand each other. Deaf folks can sign and hearing folks can speak.

Positioned at interaction points it could feature;

• Compact processor with microphone, adjustable camera, adjustable LCD screen and a volume controlled speaker
  
• Bi-directional transliteration between ASL, PSE, SEE, finger spelling <<<<and>>>> spoken and written English
• Configurable conversation output to speaker, LCD screen or both.
• Deaf persons side of the conversation is output to LCD screen as English text, or output to the speaker
• Hearing persons side of the conversation is output to LCD screen as English text or ASL signer.
• Available on mobile devices that are equipped with microphones and cameras

The technology to put this together exists. The programming for bullet point 2 is the real challenge. The program would also have to factor in differentiating between someone signing or someone making random hand movements. To mitigate that you could have the ability to invoke the system on demand.

Hopefully that addresses your questions and gives you some ideas. If not, let me know. Best of luck!!