Indian-origin techie develops app that enables home speakers to read sign language

The app uses a camera to read sign language. Once it registers the instruction by the user, it transmits a voice command to a home speaker through a laptop. When Echo responds, the app registers it and consequently types it out for the user.

An Indian origin techie, Abhishek Singh, has developed an app which will enable the hearing-impaired community to interact with a home speaker using sign language.

The app uses a camera to read sign language. Once it registers the instruction by the user, it transmits a voice command to a home speaker like Amazon Echo that runs on Amazon Alexa. When the speaker responds, the app registers it and consequently types it out for the user.

“The project was a thought experiment inspired by observing a trend among companies of pushing voice-based assistants as a way to create instant, seamless interactions. If these devices are to become a central way we interact with our homes or perform tasks, then some thought needs to be given to those who cannot hear or speak. Seamless design needs to be inclusive in nature,” Singh was quoted in a report by Fast Company.

To build the system, Singh, who first gained recognition for creating popular game Super Mario Bros. in augmented reality, trained an AI in the popular machine learning platform Tensorflow.With repeated hand gestures that articulate sign language, Singh was able to teach the system what the language "looked" like. He then deployed Google’s text-to-speech functionality to read the words out loud to Amazon Echo.