With Live Transcribe app, Google makes AI help people with hearing loss communicate

13.07.2019
HIGHLIGHTS

• Google this year launched an app called the Live Transcribe.
• The app uses a phone's microphones to listen to sounds and then transcribe that into text.
• This app is already available in 150 countries across the world.

Hearing is one of the key senses that enables us to connect with the world around us. It not only helps in understanding the message that is being conveyed to us but it also empowers us to respond to those messages effectively. But many-a-times, factors such as ageing, infectious diseases and ear infections, recreational activities or accidents, affect an individual's ability to hear.

The result is that individuals with such impairment are unable to communicate and hence connect with people around them. While there are devices such as cochlear implants and other assistive device that can be used for help such individuals, none of them are easy to use. Google is hoping to change that.

The tech giant earlier this year launched an app called the Live Transcribe app that enables people connect with the world around them by transcribing conversations in real-time. The app, which is available for Android devices, uses a host of tools, both hardware and software, to enable people to follow a communication that they otherwise would have been excluded from.

"It brings out the captions in the real world and helps you understand what is being spoken around the world," Google product manager Sagar Savla said during his presentation at the Solve with AI Conference in Tokyo on July 10. At the conference, Savla, besides explaining the technology behind the app, also gave a live demo of how the Live Transcibe app.

So how does this app work?

The app as it is simple to use. It uses a phone's microphones to listen to the sounds around an individual, it analyses the audio and it then transcribes that sound and shows the users the relevant text. In the heart of the app are three complex machine learning (ML) models that analyse the sound for words and context. While the first model - called the Acoustic Model - converts the sounds to Phonemes waveform, the second model - called the Pronunciation model - converts phonemes into words. The third and the final model - also called the Language model - then adds context of the words to make sense of the entire conversation.

"It understands context. So if you are talking about buying a new jersey in New York, it actually understands the capitalization and makes the jersey smaller so that it knows that you are talking about the shirt and not the place," added Savla.

The interesting thing about this app is that is already available in 150 countries across the world and that it can transcribe up to 70 different languages including Hindi, Portuguese, Spanish and Italian in real time. In addition to that, the Live Transcribe app can also recognise 40 different sound effects like car sirens horns, clapping, music, applause, people shouting at you, sound of a speeding vehicle and a baby crying among other things.

The use-cases of this app are not limited to people with hearing impairment. The Live Transcribe app can also be used by students and journalists to take notes. Users can also copy and save transcripts of their conversation on their devices for three days.