For hearing impaired, life can come with subtitles

Smart glasses with live captions enable the deaf to see, and even recap conversations

When Madhav Lavakare was in high school, he had a friend with hearing loss who found it near impossible to understand what teachers and classmates were saying. His friend ded up choosing to be homeschooled. "It made me look into solutions and I found that hearing devices are very expensive and don't work very well in noisy settings or when multiple people are talking.

Then I looked into automatic speech recognition technology. I asked my friend why he didn't use an app on his phone to transcribe in real time," he says. For one, his friend told him, schools don't allow cellphones and more importantly, it would prevent him from picking up on non- verbal cues like facial expressions, having to look up and down like watching a tennis match. This led Lavakare, a Yale alumni to team up with Tom Pritsky, who has had bilateral hearing loss since the age of three, and create TranscribeGlass, a lightweight augmented reality (AR) device that simply clips on to reading glasses or empty frames. It can subtitle conversations as you're having them, changing the way deaf and hard of hearing people communicate.

TranscribeGlass is not the only live-captioning glasses startup. XRAI's glasses are another such product where text floats in your vision as you engage in conversation or watch a movie. It also allows users to save a searchable transcript of conversations that they can read later, Wired reports. The company claims to have 95% accuracy within a two-metre range. With 2.5 billion people projected to have some form of hearing loss by 2050, there is plenty of need for such technology.

For Lavakare, who began working on the project at only 16, it was a long, iterative process. He'd been teaching himself coding as well as hardware skills for years. "I started looking into how to build AR glasses," he says. His first prototype was crude a transparent CD case, phone and magnifying glass. As his prototypes improved, he began to attend community meetings held by the National Association of the Deaf in Delhi to gather user feedback. This led to changes in the product for Instance, one can decide how close or far from their face they want the text to be.

How the glasses work is simple - they connect to one's phone via Bluetooth. You download their app which recognises speech. That is sent to an external service which uses machine learning models to transcribe the audio into text. The text is then projected onto the glasses, with the text appearing mid-air. They use third- party speech recognition services. "It doesn't make sense to reinvent the wheel when companies like Apple, Google and Microsoft have services that support hundreds of languages. They're also comfortable with different accents," says Lavakare.

This technology has the potential to be transformative to the lives of deaf people, but it can be quite pricey. XRAI's AR Glasses cost $354, while TranscribeGlass, set to launch in the US by the year-end, is still determining how to balance the high cost of producing the technology with affordability. Lavakare adds that they already have pre- orders in the "tens of thousands".

Raising funding was challenging for Lavakare, in part, because it's easy to laugh a teenager out of rooms. "What I was doing falls into a niche where it's not charity but also not a purely profit-making business. The investors who come in looking at a three-year timeline and a profitable exit, they say, 'Why are you targeting this niche? It isn't the easiest way to make money." But eventually, the startup was incubated through IIT Delhi and through that, the company got a Pfizer grant. Along with that came grants from SBI and the department of science and technology, US State Dept.

"Our demo went viral on TikTok and I would get hundreds of emails a month from people telling me their stories. The word they keep using to describe it is 'life-changing". That's what keeps me going."