Talking with your hands

24.07.2016
A team of engineering undergraduates has developed a prototype of a 'smart glove' that converts hand gestures based on sign language into spoken English

Students at the Bengaluru-based Amrita School of Engineering’s Amrita Robotics Research Lab have developed a prototype of a “smart glove” called MUDRA that converts hand gestures based on Indian Sign Language (ISL) into spoken English. The glove is made of plastic and much larger than the kind you’d slip on before a bikeride.

The glove is less ungainly than the jangle of wires protruding from it which are connected to a personal computer, from where various gestures are electronically decoded and translated into speech using the machine’s speakers. Here the device works because of sensors placed on each finger. Various signs which correspond to letters and concepts are interpreted based on how these sensors interpret the varied shapes formed by the fingers.

The glove can recognise numbers from 1 to 10, and ISL gestures corresponding to words such as ‘morning’, ‘night’, ‘goodbye’, ‘thank you’, etc. It can detect four different states of each finger and configure as many as 70 gestures.

“Going ahead, these speakers will be in-built into the glove,” says H.R. Nandi Vardhan, who led a team of engineering undergraduates to develop the apparatus. “The big challenge, however, is to teach the system to recognise a much larger database of gestures and adapt it to be used in regional languages,” he adds.

Slow growth for sign language

According to various government and independent reports, there are 5-7 million hearing-impaired people in India though there aren’t firm estimates of how many of them are formally trained in sign language. There has been a lot of work over the years to standardise signs and gestures, used across several Indian languages, and use them to develop educational material for the hearing-impaired. However, there has been relatively little progress in enabling users of sign language to reach out to a wider audience, which is where applications like a smart glove can be handy.

In April, two students at the University of Washington bagged a $10,000 prize for designing a set of gloves that translated gestures into speech using a sensor system similar to the one Nandi Vardhan’s team used. ‘MotionSavvy Uni’ is another gesture-driven technology tool for sign language-users that can translate nearly 300 of the American Sign Language’s 50,000 signs to text via a tablet interface.

The students associated with the Amrita project, Abhijith Bhaskaran, Anoop G. Nair, Deepak Ram and Krishnan Ananthanarayanan, say that affordability and the efficiency of the underlying algorithms powering the system are a key differentiator for their glove. “The glove is much cheaper compared to similar gesture-sensing products available today. The prototype took us 16 weeks to build and costs Rs.7,500,” says Mr. Bhaskaran.

India has its very own alphabet of signs and symbols called ISL. In 2001, the Ramakrishna Mission released the first Indian Sign Language Dictionary, which documented over 2,500 signs from 12 States, to provide a common sign language code. The Ramakrishna Mission and other organisations involved with hearing-impaired communities have since systemised ISL teaching materials, degree programmes and training sessions to popularise it in India.

Institutions such as the Indian Institute of Technology, Guwahati, and the Indian Institute of Information Technology, Allahabad, have dedicated laboratories to use technology that can help sign language users communicate more effectively with those who use spoken language.

Last September, the Union Cabinet cleared a proposal to set up a dedicated research and training centre for sign language users.