Gboard Prepares to Interpret Sign Language Through Your Phone’s Camera

Google’s Gboard aims to facilitate seamless communication by converting thoughts into written words effortlessly. There are various input methods available, from traditional typing to voice recognition, but the company is on the verge of a significant breakthrough in accessibility with its new initiative to recognize sign language.

The ability to translate sign language into written text may sound ambitious, but advancements in machine vision and artificial intelligence are making this a reality. Last year, Google DeepMind introduced its SignGemma model aimed at recognizing sign language, and this may soon come to fruition in Gboard.

In version 17.8.3 of the Gboard app, we’ve discovered an early glimpse into a forthcoming Sign-to-Text feature:

Although the tool isn’t fully operational yet, an introductory message provides insight into its expected functionality.

This feature will utilize a combination of on-device processing and cloud services. Your video input will be processed locally to detect gestures, while only gesture data will be sent to Google for word extraction, enhancing privacy by keeping the video private.

While Google will take care of the processing in the cloud, there’s curiosity surrounding which devices will support the Sign-to-Text feature.

While there’s a clear understanding of how the Sign-to-Text feature will function, questions remain about its implementation. It seems logical that American Sign Language would be prioritized, but the inclusion of other regional sign languages is yet to be clarified.

Though many details are still pending, the potential of this feature is exciting. It’s an innovative input solution that smartphones can uniquely provide, poised to enhance communication for users who need it most, showcasing Google’s commitment to improving tech accessibility.