The Emergence of Automated Sign Language Translation

Millions globally utilize sign language for communication; however, unlike widely translated languages such as Spanish or Mandarin, automated translation tools have historically been unavailable for those unable to use it. SLAIT proposes the first generally accessible tool capable of translating approximately 200 words and simple sentences, requiring only a standard computer and webcam.

Addressing a Significant Communication Barrier

Individuals with hearing impairments, or conditions hindering vocal speech, represent hundreds of millions of people who rely on common technologies alongside the hearing population. While email and text chat are now prevalent, they cannot fully replicate face-to-face interactions. Currently, there is no straightforward method to convert signing into written or spoken language, creating a substantial obstacle to seamless communication.

A History of Sign Language Translation Attempts

Efforts toward automatic sign language translation, particularly American Sign Language (ASL), have been ongoing for years. In 2012, Microsoft’s Imagine Cup recognized a team developing a glove-based hand-tracking system.

Later, in 2018, SignAll was highlighted for its work on a sign language translation booth utilizing multiple cameras for 3D positioning. Furthermore, a 2019 hand-tracking algorithm, MediaPipe, developed by Google’s AI labs, was predicted to foster advancements in sign detection – a prediction that has largely come to fruition.

SLAIT: Built on Innovative Research

SLAIT is a startup originating from research conducted at the Aachen University of Applied Sciences in Germany. Co-founder Antonio Domènech initially created a small ASL recognition engine leveraging MediaPipe and bespoke neural networks.

Following successful initial validation, Domènech collaborated with Evgeny Fomin and William Vicars to establish the company. Their focus shifted to developing a system capable of recognizing 100, and subsequently 200, distinct ASL gestures and basic sentences. The translation process occurs offline and in near real-time on modern phones or computers.

Future Plans and Community Involvement

The company intends to release the technology for educational and developmental purposes, aiming to expand its dataset and refine the model before pursuing broader consumer applications.

The Challenge of Data Acquisition

Developing the current model presented considerable challenges, despite its relatively rapid completion by a small team. While MediaPipe provided an effective, open-source solution for tracking hand and finger movements, the critical element for any robust machine learning model is data – specifically, video data of ASL usage.

Unfortunately, a substantial amount of such data is not readily available.

Achieving High Accuracy Through Data Refinement

As detailed in a presentation at the DeafIT conference, the team initially assessed an older Microsoft database but discovered a newer Australian academic database containing more extensive and higher-quality data. This allowed them to create a model achieving 92% accuracy in identifying 200 signs in real-time.

They have supplemented this with sign language videos sourced from social media (with appropriate permissions) and government speeches featuring sign language interpreters, but acknowledge the ongoing need for more data.

Empowering the Deaf and ASL Learning Communities

The platform will be made accessible to the deaf and ASL learning communities, with the expectation that their usage will contribute to further system improvements.

Transformative Potential for Accessibility

Even in its current state, the tool holds immense potential, as the company’s translation model, despite being a work in progress, could significantly improve the lives of many. With the increasing prevalence of video calls, accessibility features are often overlooked – few platforms offer automatic captioning, transcription, or sign language recognition.

SLAIT’s technology could enable individuals to sign naturally during video calls, bypassing the often-neglected chat function.

SLAIT’s Vision for the Future

“In the short term, we’ve demonstrated that 200-word models are viable and our results are continually improving,” stated Evgeny Fomin of SLAIT. “Our medium-term goal is to launch a consumer-facing app for sign language tracking. However, building a comprehensive library of all sign language gestures requires substantial effort. We are dedicated to realizing this future.”

A Practical Approach to Translation

Fomin emphasized that the translation will not be perfect – mirroring the inherent approximations in translation and transcription between any languages. The focus is on delivering practical benefits to millions, and even a few hundred words represent a significant step forward.

As more data becomes available, the vocabulary can be expanded, multi-gesture phrases can be incorporated, and the performance of the core set of signs will be enhanced.

Seeking Funding and Future Development

The company is currently seeking initial funding to deploy its prototype and expand the team. Fomin indicated they have received some interest but are prioritizing investors who share their vision and understanding of the project.

Once the engine’s reliability is improved through data augmentation and machine learning refinement, the team will explore further development and integration with other products and services. Currently, the product serves as a proof of concept, but a compelling one – with further development, SLAIT has the potential to surpass existing solutions and fulfill a long-standing need for both deaf and hearing communities.