Android Accessibility: Google Adds Facial Gesture Controls

Expanding Accessibility with Google’s New Features

Enhancing smartphone accessibility remains a crucial goal, and Google’s recent advancements offer streamlined actions and navigation for individuals who primarily interact with devices through facial expressions. Project Activate and Camera Switches empower users to execute tasks, such as uttering a personalized phrase, or navigating via a switch interface, utilizing solely facial gestures.

Leveraging Facial Expressions for Control

These innovative features utilize the smartphone’s front-facing camera to monitor the user’s face in real-time, detecting six distinct expressions: a smile, raised eyebrows, an open mouth, and looks directed left, right, or upward. All processing occurs locally on the device; no image data is stored, and the technology does not employ conventional “facial recognition.”

Instead, this system utilizes a specific type of machine learning that focuses on identifying individual facial features – like the eyebrows – and triggering a signal when they surpass a user-defined threshold.

Customizing Gestures for Diverse Needs

Each facial expression can be assigned a unique function. Camera Switches integrates seamlessly with Android’s existing switch compatibility, which already allows individuals using assistive technologies like joysticks or blow tubes to navigate the operating system.

Now, this navigation can be achieved without any external devices, enabling users to select facial gestures for actions like cycling through options, confirming selections, or backing out of menus.

Project Activate: Triggering Actions with Expressions

Image Credits: Google

Project Activate allows expressions to be linked to independent actions, such as vocalizing a phrase. For many individuals with disabilities who rely on caregivers, gaining their attention can be challenging. Assigning an extended eyebrow raise to trigger the device to say “hey!” or “I need assistance,” or even “thank you!” provides a valuable solution.

Expanding Capabilities and Language Support

The gestures can also initiate audio playback, send text messages, or dial a pre-set phone number. Further expressions and functionalities are planned, alongside expanded language support. While faces themselves aren’t language-specific, the application and its documentation will initially support English-speaking regions before expanding globally.

Camera Switches, however, will launch with support for 80 languages.

It’s important to note that these features cannot be used simultaneously, as both require camera access and expression recognition. Users should therefore maintain an alternative navigation method. Both features are designed to function on most Android phones released within the last five years.

Enhancements to Google’s Lookout App

An update to Google’s Lookout app, which assists individuals with visual impairments by reading labels, now includes the ability to scan and vocalize handwritten text, mirroring its existing capability with printed materials. This is particularly useful for reading sticky notes, handwritten signs, and personalized messages in greeting cards.

The app has experienced significant growth in usage over the past year, prompting further development. The addition of support for identifying euro and Indian rupee banknotes is expected to further increase its adoption.

Availability

These new features will be available free of charge later this week.