Apple Live Text: Interact with Text in Photos

Apple's Live Text: A New Camera Feature

Apple has unveiled a novel capability for its camera systems. This new feature automatically identifies and converts text within images, ranging from contact details on business cards to handwritten notes on whiteboards.

How Live Text Works

Dubbed Live Text, this functionality operates without requiring any user initiation or specific actions. Simply tapping an icon activates the feature, streamlining the process.

The announcement was made by Craig Federighi during WWDC. He showcased Live Text using images of a post-meeting whiteboard and photographs featuring restaurant signage.

Text Selection and Actions

Upon activating the Live Text button, detected text is subtly underlined. A subsequent swipe enables selection and copying of the identified text.

The system successfully extracted multiple sentences, including bullet points, from the whiteboard image. From a restaurant sign, it accurately captured the phone number, allowing for direct calling or saving.

Advanced Text Recognition

The feature extends beyond simple text recognition. Specific text formats, such as tracking codes, are identified and linked directly to their corresponding URLs.

Furthermore, rapid translation is supported, facilitating conversions to and from any language compatible with Apple’s existing translation services.

Comparison to Google Lens

This functionality bears resemblance to features found within Google’s Lens application. The Pixel 4 also introduced enhanced scanning capabilities in 2019.

However, Live Text distinguishes itself by passively capturing text in every photograph taken with a compatible iPhone. A dedicated scanner mode or separate application launch is not necessary.

Accessibility Benefits

This feature offers significant advantages for all users, but particularly those with visual impairments. It allows for easy dictation or saving of text that might otherwise be difficult to read.

Privacy and Performance

All processing occurs directly on the device, ensuring user data remains private and is not transmitted to external servers.

While the speed of the process is swift, the exact processing time – whether instantaneous or occurring in the background – requires further testing. Existing photos will be updated with Live Text capabilities during periods of device inactivity.