Apple's New Child Safety Feature: Protecting Kids from Explicit Content

Apple to Introduce New Safety Tools for Children

Later this year, Apple will implement new features designed to safeguard children online. These tools will alert both children and their parents if sexually explicit photos are sent or received through the Messages application.

This initiative is part of a broader effort by Apple to curtail the proliferation of Child Sexual Abuse Material (CSAM) across its platforms and services.

Detecting and Addressing CSAM

Apple will be capable of identifying known CSAM images on devices like iPhones and iPads. The system will also scan photos uploaded to iCloud, all while prioritizing user privacy.

The company emphasizes that this detection process will not compromise consumer privacy.

Enhanced Messages Feature for Parental Guidance

A new feature within Messages aims to empower parents with greater insight into their children’s online interactions. This will enable a more informed approach to online safety education.

Utilizing on-device machine learning, Messages will analyze image attachments to determine if they contain sexually explicit content. Crucially, this analysis occurs directly on the device, ensuring Apple does not access or review private communications.

How the Warning System Works

If a potentially sensitive photo is detected, it will be blocked, and a label reading “this may be sensitive” will appear. This label includes a link allowing the child to view the image if they choose.

Upon selecting the link, an additional screen will display further information. This screen explains that sensitive images often depict private body parts and clarifies that receiving such content is not the child’s fault.

The message also acknowledges the possibility that the image was shared without the subject’s consent.

Guiding Children Towards Safe Choices

The warnings are designed to encourage children to refrain from viewing potentially harmful content.

Should a child proceed to view the photo, a subsequent screen will inform them that their parents will be notified. This screen reinforces the parents’ concern for their child’s safety and suggests seeking support if feeling pressured.

Links to relevant resources for assistance are also provided.

The interface is intentionally designed to highlight the option to not view the photo, rather than making it the default action.

Protecting Children from Online Predators

These features are expected to offer protection against sexual predators. The technology interrupts communications, provides guidance, and alerts parents to potential risks.

Often, predators manipulate children and isolate them from their families, concealing their interactions. Apple’s technology aims to disrupt this pattern by intervening and alerting to explicit material exchanges.

Addressing Self-Generated CSAM

A significant portion of CSAM now consists of self-generated imagery, such as photos taken and shared by children themselves. This includes instances of sexting or sharing “nudes.”

A 2019 survey by Thorn revealed that 1 in 5 girls (ages 13-17) and 1 in 10 boys have shared nude images. However, children may not fully grasp the risks associated with sharing such content.

The new Messages feature will also address this scenario. Children attempting to send explicit photos will receive a warning before transmission.

Parents will also be notified if their child chooses to send the photo despite the warning.

Availability and Additional Safety Resources

These updates will be rolled out later this year as part of a software update for accounts configured as families in iCloud. The update will be available for iOS 15, iPadOS 15, and macOS Monterey in the U.S.

The update will also enhance Siri and Search functionality. Users will be able to ask Siri how to report CSAM or child exploitation.

Furthermore, Siri and Search will proactively offer guidance and resources when users search for CSAM-related terms, explaining the harmful nature of the topic and providing access to help.