RealityKit 2: Create AR Models from iPhone Photos - Apple

Apple Enhances RealityKit for Augmented Reality Development

During the Worldwide Developers Conference, Apple unveiled a substantial update to RealityKit, its comprehensive set of tools designed to facilitate the creation of AR (augmented reality) experiences for developers.

RealityKit 2 introduces expanded control over visual elements, audio integration, and animation capabilities within AR applications, according to Apple.

Revolutionizing 3D Model Creation with Object Capture API

The most significant advancement within this update is the introduction of Apple’s new Object Capture API. This innovative feature empowers developers to generate 3D models in a matter of minutes, utilizing only an iPhone.

Apple highlighted that the creation of 3D models has historically been a challenging and costly aspect of AR app development, often requiring considerable time and financial investment.

Now, developers can capture 2D images of an object from multiple angles – including underneath – using an iPhone, iPad, DSLR camera, or even a drone.

Subsequently, employing the Object Capture API on macOS Monterey requires only a few lines of code to produce a fully realized 3D model, as explained by Apple.

Implementing the Object Capture Workflow

The process begins by initiating a new photogrammetry session within RealityKit, directing it to the folder containing the captured images.

Then, invoking the process function generates the 3D model at the desired level of detail.

Object Capture enables the creation of USDZ files, which are optimized for AR Quick Look – a feature allowing developers to embed interactive 3D objects within apps and websites on iPhones and iPads.

These 3D models can also be seamlessly integrated into AR scenes using Reality Composer within Xcode.

Real-World Applications and Industry Adoption

Companies such as Wayfair and Etsy are already utilizing Object Capture to create 3D models of physical products.

This suggests a significant upgrade is on the horizon for online shopping experiences, with increased AR integration.

Wayfair, for instance, is developing tools for its manufacturers to generate virtual representations of their merchandise.

This will allow Wayfair customers to preview a wider range of products in AR before making a purchase.

Furthermore, developers at Maxon and Unity are incorporating Object Capture into their 3D content creation applications, including Cinema 4D and Unity MARS.

Additional Enhancements in RealityKit 2

Beyond Object Capture, RealityKit 2 introduces several other improvements:

• Custom Shaders: Providing developers with greater control over the rendering pipeline for fine-tuning the visual appearance of AR objects.
• Dynamic Loading: Enabling efficient asset management.
• Entity Component System: Allowing developers to build custom systems for organizing assets within AR scenes.
• Player-Controlled Characters: Facilitating the creation of interactive AR games where users can navigate and explore virtual worlds.

Mikko Haapoja from Shopify has been testing the new technology and shared his findings on Twitter, demonstrating real-world applications using an iPhone 12 Max.

Developers interested in experimenting with the new features can utilize Apple’s sample app and install Monterey on their Mac.

They can employ the Qlone camera app or any preferred image capturing application from the App Store to acquire the necessary photos for Object Capture.

The Qlone Mac companion app will also integrate the Object Capture API in the fall.

The Expanding AR Ecosystem

Apple reports that over 14,000 ARKit apps are currently available on the App Store, created by more than 9,000 developers.

With over 1 billion AR-enabled iPhones and iPads in use worldwide, Apple asserts its position as the provider of the world’s largest AR platform.