Luma AI Raises $4.3M to Simplify 3D Modeling with Your Phone

Revolutionizing Product Photography with Neural Rendering

When shopping online, interactive product images allowing rotation from all angles are increasingly common. These are typically created through animation of numerous photographs. Luma, a company formed by former Apple AR and computer vision engineers, is introducing a disruptive technology aimed at streamlining this process.

A New Approach to 3D Modeling

Luma has developed a novel neural rendering technology capable of generating photo-realistic 3D models from a limited number of photographs. This innovation intends to significantly accelerate product photography for e-commerce, while simultaneously enhancing the consumer experience by enabling comprehensive product visualization.

Crucially, the resulting image is a genuine 3D interpretation, allowing rendering from any perspective, and even in 3D with dual viewports offering slightly differing angles. This means consumers can view a product in 3D using a VR headset.

Addressing the Challenges of Existing 3D Representations

Previous attempts at creating 3D representations using consumer cameras and basic photogrammetry have often yielded unsatisfactory results. However, advancements in technology present new possibilities, and Luma aims to capitalize on these.

The Power of Neural Rendering

“The current shift is driven by the emergence of neural rendering techniques,” explains Luma AI founder Amit Jain. “Traditional photogrammetry involves image capture followed by extensive processing to create point clouds and reconstruct 3D models. Achieving high-quality 3D imagery requires constructing high-quality meshes from imperfect, real-world data – a problem that remains largely unsolved.”

Luma has adopted a different strategy, acknowledging the difficulty of obtaining accurate meshes from point clouds. Instead, the company leverages the principles of Physics Based Rendering (PBR), but adapts it to work with photographic input.

How Luma's Technology Works

“We operate under the assumption that a precise mesh from a point cloud is unattainable. We utilize a novel approach. With perfect shape data – essentially the rendering equation – PBR is achievable. However, starting from photographs limits the available data for this type of rendering.

Therefore, we developed a new method. We input 30 photos of an object, presenting 20 to the neural network. The remaining 10 photos serve as a ‘checksum’ – the expected outcome. If the network can accurately predict the final 10 images based on the initial 20, a robust 3D representation has been created.”

Real-World Applications and Cost Reduction

This technology has significant implications for e-commerce. Luma’s solution promises to transform how consumers interact with products online, enabling features like rotation, zooming, and virtual movement from any angle, even those not originally photographed.

The top two images are photographs, which formed the basis of the Luma-rendered 3D model below. Image Credits: Luma

“Currently, showcasing products in 3D often necessitates the involvement of 3D artists for adjustments to scanned objects, significantly increasing costs,” Jain notes. “This limits 3D renders to high-end, premium products.” Luma’s technology aims to reduce the cost of 3D asset capture and display to a fraction of the current expense.

Luma’s co-founders, Amit Jain (CEO) and Alberto Taiuti (CTO). Image Credits: Luma

The company intends to develop an embeddable player, similar to YouTube, allowing retailers to easily integrate these three-dimensional images into their product pages.

Investment and Future Outlook

Matrix Partners, South Park Commons, Amplify Partners, RFC’s Andreas Klinger, Context Ventures, and several angel investors have contributed $4.3 million in funding, with Matrix Partners leading the investment round.

“It’s clear that the next generation of computing will be built on 3D,” states Antonio Rodriguez, general partner at Matrix. “However, scaling the creation of 3D environments requires a simpler process than current labor-intensive methods. Capturing and sharing 3D content should be as easy as taking a picture!”

The company has released a video demonstrating the capabilities of its technology.