Facebook & Matterport: AI Training in Realistic Virtual Environments

Training Robots with Virtual Worlds: A Collaborative Effort

Developing navigational skills in robots necessitates extensive training, achievable through either real-world experience in numerous locations or substantial virtual time within simulated environments. The latter approach is demonstrably more efficient, and a partnership between Facebook and Matterport is poised to provide researchers with access to thousands of interactive, digital replicas of actual spaces for training their artificial intelligence systems.

Facebook's Advancements: Habitat 2.0 and ReplicaCAD

Facebook’s contributions center around two key innovations: the Habitat 2.0 training environment and the accompanying dataset designed to facilitate its use. Habitat was initially introduced a few years ago as part of Facebook’s “embodied AI” initiative, focused on creating AI models capable of interacting with the physical world.

Previously, many robots and AI systems learned fundamental skills like movement and object recognition in unrealistic, game-like environments. A genuine living room presents complexities far exceeding those of a reconstructed virtual space. Training within a reality-mimicking environment allows an AI’s acquired knowledge to transfer more effectively to practical applications, such as home robotics.

Enhanced Interactivity and Physical Simulation

Earlier virtual environments lacked depth, offering minimal interaction and no realistic physical simulation. For example, a robot colliding with a table wouldn’t experience a realistic response, like items falling. Basic actions, such as opening a refrigerator or retrieving an object from a sink, were beyond the capabilities of these systems. Habitat 2.0 and the new ReplicaCAD dataset address these limitations by introducing increased interactivity and utilizing fully 3D objects rather than merely interpreted surfaces.

Robots operating within these new, apartment-scale environments can navigate as before, but now they can actively interact with objects. Consider a task requiring a robot to pick up a fork from a dining table and place it in the sink; previously, this action was simply assumed. Now, the fork, the table, and the sink are all subject to physical simulation, increasing computational demands but significantly enhancing the system’s utility.

Competition and Speed

While not the first to achieve this level of simulation, the field is progressing rapidly, with each new system building upon its predecessors and identifying new challenges. Habitat 2.0’s primary competitor is AI2’s ManipulaTHOR, which also combines room-scale environments with physical object simulation.

Habitat excels in processing speed; the simulator operates approximately 50-100 times faster than its competitors, allowing robots to accumulate significantly more training data per unit of computation. Although direct comparisons are complex due to the systems’ distinct characteristics.

ReplicaCAD: Meticulously Recreated 3D Models

The dataset powering Habitat 2.0 is called ReplicaCAD, consisting of original room scans recreated using custom 3D models. Facebook acknowledges this is a labor-intensive manual process and is exploring methods for scaling it, but the resulting product is highly valuable.

Further enhancements, including greater detail and more sophisticated physical simulations, are planned. Currently, the system supports basic objects, movements, and robotic presence, but speed necessitated some compromises in fidelity at this stage.

Matterport's Contribution: HM3D Dataset

Matterport is also playing a crucial role through its partnership with Facebook. Having significantly expanded its platform in recent years, the company has amassed a vast collection of 3D-scanned buildings. After prior collaborations with researchers, Matterport has decided to make a substantial portion of its data available to the wider research community.

“We’ve Matterported virtually every type of physical structure imaginable – homes, skyscrapers, hospitals, offices, cruise ships, even fast-food restaurants. The information contained within these digital twins is incredibly valuable for research,” stated CEO RJ Pittman. “We recognized the potential for applications in computer vision, robotics, and object identification, and Facebook readily agreed – it aligns perfectly with their Habitat and embodied AI initiatives.”

To this end, Matterport created HM3D, a dataset comprising a thousand meticulously 3D-captured interiors, ranging from residential homes to commercial and public spaces. This is currently the largest publicly available collection of its kind.

The environments, scanned and interpreted by AI trained on precise digital twins, are dimensionally accurate, enabling calculations of parameters like window surface area or closet volume. This realism provides a valuable training ground for AI models, and while the dataset is not yet interactive, it accurately reflects the diversity of the real world. It is distinct from Facebook’s interactive dataset but could potentially serve as a foundation for future expansion.

Data Diversity and Ethical Considerations

“We specifically focused on creating a diversified dataset,” Pittman emphasized. “A wide range of real-world environments is essential for maximizing the effectiveness of AI and robot training.”

All data was voluntarily provided by the property owners, ensuring ethical data collection practices. Matterport aims to develop a larger, more parameterized dataset accessible via API, essentially offering realistic virtual spaces as a service.

“Imagine building a hospitality robot for bed and breakfasts of a specific style in the U.S. – wouldn’t it be beneficial to have access to a thousand such environments?” Pittman proposed. “We intend to assess the potential of this initial dataset, incorporate the learnings, and collaborate with the research community and our developers to further refine our offerings. This represents a significant starting point for us.”

Both datasets will be openly accessible to researchers worldwide.