RunwayML Launches First World Model & Adds Audio to Video Model

The Emergence of World Models in AI

A significant advancement in artificial intelligence is unfolding as Runway, a leading AI image and video generation company, introduces its first world model. This development places them among a growing number of both startups and established technology giants actively pursuing this technology.

Understanding World Models

A world model represents a sophisticated AI system designed to learn and internally simulate the workings of the real world. This capability allows for reasoning, planning, and action without requiring extensive training for every conceivable real-life situation.

Runway’s newly launched model, designated GWM-1, operates by predicting outcomes frame-by-frame. This process creates a simulation grounded in an understanding of physics and the natural behavior of objects over time, according to the company.

GWM-1: A More General Approach

Having recently released Gen 4.5, a video model that outperformed competitors from Google and OpenAI on the Video Arena leaderboard, Runway asserts that its GWM-1 world model is more versatile than alternatives like Google’s Genie-3.

The firm envisions GWM-1 as a tool for generating simulations used to train agents across diverse fields, including robotics and life sciences.

“Our approach to building a world model began with developing a highly capable video model,” explained Anastasis Germanidis, Runway’s CTO, during a live presentation. “We believe that directly predicting pixels at scale, with appropriate data, is the most effective path to achieving general-purpose simulation and a robust understanding of the world.”

Specialized Versions of GWM-1

Runway has released several specialized versions of its new world model, including GWM-Worlds, GWM-Robotics, and GWM-Avatars.

Exploring GWM-Worlds

GWM-Worlds is an application that enables the creation of interactive projects. Users can define a scene using text prompts or image references.

As users navigate the generated environment, the model constructs the world, incorporating elements of geometry, physics, and lighting. The simulation operates at 24 frames per second with a 720p resolution. Runway suggests that, beyond gaming applications, GWM-Worlds is valuable for training agents to navigate and interact with the physical world.

GWM-Robotics and Synthetic Data

The GWM-Robotics version focuses on utilizing synthetic data, enhanced with parameters such as fluctuating weather conditions and dynamic obstacles. Runway believes this approach can identify potential policy violations and instructional failures in robotic systems.

GWM-Avatars: Simulating Human Behavior

Runway is also developing realistic avatars through GWM-Avatars to simulate human behavior. This area has seen significant activity from companies like D-ID, Synthesia, Soul Machines, and Google, all focused on creating lifelike avatars for communication and training purposes.

While Worlds, Robotics, and Avatars currently function as distinct models, Runway intends to integrate them into a unified system in the future.

Updates to the Gen 4.5 Model

In addition to the new world model, Runway is enhancing its Gen 4.5 model. This update introduces native audio support and long-form, multi-shot generation capabilities.

Users can now generate one-minute videos with consistent characters, integrated dialogue, background audio, and complex camera angles. The model also allows for editing existing audio and incorporating new dialogue, as well as editing multi-shot videos of any length.

This update positions Runway closer to competitors like Kling, particularly regarding native audio and multi-shot storytelling. It also indicates a shift in video generation models towards production-ready tools. The updated Gen 4.5 model is available to all paid subscribers.

Future Availability and Collaboration

Runway plans to make GWM-Robotics accessible through a Software Development Kit (SDK). The company is currently engaged in discussions with several robotics firms and enterprises regarding the implementation of GWM-Robotics and GWM-Avatars.