DeepMind's MuZero Masters Games Without Prior Knowledge

DeepMind is dedicated to demonstrating that artificial intelligence can achieve true mastery in games, even without prior knowledge of the governing rules. Their latest AI agent, named MuZero, achieves this capability not only in games characterized by straightforward visuals and intricate strategies, such as Go, Chess, and Shogi, but also in the more visually demanding realm of Atari games.

The accomplishments of DeepMind’s previous AI systems were, in part, attributable to their efficient exploration of the extensive decision trees that define the potential actions within a game. In games like Go or Chess, these trees are structured by precise rules, dictating piece movements and the consequences of those movements.

AlphaGo, the AI that defeated professional Go players, utilized these rules as a foundational element while analyzing games played by both humans and itself, ultimately developing a collection of optimal strategies and techniques. Its successor, AlphaGo Zero, refined this process by learning solely through self-play. AlphaZero extended this approach to Go, Chess, and Shogi in 2018, resulting in a unified AI model capable of proficiently playing all three games.

However, in each of these instances, the AI was provided with a fixed set of established rules for the games, establishing a framework for strategy development. Consider this: knowing a pawn can promote to a queen influences planning from the outset, whereas discovering this possibility could lead to entirely different strategic approaches.

This helpful diagram shows what different models have achieved with different starting knowledge. Image: DeepMind

As detailed in a recent blog post, the company explains that pre-defined rules can hinder an AI’s application to real-world challenges, which are often complex and difficult to reduce to simple rules.

DeepMind’s newest innovation, MuZero, addresses this limitation by excelling in not only the previously mentioned games but also a diverse selection of Atari games, all without receiving a rulebook. The final model mastered these games through independent experimentation (without human data) and without being informed of even the most fundamental rules.

Rather than relying on rules to identify optimal outcomes (as it has no rules to rely on), MuZero learns to consider all aspects of the game environment, independently determining their relevance. Through countless games, it acquires an understanding of not only the rules themselves but also the inherent value of different positions, effective strategies for gaining an advantage, and a method for evaluating its actions retrospectively.

This capacity for self-assessment allows it to learn from errors, revisiting and replaying games to explore alternative approaches that further refine its understanding of position and strategy values.

Agent57, another successful DeepMind AI specializing in 57 Atari games, may come to mind. MuZero integrates the strengths of both Agent57 and AlphaZero. MuZero distinguishes itself from Agent57 by focusing on elements of the game environment that directly impact its decision-making, rather than modeling the entire environment. It differs from AlphaZero in that its understanding of the rules is derived solely from its own experimentation and direct experience.

By comprehending the game world, MuZero can effectively plan its actions, even when the game environment, as is common in many Atari games, incorporates randomization and visual complexity. This advancement brings it closer to an AI capable of interacting safely and intelligently with the real world, learning to interpret its surroundings without requiring explicit instruction on every detail (although certain principles, such as “avoid harming people,” will likely be firmly established). According to a researcher interviewed by the BBC, the team is currently investigating MuZero’s potential to enhance video compression—a significantly different application than Ms. Pac-Man.

The specifics of MuZero’s development were published today in the journal Nature.