Anthropic Benchmarks AI Model with Pokémon - AI News
Anthropic's AI Model Benchmarked with Pokémon
Anthropic has employed the classic Game Boy game, Pokémon, as a benchmark for its most recent AI model. This unconventional approach was recently confirmed by the company.
Testing Claude 3.7 Sonnet on Pokémon Red
In a blog post released on Monday, Anthropic detailed its testing of Claude 3.7 Sonnet on Pokémon Red. The model was provided with fundamental memory capabilities, screen pixel input, and the ability to simulate button presses for in-game navigation.
This setup allowed the AI to play the game autonomously and continuously.
The Power of "Extended Thinking"
A key characteristic of Claude 3.7 Sonnet is its capacity for “extended thinking.” Similar to models like OpenAI’s o3-mini and DeepSeek’s R1, this model can tackle complex challenges by allocating additional computational resources and extending processing time.
Improved Performance Compared to Previous Versions
This capability proved beneficial during gameplay. While a prior iteration, Claude 3.0 Sonnet, was unable to progress beyond the initial town of Pallet, Claude 3.7 Sonnet successfully defeated three Pokémon gym leaders and obtained their respective badges.
Computational Effort and Future Exploration
The precise amount of computing power required for Claude 3.7 Sonnet to achieve these milestones, along with the time taken for each step, remains unspecified. Anthropic indicated that the model executed 35,000 actions to reach the final gym leader, Surge.
It is anticipated that developers will soon investigate and quantify these performance metrics.
Pokémon as a Benchmark – A Growing Trend
While Pokémon Red serves as a relatively simple benchmark, the use of games for AI evaluation has a notable history. Numerous new applications and platforms have emerged in recent months to assess models’ gaming skills across a variety of titles, including Street Fighter and Pictionary.
This demonstrates a growing interest in utilizing game environments for rigorous AI testing.
