AI Benchmarking Reaches Pokémon - A New Debate

AI Benchmarking and the Pokémon Controversy

The realm of artificial intelligence isn't immune to debate, even when it comes to seemingly lighthearted tests. Recent discussions have arisen concerning the benchmarking of AI models, specifically involving the classic Pokémon video games.

Gemini and Claude: A Viral Claim

A post circulating on X (formerly Twitter) gained significant attention, suggesting that Google’s Gemini model outperformed Anthropic’s Claude model in navigating the initial Pokémon trilogy. The claim indicated that Gemini had progressed to Lavender Town during a live Twitch stream, while Claude remained stalled at Mount Moon as of February’s end.

However, a crucial detail was omitted from the initial report.

The Advantage Given to Gemini

Reddit users quickly identified that the developer responsible for the Gemini stream had implemented a custom minimap. This tool aided the model in recognizing specific game elements, such as trees that could be cut down. Consequently, Gemini required less intensive screenshot analysis before executing gameplay actions.

Pokémon as a Benchmark: Limitations

While Pokémon has emerged as a benchmark for AI, its value as a truly informative test of a model’s capabilities is debatable. Nevertheless, it serves as a valuable illustration of how variations in benchmark implementation can skew results.

SWE-bench Verified and Anthropic's Sonnet

Anthropic itself demonstrated this principle with its Anthropic 3.7 Sonnet model and the SWE-bench Verified benchmark, designed to assess coding proficiency. The model initially achieved 62.3% accuracy.

However, with the addition of a “custom scaffold” developed by Anthropic, the accuracy increased to 70.3%.

Meta's Llama 4 Maverick and LM Arena

Further evidence of this trend comes from Meta, which fine-tuned a version of its Llama 4 Maverick model specifically to excel on the LM Arena benchmark. The standard version of the model yielded a considerably lower score on the same evaluation.

The Growing Challenge of Comparison

Considering that AI benchmarks, including Pokémon, are inherently imperfect, the introduction of custom and non-standard implementations further complicates the process of accurate evaluation.

As a result, comparing the performance of different AI models as they are released is likely to become increasingly difficult.

• Key takeaway: Benchmark implementations significantly impact results.
• Implication: Direct model comparisons are becoming more challenging.