DeepMind AI Outperforms Math Olympiad Gold Medalists

Google DeepMind’s AI Achieves Gold Medal-Level Geometry Problem Solving

A newly developed artificial intelligence system originating from Google DeepMind, the company’s dedicated AI research division, has demonstrated performance exceeding that of typical gold medalists in international geometry competitions.

AlphaGeometry2: An Advancement in AI Problem Solving

This system, known as AlphaGeometry2, represents a significant upgrade from its predecessor, AlphaGeometry, which was initially released in January of the previous year. Researchers at DeepMind assert, in a recently published study, that their AI is capable of successfully solving 84% of geometry problems featured in the International Mathematical Olympiad (IMO) over the past 25 years.

The Significance of Geometry for AI Development

The IMO is a prestigious mathematics contest designed for high school students. DeepMind’s focus on this level of mathematical challenge stems from the belief that breakthroughs in solving complex geometry problems – particularly those within Euclidean geometry – may unlock pathways to more advanced and versatile AI capabilities.

The ability to prove mathematical theorems, requiring both logical reasoning and strategic decision-making, could prove to be a valuable asset in the development of future general-purpose AI models.

Combining AI Models for Enhanced Performance

This past summer, DeepMind showcased a system integrating AlphaGeometry2 with AlphaProof, an AI model specializing in formal mathematical reasoning. This combined system successfully solved four out of six problems presented at the 2024 IMO. The principles behind these approaches could potentially be applied to other scientific and mathematical disciplines, such as aiding in intricate engineering calculations.

Core Components of AlphaGeometry2

AlphaGeometry2’s architecture incorporates several key elements. These include a language model derived from Google’s Gemini family of AI models, and a dedicated “symbolic engine.” The Gemini model assists the symbolic engine, which utilizes established mathematical rules to deduce solutions, in formulating viable proofs for geometry theorems.

Constructs and Deductions in Geometry

Geometry problems presented at the Olympiad level often require the addition of “constructs” – such as points, lines, or circles – to the diagrams before a solution can be found. AlphaGeometry2’s Gemini model predicts which constructs would be most beneficial to add, providing information that the symbolic engine uses to draw logical conclusions.

Essentially, the Gemini model proposes steps and constructions using a formal mathematical language, which the engine then validates for logical consistency based on predefined rules. A search algorithm enables AlphaGeometry2 to explore multiple solution paths concurrently and store potentially useful information in a shared knowledge base.

Defining a “Solved” Problem

AlphaGeometry2 considers a problem solved when it generates a proof that integrates suggestions from the Gemini model with the established principles of the symbolic engine.

Addressing the Challenge of Training Data

The scarcity of readily available training data for geometry proofs presents a challenge. To overcome this, DeepMind created its own synthetic data set to train AlphaGeometry2’s language model, generating over 300 million theorems and proofs of varying complexity.

Performance Evaluation and Results

The DeepMind team selected 45 geometry problems from IMO competitions spanning the years 2000 to 2024, including those involving linear equations and geometric transformations. These were expanded into a set of 50 problems for testing purposes.

AlphaGeometry2 successfully solved 42 of these 50 problems, surpassing the average score of 40.9 achieved by gold medalists.

Limitations and Future Improvements

Despite its success, AlphaGeometry2 has certain limitations. It currently struggles with problems involving a variable number of points, nonlinear equations, and inequalities. Furthermore, it is not the first AI system to achieve gold-medal-level performance in geometry, although it is the first to do so with a problem set of this magnitude.

Performance on a more challenging set of IMO problems was also lower. The team selected 29 problems nominated for IMO exams but not yet used in competition. AlphaGeometry2 solved only 20 of these.

The Debate: Symbolic Manipulation vs. Neural Networks

These results are expected to contribute to the ongoing discussion regarding the optimal approach to AI development: whether to prioritize symbol manipulation – using rules to process knowledge – or neural networks, which mimic the structure of the human brain.

AlphaGeometry2 employs a hybrid approach, combining the neural network architecture of the Gemini model with the rules-based symbolic engine.

The Strengths of Each Approach

Advocates of neural networks contend that intelligent behavior can emerge solely from vast amounts of data and computational power. In contrast, symbolic systems define specific rules for dedicated tasks. Neural networks learn through statistical approximation and examples.

Neural networks are central to powerful AI systems like OpenAI’s o1 model. However, proponents of symbolic AI argue that it may be more effective for encoding knowledge, reasoning through complex scenarios, and providing explanations for its conclusions.

Expert Commentary and Future Outlook

Vince Conitzer, a computer science professor at Carnegie Mellon University, stated to TechCrunch, “It is striking to see the contrast between continuing, spectacular progress on these kinds of benchmarks, and meanwhile, language models, including more recent ones with ‘reasoning,’ continuing to struggle with some simple commonsense problems.” He emphasized the need for a deeper understanding of these systems and their potential risks.

AlphaGeometry2 suggests that a combined approach – integrating symbol manipulation and neural networks – may be a promising path toward achieving more generalizable AI.

Potential for Self-Sufficiency

The DeepMind paper indicates preliminary evidence that AlphaGeometry2’s language model can generate partial solutions independently of the symbolic engine.

However, the team notes that improvements in model speed and the resolution of “hallucinations” (generating incorrect or nonsensical information) are necessary before the language model can function entirely independently in mathematical applications.