Mathematical reasoning is often considered one of the most critical abilities of foundational AI models and serves as a proxy for general problem-solving. Over the past few years, we have witnessed large language models (LLMs) push the boundaries of math benchmarks, scoring competitively on International Math Olympiad (IMO) problems and advancing discoveries in various areas of mathematics. From this perspective, it might seem as though LLMs are inching towards “super math powers,” but that is not entirely the case.

Much of AI’s impressive performance in math benchmarks relies on scenarios where the problem is perfectly articulated within a prompt. However, most foundational models struggle when they need to combine different ideas creatively or use “common sense” to structure and solve a problem. Can we develop benchmarks that measure these deeper reasoning capabilities?

FrontierMath is a newly developed benchmark specifically designed to gauge the capabilities of AI systems in tackling complex mathematical problems. The hallmark of this benchmark lies in its exceptional difficulty, encompassing problems that typically require hours or even days of effort for expert mathematicians to solve. This stands in stark contrast to pre-existing mathematical benchmarks like GSM8K and MATH, which largely focus on elementary to undergraduate-level problems and are approaching saturation in terms of AI performance...