Pipeline for Verifying LLM-Generated Mathematical Solutions

Researchers Varvara Sazonova, Dmitri Shmelkin, Stanislav Kikot, and Vasily Motolygin introduced a pipeline for verifying Large Reasoning Models' mathematical solutions on February 24, 2026, addressing the need for more accurate benchmarking as these AI systems increasingly solve complex mathematical problems. The paper, now available on arXiv under the category of Artificial Intelligence, presents a novel approach to evaluating the capabilities of Large Reasoning Models in mathematics. Unlike current methods that primarily check only the final answer, the new pipeline offers both automatic and interactive verification processes, providing a more comprehensive assessment of these AI systems' reasoning abilities. The researchers designed their system to generate correct solutions in both formal and informal languages, making it versatile for various mathematical contexts. The key innovation lies in the use of specialized prompts to guide LLMs to produce solutions in specific formats that can be more easily verified using proof assistants. This approach allows for the use of smaller models (with parameters of 8 billion or fewer) while maintaining verification accuracy. The pipeline incorporates three distinct AI agents that can be selected based on benchmarking requirements, giving researchers flexibility in their evaluation methods. According to the authors, experiments conducted across multiple datasets demonstrated a low probability of false positives, indicating high reliability in the verification process. The researchers have made their work openly accessible by providing an open-source implementation along with detailed instructions for setting up a server. This transparency enables other researchers to replicate and build upon their findings, potentially accelerating advancements in AI verification methodologies. As Large Reasoning Models continue to evolve and tackle increasingly complex mathematical problems, robust verification systems like this pipeline will become essential for accurately assessing their capabilities and limitations.