On Randomness in Agentic Evals

Researchers specializing in artificial intelligence evaluation published a technical paper on the arXiv preprint server this week challenging the reliability of current agentic system benchmarks. The study highlights significant statistical volatility in AI performance metrics after the team collected and analyzed 60,000 agentic trajectories on the SWE-Bench-Verified framework, involving three major large language models and two distinct scaffolds. The investigation was prompted by concerns that the industry standard of reporting 'pass@1' scores based on a single trial per task fails to account for the inherent randomness of autonomous agent behavior, potentially leading to inaccurate leaderboards and overstated progress in the field. The core of the research focused on the variance found when an agent attempts the same task multiple times. Conventionally, AI developers report a single-run success rate, assuming it serves as a stable proxy for the model's actual capability. However, the researchers discovered that these single-run estimates can fluctuate by as much as 2.2 to 6.0 percentage points solely due to chance. This margin of error is significant enough to rearrange the rankings of top-tier models, suggesting that many current performance claims in AI research may lack the statistical rigor necessary for objective comparison. To ensure a comprehensive analysis, the team utilized SWE-Bench-Verified, a rigorous benchmark designed to test an AI's ability to resolve real-world software engineering issues. By scaling their tests to tens of thousands of internal trajectories, the authors demonstrated that what often appears to be a 'superior' model or architecture might simply be the beneficiary of a lucky run. This findings advocate for a shift in how the AI community evaluates agentic systems, suggesting that multiple trials and confidence intervals should become the new standard for reporting results to ensure scientific validity and reproducibility.