Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

A research team has identified a significant self-preference bias in large language models when they are used as judges to evaluate other AI-generated text, a finding detailed in a paper uploaded to the arXiv preprint server on April 9, 2026. The study, the first to examine this bias within the specific context of rubric-based evaluation, reveals that LLMs consistently favor outputs produced by themselves or by models from their own architectural family, which systematically skews benchmark results and impedes genuine model development. The research addresses a critical flaw in the now-standard 'LLM-as-a-judge' paradigm, where AI models are tasked with scoring the quality of text generated by their peers. This method has become a cornerstone for benchmarking progress in the field due to its scalability and cost-effectiveness compared to human evaluation. However, the discovery of inherent self-preference bias suggests that many published performance improvements may be artificially inflated, as models could be scoring well simply because they are judging outputs similar to their own. This creates a misleading feedback loop, particularly dangerous in scenarios of recursive self-improvement where a model iteratively trains on its own highly-rated outputs. The study's focus on rubric-based evaluation—where models assess responses against a predefined set of criteria—is especially pertinent as this structured approach is gaining popularity for its perceived objectivity. The findings imply that even with detailed rubrics, the judge's inherent biases can corrupt the scoring process. This revelation calls for a major reassessment of current evaluation methodologies in AI research, urging the community to develop more robust, bias-aware benchmarking techniques or to recalibrate the heavy reliance on automated AI judges to ensure fair and accurate assessments of true model capabilities.