Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework
#HUMAINE Framework #LLM evaluation #human preference #demographic awareness #AI bias #large language models #equitable AI
📌 Key Takeaways
- The HUMAINE Framework introduces a demographically aware evaluation method for LLMs.
- It aims to uncover how human preferences for LLMs vary across different demographic groups.
- The framework addresses biases in LLM performance and alignment with diverse user needs.
- This approach could lead to more equitable and effective AI systems tailored to varied populations.
📖 Full Retelling
arXiv:2603.04409v1 Announce Type: cross
Abstract: The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic co
🏷️ Themes
AI Evaluation, Demographic Bias
📚 Related People & Topics
Algorithmic bias
Technological phenomenon with social implications
Algorithmic bias describes systematic and repeatable harmful tendency in a computerized sociotechnical system to create "unfair" outcomes, such as "privileging" one category over another in ways that may or may not be different from the intended function of the algorithm. Bias can emerge from many f...
Entity Intersection Graph
No entity connections available yet for this article.
Mentioned Entities
Original Source
--> Computer Science > Computation and Language arXiv:2603.04409 [Submitted on 3 Feb 2026] Title: Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework Authors: Nora Petrova , Andrew Gordon , Enzo Blindow View a PDF of the paper titled Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework, by Nora Petrova and 1 other authors View PDF HTML Abstract: The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson model, with post-stratification to census data, and our analysis reveals three key insights. \textbf{(1)} We establish a clear performance hierarchy where \texttt{google/gemini-2.5-pro} ranks first overall, with a 95.6\% posterior probability of being the top-ranked model. \textbf{(2)} We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model's perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. \textbf{(3)} We quantify the vast difference in discriminative power across evaluation dimensions, with ambiguous qualities like \textit{Trust, Ethics \& Safety} showing a 65\% tie rate, in stark contrast to the decisive 10\% tie rate for \textit{Overall Winner}. Our work emphasises the need for a more multidimen...
Read full article at source