Scalable Delphi: Large Language Models for Structured Risk Estimation

Researchers specializing in artificial intelligence published a paper titled "Scalable Delphi: Large Language Models for Structured Risk Estimation" on the arXiv preprint server on February 14, 2025, to determine whether AI can replace human specialists in high-stakes risk assessment. The study addresses the logistical bottlenecks of the traditional Delphi method, which currently requires months of coordination among human experts to estimate unobservable risks in sectors like finance, public health, and security. By automating the expert elicitation process, the team aims to make rigorous, auditable risk analysis accessible to a broader range of applications that cannot afford the high costs of human-led studies. The core of the research focuses on the Delphi method, long considered the gold standard for reaching consensus among experts. Traditionally, this process involves multiple rounds of surveys where specialists provide independent estimates and then refine them after seeing anonymized feedback from their peers. While highly effective, this method is notoriously difficult to scale because of the intensive time demands placed on human specialists. The researchers propose utilizing Large Language Models (LLMs) as scalable proxies, mimicking the iterative reasoning and collaborative refinement of the Delphi process at a fraction of the time and cost. By leveraging the vast datasets and reasoning capabilities of current LLMs, the study investigates whether AI can produce calibrated and structured risk estimates that match the quality of human experts. If successful, this approach could revolutionize how organizations manage uncertainty in high-stakes environments. The integration of AI into this domain would allow for real-time risk modeling, enabling quick responses to emerging threats while maintaining the auditability and structure required for institutional accountability.