MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

A team of AI researchers has introduced MedDialBench, a new benchmark designed to rigorously test the diagnostic robustness of Large Language Models (LLMs) when interacting with adversarial patient behaviors, as detailed in a recent paper published on the arXiv preprint server. This development, announced on April 26, 2024, addresses a critical gap in existing medical AI evaluation, where current benchmarks fail to systematically measure how LLM performance degrades when patients are uncooperative, misleading, or provide incomplete information. The core motivation is to create a more realistic and controlled testing environment that moves beyond simplistic pass/fail assessments. Current interactive medical dialogue benchmarks have demonstrated a significant drop in LLM diagnostic accuracy when models converse with non-cooperative patients. However, prior approaches have been limited. Some apply adversarial behaviors—like a patient being evasive or providing contradictory symptoms—without considering varying levels of severity or grounding them in specific medical cases. Others oversimplify the challenge by reducing patient non-cooperation to a single, ungraded axis, such as general unhelpfulness. Crucially, no existing framework analyzes the complex interactions between different dimensions of adversarial behavior, such as how being verbose might compound the effects of being inaccurate. MedDialBench is engineered to overcome these limitations by enabling a "dose-response" characterization of LLM robustness. This means it can systematically vary the intensity or "dose" of specific adversarial traits—like the degree of symptom omission or the level of contradictory information—and measure the corresponding "response" in the model's diagnostic accuracy. The benchmark provides case-specific grounding, ensuring that adversarial behaviors are tested within the context of realistic clinical scenarios. By offering this granular, multi-dimensional analysis, MedDialBench aims to provide developers and healthcare AI evaluators with a powerful tool to identify specific weaknesses in LLMs, ultimately guiding the creation of more reliable and trustworthy diagnostic assistants capable of handling the complexities of real-world patient interactions.