The Detection--Extraction Gap: Models Know the Answer Before They Can Say It

A team of AI researchers has identified a significant inefficiency in modern reasoning models, revealing that large language models continue generating lengthy chain-of-thought explanations long after they have already determined the correct answer. According to a research paper published on arXiv on April 26, 2024, this phenomenon, termed the "detection-extraction gap," was observed across five model configurations, two model families, and three reasoning benchmarks, where 52-88% of chain-of-thought tokens were produced after the answer was already recoverable from a partial prefix of the model's own generation. The study demonstrates that if researchers interrupt a model's reasoning process early and ask it to freely continue from that partial prefix, it consistently recovers the correct final answer. This indicates that the model's internal reasoning process reaches a conclusion much earlier than its verbose textual output suggests. The detection-extraction gap represents the disparity between when a model first "knows" the answer (detection) and when it finally articulates that answer in its complete output (extraction), with substantial computational resources wasted on post-commitment generation that serves little functional purpose. This finding has important implications for AI efficiency and interpretability. The research suggests current chain-of-thought prompting, while valuable for making reasoning steps explicit, may be unnecessarily verbose and computationally expensive. Identifying this gap opens avenues for developing more efficient reasoning methods that could reduce inference costs and latency. Furthermore, understanding when models commit to answers could improve trust and verification systems, allowing developers to identify reasoning shortcuts or potential biases earlier in the generation process.