Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance

Researchers specializing in Large Language Models (LLMs) introduced a novel framework called Variational Speculative Decoding (VSD) in a study released on the arXiv preprint server on February 10, 2025, to bridge the efficiency gap in speculative decoding. This new methodology addresses the persistent 'training-decoding discrepancy' by shifting the focus from individual token likelihood toward the optimization of entire sequence acceptance. By formulating draft training as a process of variational inference, the team aims to accelerate inference speeds of advanced models significantly more effectively than traditional methods allow. At the core of this breakthrough is the recognition that conventional speculative decoding techniques often fail to reach their full potential because they optimize for single, greedy trajectories. In real-world scenarios, however, decoding involves the verification and ranking of multiple sampled draft paths simultaneously. The researchers argue that traditional training objectives do not align with the actual behavior of the target model during the verification phase, leading to suboptimal performance in high-stakes computational environments. To solve this, the VSD approach treats draft paths as latent variables within a variational framework. Instead of merely predicting the next likely token, the auxiliary 'draft' model is trained to maximize the marginal probability that its proposed sequences will be accepted by the larger, more powerful target model. This shift from token-level accuracy to sequence-level acceptance ensures that the draft model serves as a more reliable proxy, thereby reducing the computational overhead and increasing the overall throughput of generative AI tasks.