Speculative decoding

Speculative decoding is an inference speedup for autoregressive models that trades cheap extra computation for fewer expensive steps. The core idea: a small, fast draft model proposes several tokens ahead, then the large target model verifies them all in a single forward pass. Because verification runs in parallel over the proposed block, you get multiple tokens for roughly the cost of one target step, as long as the draft's guesses are accepted.

Think of it like branch prediction in a CPU, except the misprediction penalty is bounded: any accepted prefix is kept, and the first rejected token is resampled from the target's own distribution. Crucially, the output is provably identical in distribution to plain target-model sampling, so there is no quality loss.

Concretely, if the draft proposes a block of k = 5 tokens and the target accepts the first 3, you advance 3 tokens plus 1 corrected token in one target pass, then draft again from there. Higher acceptance rates mean bigger effective speedups; a poor draft that rarely agrees can even slow you down.

The block size k is the main knob. Too small wastes the parallel verification; too large wastes draft compute on tokens that will be rejected. Most systems fix k during training and reuse it everywhere, which is suboptimal because acceptance varies by prompt.

This is where instance-adaptive methods come in. BlockPilot, for diffusion-based speculative decoding, feeds the target model's last-token predictive distribution (available right after prefilling) into a two-layer MLP that picks a per-sample k from a small range around the trained default. A confident, low-entropy distribution suggests easy tokens ahead, so a larger block pays off.