Why the bandit can't win, and a free signal can
A small draft model guesses several tokens at once. The big target model checks them all in one pass. So the question is simple: how many tokens should you draft each round? This demo shows why a bandit that "learns" the answer hits a ceiling. It also shows why a free signal, read before you draft, breaks through it.
The world: some rounds are easy, some are hard
Some spots are easy for the draft model. On code syntax, numbers and boilerplate, its guesses land about 90% of the time. Other spots ask for meaning, like names and turns in reasoning. There it drops to about 30%. Each drafted token that gets thrown out iswasted compute. Each one that is kept isfree speed.
Three players, one race
Fixed always drafts 8 tokens. Bandit tries different counts. It learns which one pays best on average. Adaptive peeks at onefree signal first. That signal is the draft model's own confidence, which it works out before it drafts a thing. Adaptive then drafts long when the model is sure, and short when it is not.
Efficiency is kept tokens per unit of draft and check work. Higher means faster decoding for the same output. The text is bit-identical in all three lanes.
Watch what the bandit learns
The bandit's estimate for each arm settles on k = 8, the best fixed count. That is the ceiling. A bandit gets its reward after the round. So all it can learn is which count is best on average. It has no way to know that this round is a hard one.
The bandit's estimated payoff per arm. Run it and watch every arm settle, with k = 8 on top. That is the fixed baseline, exactly.
The idea in one line: a bandit tunes the average round. The free signal lets you tune each round. What you know before the choice beats any amount of learning that shows up after it.
Why "cheaper" matters
The confidence signal costs nothing. The draft model already gives you its own token odds as a by-product of drafting. In the study, these free signals predict acceptance at about 0.87 AUROC. Probing the draft model's own hidden states adds at most +0.006 on top of them. So for the drafter, you do not need to open the model up. The cheap signal it already emits is enough. (The full study finds the opposite on the target side, where a probe read before the round does add real signal.)
On real traces this rule beat the best fixed count by+11.2% efficiency. It also cut wasted draft tokens by62%. That is over 302k tokens and 322 held-out prompts. And the output text stays identical to what the target model would write alone.Full study →