Speculative Decoding Speedup Calculator

Autoregressive decoding runs the large target model once per token, and each run is limited mostly by memory bandwidth, not arithmetic. Speculative decoding lets a small draft model propose gamma tokens, then the target model scores all of them in one parallel forward pass. Drafted tokens are accepted left to right with a rejection sampling rule that keeps the output distribution identical to the target model, so quality does not change.

Every step yields at least one token, because when a draft token is rejected the target model samples a corrected token in its place. When all gamma drafts are accepted, the step yields gamma + 1 tokens. The calculator turns three numbers into an expected speedup: the acceptance rate alpha, the draft length gamma, and the cost coefficient c, which is the time for one draft forward pass divided by the time for one target forward pass.

Acceptance rate. Log how many draft tokens are accepted per step on prompts like your production traffic. If a run of steps averages n accepted drafts plus the one bonus or corrected token, solve E = n + 1 for alpha, or use the fraction of accepted draft tokens as a first estimate. Alpha rises when the draft model is trained on the same data as the target, and it is usually higher for greedy decoding than for temperature 1 sampling. In the paper, a T5-small draft reached alpha = 0.75 at temperature 0 but 0.62 at temperature 1.

Cost coefficient. Time a single decoding forward pass of each model on the same hardware and divide draft by target. The paper estimated c = 0.02 for T5-small, 0.04 for T5-base and 0.11 for T5-large drafting for an 11B target. A trivial bigram draft has c close to 0 and still gave alpha of about 0.2, a 1.25x gain at gamma = 3.