Speculative decoding: draft models, n-grams, MTP, and EAGLE

The problem: autoregressive decoding is slow

LLMs generate text 1 token at a time. Each token requires a full forward pass through the model — every weight, every layer — before the next token can begin. As How inference works explains, this decode phase is memory-bandwidth-bound: the GPU reads gigabytes of weights from memory to produce a single token, then does it again for the next one. The compute units sit idle most of the time, waiting for data to arrive. Your GPU is not doing useful work — it is waiting on memory. Speculative decoding exploits this idle capacity. Instead of generating 1 token per forward pass, it guesses several tokens ahead and verifies them all at once.

The core idea

Speculative decoding uses 2 models working together:

Draft model — a small, fast model proposes N tokens ahead (typically 4-8 tokens).
Target model — the large model you actually want to use verifies all N proposed tokens in a single forward pass.

If the target model agrees with K of the proposed tokens, you get K+1 tokens for the cost of 1 forward pass of the large model (plus 1 cheap forward pass of the small model). Wrong guesses are discarded and generation continues from the last accepted token. The math is simple: with a 75% acceptance rate and 5 draft tokens, you average roughly 2-3 tokens per large-model forward pass instead of 1. That translates directly into 2-3x faster generation.

Speculative decoding does not change output quality. The target model still makes every acceptance decision, so the output distribution is mathematically identical to normal autoregressive decoding.

N-gram speculative decoding

How it works

Instead of using a separate draft model, n-gram speculative decoding looks at the last N tokens in the current sequence, searches for matching patterns earlier in the context, and proposes the tokens that followed that pattern. It is pure pattern matching within the existing text.For example, if the last 4 tokens are def calculate_total( and the context earlier contains def calculate_total(items): return sum(, the engine proposes items, :, return, sum as the next 4 tokens. The target model then verifies them.

When to use it

Best for: code generation, structured output, repetitive text, templates, log parsing.Worst for: creative writing, diverse open-ended responses, short contexts with no repeating patterns.N-gram speculation shines when your output contains repeated structures — function signatures, JSON schemas, boilerplate code. It fails silently (low acceptance rate, falls back to normal decoding) when the text is too diverse for pattern matching.

Implementation

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model ngram \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 5 \
  --gpu-memory-utilization 0.90

./llama-server \
  --model models/llama-3.1-70b-q4_k_m.gguf \
  --draft ngram \
  --n-predict 128 \
  --lookup-cache-static

No extra model download, no extra VRAM. The only cost is CPU time spent searching the context for matches — negligible compared to GPU forward passes.

Draft model speculative decoding

How it works

A smaller model from the same family generates candidate tokens. For example, Llama-3.1-8B drafts for Llama-3.1-70B. The small model runs its own autoregressive generation to propose 4-8 tokens, then the large model verifies them all in 1 forward pass.The acceptance rate — the fraction of draft tokens the target model agrees with — typically falls between 70% and 85% for well-matched model pairs. Higher acceptance means more speedup.

When to use it

Best for: general-purpose generation when you have VRAM headroom for the smaller model.Memory cost: both models must be loaded simultaneously. An 8B draft model adds roughly 4-5 GB (at Q4 quantization) on top of the target model. Plan your model fit accordingly.Speedup: 1.5-2.5x depending on acceptance rate and draft token count. Diminishing returns past 6-8 draft tokens because acceptance drops for later tokens in the sequence.

Implementation

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --gpu-memory-utilization 0.90

./llama-server \
  --model models/llama-3.1-70b-q4_k_m.gguf \
  --draft models/llama-3.1-8b-q4_k_m.gguf \
  --n-predict 128

Use the same model family for draft and target. Cross-family pairs (for example, Mistral drafting for Llama) have much lower acceptance rates because their token distributions diverge.

Multi-token prediction (MTP)

How it works

Some model architectures are trained to predict multiple future tokens simultaneously from a single forward pass. The model itself produces several candidate tokens at once — no separate draft model, no pattern matching. The model’s own internal representations generate the speculative tokens.This is fundamentally different from external speculation: the multi-token heads are trained jointly with the base model, so they share the same understanding of context.

When to use it

Best for: models that natively support it. There is no configuration choice — if the model was trained with MTP heads, you enable them; if not, you cannot add them.Models with MTP support: DeepSeek-V3, DeepSeek-R1, and select Qwen3 variants.Advantage: zero extra memory, zero quality loss, no model pair tuning.Disadvantage: requires specific model training. You cannot apply MTP to an arbitrary checkpoint.

Implementation

vllm serve deepseek-ai/DeepSeek-V3 \
  --enable-mtp \
  --num-speculative-tokens 1

The --num-speculative-tokens flag controls how many MTP heads to chain. Most MTP-trained models support 1 additional token per head; chaining heads gives diminishing returns past 2-3.

EAGLE (extrapolation algorithm for greater language-model efficiency)

How it works

EAGLE trains a small “head” network on top of the target model’s hidden states. Instead of loading a full draft model, it attaches a lightweight neural network to the intermediate representations of the target model and uses those to predict future tokens.Because the head network reads the target model’s own internal state, it achieves higher acceptance rates (80-90%) than naive draft models. The head network is tiny compared to a full model — typically 5-10% of the target model’s size.

When to use it

Best for: when available, EAGLE offers the best speedup-to-memory ratio of any technique.Current status: research stage with limited runtime support. vLLM and SGLang have experimental EAGLE integration, but it requires pre-trained EAGLE head checkpoints that are not yet widely published for most model families.When EAGLE support matures, expect: 2-3x speedup with only 5-10% memory overhead — significantly better than full draft models.

Implementation

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model yuhuili/EAGLE-LLaMA3.1-Instruct-70B \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1

EAGLE head checkpoints are model-specific and architecture-specific. Verify that a published EAGLE checkpoint exists for your exact target model before attempting to use this technique.

Comparison

N-gram

Extra memory: none Speedup: 1.2-1.5x Requires training: no Runtime support: vLLM, llama.cpp Best for: repetitive content, code, structured output

Draft model

Extra memory: 20-30% extra Speedup: 1.5-2.5x Requires training: no (use a smaller model from the same family) Runtime support: vLLM, llama.cpp Best for: general-purpose generation with VRAM headroom

Multi-token prediction

Extra memory: none Speedup: 1.5-2x Requires training: yes (model-specific) Runtime support: vLLM (select models only) Best for: DeepSeek-V3, supported Qwen3 variants

EAGLE

Extra memory: 5-10% Speedup: 2-3x Requires training: yes (head network) Runtime support: experimental Best for: best speedup-to-memory ratio when available

Technique	Extra memory	Typical speedup	Requires training	Runtime support	Best for
N-gram	None	1.2-1.5x	No	vLLM, llama.cpp	Repetitive content
Draft model	20-30%	1.5-2.5x	No	vLLM, llama.cpp	General use
MTP	None	1.5-2x	Yes (model-specific)	vLLM (select models)	Supported models only
EAGLE	5-10%	2-3x	Yes (head network)	Experimental	When available

When to use speculative decoding

Use it when:

Your workload is decode-bound — long generations where most time is spent producing tokens, not processing the prompt
You have VRAM headroom for a draft model (at least 5-10 GB free beyond the target model)
You generate structured or repetitive output — code, templates, JSON, logs
You serve a single user or low-concurrency workload

Do not use it when:

Your workload is prefill-bound — short outputs with long inputs (summarization, classification). Speculative decoding only accelerates the decode phase.
You are already at your VRAM limit. The draft model or extra KV cache entries may push you into OOM. See DGX Spark memory tuning for capacity planning.
You use continuous batching with high concurrency. Speculative decoding conflicts with efficient batch scheduling in most runtimes — the verification pass assumes a fixed draft length per request, which complicates batch composition.

Practical setup guide

N-gram speculation in vLLM

The simplest starting point — no extra downloads, no extra memory:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model ngram \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 5 \
  --gpu-memory-utilization 0.90

The --ngram-prompt-lookup-max flag sets the window size for pattern matching. Larger values find more matches in long contexts but slow the draft step slightly. Start with 5 and increase if acceptance is low.

Draft model in vLLM

For higher speedup when you have the VRAM:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Reduce --max-model-len if you hit OOM. Both models share the same VRAM pool, and the draft model’s KV cache adds to the total.

N-gram speculation in llama.cpp

./llama-server \
  --model models/llama-3.1-70b-q4_k_m.gguf \
  --draft ngram \
  --n-predict 128 \
  --lookup-cache-static \
  --port 8080

The --lookup-cache-static flag caches n-gram lookups across tokens, improving draft speed for repetitive patterns.

How to measure the speedup

Run the same prompt twice — once with speculative decoding and once without — and compare tokens per second:

# Baseline (no speculation)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 &

# Send a test request and note tok/s in the response headers
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Write a Python class that implements a binary search tree with insert, delete, and search methods."}],
    "max_tokens": 512
  }'

Stop the baseline server, start the speculative server, send the same request, and compare. Look for the x-tokens-per-second header or calculate from usage.completion_tokens divided by wall-clock time. For systematic benchmarking, use vLLM’s built-in benchmark tool:

python -m vllm.benchmarks.benchmark_serving \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --num-prompts 10 \
  --request-rate 1

A well-tuned speculative setup shows 1.5-2.5x improvement in mean decode latency with no change in output quality or token count.

Go deeper

Decode and prefill power

Tune prefill and decode phases for maximum throughput on your hardware.

How inference works

Understand prefill, decode, and why memory bandwidth is the bottleneck.

vLLM on DGX Spark

Configure vLLM with optimal settings including speculative decoding support.

Model fit

Calculate whether your target model plus a draft model fit in VRAM.

Benchmark your deployment

Systematic benchmarking methodology for local LLM deployments.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Speculative decoding: draft models, n-grams, MTP, and EAGLE

The problem: autoregressive decoding is slow

The core idea

N-gram speculative decoding

Draft model speculative decoding

Multi-token prediction (MTP)

EAGLE (extrapolation algorithm for greater language-model efficiency)

Comparison

N-gram

Draft model

Multi-token prediction

EAGLE

When to use speculative decoding

Practical setup guide

N-gram speculation in vLLM

Draft model in vLLM

N-gram speculation in llama.cpp

How to measure the speedup

Go deeper

Decode and prefill power

How inference works

vLLM on DGX Spark

Model fit

Benchmark your deployment

​The problem: autoregressive decoding is slow

​The core idea

​N-gram speculative decoding

​Draft model speculative decoding

​Multi-token prediction (MTP)

​EAGLE (extrapolation algorithm for greater language-model efficiency)

​Comparison

N-gram

Draft model

Multi-token prediction

EAGLE

​When to use speculative decoding

​Practical setup guide

​N-gram speculation in vLLM

​Draft model in vLLM

​N-gram speculation in llama.cpp

​How to measure the speedup

​Go deeper

Decode and prefill power

How inference works

vLLM on DGX Spark

Model fit

Benchmark your deployment

The problem: autoregressive decoding is slow

The core idea

N-gram speculative decoding

Draft model speculative decoding

Multi-token prediction (MTP)

EAGLE (extrapolation algorithm for greater language-model efficiency)

Comparison

When to use speculative decoding

Practical setup guide

N-gram speculation in vLLM

Draft model in vLLM

N-gram speculation in llama.cpp

How to measure the speedup

Go deeper