EAGLE-4 Technical Deepdive

In collaboration with Fireworks and fal, we are releasing a technical analysis of EAGLE-4 alongside a public dataset for benchmarking speculative decoding on creative tasks.

Why Speculative Decoding?

LLMs generate text auto-regressively, predicting one token at a time. At small batch sizes, decoding relies on "thin" matrix multiplications that underutilize GPU compute because they are limited by memory bandwidth. Batching requests from multiple users turns these operations into larger matrix multiplications and improves overall throughput, but it can reduce the token generation rate experienced by each user.

Speculative decoding addresses this need for faster responses by using a lightweight model to draft several tokens, which the target model then verifies together. Verifying multiple tokens increases the amount of computation performed for each read of the model weights, making better use of otherwise idle GPU compute. With the appropriate verification procedure, speculative decoding preserves the target model's output distribution.

The EAGLE family is one approach to speculative decoding used in production. Its lightweight transformer drafter takes hidden states and token embeddings from the target model and is trained to predict the target model's output tokens. EAGLE-3 uses hidden states from three layers at different depths, combined with training techniques designed to improve acceptance length.

In the year since EAGLE-3 was released, non-autoregressive approaches such as DFlash and DSpark have also emerged. These methods use masked diffusion to draft multiple tokens in parallel. This can make larger drafters practical, because drafting does not require a separate sequential step for every predicted token.

Why Image / Video models need fast decoding

Many recent image and video generation systems process the user's prompt before passing it to the generative model. This step, called "prompt enhancement" or "prompt expansion," turns a brief request into a more detailed scene description. It also adapts the prompt to the target model's expected format, helping produce more appealing results.

As image and video models become faster, prompt enhancement can account for a larger share of end-to-end latency. For example video models can generate videos realtime, but expansion stage without any optimizations would take more time than the video generation itself. So these systems need fast responses at small batch sizes to keep the experience interactive, making prompt enhancement a useful application for speculative decoding.

To bring these creative workloads into speculative decoding evaluation, we are releasing Prompt Enhancement SpecBench. It contains 400 examples drawn from 18 unique datasets across four categories: text-to-video, image-to-video, text-to-image, and image-to-image. Each category includes 100 examples.

EAGLE-4 architecture

EAGLE-4 builds on EAGLE-3's approach: a lightweight drafter combines the target model's hidden states with token embeddings to predict possible continuations. Its effectiveness depends on both the quality of those predictions and how cheaply it can produce them.

The released Qwen 3.8 27B model makes two changes to this drafting path. Sliding-window attention bounds the context used by the drafter's attention layer. A low-rank candidate head first scores the vocabulary in a smaller representation, then computes full logits only for the selected candidates. These changes target the cost of attention and next-token prediction, respectively.

The diagram below shows the broader architecture, including optional components. Multiple draft layers, mixture-of-experts (MoE) support, and Latent ReLU are not yet included in the released model. We expect future models to use these features.

EAGLE-4 verification trees and harness

The architecture affects the drafter prediction; the verification tree expands which predictions the target model checks. A single chain commits to one continuation at each position, so an early mismatch prevents later tokens in that chain from being accepted. Branching gives the verifier alternative continuations to consider, which is useful when several next tokens are plausible.

EAGLE represents these alternatives as a tree and evaluates selected nodes in a single target-model forward pass. The example below shows how verification follows an accepted path, then continues with a token from the target model when the draft no longer matches.

To evaluate these branches together, the target model needs to keep their contexts separate. A custom attention mask lets each token attend to the shared prefix, its ancestors, and itself, while hiding sibling tokens and other branches. This preserves the context of each candidate continuation within the combined forward pass.

A larger tree offers more alternatives, but also costs more to draft and verify. EAGLE-4's harness manages this tradeoff using latency measurements from the GPU. It expands the tree level by level, decides when to stop drafting, and selects the candidate nodes to verify based on the verification cost. After verification, it keeps the accepted continuation, frees rejected KV-cache entries, and starts the next round.

Before serving, a one-time offline calibration on the target GPU type and parallelism builds a cost table: verify latency across batch size, context length and number of candidates. At each draft level the harness estimates the cumulative-probability gain divided by the measured marginal cost per token. It widens or deepens the tree while that ratio stays above a threshold, then stops. The top candidates are verified in one target forward pass with a tree attention mask, and rejection sampling accepts the longest matching prefix, so the output distribution is unchanged. The result is a wide, deep tree at batch size 1 or long context, and a narrow tree or plain chain at large batch, from one drafter with no fixed depth or top-k settings.

Benchmarks

All experiments were run on NVIDIA B300 GPUs with tensor parallelism set to 4 (TP=4).

Evaluation datasets

We evaluated EAGLE-4 on two curated datasets:

• fal prompt expansion dataset: 400 prompts across text-to-video, text-to-image, image-to-video, and image-to-image, with 100 prompts per category.
• Fireworks curated speculative decoding dataset: chat-30, code-30, stem-30, math-30, and coding-long-20. Of the 20 long coding prompts, 18 were usable, giving 138 prompts in total.

The chain-versus-tree comparisons below use all 138 Fireworks prompts and the 200 text-only fal prompts. We report acceptance length (AL) to examine drafting behavior, followed by decoding throughput to show the runtime effect.

Acceptance length and workload difficulty

Creative prompt expansion involves open-ended continuations that can be difficult for a drafter to predict. The entropy analysis below shows lower acceptance length for prompts in the higher-entropy groups: 2.66 in the highest-entropy quartile, compared with 2.97 in the lowest-entropy quartile.

Tree gains across tasks

At the configurations shown below, the tree improves acceptance length across every task split. The aggregate relative gain is 9.6% on the Fireworks dataset and 12.4% on the fal text-only subset.

Scaling the verification budget

At every tested verification budget, both methods have lower acceptance lengths on the fal creative prompts than on the Fireworks dataset as expected. These workloads are harder for the drafters in this evaluation, and a larger verification budget improves acceptance on both datasets.

The tree's advantage grows as the verification budget increases. Both methods improve, but the chain shows diminishing returns, particularly on the fal dataset, while the tree continues to gain acceptance length.

At a verification budget of 16 tokens, the largest tested, the tree improves acceptance length by 21.1% on the Fireworks dataset and 26.1% on the fal text-only subset relative to the chain.

Where the tree gains acceptance

The per-position hit rates help explain these gains. The following chart compares the tree at L=8 with the chain at N=7 on the Fireworks dataset. Each value is the probability that the drafted token at that position is accepted along the path.

The tree's advantage is concentrated in the early positions. On the Fireworks dataset, the first-token hit rate rises from 72.5% to 85.3%, an increase of 12.8 percentage points. Exploring alternative continuations near the start of the tree is a likely explanation.

At a fixed verification budget, branching near the start can improve overall acceptance length even if fewer deep tokens are accepted. A long chain spends more of its budget on later positions that are less likely to be reached.

Decoding throughput

Acceptance length helps explain drafting behavior, while decoding throughput captures the combined cost of drafting and verification. We compare DSpark, DFlash2, the EAGLE-4 static chain, and EAGLE-4 Harness at L=8 on Fireworks Bench and Prompt Enhancement SpecBench, at C=1 and C=8.

EAGLE-4 Harness has the highest throughput in all four comparisons. At C=1, it reaches 660.6 tokens per second on Fireworks Bench and 622.9 on Specbench, or 1.63× and 1.85× the throughput of the fastest alternative in each benchmark. At C=8, it reaches 3566.4 and 3551.9 tokens per second, respectively, with corresponding advantages of 1.36× and 1.62×.

The advantage is larger on Specbench at both settings. At C=8, the Harness achieves nearly the same throughput on Specbench as on Fireworks Bench, while the other methods show a larger gap between the two workloads.

Increasing the tree verification budget

The earlier throughput experiment also tested L=16. In that comparison, increasing the tree budget from L=8 to L=16 raised throughput from 660.1 to 752 tokens per second, with a derived speedup of 3.77× over decoding without speculation.

Conclusion

EAGLE-4 shows how an efficient drafter, branching verification, and a harness that accounts for hardware costs can work together to improve decoding throughput. In our evaluation, the Harness leads across both benchmarks at C=1 and C=8. Its advantage is especially strong on creative prompt enhancement: on SpecBench, it delivers 1.85× the throughput of the fastest alternative at C=1 and 1.62× at C=8.

Creative workloads remain harder for drafters to predict, but the tree recovers more useful continuations from that uncertainty. As image and video generation gets faster, reducing the time spent on prompt enhancement becomes increasingly valuable. We are releasing Prompt Enhancement SpecBench to help make these workloads a regular part of speculative decoding evaluation.

Thank you to our collaborators at Fireworks for working with us on this analysis and evaluation.