How We Achieved ~1000 tok/s and 16x Throughput with DSpark for Ideogram V4 Prompt Expander

At @fal , we've achieved 16 times higher throughput on Qwen3.6 for a use-case that required high interactivity per user using DSpark on SGLang. But why did we need it? And how did we achieve it?

Intro

Text-to-image models like Ideogram V4 can generate stunning, highly detailed images. However they're only as good as the prompt you give them, and most users don't write detailed prompts. If a given prompt is not detailed enough, it might not look as impressive to the user. That's why a feature called the prompt expander exists to enhance the user's prompt. The prompt expansion applies the LLM's world knowledge to the image model by describing the scene from the user's prompt in detail, with hints about the style and where everything is placed. This feature also makes the model very precise about where everything is placed, which lets users carefully edit any part of the scene, since the model can easily follow those instructions.

However, this prompt expansion stage is slow, especially compared to the time it takes for distilled models to generate images. The bottleneck might quickly become the prompt expansion rather than the image generation itself. We generate around 600 tokens to expand the prompt and describe the scene in detail. If you are using some API provider, it might take anywhere from 15 seconds up to a minute, whereas your image model only takes a couple of seconds to run.

Model Choice

First of all, choosing a small model is not ideal; the prompt expansion stage is sensitive to the model's world knowledge. We have experimented with Qwen 3.5 4B model, and its output was significantly worse. On the other hand, a big model can create very good looking images; however, running it is more expensive, and getting high throughput is challenging. Unlike image models, a big model might require 4 or even 8 GPUs with significant VRAM to run.

We identified a big model, Qwen 3.5 397B MoE, as our baseline. We compared it to 0.8B, 4B, 9B, and 27B dense variants and a 35B MoE variant. Despite the 35B model having only 3B active parameters, it outperformed them all, which we believe comes from its larger world knowledge.

One thing to note is that size is not linear with serving speed. LLM inference is heavily dominated by the memory bandwidth bottleneck. Serving a 0.8B model on a B200 doesn't guarantee 10x throughput over the 9B version. What we found is that these models are almost equally slow at low concurrency settings across all inference engines, which is where we aim to serve in order to give the most interactivity per user. This led us to the decision to use the 35B MoE model (FP8), which we found to be the sweet spot for inference speed without sacrificing significant image performance.

Images generated by the smaller model had some issues; it did not carry the same artistic style the bigger model was able to achieve. It was also making formatting errors with the JSON output of the expanded prompt, despite the few-shot examples. So next, we distilled our 397B model's output style and format onto our 35B version. We tried online distillation, offline distillation, and PEFT. Distillation took a lot longer and required more data; however, we found that all of them resulted in a similar model. We stuck to PEFT since it was much easier and faster to fine-tune. We froze the MoE weights during fine-tuning, as we found that training them creates more instability: the model sometimes tries to do more ambitious things beyond its capability, breaking the coherency in the image.

How to Serve Faster?

Since we now have our 35B model, the question is: how do we serve it efficiently? We can't rely on other providers since we have fine-tuned it. Also, prompt expansion requires high interactivity, so we aimed to get maximum speed at low concurrency. We tested both SGLang and vLLM, and we were only able to achieve 328 tok/s at concurrency=1 with SGLang's FlashInfer backend and FlashInfer + TRTLLM MoE kernels. This allowed us to get sub-2-second prompt expansion. However, if we increased concurrency to 8, our throughput dropped to <200 tok/s, and at 16 and 32 we were getting near ~100 tok/s, which is not fast enough for a highly interactive use case.

The Qwen 3.6 family comes with an MTP head, so the first natural thing to do was to enable this MTP head. Unfortunately, the PEFT framework did not fine-tune MTP heads, so our accuracy was slightly degraded; however, acceptance length was pretty close to the base model. We were able to increase our throughput to 500 tok/s by enabling speculative decoding using the MTP head with around ~2.5 acceptance length.

This wasn't enough, so we looked for other solutions such as DFlash, a diffusion-based speculative decoding model that predicts each token in parallel. It can get slightly better acceptance than MTP, and drafting overhead is lower for low batch size serving. However, we were only able to reach 468 tok/s with ~2.4 acceptance length using z-lab's checkpoints. This checkpoint was trained for the base model, so we expected lower than ideal acceptance, similar to the MTP head. The next natural step was to train our own speculative decoding model, specifically for this task and for the fine-tuned model.

We retrained a DFlash model using TorchSpec, a framework for training speculative decoding models with inference/training disaggregation, on our 250K prompt expansion samples, regenerated by our distilled model. This helped us match the output distribution better. However, the retrained model fell short: 250K samples were not diverse enough to give very high acceptance, and the training took a long time. So instead, we decided to warm-start the training by pre-loading the weights and just kept training on top of them using our 250K samples. This worked out great: we were able to get ~3.9 acceptance length @ block size 8 and 700 tok/s throughput.

While we were working on speeding up the prompt expander, DSpark was released by the DeepSeek team. It combines a diffusion-based block predictor (DFlash) with Markovian heads that raise the conditional acceptance rates of later positions. It is almost like a small MTP-like autoregressive refining part attached to the DFlash drafter to improve the acceptance rate. The trainer for DSpark was released, but it required all hidden states to be stored on disk, something we did not want to deal with. Storing ~38TB of hidden states wasn't easy, especially if the disk is networked, where it might quickly become the bottleneck. So we implemented DSpark in TorchSpec.

Since DSpark uses DFlash as the backbone, we warm-started our training from our own DFlash checkpoint by loading its weights and skipping the Markov heads. This allowed the model to quickly recover the accuracy of the DFlash backbone and compound more acceptance on top of it. It also let us train a DSpark model without going through a long training loop.

The next challenge was serving the DSpark model efficiently. First, we patched both vLLM and SGLang to support DSpark, since neither did yet given how recent it was. We eventually found that SGLang was faster for low batch serving, so we settled on it. We also wanted to know whether we could reach 1000 tok/s, so we tried Tensor Parallelism, but it caused a slowdown: the Markovian heads were doing collective operations because they were sharded across the ranks. We fixed it by replicating the weights instead, which finally pushed our serving speed past 1000 tok/s at TP=4. That said, we decided to keep TP=1 at 830 tok/s with 4.6 acceptance length in production, since TP=4 was heavily underutilizing the GPUs and wasn't cost-effective.

Finally, we ran experiments comparing our final serving config to where we started. First, at single-user serving for maximum interactivity, we got 2.6x throughput with under 1 second prompt expansion. But for a realistic serving setting, we targeted 300 tok/s per user, and under that constraint, on the same hardware, we were able to increase our total throughput by 16 times on a single B200 while keeping prompt expansion under 2 seconds.