Cloud AI

Attention-FFN disaggregation (AFD) is the next architectural split beyond prefill-decode disaggregation for large mixture-of-experts (MoE) models: it separates the attention modules from the feed-forward expert modules onto distinct GPU pools inside the decode phase, fusing the extra activation transfer with the all-to-all dispatch that MoE expert parallelism already requires. The technique is not theoretical. Step-3, StepFun’s 321B-parameter model, implements AFD and reports 4,039 decode tokens per second per GPU under a 50 ms TPOT service-level objective at 4K context in FP8, against 2,324 for DeepSeek-V3 in the same configuration. In other words, AFD decouples attention and FFN layers into specialized subsystems inside the decode phase, adding a second split after prefill-decode disaggregation.

Why Prefill-Decode Is Not Enough

After prefill and decode run on separate pools, the decode pool itself still blends two sub-phases with opposing bottlenecks. Attention is memory-bandwidth-bound because each generated token must read key-value state for every prior token, while the feed-forward expert network is compute-bound and scales well with batch size. In a dense model one FFN per layer sees enough tokens at the maximum batch size to saturate the GPU, but in a sparse MoE model each expert receives only a fraction of the batch — often less than a quarter — so FFN utilization collapses. MegaScale-Infer, built by ByteDance and Peking University, measured this collapse directly and showed that disaggregating attention from experts and consolidating requests across attention replicas restores FFN compute intensity, delivering up to 1.9× higher per-GPU decode throughput than prior systems. The same study reports 1.7× higher throughput per unit cost on a heterogeneous cluster and notes the design is already deployed in production inference, cutting serving cost by 1.5–2.0×. That production footprint is the strongest signal, because it shows the FFN-utilization fix holds under live traffic rather than only in controlled benchmarks.

How Attention-FFN Disaggregation Works

The mechanical change is to stop co-locating attention and FFN on the same device. Attention modules are replicated with data parallelism so each replica owns the full KV cache for its requests, while FFN experts are distributed with expert parallelism across a separate set of GPUs. Requests are partitioned into micro-batches and shuttled between the two pools in a ping-pong pipeline so neither sits idle while the other computes. MegaScale-Infer pairs this ping-pong schedule with a custom M2N communication library that removes GPU-to-CPU copies and group-init overhead, reporting 4.2× higher throughput and 68.2% lower latency than NCCL for the many-to-many token-dispatch pattern. The split also enables heterogeneous hardware: attention replicas can run on GPUs chosen for memory bandwidth and capacity, while expert pools run on GPUs chosen for raw FLOPS. That flexibility is what drives the cost-per-token gains, because you stop paying for HBM you do not need on the expert side.

The All-to-All Trick Behind AFD

AFD introduces additional data transfer between the attention and FFN pools, so the split pays off only when that traffic can be fused with communication the MoE stack already carries. That calculus changed when large MoE models such as DeepSeek-R1 and Qwen3-235B began serving with wide expert parallelism, which already performs two all-to-all communications per decoding layer to dispatch and combine expert tokens. By aligning the attention-FFN split with those existing all-to-all patterns and fusing the traffic, the incremental cost of AFD becomes close to free. DeepSeek’s own inference system, which adopted prefill-decode disaggregation early, runs decode with routed-expert EP144 across 18 nodes per deployment unit and hides the dispatch and combine communication behind a five-stage compute pipeline. The same overlap principle is what makes AFD viable: the attention-FFN transfer rides communication that was already budgeted.

What Production Numbers Actually Show

The numbers from shipped systems tell the same story as the papers. SGLang’s open-source DeepSeek deployment, using twelve eight-GPU H100 nodes with prefill-decode disaggregation and large-scale expert parallelism, reaches 52.3k input tokens and 22.3k output tokens per second per node, about one-fifth the cost of the official DeepSeek API and the first open-source stack to match DeepSeek’s reported throughput. Step-3 shows where the next split lands: it activates 38 billion parameters per token, more than DeepSeek-V3 or Qwen3 MoE 235B, yet still cuts theoretical decoding cost, which StepFun credits to hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD together. Hao AI Lab, which introduced prefill-decode disaggregation with DistServe, now reports that nearly every production-grade framework — NVIDIA Dynamo, llm-d, Ray Serve LLM, SGLang, vLLM, LMCache, Mooncake — runs on disaggregation, and names attention-FFN disaggregation as the natural next step that today’s MoE models make practical. If you are choosing an engine to host this, our vLLM versus SGLang comparison breaks down how each handles the disaggregation primitives.

When Attention-FFN Disaggregation Pays Off

AFD is not a free upgrade. MegaScale-Infer’s authors caution that ping-pong pipeline efficiency depends on attention and FFN taking roughly similar compute time. Current AFD implementations demonstrate effectiveness on MoE models, while dense models remain an open challenge because communication overhead is higher and harder to overlap with computation. Step-3, by contrast, was co-designed end to end — its Multi-Matrix Factorization Attention shrinks both KV cache and attention compute so the attention pool hits the arithmetic intensity the hardware wants, which is what lets AFD actually win. Current evidence puts AFD on the roadmap for wide-EP MoE serving after prefill-decode disaggregation: current implementations demonstrate effectiveness on MoE models, while communication overhead for dense models remains an open challenge.

Sources

• Step-3 technical report (StepFun) — arXiv:2507.19427
• MegaScale-Infer, ACM SIGCOMM 2025 (ByteDance & Peking University) — full paper
• DeepSeek-V3/R1 inference system overview (DeepSeek) — GitHub
• Disaggregated Inference: 18 Months Later (Hao AI Lab @ UCSD) — blog
• Deploying DeepSeek with PD Disaggregation on 96 H100 GPUs (SGLang / LMSYS) — blog