Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X

The MiniMax M3 day-0 post described the first working vLLM implementation: MiniMax Sparse Attention (MSA), multimodal inputs, reasoning and tool outputs, MXFP8 weights, and EAGLE3 on AMD Instinct MI355X.

This follow-up is about what happened after the model ran. The useful result is not only a higher throughput number. It is a way to decide what to optimize next when the bottleneck keeps moving.

Results in one minute

Public results from the SemiAnalysis InferenceX benchmark show what MiniMax-M3 now delivers on AMD Instinct MI355X:

• At concurrency 32, fixed-topology MXFP8 standard serving rose from 109.1 to 342.4 output tokens/s/GPU, or 3.14× the day-0 result. Median TTFT fell from 1.46 to 0.67 seconds, and mean TPOT fell from 69.1 to 22.1 milliseconds.
• At concurrency 128, the same TP4/EP1 four-GPU path rose from 297.8 to 623.7 output tokens/s/GPU, or 2.09×. Median TTFT fell from 3.53 to 1.54 seconds, and mean TPOT fell from 100.7 to 48.8 milliseconds.
• MXFP4 first rose from 212.1 to 716.8 output tokens/s/GPU at concurrency 128 under the same TP4/EP1 four-GPU contract. A later TP2/EP1 result reached 943.5 output tokens/s/GPU, 31.6% above that TP4 checkpoint and 4.45× the initial per-GPU result.
• We added EAGLE3 speculative decoding, reaching 682.4 output tokens/s/GPU at concurrency 128 on TP4/EP1.
• We added P/D disaggregation and retuned the prefill/decode topology, reaching 6,370.5 total tokens/s/GPU at concurrency 512 with 1.32 seconds median TTFT.

Figure 1. Standard decoding at 8K input and 1K output. Compare points within a row. MXFP8 stays on TP4/EP1; the last MXFP4 point moves from TP4 to TP2, so it shows higher deployment density, not a fixed-topology speedup.

Each checkpoint is cumulative and can bundle several changes. The sections below use isolated PR measurements to explain individual optimizations.

One decode step, five questions

The optimization work followed a familiar workflow: estimate the dominant costs, measure them, batch repeated work, precompute invariants, and move up the stack when leaf profiles flatten.

MiniMax M3 has 60 decoder layers; 57 use sparse MoE and sparse attention. For a 1K-token response, a small per-layer cost can appear tens of thousands of times in one request. That makes five questions useful:

1. What local shape reached this rank?
2. What work repeats at every layer or token?
3. Which bytes move, and can metadata move instead?
4. Did the intended fast path run, with the intended math?
5. When kernels are no longer dominant, which queue grows?

Figure 2. The top lane follows one sparse decoder layer in execution order; AR marks the tensor-parallel collectives after attention and MoE. The lower lane applies the same reasoning to P/D: validate the KV handoff, then add capacity where requests wait.

The rest of this post answers those questions with code and benchmark evidence.

1. What shape reached this rank?

Model diagrams show global dimensions, but kernels run on local M, N, and K after tensor parallelism, head replication, padding, and token routing. At TP8, MiniMax M3’s 64 query heads shard to eight per rank, while its four KV and four index heads replicate to one per rank. The fused QKV projection therefore sees local N=1536—not global N divided by eight.

Prefill and decode also arrive with different M. Prefill processes many tokens at once; decode often has only a few rows. vLLM #45725 split the launcher into large-M and small-M regimes, improving TP8 8K/1K output throughput by 7.8%–9.4%. vLLM #46117 then selected tiles from the full local shape: narrower N tiles exposed more independent work in decode, while a larger K step reduced loop iterations. Prefill used wider tiles when M already supplied enough parallelism.

Figure 3. TP sharding and head replication determine local N; the serving phase determines M. The launcher selects tiles for the shape each rank actually runs.

The same PR reordered grouped-MoE programs so neighboring programs could reuse activation rows and expert-weight tiles from GPU cache instead of fetching them again from HBM. Across its TP4 end-to-end tests, the combined changes produced 1.08×–1.46× gains. The benefit was largest at low concurrency, where the original launch had the least parallel work.

Local shape also decides which backend is legal. The AITER sparse-attention path used in the final MXFP8 recipe requires one KV head per TP rank. TP4 satisfies that condition. TP2 uses vLLM’s Triton fallback. Changing TP therefore changes more than collective size; it can change the operator graph.

There was no universally fastest backend, either. InferenceX #2003 initially selected an emulated linear backend for the whole sweep. Later measurements in InferenceX #2187 showed that native MXFP8 linear was faster at low and middle concurrency, while emulation won only for long-input, high-concurrency runs. Sparse paged attention had a similar crossover. The final recipe enables both only for 8K input at concurrency 64 or higher.

That is the first reusable lesson: tune and dispatch on the shape distribution that actually runs. “Prefill,” “decode,” or “TP4” is only a label.

2. What work repeats?

The first easy-to-miss repeated cost was launch overhead. The day-0 recipe ran eagerly. InferenceX #1754 and #1755 enabled graph execution for standard and EAGLE3 serving. That change is part of the cumulative history, although the published results do not isolate its gain.

The larger structural win was the shared expert. Originally, every sparse-MoE layer ran it as a separate dense MLP: gate/up projection, activation, down projection, intermediate storage, and addition. The math was required. The separate path was not.

vLLM #46545 appended the shared expert to the routed expert table and selected it for every token. The grouped GEMMs then handled routed and shared experts together. This removed launches and intermediate traffic without removing model FLOPs. Output throughput improved 30.2% at concurrency 1 and 5.6% at concurrency 128. The shrinking gain is useful evidence: it is what launch amortization looks like.

Figure 4. Fusion appends the shared expert as a slot selected by every token. Routed and shared experts then use the same grouped GEMMs, preserving the model math while removing a separate MLP path and its intermediate traffic.

The AITER path applied the same idea in vLLM #46474. vLLM #46184, backed by AITER #3811, also moved MXFP8 weight and scale reshuffling to model load. AITER carries tuned MoE configurations from one to 32,768 tokens and for the local intermediate widths produced by TP4 and TP8. Layout conversion happens once; the serving loop consumes the prepared form.

Speculative decoding supplied a clean example of batching repeated work. The original MSA indexer launched one workgroup per speculative token. vLLM #45743 launched one workgroup per request and processed all draft positions together, reusing key loads. It also removed a positive score scale because only top-k order matters and multiplying every score by the same positive constant cannot change that order.

The index kernel improved by as much as 48.9%, while end-to-end serving improved by about 3.3% in the PR tests. That gap is Amdahl’s law doing its job: a large kernel win can be real without being the application’s whole critical path.

3. Which bytes move?

Sparse attention reduces attention math, but it adds a control plane: score blocks, select top-k, map logical blocks to physical pages, and pass that metadata to the attention kernel.

vLLM #47269 observed that adjacent sparse layers often selected nearly the same blocks. With index sharing enabled, one layer computes the top-k decision and later layers reuse it. Mean TPOT fell by about 10% at concurrency 1 and about 4% at high concurrency.

Skipping the selector was only half the job. The fused projection still produced index Q/K values, normalized them, applied RoPE, and wrote the index cache. vLLM #47287 made the reuse decision visible to that fused kernel, so the unused producer branch is compiled away.

The same PR integrated AITER sparse paged attention across a layout mismatch. MiniMax M3 selects logical 128-token blocks; AITER consumes 16-token pages. Instead of copying KV data, vLLM turns each selected block ID into eight page IDs and builds a compact page table. The KV cache stays where it is. This is the serving equivalent of passing a view instead of copying a container.

Figure 5. Each selected 128-token block resolves to a physical block, then expands into eight 16-token page entries. AITER reads them through a view of the existing KV allocation; only the table is rebuilt.

At TP4, concurrency 256, the PR improved output throughput by 6.93% for MXFP4 and 5.56% for MXFP8 in its isolated A/B.

There is an important benchmark boundary here. InferenceX’s fixed 8K/1K review policy excluded cross-layer index reuse because it reduces architecture work. The fixed-shape recipe uses the page adapter but not top-k reuse. AgentX enables reuse under its workload rules. We do not credit the fixed-contract curve with work it did not run.

Quantization makes the “which bytes?” question even more important. It is helpful to separate three planes:

Byte plane	MiniMax M3 example	What must be proved
Weights and activations	MXFP8 or MXFP4 GEMM/MoE	Packing, scales, activation math, backend layout
Persistent state	FP8 KV and sparse index cache	Platform dtype, page layout, read/write geometry
Communication	Quantized all-reduce or KV transfer	Eligibility, selected codec, ownership, completion

These planes have independent dispatch and correctness contracts. “The model is MXFP4” does not tell us the KV dtype or the collective path.

4. Did the fast path run—and was it correct?

This audit changed one claim in the post. We initially believed a roughly 1.5 MB decode collective used INT4 QuickReduce. The available evidence does not prove it.

InferenceX #2104 configured INT4 and a 256 KB codec threshold, but not QuickReduce’s separate eligibility threshold. For BF16 at TP4, the pinned built-in table requires 16 MB for INT4. The 1.5 MB collective therefore does not reach codec selection; the 256 KB threshold is consulted only after QuickReduce is eligible.

Figure 6. QuickReduce checks eligibility before choosing FP or INT4. Here the collective falls below the built-in eligibility gate, so configuration alone cannot establish execution.

The available logs prove that INT4 was configured, not that the QuickReduce kernel ran. We therefore treat #2104 as a cumulative image and recipe checkpoint and do not attribute its curve to INT4 all-reduce.

This distinction generalizes:

configured  !=  eligible  !=  executed

Use a dispatch trace or profiler before assigning a gain to a backend.

Correctness needs the same discipline. Three examples caught different broken contracts:

• vLLM #45794 mapped packed MXFP4 Q/K/V and gate/up checkpoint tensors into the correct fused-parameter slices and passed MiniMax M3’s SwiGLU-OAI parameters into MoE.
• vLLM #45720 fixed the FP8 KV view on FNUZ ROCm devices. On MI300X, the unpatched path scored 0.0099 strict match on GSM8K; the patched path scored 0.9575. This was a correctness fix, not a claimed MI355X speedup.
• vLLM #47158 fixed the expert-parallel mask passed to AITER. The buggy path had cosine similarity 0.527; the corrected path reached 1.0 and restored GSM8K accuracy.

The last two examples do not explain the TP4/EP1 hero curve; they expose contracts that other configurations must satisfy. For performance work, “passed” should mean three things: the output is correct, the intended path executed, and the end-to-end metric improved under the same contract.

EAGLE3 adds a second decode loop

EAGLE3 adds a draft model, multi-token verification, acceptance behavior, and a second set of attention metadata. It cannot be treated as a flag on the standard curve.

vLLM #45546 connected the AMD model to the EAGLE3 interface. Then vLLM #45564 fixed a subtle cache-key bug: the target and draft use different query-head counts, so they must not share an attention-group builder merely because their backend and KV type match.

That is a general cache rule: the key must contain every invariant that changes the cached object.

After the request-level index batching described above, InferenceX #2107 found that the target’s attention-backend setting did not configure the draft. Pinning TRITON_ATTN inside the speculative config avoided the draft’s slower fallback.

Finally, vLLM #47984 extended AITER sparse paged attention from one-token decode to multi-token verification. It maps each flattened query row back to its request and local speculative position, reuses the existing page-table builder, and preserves the one-token fast path. Its TP4 tests improved output throughput by 8.32% for MXFP4 and 7.90% for MXFP8 without materially changing acceptance.

Together, this work produced the separate 682.4 output tok/s/GPU EAGLE3 result at concurrency 128.

5. Which queue grows?

Prefill/decode disaggregation moved the bottleneck above one process. Before tuning worker counts, the KV boundary had to be correct.

The initial MoRIIO path assumed that the first layer’s KV layout represented every layer. MiniMax M3 has separated K/V tensors, interleaved K/V tensors, and a key-only index cache. The transfer completed and throughput looked healthy, but GSM8K fell to roughly 0.0008—effectively token salad.

The repair came in three steps:

• vLLM #46039 derived transfer geometry and byte offsets per layer.
• vLLM #46290 counted the writes actually scheduled for each request, sealed that count after forward, and released buffers only after those writes completed.
• vLLM #46332 added heterogeneous-TP rank mapping and acknowledgment fan-in. With prefill TP4 and decode TP8, two decode ranks can consume one producer rank, so both must acknowledge before its blocks are reused.

Only then was worker allocation worth tuning.

Figure 7. Two 8K/1K P/D operating points. The endpoints use different GPU counts and concurrency, so this is system evolution, not a controlled speedup.

The first public profile used one TP8 prefill worker and one TP8 decode worker. At concurrency 1024 it reached 2,084.6 total tok/s/GPU, but median TTFT was 223.20 seconds (InferenceX #1762). The throughput number did not make that system usable; the prompt queue was the signal.

InferenceX #2144 moved every worker to TP4, synchronized them with the faster single-node recipe, and searched the prefill/decode ratio. For 8K/1K, two TP4 prefill workers fed one TP4 decode worker. At concurrency 512, the result reached 6,370.5 total tok/s/GPU and 1.32 seconds median TTFT.

Mean TPOT moved from 31.26 to 54.60 milliseconds. That is not a contradiction. Added prefill capacity cleared the admission queue, while the selected decode operating point produced each active sequence more slowly. P/D has at least two latency objectives. Publish both.

One high-concurrency run also exhausted the container’s file-descriptor limit. Raising nofile fixed the TCP failures. Once a profile moves up the stack, an OS limit can be as real as a GEMM tile.

AgentX shows the next bottleneck

Fixed 8K/1K is excellent for controlled comparisons. Agentic coding is not fixed-shape traffic: it has long, multi-turn traces, reusable prefixes, irregular outputs, and a KV-capacity knee.

InferenceX #2487 is the first MI355X MiniMax M3 AgentX point with MXFP4, EAGLE3-GQA, prefix caching, optional TP-sharded LMCache, and cross-layer index reuse. Throughput replay uses a committed synthetic acceptance length so compared systems do the same speculative work; eval uses real target verification.

In the successful run, TP4 at concurrency 28 delivered 127.4 output tok/s/GPU, 509.5 total output tok/s, 0.582 mean QPS, 645 ms p50 TTFT, and 41.3 ms p50 TPOT.

The service metrics make this more than another score:

• Theoretical prefix-cache hit rate: 96.7%
• Realized GPU cache hit rate: 92.1%
• GPU KV-cache use: 88.5%
• GPU KV capacity: 6,264,960 tokens

At this point, another GEMM is not automatically the best next project. The 4.6-point cache-realization gap and the near-capacity operating point direct attention toward prefix alignment, admission and eviction policy, scheduling, and offload. These are observations from one run, not yet an optimization claim. We will use this point as the baseline for the next round of agentic optimization.

A checklist for the next model

When a new serving path works but is not yet fast:

1. Record local shape histograms after sharding. Include replicated heads and routed-token counts.
2. Estimate repetition. Multiply per-layer work by layers, output tokens, and active requests.
3. Separate the byte planes: compute tensors, persistent state, and communication.
4. For every fast path, record its eligibility condition and verify its execution.
5. Put a correctness gate beside every performance gate.
6. After each win, profile again. If leaf kernels flatten, inspect queues, ownership, cache capacity, and OS limits.

That is the main result of this work. MiniMax M3 became faster because the team kept changing the level of the question—from tiles, to repeated paths, to sparse metadata, to distributed state, and finally to workload queues.

Reproduce the fixed-shape result

The public InferenceX runs record the container images, arguments, and artifacts. The final MXFP8 TP4 checkpoint used:

vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f

export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1

vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
  --tensor-parallel-size 4 \
  --block-size 128 \
  --no-enable-prefix-caching \
  --language-model-only \
  --moe-backend aiter \
  --max-model-len 10240 \
  --max-num-batched-tokens 32768 \
  --kv-cache-dtype fp8 \
  --attention-backend TRITON_ATTN \
  --tool-call-parser minimax_m3 \
  --reasoning-parser minimax_m3 \
  --enable-auto-tool-choice

For the exact high-concurrency dispatch used by Figure 1, use the gated recipe in InferenceX #2187. For MXFP4 TP2, use #2446; for P/D, use #2144. Copying only the flags above does not reproduce a different image, topology, or workload.

The concurrency-128 benchmark command is:

vllm bench serve \
  --backend vllm \
  --model MiniMaxAI/MiniMax-M3-MXFP8 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.8 \
  --num-prompts 1280 \
  --max-concurrency 128 \
  --request-rate inf \
  --ignore-eos \
  --num-warmups 256 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --save-result

Acknowledgements

We thank MiniMax for releasing MiniMax M3 and everyone who built, optimized, and validated this serving path: Aakif Nawaz, Ajith Sirra, Bryan Shan, Bugen Zhao, Cameron Quilici, Chun Fang, Duyi Wang, Ethan Yang, Fangzhou Ai, Felix Marty, functionstackx, Hongxia Yang, Isotr0py, Jun Kang Chow, Pin Siang Tan, Qiang Li, Seung Rok Jung, Sun Peng, Tian Di, Tun Jian Tan, Uma Kannikanti, wangjiaxin99, Ye Hur Cheong, youkaichao, Yue Liu, and Zheng Gong.

We also thank the broader vLLM, AMD, Embedded LLM, Inferact, and SemiAnalysis InferenceX reviewer communities.