NVIDIA 最硬的护城河，从来不是 GPU。

而是 CUDA。

但这个护城河，可能正在被 AI Agent 一点点凿穿。

最近 Wafer 把 GLM-5.2 跑在 AMD MI355X 上，单节点做到 2626 tokens/s，成本比 Blackwell 低 2 倍以上。

这件事真正危险的地方不是「AMD 性能很强」。

而是它证明了：

AMD 缺的可能不是硬件。

缺的是软件生态、模型适配、kernel 优化和工程经验。

问题来了。

这些东西，恰好是 AI Agent 最擅长加速的：

修框架 bug 调 kernel 做量化 适配新模型 优化推理配置 自动跑 benchmark

过去 CUDA 的优势，是十几年工程生态堆出来的。 但如果 AI Agent 能把这些工程适配成本压低 10 倍，NVIDIA 的高溢价还能持续多久？ 我越来越觉得，真正挑战 NVIDIA 的不是 AMD。 而是： AMD + 开源推理框架 + AI Agent 工程自动化。 CUDA 护城河会不会还在？

会。

但它可能没以前那么深了。

你觉得 NVIDIA 的 CUDA 护城河还能守几年？ 1 年、3 年、5 年，还是继续无敌？

Performance per dollar is getting faster and cheaper | Wafer

Source: https://www.wafer.ai/blog/glm52-amd Have you noticed we like AMD?

The demand for inference is skyrocketing and outpacing supply. With frontier models being released almost every other week — Claude Fable, GLM5.2, and Minimax M3, to name a few — the token craze is only getting crazier, and there aren’t enough Blackwells going around to support it. Thus, NVIDIA GPU prices are climbing fast, and tokens are getting really expensive.

In comes AMD. At around 2.75x cheaper per GPU on average (MI355X vs B300) with comparable hardware specs, the solution to cheap inference is hiding in plain sight — a message we at Wafer have been preaching for months. But although AMD’s Instinct MI350 series competes with Blackwells at the silicon level, NVIDIA’s software advantage and day-0 support typically allows providers to serve inference much faster on their hardware with much less friction.

Conversely, on the MI355X / ROCm stack SOTA performance rarely comes out of the box for these frontier models (sometimes it does!). In fact, you’re lucky if you can find an image that runs them at all. Without this day-0 support, building and optimizing for the newest models can require weeks of engineering and compute. By then, the newest model has already been released, making it so AMD is always playing catch-up.

But as agents improve at kernel and model optimization, this gap is closing in real time. At Wafer, we’ve proven this time and time again.

And again — on a 20k in / 1k out, 60% cache hit rate workload, we hit an aggregate throughput of 2626 tok/s/node @ 2.4 rps with a defined knee of ≤5s TTFT — only 80% of the performance measured on a B200, despite being over 2x cheaper.

Sustained RPSAggregate tok/s/nodeTTFT p50 / p95Success0.54490.59s / 0.60s100%1.09740.60s / 0.81s100%1.519130.62s / 1.03s100%2.019440.62s / 1.05s100%2.2520890.63s / 1.23s100%**2.4 (saturation)26260.81s / 2.22s****100%**We also hit 213 tok/s on GLM5.2 on 10k input tokens / 1.5k output tokens single stream, followingArtificial Analysis standards, served on AMD MI355X capacity from TensorWave. Though this number doesn’t top the AA leaderboard, it still wins on performance per dollar.

How we did it

The first step with any model work is to choose a quantization and framework. We quantized the base bf16 GLM-5.2 to MXFP4 with AMD Quark. In comparison to z-ai’s official FP8 quantization, our MXFP4 was lossless (GPQA-Diamond, tau2, GSM8K).

EvalFP8 baselineMXFP4Δ (MXFP4 − FP8)GSM8K (200q, 5-shot, greedy)0.965 ± 0.0130.955 ± 0.014−0.010GPQA-Diamond (198q × 2 seeds, temp 1.0)0.9217 ± 0.0270.9026 ± 0.029−0.019tau2 macro0.8190.834+0.015As for the inference framework, we had three options — vLLM, ATOM, and sglang. Among the three, we chose sglang — vLLM had no working MXFP4 + GlmMoeDsa path so the MXFP4 weights provided no benefit, and ATOM’s output degraded at long context. Sglang was the inference engine with the least friction to native support, able to take advantage of the quantization while remaining coherent.

The next natural step to improving throughput was enabling speculative decode on sglang. However, the sglang ROCm image does not support this out of the box. There were two fixes needed before MTP worked properly.

First, the MTP head, like every other layer, keeps its single shared expert stored in bf16, not MXFP4. However, the MTP head is registered under a different module prefix than the main decoder stack (Quark names its bf16 shared expertmodel\.layers\.78\.mlp\.shared\_experts\.\*, while the MTP layer’s real prefix ismodel\.decoder\.\*). Because of the mismatch, sglang’s quantization lookup fails and defaults to building that shared expert as MXFP4. At load it then tries to read a full-width bf16 weight into a half-width 4-bit slot and the init crashes on a shape mismatch. Quark records which weights to leave un-quantized as a list of layer names, so we copied over the layer 78 entries to that list a second time under the decoder name sglang actually uses. This fix unblocked speculative decode, netting us close to a 3x gain in single stream throughput.

Second, deep speculative decode (such as the 5/1/6 config z-ai suggests) was still blocked. The fused multi-step metadata kernel needed for draft depth ≥4 writes\#include <cuda\_runtime\.h\>with no ROCm guard. Fix: one\#ifdef USE\_ROCMguard.

Two trivial, but necessary changes to take full advantage of speculative decode. With spec dec working properly, alongside a few config optimizations (such as\-\-kv\-cache\-dtype fp8\_e4m3and\-\-enable\-aiter\-allreduce\-fusion), we reached our headline single stream decode number at 213 tok/s.

But for aggregate throughput, especially with our defined workload, decode optimizations are necessary but insufficient. At 20k in @ 60% cache, the workload is primarily prefill bound.

At TP8, which was the configuration optimized for single stream decode, the MI355X can run GLM5.2-MXFP4 at 1461 tok/s/node. Switching to TP4×DP2 netted a massive improvement on this workload, getting us to 1944 tok/s/node at 2.0 RPS — still relatively slow compared to our measured Blackwell performance, which hit 3192 tok/s/node at 3.0 RPS. A big reason for the poor prefill performance on the MI355X is that on the sglang image, GLM-5.2’s fp4 MoE was silently on a slow FlyDSL heuristic fallback (aiter only shipped tuned configs for the a8w8/fp8 path). We tuned the MoE kernel selection ourselves on GLM’s fp4 shapes (model\_dim 6144, moe\_inter 2048, E=256, topk=8), which allowed us to reach 2626 tok/s/node at 2.4 RPS. Much better.

Why this matters

Although there was some degree of friction, achieving the best performance per dollar ratio on the MI355X wasn’t particularly hard — though there were some framework related bugs, unlike our work with Qwen3.5 397B, you’ll notice that we didn’t actually write any custom kernels this time. Though this study doesn’t take multi-node performance into consideration, single-node deployments still remain highly prevalent in practice.

SOTA on AMD is becoming more a matter of support, not software. The CUDA moat is eroding in real time.