@SuJinYan123: NV72: Each tray has 4 GPUs, each tray is equipped with 2 NVIDIA Grace (ARM Neoverse V2, 72 cores, total 144 cores, 3.39 GHz), single CPU ~480 GB LPDDR5X, so 4 GPUs...

X AI KOLs Timeline 07/12/26, 05:18 AM News

nv72 nvidia-grace gpu-tray inference-architecture mixture-of-experts parallel-strategy kv-cache coding-traces

Summary

The article provides a detailed analysis of the tray design in the NV72 architecture, where every 4 GPUs are paired with 2 Grace CPUs. It discusses optimizing inference performance under coding workloads using parallel strategies such as EP64 and TP8, as well as the complexity of KV cache management and routing algorithms.

NV72: Each tray has 4 GPUs, each tray is equipped with 2 NVIDIA Grace (ARM Neoverse V2, 72 cores, total 144 cores, 3.39 GHz), single CPU ~480 GB LPDDR5X, so 4 GPUs get ~956 GB CPU memory (~239 GB/GPU). Each tray has 4 CX8 network cards, PCIe 6.0 (128 GB/s). So a common concern about traffic conflict between p/d (prefill/decode) and EP all-to-all can be fully resolved by routing P/D traffic over RDMA and EP traffic over NVLink. For NV72, but for example GLM5.2 has 256 experts, which is not divisible. Forcing 72 can use something like redundant experts for EPLB, adding some complexity. On the other hand, should prefill and decode be colocated? If colocated, how to handle very long prefill? Can NV72 communicate with another NV72? (Seems complicated). If colocated, something like green ctx can resolve some conflicts, but performance degrades when prefill is long. Another approach is to separate attention and FFN, which is also complicated, so we won't consider it for now. EP64 is just right in form factor; the remaining 8 cards can be used as something like TP8 to handle long prefill, while the remaining short prefill and decode run together. We need a specific quantitative number, assuming we serve coding traffic. https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces… Let's look at real coding traces from vllm's open-source codex running on swebench-pro, Assuming infinite KV cache, among requests that need prefill, 50% require prefill < 357 tokens, 72% require prefill < 2048 tokens, average context length is 63K. Coding agents are workloads with heavy KV cache hits (requiring excellent KV cache system support). Even if we use a 2K threshold and run 72% of requests together with decode, the remaining 28% are long prefill, supported only by TP8 (or other methods may not be sufficient). We can only hope they don't arrive all at once (this requires real traffic traces). For GLM5.2, let's look at KV cache capacity. After DP64 and EP64, a single B300 has close to 230GB of GPU KV cache memory. According to previous estimates, each GPU actually has only about the same CPU KV cache size, meaning that if the CPU uses a write-through strategy, it can barely bring any performance improvement. The B300 GPU memory is just too large. Assuming average context = 256K, FP8 KV cache, the GPU memory can hold 17 requests. If we want to do sparse batch sizing, then the CPU memory seems a bit small. So we must think about the significance of using CPU memory as KV cache, but that can be deferred. So the traffic might end up like this: for cold-start prefill, it will be routed to a prefill worker (supported by 8 cards), and P/D separation with the remaining 64 cards. Short prefill will be routed to the 64 cards. Here we need to select a DP worker (whether to enable TP on this DP worker is also a question; we need a sweet spot). The routing algorithm here will be quite complex, considering two aspects: one is decode balance, the other is prefill hot spots (who has the KV cache). On the other hand, if the 64 GPUs' KV cache is P2P, or the entire 64 can be logically seen as a single KV cache pool, then scheduling doesn't need to be cache-aware, leaning more towards decode balance. So we need a GPU P2P KV cache based on NVLink, which might complicate things. Short-term goal can utilize CPU P2P (already supported by Pegaflow), including prefill worker to normal worker (64 cards), also using Pegaflow P2P. The remaining part is the engine architecture on both sides: Prefill worker: TP8? TP8 + EP8? We need to answer which parallel strategy provides extreme throughput under long context. Normal worker: EP64, but whether the attention part is pure DP? If TP is enabled, it would cut heads (cutting by 64 is not friendly), but there are other ways to solve. How to write MoE all-to-all? It seems NVIDIA has written it, but not tested yet. Some scheduling, DP collaboration, MTP handling etc., are still quite complex. After building this, 72 cards should provide very amazing performance. To be continued.

Original Article

View Cached Full Text

Cached at: 07/12/26, 04:59 PM

NV72: 1 tray per 4 GPUs, each tray with 2 NVIDIA Grace (ARM Neoverse V2, 72 cores, 144 cores total, 3.39 GHz), single CPU ~480 GB LPDDR5X, which means 4 GPUs get ~956 GB CPU memory (~239 GB/GPU). NIC: 4 CX8 per tray, PCIe 6.0 (128 GB/s).

So a commonly mentioned issue, the conflict between p/d traffic and ep all-to-all traffic, can be completely resolved by having P/D go through RDMA and ep go through NVLink.

NV72, but for example, GLM5.2 has 256 experts, which doesn’t divide evenly. Forcing it onto 72 can use something like redundant experts for EPLB, which adds some system complexity. On the other hand, whether to put prefill and decode together; if together, how to handle very long prefills, and can an NV72 communicate with another NV72? (Seems complex.) If together, something like green ctx can resolve some conflicts but performance suffers with long prefills.

Another idea is to separate attn from ffn, which is also complex and not considered for now.

EP64 fits well morphologically; the remaining 8 cards can handle long prefill with something like TP8, while short prefills and decode run together. We need a specific quantified number, assuming we serve coding traffic.

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces…

Take a real coding trace from vLLM’s open-source codex running on SweBench-Pro.

Assume KV cache is infinite; among requests needing prefill, 50% need prefill < 357 tokens, 72% need prefill < 2048 tokens, average ctx length is 63K. Coding agents are heavy KV cache hit workloads (require excellent KV cache system support).

Even if we set a 2K threshold and run 72% of requests with decode, leaving 28% as long prefills handled only by TP8 (or other methods might not be sufficient), we can only hope they don’t arrive all at once (this requires real traffic traces).

For GLM5.2, look at KV cache capacity again. After DP64 EP64, a single B300 has nearly 230GB of GPU KV cache memory. According to previous estimates, each GPU also has roughly the same CPU KV cache size, meaning if CPU uses a write-through strategy, it brings almost no performance improvement. B300 memory is just too large.

Assume average ctx=256k, fp8 KV cache, memory can hold 17 requests. If we want to do sparse to increase BS, CPU seems a bit small.

So we must think about the significance of using CPU memory for KV cache, though this can be addressed later.

So the traffic might end up like this: for cold-start prefills, they route to prefill workers (supported by 8 cards), and the remaining 64 cards do P/D separation. Short prefills route to the 64 cards, where we need to select a DP worker (whether this DP worker uses TP is also a question; we need a sweet spot). The routing algorithm here will be complex, needing to consider two aspects: decode balance and prefill hot spots (who has the KV cache).

On the other hand, if the 64 GPU KV caches are P2P, or the entire 64 can be logically seen as one KV cache pool, then scheduling doesn’t need to be cache-aware, leaning more towards decode balance.

So we need a GPU P2P KV cache based on NVLink, which might complicate things. Short-term goal can leverage CPU P2P (PegaFlow already supports it), including prefill worker to normal worker (64 cards), also via PegaFlow P2P.

The remaining part is the engine form on both sides. Prefill worker: TP8? TP8 + EP8? The question is what parallelism strategy provides maximum throughput under long context. Normal worker: EP64, but for the attention part, is it pure DP? If using TP, we need to split heads (64-way split is not friendly), though there are other ways to solve it. How to write MoE all-to-all? NV might have written it, but not tested yet. Some scheduling, DP coordination, MTP handling—pretty complex.

After building this, the 72 cards should provide very impressive performance.

To be continued.

Inferact/codex_swebenchpro_traces · Datasets at Hugging Face

Source: https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces This is a dataset generated by real swebenchpro agentic workload trace + codex agent.

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#1-eval-result-summary1. Eval Result Summary

Metric	Value
Total trials	731
Successful trials	610
Failed trials	120
No data (skipped)	1
Passed	329
Pass rate (of successful)	53.9%

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#per-repo-breakdownPer-Repo Breakdown

Repo	Total	Success	Failed	Passed	Pass%
ansible/ansible	96	93	3	60	65%
internetarchive/openli	91	88	3	52	59%
flipt-io/flipt	85	82	3	26	32%
qutebrowser/qutebrowse	79	78	1	56	72%
gravitational/teleport	76	40	36	12	30%
protonmail/webclients	65	0	65	0	0%
future-architect/vuls	62	56	5	31	55%
navidrome/navidrome	57	55	2	25	45%
element-hq/element-web	56	54	2	24	44%
nodebb/nodebb	44	40	3	17	70%
tutao/tutanota	20	20	0	1	260%

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#2-per-trial-statistics2. Per-Trial Statistics

Based on successful trials only.

Metric	Mean	P50	P90	P99
LLM calls per trial	33	30	57	90
Total input tokens	2,266,055	1,637,000	4,750,691	9,030,908
Total cached tokens	2,133,687	1,525,376	4,580,224	8,781,952
Total computed tokens	132,368	116,991	230,677	512,315
Total output tokens	17,239	15,666	29,968	40,542
Starting context (1st call)	12,367	12,278	12,813	13,744
Ending context (last call)	84,482	80,488	130,165	180,943
Max context length	84,513	80,488	130,165	180,943
Context growth per turn	2,242	880	5,964	15,930

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#3-per-llm-call-statistics3. Per-LLM-Call Statistics

20,230 total LLM calls across 610 successful trials.

Metric	Mean	P50	P90	P99
Input tokens	68,329	63,917	114,888	166,322
Cached tokens	64,338	60,928	112,512	162,048
Computed (uncached) tokens	3,991	758	8,736	53,323
Output tokens	520	246	1,133	4,845

Context length per trial:

Metric	Mean	P50	P90	P99
Starting context (1st call)	12,367	12,278	12,813	13,744
Ending context (last call)	84,482	80,488	130,165	180,943
Max context	84,513	80,488	130,165	180,943

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#4-caching-analysis4. Caching Analysis

Overall cache hit rate: 94.2% (1,301.5M of 1,382.3M input tokens served from cache)

Only 80.7M tokens (5.8%) of all input required actual KV compute; the rest were cache hits.

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#41-intra-trial-cache-turn-by-turn4.1 Intra-Trial Cache (Turn-by-Turn)

Turn	Avg Cache Rate	Median	N Trials
1	87.4%	93.6%	610
2	65.5%	66.2%	610
3	76.2%	77.8%	610
4	85.3%	87.2%	610
5	89.5%	91.5%	610
6	92.7%	95.2%	610
7	94.2%	96.3%	609
8	92.4%	97.3%	607
9	71.3%	95.1%	604
10	84.6%	96.2%	598
11	89.2%	97.2%	588
12	93.7%	98.1%	576
13	94.2%	98.5%	566
14	94.5%	98.8%	557
15	95.2%	99.0%	540
16	94.9%	99.3%	524
17	95.6%	99.3%	503
18	96.3%	99.4%	489
19	95.8%	99.4%	477
20	95.7%	99.4%	461
25	96.1%	99.5%	388
30	96.4%	99.5%	310
35	96.3%	99.5%	241
40	98.1%	99.6%	183
45	96.5%	99.7%	133
50	97.8%	99.6%	93
55	94.6%	99.7%	74
60	94.9%	99.7%	52
65	98.4%	99.8%	33
70	98.4%	99.7%	31
75	98.2%	99.7%	22
80	99.3%	99.8%	15
85	98.1%	99.7%	10

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#42-cross-trial-cache-1st-call-analysis4.2 Cross-Trial Cache (1st-Call Analysis)

The 1st LLM call of each trial can only have cache hits from other concurrent trials sharing the same system prompt prefix.

Metric	Value
Total trials	610
Trials WITH 1st-call cache hit	572 (93.8%)
Trials WITHOUT 1st-call cache hit	38 (6.2%)

Among trials with 1st-call cache hit:

Metric	Value
Mean cache rate	93.2%
Median cache rate	93.8%

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#prefix-group-analysisPrefix Group Analysis

Cached Tokens	# Trials	Notes
11,520	568	Main shared prefix
0	38	Cache miss on 1st call
12,032	2	Variant
12,672	1	Variant
12,160	1	Variant

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#5-turn-timing–inter-call-delays5. Turn Timing & Inter-Call Delays

Time between consecutive LLM calls within a trial. Includes model response time + tool execution + agent processing.

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#per-trial-duration-first-to-last-llm-callPer-Trial Duration (first to last LLM call)

Metric	Mean	P50	P90	P99
Trial duration (seconds)	336.8	273.7	637.8	1160.5
Avg inter-call delay per trial (s)	10.2	9.7	14.5	22.8

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#inter-call-delay-distributionInter-Call Delay Distribution

Metric	Mean	P50	P90	P99
Inter-call delay (seconds)	10.5	5.2	23.0	81.4

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#delay-by-turn-numberDelay by Turn Number

Turn	Avg Delay (s)	Median (s)	N Trials
2	4.8	4.6	610
3	5.7	4.6	610
4	6.6	4.6	610
5	8.7	4.9	610
6	10.0	4.9	610
7	11.4	4.9	609
8	11.3	4.5	607
9	13.0	5.3	604
10	10.9	5.0	598
11	13.0	5.3	588
12	11.5	5.0	576
13	11.7	5.4	566
14	11.8	5.2	557
15	11.1	5.0	540
16	12.2	5.2	524
17	12.3	5.0	503
18	11.3	5.0	489
19	10.7	5.4	477
20	11.1	5.4	461
25	11.0	4.6	388
30	10.3	5.1	310
35	11.1	6.0	241
40	14.1	6.4	183
45	12.4	6.3	133
50	9.9	5.9	93
55	11.0	5.6	74
60	13.0	9.5	52
65	9.0	4.7	33
70	8.3	6.5	31
75	11.5	5.0	22
80	12.7	7.2	15
85	10.5	7.3	10

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#6-workload-characteristics6. Workload Characteristics

131:1 input:output ratio — massively prefill-dominated
Context grows ~2,242 tokens/turn on average
Cache:New-input ratio = 16.1:1 — for every 1 token of new compute, 16.1 tokens are served from cache
20,230 total LLM calls across 610 trials, avg 33 sequential calls per trial
Avg output per call: 520 tokens (median 246)

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#compute-distribution-heavy-tailCompute Distribution (Heavy-Tail)

Uncached prefill compute is concentrated in a small fraction of calls:

Top % of Calls	# Calls	% of Uncached Compute
1%	202	20.5%
5%	1,011	49.6%
10%	2,023	64.2%
20%	4,046	80.0%

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#context-growth-by-phaseContext Growth by Phase

Phase	Turns	Avg Growth/Turn
Exploration (turns 2-5)	2-5	~6,166
Active coding (turns 6-12)	6-12	~2,535
Iteration (turns 13+)	13-100	~1,410

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#9-failed-trials-analysis9. Failed Trials Analysis

120 total failures out of 731 trials.

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#by-root-causeBy Root Cause

Root Cause	# Trials	Repos Affected
Package not found	108	protonmail/webclients (65), gravitational/teleport (35), future-architect/vuls (5), element-hq/element-web (1), flipt-io/flipt (1), navidrome/navidrome (1)
Setup timeout	7	ansible/ansible (2), internetarchive/openli (2), element-hq/element-web (1), flipt-io/flipt (1), gravitational/teleport (1)
NonZeroAgentExitCodeError: Command failed (exit 1): set -euo pipefail; if ldd –version 2>&1	grep -qi mus	3
DownloadVerifierDirError: Failed to download verifier directory from environment	2	ansible/ansible (1), qutebrowser/qutebrowse (1)

https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces#by-repoBy Repo

Repo	Errors	Total Trials	Failure Rate
protonmail/webclients	65	65	100%
gravitational/teleport	36	76	47%
future-architect/vuls	5	62	8%
ansible/ansible	3	96	3%
flipt-io/flipt	3	85	4%
internetarchive/openli	3	91	3%
element-hq/element-web	2	56	4%
navidrome/navidrome	2	57	4%
qutebrowser/qutebrowse	1	79	1%

Downloads last month | 1,197