The main thing I had fable doing was routing moa and rlm experiments spanning local api and cerebras. Get your agent to summarise I think some of it was interesting. https://services.turquoisebay.ai/share/moa-next-phase/index.html… @DJLougen @no_stp_on_snek @Teknium

Hermes MoA — Findings, Verdicts & Roadmap

Source: https://services.turquoisebay.ai/share/moa-next-phase/index.html

Hermes Mixture-of-Agents: what we tested, what we found, when to use it

Branchfeature/moa-openai-proxyon the public forkgithub.com/hughmadden/hermes-agent· 2026-07-03 · ~330 benchmark turns on OpenRouter + Cerebras, plus local GPU measurements

Changelog

VerWhen (UTC)What was done / what changed in this documentv3507-06 ~01:45Voter-upgrade attempt REFUTED (iter 43): qwen3.6-27b as RLM voter measured from BOTH routes — local FP8 partial 16/25 (64%) @384 s median; OpenRouter 20/30 (67%) @382 s — slower AND less accurate than gemma31-RLM (82% @2.5 s). The thinking-model law reconfirmed a third time: heavy reasoners fight the RLM loop’s short-turn structure (qwen27’s earlier 80% HMMT came from native long-form single passes). gemma31-RLM keeps the voter crown; qwen27’s local role = general mid-tier lane. Side-finding documented: qwen3.6-27b OR solo collapsed to 10% (provider pathology consistent with its 0%-caching/erratic-routing profile from the cache study) — OR routing quality varies wildly per model. GPU work paused for Hugh’s meeting mid-iteration (partials preserved); GLM-4.7-IQ3 download continues for the smartest-open head-to-head hand-off.v3407-06 ~18:302×RTX-6000-Blackwell topology study (iter 42)— the one-big-vs-several-small question, measured (no public benchmark existed): for anything that fits one 96 GB card, TWO INDEPENDENT INSTANCES dominate every axis — full speed simultaneously (176+177 tok/s; 1,434 tok/s aggregate @16+16) vs TP=2’s 136 tok/s single-stream (23% SLOWER than one GPU — the PCIe/no-NVLink tax) and half the aggregate. TP=2 over PCIe is purely a FIT mechanism for >96 GB weights. Context answer: 262k native on ONE card (Qwen3-Next-80B-FP8 instrument); caching answer: a 180k-token agent-session turn costs 0.38 s TTFT warm vs 35.3 s cold (99% cut) — local prefix caching makes long sessions essentially free after turn one. Permanent 262k endpoint now serving on bench GPU 1. Dogfood tally: our cascade drafted this section’s first version (tier-2, 21.5 s, $0, styling-only edits). New section 15.v3307-06 ~13:30Serving-cache economics measured (iter 41)— agentic sessions re-send a growing prefix every turn, so provider prompt-caching is a first-order cost lever. OpenRouter study (30k-token prefix, 4-call sessions): deepseek-v4-pro 99.5% cached / ~40% prompt-cost cut; glm-5.2 92% / 40% + fastest cold (3.8 s); GPT-5.5 caches upstream but NO billed saving through OR; kimi hits 1-in-3 (multi-backend routing defeats caching); qwen3.6-27b zero. Cerebras probe: gemma-4-31b real prefix cache (warm TTFT 3× lower); wafer prefill so fast (~1.5 s cold @30k) that voter-tier session economics are fine even cache-cold. New section 14; guidance: pin cache-critical lanes to deepseek/glm, volatile prompt fields LAST. 2×Blackwell topology study in progress (results as v34).v3207-06 ~03:00Session-aware tool turns SHIPPED (iter 40) + a double bug hunt. (1) cascade.tool_turns: detect (default): mid-tool-loop turns stay acting-solo; fresh user turns run a tool-aware voter gate — real-answer consensus REVERTS the session to tier-0 cascade; TOOL_TURN vote / no consensus → acting-solo with tools. Live 4-turn state-machine walk verified:[email protected]→ tool call →[email protected]→ tier-0[email protected]. (2) Found via live candidate inspection: RLM voters emitting “FINAL: ANSWER: x” split every mixed pool 2-vs-2 (silently costing tier-0 hits since the RLM voters shipped) — extraction now peels stacked prefixes; the fix then exposed the SAME bug in the bench grader (RLM winners’ text was mis-graded) — also fixed. (3) Grader-corrected rebench: results now published as run-to-run BANDS — cascade-rlm AIME 2026 93–97%, HMMT Feb 2026 90–95% (3 runs each; still ties GPT-5.5 on AIME, beats it on HMMT in every run). Stable service on v28.5; 179 tests.v3107-05 ~17:30All-open ladder (iter 39, honest negative with a law): RLM-capable clean arbiters shipped (open models re-solve disagreements with the voter loop; 171 tests) and the zero-proprietary-API ladder benched twice on 2026 sets: wafer tier-0 stayed superb (23/24, 13/13 @3–5 s) but the deepseek-v4-pro-RLM arbiter went 0/10 in-proxy despite 90% standalone —the hard-residue budget law: an arbiter tier inherits only the hardest problems, so it needs MORE rounds/time than standalone averages suggest (capping to 3 rounds guaranteed failure; tail solves need 500–900 s via OpenRouter). Verdict: all-open interactive today = wafer consensus alone (77% AIME26 @ ~4 s); all-open arbitration is a batch tier until the big open arbiter is locally hosted (the 96 GB box thesis, full circle). Plan-frontier cascade-rlm (97/95%) remains the production recommendation.v3007-05 ~15:30PRODUCTION CASCADE SHIPPED (iter 37)— the three real-world gaps closed: (1) streaming cascade — voter progress as reasoning deltas, tier-0 winner streamed (verified live: 1.3 s), tier-1 live-streams when no escalation inspection needed; (2) tool-carrying requests = acting-model solo (cheaper AND better than the old fanout fallback — advisory context hurts tool work); (3) context-solo guard (cascade.max_context_tokens, default 100k) — oversized requests bypass the ~128k wafer voter pool to the acting slot instead of erroring (long-context ladder: cascade <100k → plan-GPT-5.5 to ~400k at $0 → 1M open lanes when truly needed). Built by Sonnet builders + adversarial review (2 event-loop-blocking defects fixed pre-merge); 170 tests; stable service on v28.3; hermes turbo profile now gets real cascade economics on live agent traffic.v29.107-05 ~13:30Modern-open RLM (iter 38, AIME 2026): the loop lifts EVERY modern open model massively — deepseek-v4-pro 53→90% (+37 pp), glm-5.2 53→80% (+27), minimax-m3 40→60% (+20). The earlier “thinking models don’t benefit” law was a gpt-oss format artifact — REVISED: the reason→python→observe loop is a general quality multiplier for open models on novel problems (at OR latency ~80 s median: a quality tier, not a speed tier). Open solos scored 40–53% fresh — far below reputation (contamination, again). deepseek-v4-pro-RLM = strongest fully-open single-model result (90%); queued: all-open escalation ladder (wafer cascade → deepseek-RLM tier, zero proprietary APIs).v2907-05 ~12:00POST-CUTOFF REVALIDATION (AIME 2026 + HMMT Feb 2026, per Hugh’s contamination concern) — the architecture’s biggest win yet: frontier solos DROPPED on fresh problems (Fable 100→97/90%, GPT-5.5 98/95→97/85% — old ceilings were partly contamination) while cascade-rlm HELD: 97% AIME26 (ties both frontiers) and 95% HMMT26 (beats GPT-5.5 +10pp, Fable +5pp) at 3.9–5.7 s vs their 22–38 s, frontier touched 3–20%. Champion swap: always-frontier arbitration (rlm-conv 87/90%) falls behind escalation-on-discord on fresh sets — cascade-rlm is the new default (stable service + turbo switched). Also this release: STABLE SERVICE live (pinned image, restart-always, :8655) wired as hermes turbo’s default model; streaming bug found via turbo and fixed (openai-codex ignores stream=True — adapter shipped, 162 tests); GPQA: RLM law generalizes to science (gemma31 +16pp); dedicated showcase page with charts at/share/moa-cascade/. Fresh-set section added below; every result table now flags old-set numbers as contamination-suspect.v2807-05 ~08:00Consensus space closed (iters 33–34): champion validated cross-set — cascade-rlm-conv HMMT 18/20 (90%) @5.5 s (clean arbiter 7/7). Unanimity retest on the RLM pool: same 90% at 37 s — and BOTH residual misses were 4-of-4 agreed-wrong:the last errors are systematic model-class errors that no consensus rule can filter— only a stronger arbiter or a genuinely different family. mc3 dominates mc4 everywhere; final champion profile: 97% AIME / 90% HMMT @4–6 s pure cloud, vs local-hybrid convergence 95%/95%. Recipe book updated with the choose-by-set guidance.v2707-05 ~06:00ITERATIONS 31–32 — CAMPAIGN RECORD: 97% AIME, pure cloud. RLM voter slots shipped (agent: rlm on any reference slot; 88 tests). cascade-rlm (all-wafer): 92% @3.6s at just 1.7% frontier. Then the composition of every law — cascade-rlm-conv (RLM voters + clean $0-GPT-5.5 arbitration):58/60 (97%) @3.9s median, tier-0 covers 83% of traffic at 98% precision, 17% $0-frontier, zero errors, zero local hardware— essentially frontier-solo accuracy (98% @40s) at 10× lower latency. New champion; recipe book, config reference, family map updated.v2607-05 ~04:00ITERATION 30 — RLM agent on wafer (adventurous lane, per Hugh): a reason→python→observe loop (≤12 rounds, sandboxed exec) on Cerebras models vs same-model single-pass, AIME-60. Discovery:gemma-4-31b RLM 49/60 (82%) vs 41/60 solo (+14 pp) at 2.5 s mean— best small-model result of the campaign; gpt-oss-120b RLM showed no benefit (48% vs 55%, with FINAL-extraction artifacts noted). New law: externalized reasoning lifts models without internal reasoning; thinking models don’t need it. Recipe book + scorecard + tally updated. Queued: RLM lane as a cascade voter (needs proxy agent-loop support).v2507-05 ~02:00Full document overhaul (per Hugh): new executive summary with the proven-recipes list; architecture section expanded to explain every mechanism; sections renumbered into a coherent flow; new tooling & execution tally (section 13, incl. the honest “are we dogfooding?” answer); pros/cons rewritten for the cascade era. ITERATION 29 folded: cascade-convergence VALIDATES on HMMT — 19/20 (95%), tier-1 clean $0-frontier arbitration 11/11; fair mc4 retest (per-slot 15k voter cap, feature shipped) stays degenerate (87%, tier-0 never fires) — unanimity dials with thinking voters are settled as impractical; mc3 convergence is the standing optimum on BOTH sets (95%/95%). OpenRouter budget raised to $500.v2407-05 ~00:15Frontier curve completed (iteration 28): unanimity over the diverse pool degenerated to frontier-solo (tier-0 never fired — the local 32B’s thinking exhausted its voter cap before the ANSWER line; 55/56 at tier-1 = GPT-5.5-class). Curve: 93%@5% · 95%@35% (optimum) · 98%@~100% frontier. Mechanism lesson added to section 1d000.v23.107-04 ~23:15Configuration reference added (section 0, per Hugh): every tested config now defined — models, hosts, quantization, roles (voter/reference/aggregator/classifier/judge/verifier/escalation), consensus settings, routing lanes, learning-loop setup. All results-table config names resolve there. No result values changed.v2307-04 ~22:30CONVERGENCE (iteration 27, section 1d000): 95% AIME, ceiling broken by design— Qwen3-32B-AWQ on the local 5090 joined the voter pool (tier-0 39/39 PERFECT, first zero-false-consensus run) + clean $0-frontier arbitration (18/21 on the hard tail). Every measured law contributed causally.v2207-04 ~21:00Full circle (iteration 26): first hybrid local+wafer consensus pool live (Qwen3-8B on the 5090 voting beside Cerebras models; 0 errors) — 88% AIME because a weak voter’s dissent is noise. Closing law: diversity value = independence × competence; the box needs 27B+ local voters. Batch closed at 26 measured iterations.v2107-04 ~19:30Arc closed with iteration 25 (clean_arbiter shipped + measured): clean frontier arbitration confirms anchoring directionally (disagreement subset 78%→86%) but the family ceiling stands at 92.1% mean across SEVEN AIME-60 variants — stochastic false consensus is the binding term, and voter DIVERSITY (local vLLM families) is the identified next lever. Section 1d00 updated.v2007-04 ~18:15Cascade family synthesis (section 1d00): six AIME variants measured — 92±1.5% invariant; knobs trade latency/frontier-fraction only; unanimous consensus perfect cross-set (53/53). New deepest finding: voter context ANCHORS even a frontier arbiter (GPT-5.5 7/9 on disagreements vs 98% solo) — clean-escalation queued as iteration 25.v1907-04 ~17:00Dial confirmation at n=60 — honest non-monotonicity: mc4 on AIME 88% < mc3 93% (strictness demoted reliable weak consensus to a weaker tier-1 aggregator). Robust cross-set findings: unanimous tier-0 ≈ perfect (53/53 across sets), tier-2 near-perfect; tier-1 aggregator is the weak link → queued upgrade. Dial guidance rewritten task-relative.v1807-04 ~16:00Frontier dial measured (section 1d0)— strict-unanimity cascade: 20/20 HMMT (= Fable, > GPT-5.5 solo) at 2 frontier calls; dial direction confirmed: consensus strictness buys accuracy with wafer aggregation, not frontier spend. mc3 = latency default, mc4 = quality default.v1707-04 ~15:00moa:omni assembled and showcased (section 1e): one model id, four lanes, 10/10 mixed-traffic prompts routed correctly at 2.6 s median. Verified cascade (iter 19) measured honestly: neutral on AIME (92% vs 93%) — verifier independence is the binding constraint; WRONG verdicts did force correct escalations (tier-2 6/7). The quality/cost dial (escalation rate) replaces the free-lunch framing. Review-found returncode defect fixed pre-bench.v1607-04 ~13:00Judge gate measured — honest split verdict: extending tier-0 to freeform via a 300 ms wafer consistency judge gives 2× speed (2.6 s vs 5.3 s median, tier-0 on 30/30 prompts) but LOSES blind Fable-graded quality 2/2/23 vs always-on MoA — consensus transfers correctness, not polish; a raw voter answer can’t match an aggregated synthesis on subjective prose. Verdict: cascade exact-gate stays the default for verifiable work; `gate: judge` is a latency-first option; freeform quality lanes keep full aggregation. Feature + 5 tests shipped (121 green); next-iteration idea logged: judge-picks-best-voter variant.v15.107-04 ~11:30k-sampled consensus tuning: 4 voters (duplicated wafer slots, 3-of-4 consensus, zero code change) = same 93% AIME with median latency 9.9→4.4 s, tier-0 rate 33%→80%, frontier fraction 13%→5%. cascade-wafer4 is the recommended default; table row added.v1507-04 ~10:30FLAGSHIP SHIPPED: cascade mode (section 1d added)— lazy MoA with consensus gating: AIME 93% @ 9.9 s median / HMMT 90% (+15 pp over always-on MoA) at ≤15% frontier fraction, tier-0 answers in ~1.4 s with 100% consensus precision on AIME. Built ultracode-style: Fable spec + 3 parallel Sonnet builders + 2 adversarial Sonnet reviewers (3 confirmed defects fixed pre-bench); worker MoA endpoint used for agent offload. Also folded: stopped iter-15 lane 1 — plan-GPT-5.5 on TB sample 8/10 at $0 in 13 min.v1407-04 ~08:00escalation.min_failures shipped + A/B’d on SWE-bench: requiring 3 failure observations before escalating kept resolution at 20/25 (frontier parity) while cutting frontier turn share 33%→18% and escalated conversations 20→16 of 25 — kimi now drives 82% of turns with GPT-5.5 still at $0 on the plan. The over-firing lesson from v13 is fixed and measured.v1307-04 ~05:40Live plan-escalation SWE-bench result: 20/25 (80%) — parity with paid GPT-5.5 solo, kimi absorbed 2/3 of turns, frontier rode the ChatGPT plan at $0 marginal. Design lesson recorded: failure-pattern escalation over-fires on debugging workloads (tracebacks are everywhere — 20/25 conversations escalated); next refinement is escalating only on the agent’s own verify-step failure or an explicit client outcome header. Scorecard SWE row updated.v12.107-04 ~04:00Step-3.5-Flash verdict corrected after 900 s retry: OpenRouter provider returns empty responses on long generations (18/40) — provider-unreliable, excluded from lanes.Next iteration launched: live SWE-bench escalation with a plan-covered frontier lane(kimi → plan-GPT-5.5 via escalation.tiers, moa:auto, 25 instances) — frontier-quality debugging at 0 frontier cost if it holds\.v1207\-04 ~03:15**New batch: frontier/wafer/local mixes on HMMT \(section 1c added\)**— GPT\-5\.5 now rides the ChatGPT plan through the proxy \(0 marginal; codex OAuth from the live hermes store); Fable saturates HMMT (councils show no-harm, no-lift-possible); local-trio-glm52agg 85% best local; qwen3.6-27b 80% sleeper; Step-3.5-Flash timeout-bound (retry running). Anthropic OAuth not provisioned in hermes — plan-Fable blocked on one login (reminder added).v11.107-03 ~22:00Methodology caveat added to the escalation results(per Hugh’s question): offline composites (aider 30/30, SWE 25/25) are oracle-detected upper bounds — the SWE one used hidden eval tests no runtime agent can see; only the live run (router pattern-matching client-produced test output) is real-world-shaped. Boundary conditions spelled out under the quality-loop section.v1107-03 ~21:15Final iteration measurements in: verifier tool +5 pp on kimi (93%) after fixing a v1 termination artifact (documented in loop rows 10); quorum straggler-dropping = 60/60, zero accuracy cost (row 11); AIME table gains both rows. All experiment containers stopped; phase complete at ~$193 OpenRouter spend of $300.v1007-03 ~16:20AIME retest complete — new primary quality table added (section 1b): composition lifts hard reasoning (+4 to +7 pp over component solos; the old “no gain” was ceiling artifact); all-Cerebras MoA = 90% AIME @ 19 s/task; frontier saturates (Fable 60/60); nano collapses (32%). Verifier-tool bench now running; quorum A/B queued.v9.207-03 ~13:30Draft-review A/B measured and refuted(50% pass@1 vs 73.3% solo — even review-only context hurts editing; loop table row 9 added). AIME retest ~45%: frontier saturates (Fable 60/60), open tier spreads (kimi 88%, v4flash 79%). Ops note: one serve crash mid-A/B destroyed by cleanup before diagnosis — rerun was clean.v9.107-03 ~11:00Four features shipped to the fork while the AIME retest runs(all unit-tested, 103 MoA tests green): quorum straggler-dropping in the fan-out (reference_quorum_grace), multi-tier escalation (router.escalation.tiers), inverted-MoA draft_review preset mode for code, per-preset evolve skills (--per-preset). Measurement pipeline queued behind the retest: verifier-tool bench → quorum A/B on AIME → draft-review aider A/B. No result tables changed yet.v907-03 ~09:30Harder benchmark adopted: AIME 2024+2025 (60 problems)— the 40-task set saturated at kimi-class (three configs 40/40), masking quality differences. Full 10-config retest running (frontier anchors, open solos, all compositions, Cerebras lanes); an AIME results section will replace the 40-task table as the primary quality axis when it lands. Changelog section added (this table).v807-03 08:58All benchmark chains complete. Final SWE-bench numbers: gpt-5.5 20/25; kimi→Fable escalation 25/25 beats every solo. Final spend $125.v707-03 07:50Loop iters 7–8: ref-cap 600 = −17% latency free; wafer models can’t edit (gemma31 10% pass@1 @2.2s) — lane map complete, measured vision config shipped to the fork.v607-03 07:00Terminal-Bench chain complete (gpt5.5 9/10; auto/flash/kimi 6/10; heavy 5/10 timeout-bound); ref-cap trade added.v507-03 06:30Escalation lane shipped as a router feature (router.escalation) + live-validated via aider: 100% pass@2 with 3/30 exercises reaching Fable.v407-03 06:00SWE-bench escalation replication: kimi 19/25 + Fable 6/6 remainder = 25/25 composite at 24% Fable calls.v307-03 05:30Quality-loop section added (routing A/B confirmed; Self-MoA no-signal at ceiling; classifier-guessed cascade refuted; verification-gated escalation discovered). Fable baseline added.v207-03 04:00Decision-focused rewrite: scorecard, measured pros/cons, “when MoA over frontier” verdict, 8-item roadmap. (Requested: clearer summary of tests/findings.)v107-03 02:30First full report on the public share: what was built, all experiment results to date, replication guide, fork link.

Executive summary — what we built, what won, when to use what

Across 29 measured iterations this campaign turned the Hermes MoA proxy into a routed, cascading, self-improving serving system and mapped its behavior on AIME, HMMT, aider polyglot, SWE-bench Lite and Terminal-Bench. The one law that survived everything:**never predict — observe.**Observed signals (answer consensus, test failures, verification verdicts, discord) route work correctly; predicted difficulty and injected advice failed every time they were tried.

Proven configurations (the recipe book — all names resolve in section 0):

• **Frontier-supplement (cut frontier cost, keep frontier accuracy):**kimi-lane + router.escalation[gpt55-plan-lane]— SWE-bench 20/25 = paid-GPT-5.5 parity with the frontier at $0 (ChatGPT plan) and only 18% of turns (min_failures=3); aider 100% pass@2 with 3/30 tasks ever reaching the frontier.
• **THE CHAMPION (post-cutoff validated) — beat the frontier at wafer speed:**cascade-rlm— all-wafer RLM voters, wafer aggregation on disagreement, $0-frontier escalation only on observed discord:**97% AIME 2026 (ties GPT-5.5 & Fable) and 95% HMMT Feb 2026 (beats GPT-5.5 +10 pp)**on post-cutoff problems, @ 3.9–5.7 s median, frontier touched 3–20%. Running as the always-on stable service and the hermes turbo default.
• **Prior old-set champion (contamination-suspect edge):**cascade-rlm-conv— RLM wafer voters + clean $0 plan-GPT-5.5 arbitration =97% AIME @ 3.9 s median(GPT-5.5 solo: 98% @ 40 s), 17% frontier fraction at $0, no local GPU. Cheapest-frontier variant:cascade-rlm92% @ 3.6 s with 1.7% frontier. Cross-set: 90% HMMT @ 5.5 s (clean arbiter 7/7 there); on the very hardest sets the local-hybridcascade-convergence(95%/95%) is the more consistent pick — the champion’s residual misses are systematic all-voters-agree-wrong errors no consensus rule filters.
• **Near-Fable intelligence without Fable:**cascade-convergence— 95% AIME (Fable 100%, GPT-5.5 98%) from 2×Cerebras models + a local RTX-5090 Qwen3-32B voter, with $0 plan-GPT-5.5 arbitrating the 35% of requests where voters disagree. Open-only alternative:cere-agg-openrefs95% AIME.
• **All-open (zero proprietary APIs) — honest status:**interactive: wafer tier-0 consensus alone (77% AIME 2026 @ ~4 s; arbiter tier omitted). Full-quality open arbitration (deepseek-v4-pro-RLM, 90% standalone) is currently a BATCH tier — tail problems need 500–900 s via OpenRouter (hard-residue budget law: the arbiter only sees the hardest problems). Structural fix on the roadmap: host the big open arbiter locally on 96 GB-class GPUs.
• **Best local-class (fits single consumer/96GB GPUs):**gpu4-composed90% AIME (simulated 4×96GB box); strongest measured small local model: qwen3.6-27b (80% HMMT); on pg today: Qwen3-32B-AWQ on the 5090 as a perfect-precision cascade voter.
• **Best wafer(+local), near-zero frontier:**cascade-wafer4— 93% AIME @ 4.4 s median with 5% frontier; pure wafer always-on:cere-moa90% AIME @ 19 s; adding the local 32B voter makes tier-0 consensusperfect(39/39,cascade-convergencetier 0).
• **Speed up heavy fan-outs with wafer:**a Cerebras aggregator merging heavy OpenRouter references (cere-agg-openrefs) = best open accuracy at the LOWEST full-fan-out latency; plus quorum straggler-dropping (no accuracy cost) and reference caps at 600 tokens (−17% latency free). Wafer advisors briefing Fable cost only +4 s and nothing in accuracy (wafer-fable).
• Make a small wafer model punch above its weight (RLM loop):a reason→python→observe agent loop lifted gemma-4-31b from 68% to82% AIME at 2.5 s mean— small-non-thinking + tools ≈ big-model territory at a fraction of latency/cost; updated iter 43: the boundary is now precise — the lift is general across modern open NON-thinking-dominant models; heavy reasoners (gpt-oss, qwen3.6-27b) fight the loop’s short-turn structure and lose speed AND accuracy in it. Updated iter 38: on post-cutoff AIME 2026 the lift is GENERAL across modern open models — deepseek-v4-pro +37 pp (53→90%), glm-5.2 +27, minimax-m3 +20; the earlier thinking-model exception was gpt-oss format friction. RLM tiers are a quality multiplier wherever the model follows the loop format.
• **One endpoint for everything:**moa:omni— classifier routes code→solo+escalation, checkable→cascade, chat→wafer MoA, trivia→self-answer: 10/10 mixed-traffic prompts routed correctly at 2.6 s median through a single model id.
• **Self-improvement for weak fleets:**the evolve loop (+8.3 pp held-out, zero leakage) pays when the aggregator is small; big models don’t need the distilled heuristics.

**When to use frontier solos instead:**precise code editing (any multi-model involvement hurt: −13 pp fan-out, −23 pp draft-review) and time-budgeted agentic terminal work (GPT-5.5 9/10 TB; fan-out latency blows agent budgets). The router exists precisely so those lanes stay solo.

1. Configuration reference — every tested config, models & roles

**Roles:**voter= answers the request directly, its answer is compared for consensus (cascade mode) ·reference/advisor= writes advice the aggregator reads (classic MoA fan-out) ·aggregator= the acting model that produces the final answer (in cascade: the tier-1 arbiter on disagreement) ·classifier= routes moa:auto requests (~300 ms) ·judge= one-word consistency check for freeform consensus ·verifier= writes a python check script (sandboxed exec) ·escalation lane= stronger preset a conversation/cascade moves to on observed failure/discord.Hosts:@cere= Cerebras wafer API (provider-served precision) ·@OR= OpenRouter (provider-default precision) ·@plan= openai-codex OAuth (ChatGPT plan, $0 marginal) ·@5090= local vLLM on the RTX 5090. Defaults unless stated: reference_max_tokens 8000 (cascade voters) / 1500 (classic fan-out), quorum straggler-dropping on where noted, save_traces off in benches.

0.1 Cascade family (mode: cascade — voting by exact-answer consensus)

ConfigVoters (tier 0)ConsensusTier-1 arbiterTier-2 escalationExtrascascade-wafergpt-oss-120b@cere, gemma-4-31b@cere2 of 2zai-glm-4.7@cere (sees voter outputs)gpt-5.5@plan on discordquorum 0.5cascade-wafer42× gpt-oss-120b@cere + 2× gemma-4-31b@cere (k-sampling via duplicate slots, temp 0.6)3 of [email protected]@planquorum 0.8cascade-wafer4-mc4same 44 of 4 (unanimity)[email protected]@plan—cascade-wafer4vsame 43 of [email protected]@planverify: python — verifier gemma-4-31b@cere writes check scripts, run sandboxed; verify_when weak (non-unanimous consensus + all tier-1)cascade-wafer4-judgesame 43 of 4, then [email protected]@plangate: judge — gemma-4-31b@cere one-word consistency check extends tier-0 to freeformcascade-wafer4-gptaggsame 43 of [email protected]@plan—cascade-consfrontiersame 43 of 4gpt-5.5@plan(frontier arbitrates every disagreement, sees voter outputs)none—cascade-consfrontier-cleansame 43 of 4gpt-5.5@plan,clean_arbiter(re-solves from scratch, voter outputs discarded)noneanchoring guardcascade-diverse2× gpt-oss-120b@cere, gemma-4-31b@cere,Qwen3-8B@5090(BF16, vLLM)3 of [email protected]@planquorum 1.0cascade-diverse322× gpt-oss-120b@cere, gemma-4-31b@cere,Qwen3-32B-AWQ@5090(4-bit AWQ, vLLM, 16k ctx)3 of [email protected]@planquorum 1.5cascade-rlm2× gpt-oss-120b@cere + 2× gemma-4-31b@cereas RLM agents(agent: rlm — reason→python→observe loop, ≤6 rounds, sandboxed exec)3 of [email protected]@planall-wafer; frontier 1.7%cascade-rlm-convsame RLM voter pool3 of 4gpt-5.5@plan, clean_arbiternonethe champion: 97% AIME @3.9 sAll cascade presets support:tool_turns: detect(default — sessions revert to cascade after tool phases via a tool-aware voter gate; “solo” pins tool-carrying requests to acting-solo),max_context_tokens(default 100k — oversized requests bypass voters), streaming (tier-0 winner streamed; tier-1 live when no escalation inspection).cascade-convergence2× gpt-oss-120b@cere, gemma-4-31b@cere, Qwen3-32B-AWQ@50903 of 4gpt-5.5@plan, clean_arbiternonethe 95% result — diversity + anchoring laws composed

0.2 Routed / escalation / omni configs (moa:auto)

ConfigClassifierLanes (route → preset)Failure escalationmoa:auto v1gemma-4-31b-it@ORcoding→open-moa-heavy · math→open-moa-alt · general→open-moa-flash · trivial→SELF (classifier answers)—moa:auto v2 (evidence-informed)gemma-4-31b@cere (paid)coding→open-kimi-solo (solo lane) · hard-reasoning→open-moa-heavy · general→cere-fast · SELF on—escalation-live (serve-v4)gemma-4-31b@cereall coding→kimi-lane (kimi-k2.6@OR solo)tiers [fable-lane = claude-fable-5@OR], min_failures 1 (failure text in latest client/tool message)plan-escalate / esc2gemma-4-31b@cereall→kimi-lanetiers [gpt55-plan-lane], min_failures 1 (v13) then 3 (v14)frontier-cascade (refuted)gemma-4-31b@cereroutine-code→zai-glm-4.7@cere · hard→fable-lane@OR (classifier-GUESSED difficulty)—moa:omnigemma-4-31b@cerecode→omni-code (kimi-k2.6@OR solo) · checkable→omni-reason (= cascade-wafer4) · chat→omni-chat (2-voter cere fan-out → zai-glm-4.7) · trivial→SELFtiers [gpt55-plan-lane], min_failures 2kimi-reviewed (draft_review, refuted)—aggregator kimi-k2.6@OR drafts solo → reviewers deepseek-v4-flash@OR + glm-5.1@OR critique → kimi revises—

0.3 Classic MoA fan-out combos (references advise → aggregator acts)

ConfigReferences/advisorsAggregatorNotesopen-moa-heavy / mixed-heavydeepseek-v4-pro, qwen3.7-max, z-ai/glm-5.2 (all @OR)kimi-k2.6@ORcap 1500; “+quorum” variant adds grace 0.5open-moa-altkimi-k2.6, minimax-m3, deepseek-v4-flash @ORdeepseek-v4-pro@ORopen-moa-flashdeepseek-v4-flash, qwen3.6-35b-a3b, glm-5.1 @[email protected], ministral-8b-2512, gemma-3-12b @ORgemma-3-27b@ORdeliberately fallible — the learning-loop subjectgpu4-composedqwen3-coder-next, gpt-oss-120b, qwen3-next-80b-a3b-thinking @ORqwen3.5-122b-a10b@ORsimulates 4× single-96GB-GPU modelscere-moagpt-oss-120b, gemma-4-31b @cerezai-glm-4.7@cerethe always-on wafer MoA baselinecere-agg-openrefs-{gemma,gptoss}heavy refs @ORgemma-4-31b / gpt-oss-120b @cerewafer fast-merge over strong refswafer-fable / wafer-gpt55plangpt-oss-120b + gemma-4-31b @cere (cap 2000)claude-fable-5@OR / gpt-5.5@planwafer advisors briefing a frontier judgecouncil-fablegpt-5.5@plan + claude-opus-4.8@OR (cap 2000)claude-fable-5@ORfrontier councillocal-trio-stepagg / -glm52aggdeepseek-v4-flash, qwen3.6-27b, glm-4.7-flash (or step-3.5-flash) @ORstep-3.5-flash / glm-5.2 @ORlocally-runnable-class stacks

0.4 Solos, learning loop & misc

ItemDetailSolo baselines (X-solo)the named model as sole aggregator through the identical code path (disabled preset, no references): fable/opus/kimi/deepseek-pro/-flash/qwen*/nemotron-550b/qwen3.5-397b/step-3.5-flash @OR; gpt-5.5 @OR and @plan; gemma-4-31b/gpt-oss-120b/zai-glm-4.7 @cereLearning loop (evolve)subject open-moa-nano; 20 train + 20 held-out tasks; traces graded with correct/expected; distiller deepseek-v4-pro@OR rewrites the ≤4KB aggregation-heuristics skill injected into every aggregator prompt; leakage-scanned; per-preset skills via --per-presetRouter/judge/verifier slotgemma-4-31b@cere throughout (100% routing acc, p50 288 ms on the paid key); free-tier key RPM-collapses (measured)Verifier sandboxpython -I, empty env, tempdir cwd, 12 s timeout, non-zero exit = inconclusive; verdicts only ever strike/escalate, never block Full machine-readable configs: the serve YAMLs and bench CONFIG dicts in thefork branch(scripts/moa_*_bench.py, docs/plans/moa-routed-config-example.yaml, docs/plans/moa-cascade-spec.md); every results table’s config name matches a row above.

2. Architecture — how each mechanism works

**The proxy (hermes moa serve).**An OpenAI-compatible/v1/chat/completionsserver where every configured preset is a model id (moa:<preset>). The CLIENT owns tools and turn termination: the proxy forwards the client’s tool definitions to the acting model and streams tool_calls back — so any agent harness (aider, mini-swe-agent, Harbor/terminus-2, OpenClaw, Claude Code via LiteLLM) can use these compositions as “a model” with zero integration work. Reference/voter thinking streams as reasoning deltas;usage.moaitemizes every upstream call. Hard per-slot timeouts (slot_timeout_s, 300 s) after real wedges. Provider slots resolve through the standard Hermes chain, which is how Cerebras (custom provider), local vLLM (custom provider), OpenRouter, and plan-OAuth models (openai-codex; anthropic once logged in) all mix freely in one preset.

Classic fan-out (mode: fanout, the original MoA).Every reference slot receives an advisory view of the conversation (with an advisor system prompt and truncated tool results), writes advice in parallel; the aggregator then acts with that advice appended as a guidance block. Measured: lifts hard reasoning +4–7 pp over its own aggregator solo; HURTS precise code editing (−13 pp) and never helps agentic coding. Latency = slowest reference → mitigations:reference_max_tokens: 600(−17% latency, free) andquorum straggler-dropping(reference_quorum_grace: once all but one reference finish at elapsed T, the straggler gets T×grace then is dropped with a note — no accuracy cost measured, 60/60).

**Cascade (mode: cascade— lazy MoA, the flagship).**Voters (the reference slots) answer the request DIRECTLY (no advisory framing) in parallel at tier 0; a candidate answer is extracted from each (ANSWER-line → balanced-\boxed{} → short last line; failure boilerplate can never vote). If ≥min_consensusnormalized candidates agree, the first agreeing voter’s full text returns immediately — no aggregation at all (~1.4–6 s, and consensus precision measured 96–100%; PERFECT 39/39 with a competent-diverse pool). On disagreement, tier 1: the aggregator acts, normally seeing the voters’ work — or, withclean_arbiter: true, re-solving from scratch (measured: voter context ANCHORS even frontier arbiters — 7/9 vs 98% solo). If the tier-1 answer agrees with no voter andescalate_tonames a preset, tier 2 escalates to that preset’s aggregator. Optional gates:gate: judge(a 300 ms wafer consistency check extends tier-0 to freeform — 2× speed, but costs blind-graded polish 2/2/23, latency-first lanes only) andverify: python(a verifier slot writes a check script, executed sandboxed — python -I, empty env, tempdir, 12 s, non-zero exit distrusted; WRONG strikes the consensus or forces escalation; only pays when checks are computational). Per-slotmax_tokensoverrides let thinking-heavy voters finish (an 8 k cap silenced a 32B voter’s candidates entirely).**Production semantics (v1.4):**streaming tool-free requests run the true cascade (voter progress as reasoning deltas; a tier-0 winner arrives as one content delta — 1.3 s measured live; tier-1 streams token-by-token unless an escalation tier must inspect the answer first); tool-carrying requests run the acting slot SOLO with tools forwarded (no voter overhead — advisory context measurably hurts tool work); requests estimated overcascade.max_context_tokens(default 100k) bypass the ~128k wafer voters to the acting slot (“context-solo”) instead of erroring — the long-context ladder being cascade <100k → plan-GPT-5.5 to ~400k at $0 → 1M-class open lanes on explicit routes.

**Router (moa:auto).**A wafer classifier (gemma-4-31b@cere: 100% accuracy, p50 288 ms) reads the request tail and picks a route-described preset; SELF class answers trivia directly. Decisions are sticky per conversation (session header or first-user-message hash) so tool loops never re-classify.Failure-gated escalation: when a sticky conversation’s latest client/tool message carries failure markers, it re-routes one tier up (escalation.tiers), gated bymin_failures(debugging prints tracebacks as normal work — first-failure escalation over-fired on 80% of SWE conversations; min_failures=3 cut frontier turns 33%→18% at identical resolution). Disabled presets with a route block are routable solo lanes — how coding traffic bypasses the fan-out penalty.

**Learning (hermes moa evolve).**Withsave_traceson, every fan-out turn records the full advisory inputs/outputs plus optional externally-graded outcomes and the routing decision. Evolve replays recent traces through a distiller LLM that REWRITES a bounded (≤4 KB) aggregation-heuristics skill, injected into every subsequent aggregator prompt;--per-presetkeeps lane heuristics separate. Measured on a weak fleet: +8.3 pp held-out across 3 cycles, zero answer leakage; cost is +40–70% latency from the verification habits it teaches.

**Plan integration.**Mounting the live~/.hermes/auth.json(single shared store — never copy it; OAuth refresh rotation would split-brain) into the serve container lets slots useprovider: openai-codex→ GPT-5.5 at $0 marginal on the ChatGPT plan. The same path unlocks plan-Fable/Opus oncehermes auth login anthropichas been run.

Test surface: 82+ hermetic MoA tests + 9 live integration tests; slim docker release (docker-compose.moa.yml) serves a fresh clone in ~3 commands.

3. Scorecard — every test we ran

TestWhat it measuresWinnerKey numbersExact-answer reasoning, 40 verified taskscomposition vs big solos (4×96 GB sim)open MoA (composed)composed 40/40 · v4-flash 38 · 122B/gpt-oss solos 37 · 397B 35 · 550B 24*Cerebras slots, same 40 taskswafer-speed classifier/aggregator/MoACerebras (all 3 roles)fast-merge agg 40/40 (fastest fan-out) · all-Cerebras MoA 38/40 @ 6.8 s · gemma31 solo 39/40 @ 1.1 sLearning loop, 3 evolve cycles, train/held-outdoes MoA improve with use?evolve (works)+1/+3/+1 per cycle · 73.3% vs 65.0% paired (+8.3 pp) · 0 leakage · +40–70% latencyRouting classifier, 48 labelled casesmoa:auto accuracy + overheadCerebras paid100% acc, p50 288 ms (OpenRouter sub: 100%, 338 ms; free Cerebras: quota-collapse, all contained)Aider polyglot, 30 Python exercisesprecise code editingfrontier soloOpus 90% ·Fable 87%@13.6s/100% pass@2· GPT-5.5 73% · kimi solo 73% · moa-heavy 60% · moa-flash 43%SWE-bench Lite, 25 instances (mini-swe-agent)agentic debuggingescalationgpt-5.5 20/25 (80%) · kimi-solo 19/25 (76%) · moa-heavy 17/23 · kimi→Fable oracle composite 25/25 (upper bound) ·LIVE kimi→plan-GPT-5.5 escalation: 20/25 (80%) = paid-frontier parity at $0 frontier cost; with escalation.min_failures=3 the same 20/25 holds at only 18% frontier turn share (kimi drives 82%)Routing A/B (aider, moa:auto v1 vs v2)does routing fix the coding penalty?v2 routingv1 (coding→fan-out) 63.3%@527s → v2 (coding→solo lane) 66.7%@235s; 71/73 turns routed correctly — kimi-parity within single-run noiseFrontier cascade (aider, classifier-guessed difficulty)fewer Fable calls via up-front routingfailedcheap lane (Cerebras GLM-4.7) 30% pass@1 — guessing difficulty up-front fails when the cheap lane can’t do the formatVerification-gated escalation (kimi → Fable on test failure)Fable quality at fewer Fable callsworkscomposite30/30 (100%)with test-feedback escalation — and replicated on SWE-bench Lite:kimi 19/25 + Fable 6/6 on the remainder = 25/25 resolved with Fable on 24% of instancesSelf-MoA vs mixed refs vs solo (40 tasks, kimi aggregator)do mixed references beat self-ensembles?ceiling — no signalall three 40/40; solo used 4× fewer tokens — composition only adds cost once the model is at ceilingTerminal-Bench sample, 10 tasks (Harbor/terminus-2)agentic terminal workfrontier soloGPT-5.5 9/10· moa:auto 6/10 · flash 6/10 · kimi 6/10 · heavy 5/10 (6 agent-timeouts — fan-out latency kills time-budgeted work)TP=2 over PCIe vs independent GPUs (5090+4090, vLLM)multi-GPU serving strategyindependent per-GPUTP=2: 0% single-stream gain, 1016 vs 2212 tok/s aggregate (2.2× worse)RLM wafer-agent loop vs single-pass (AIME-60, iter 30)gemma-4-31b: RLM 82% vs solo 68% (+14 pp) @2.5 s · gpt-oss-120b: no benefitagentic wafer lane proven * 11 of the 550B model’s 16 misses were empty/invalid responses from its OpenRouter provider — frontier-open serving is itself still flaky, which is a reliability argument for compositions that degrade gracefully (a failed reference becomes a note; a failed monolith is a failed request).

4. Primary quality axis — AIME 2024+2025 (hard benchmark)

The original 40-task set saturated at kimi-class (three configs 40/40), so AIME 24+25 (60 competition problems, exact integer answers) replaced it as the primary quality axis. Same code path, same configs:

ConfigAIME accAvg latencyTok/taskfable-solo (claude-fable-5)60/60 (100%)24 s2 191gpt55-solo59/60 (98%)48 s2 696cere-agg over heavy refs (gpt-oss merge)— best open57/60 (95%)468 s38 285mixed-heavy (3 refs → kimi)55/60 (92%)488 s42 442gpu4-composed (4 single-GPU models)54/60 (90%)223 s50 807cere-moa (all-Cerebras)— efficiency champion54/60 (90%)19 s21 545kimi-solo53/60 (88%)288 s19 052kimi-solo + python verifier tool (v2 harness)56/60 (93%)455 s—mixed-heavy + quorum straggler-dropping60/60 (100%)508 s43 240v4flash-solo47/60 (78%)190 s8 718cere-gemma31-solo46/60 (77%)1.7 s2 572open-moa-nano19/60 (32%)243 s34 727 What the hard benchmark changes:(1) compositiondoeslift hard reasoning — +7 pp for a Cerebras-merged heavy fan-out over kimi solo, +4 pp for mixed-heavy, and every composition beats its own parts (the earlier “no gain” was a ceiling artifact of the saturated 40-task set); (2)all-Cerebras MoA delivers 90% AIME at 19 s/task— gpu4-composed accuracy at 12× lower latency, the practical open fast lane; (3) frontier saturates AIME (Fable perfect), so open-tier improvements are where the measurement signal lives; (4) small-model MoA (nano) collapses on hard problems (32%) — composition cannot rescue models far below the task.

4b. POST-CUTOFF REVALIDATION — AIME 2026 + HMMT Feb 2026 (the numbers that count)

Both competitions were administered February 2026 — after the training cutoff of every model in the pool. 50 gradable problems, identical harness. This is the contamination-resistant scoreboard:

ConfigAIME 2026 (30)HMMT Feb 2026 (20)Median latencyFrontier use**cascade-rlm (new default)**93–97% (3 runs)90–95% (3 runs)3.9 / 5.7 s3% / 20%cascade-rlm-conv (old champion)26/30 (87%)18/20 (90%)3.3 / 5.0 s20% / 25%GPT-5.5 solo ($0 plan)29/30 (97%)17/20 (85%)29 / 38 s100%Fable solo29/30 (97%)18/20 (90%)22 / 28 s100%cere-moa (always-aggregate wafer)26/30 (87%)15/20 (75%)10 / 15 s0% Three findings.(1)*Contamination was real:*frontier solos dropped 1–10 pp on fresh problems (Fable 100→97/90, GPT-5.5 98/95→97/85) — the old-set ceilings flattered them. (2)*The architecture generalizes:*cascade-rlm matched its old-set numbers on problems no model had seen — tying both frontier solos on AIME 2026 and BEATING them on HMMT Feb 2026 (+10 pp over GPT-5.5, +5 pp over Fable) at 6–8× lower latency. Consensus among diverse fast agents is measuring problem difficulty, not memorization. (3)*Champion swap:*the always-frontier clean arbiter (rlm-conv) fell behind — its old-set edge was partly the arbiter’s contamination; observation-gated escalation (cascade-rlm: wafer aggregator first, frontier only on discord) is the better architecture on genuinely novel problems, and is now the default on the stable service and the turbo profile. All older sections retain AIME 24+25 / HMMT 2025 numbers — read those as contamination-suspect upper bounds for the frontier solos, and as validated-by-revalidation for the cascade family.

5.1 The cascade program — flagship: lazy MoA with consensus gating

Built 2026-07-04 (ultracode batch): the campaign’s one law is thatpredicteddifficulty fails andobservedsignals win — somode: cascadegates on the cheapest observation there is: answer agreement between wafer-speed voters. Tier 0: Cerebras gpt-oss-120b + gemma-4-31b answer directly in parallel (~1.5 s); unanimity returns instantly. Tier 1: on disagreement the aggregator synthesizes from the voter outputs already paid for. Tier 2: when the aggregator agrees withnovoter (observed discord = genuine hardness), escalate to plan-GPT-5.5 ($0). Implementation farmed to parallel Sonnet builders with adversarial review (3 confirmed findings fixed: nesting-aware boxed extraction, failure-boilerplate votes, trace attribution); 98 tests green.

MetricAIME 24+25 (60)HMMT (20, harder)cascade-wafer (2 voters) accuracy56/60 (93%)18/20 (90%)**cascade-wafer4 (4 voters, 3-of-4)**56/60 (93%) · median 4.4 s · tier-0 rate 80% · frontier 5%—vs always-aggregate cere-moa90%75% —cascade +15 ppvs plan-GPT-5.5 solo98%95%median latency9.9 s (tier-0: 1.4 s)19.6 sfrontier fraction (all $0 on plan)13%15%tier split (t0/t1/t2)20/31/84/13/3tier-0 consensus precision20/20 (100%)3/4tier-2 accuracy7/83/3 **Why this is the product:**it strictly dominates its own always-on MoA (better accuracy on both sets, half the median latency, no aggregator cost on easy traffic), reaches within 2–5 pp of a frontier solo while touching the frontier on ≤15% of requests, and every tier below the top runs on wafer/local-class models — this is the 4×96 GB box’s serving architecture, live today:mode: cascadein one preset. Smoke: 17×23 answered end-to-end in 1.16 s. The cascade self-adapts to difficulty: easy sets ride tier 0, hard sets shift to tiers 1–2 — no classifier, no prediction, just observation.

5.2 The frontier dial — measured

Consensus strictness IS the quality/cost dial. HMMT Feb 2025 (hard set), same wafer voters throughout:

Dial positionHMMT accMedian latFrontier calls2 voters, unanimity (mc2)18/20 (90%)19.6 s3/204 voters, 3-of-4 (mc3)17/20 (85%)15.2 s3/20**4 voters, unanimity (mc4)**20/20 (100%)15.5 s2/20GPT-5.5 solo (reference)19/2040 s20/20Fable solo (reference)20/2033 s20/20 The strict-unanimity cascade matched Fable and edged GPT-5.5 solo on the hard set with TWO frontier calls (both $0 on the plan)— stricter consensus routes marginal cases into tier-1 aggregation, which went 12/12 here. AIME-60 confirmation (v19) sharpened this honestly: mc4 scored 88% on AIME — BELOW mc3’s 93% — because strict unanimity demoted 22 problems to tier-1, whose aggregator went 16/22, worse than the 95.8%-precise weak consensus it replaced. The dial is task-relative: where voters are competent (AIME), trust weak consensus; where stretched (HMMT), strictness pays. The two cross-set constants: unanimous tier-0 is essentially perfect everywhere measured (33/33, 20/20), and tier-2 is near-perfect (4/4, 3/3, 7/8) — the tier-1 aggregator (glm-4.7) is the weak link and the next upgrade target.

5.3 Cascade family — complete measured map

Variant (AIME-60 unless noted)AccMedianFrontier2 voters, unanimity93%9.9 s13%4 voters, 3-of-4 (glm agg)93%4.4 s5%4 voters, 3-of-4 + verification92%4.8 s12%4 voters, unanimity88%7.2 s7%4 voters, 3-of-4 (gpt-oss agg)90%4.3 s2%cascade-rlm (RLM wafer voters, iter 31)92%3.6 s1.7%**cascade-rlm-conv (RLM voters + clean $0 arbiter, iter 32)**97% — record3.9 s17%consensus-or-frontier (GPT-5.5 arbitrates all disagreements)93%3.5 s15%4 voters, unanimity — HMMT (hard set)100% (20/20)15.5 s10% The measured laws of the cascade family:(1) it sits at 92±1.5% AIME across every knob — the knobs trade latency (3.5–10 s median) and frontier fraction (0–15%), not accuracy; (2) consensus precision is the engine (unanimous tier-0: 53/53 cross-set; 3-of-4: ~96%); (3) the residue is bounded by ~4% false consensus plus frontier-hard items; and (4) — the deepest finding —voter context anchors even a frontier arbiter: GPT-5.5 judging disagreements scored 7/9 where GPT-5.5 solo runs 98% — the same contamination measured in code editing, now in hard reasoning. Iteration 25 tested clean escalation: the clean frontier arbiter improved the disagreement subset directionally (78% → 86%, small n) — anchoring is real — but the total held at 92% because stochastic false consensus (~4–5%, different problems each run) is the family’s true ceiling. Seven AIME-60 variants land at 92.1% mean.The remaining lever is voter DIVERSITY, not architecture: Cerebras offers three model families; adding independent families (e.g. local vLLM voters on the 5090/4090 — exactly the 4×96 GB box thesis) attacks the correlated-error term directly. Iteration 26 tested it live: a LOCAL Qwen3-8B voter (vLLM on the RTX 5090) joined the Cerebras voters in one consensus pool — the hybrid ran flawlessly (60/60 turns, 0 errors, hybrid tier-0 consensus at ~5 s) but scored 88%: the weak voter’s dissent was noise, inflating tier-1 volume without cutting false consensus.**The closing law: diversity value = independence × competence.**An 8B is below the bar; the 4×96 GB box thesis holds precisely because 96 GB cards run 27B–120B-class voters (qwen3.6-27b measured 80% HMMT) that can be right when the wafer models collude wrong.

5.4 CONVERGENCE RESULTS — 95% with local silicon; 97% pure-cloud (the record)

Update (iterations 31–32):composing the RLM discovery into the convergence pattern beat the local-GPU result:cascade-rlm-conv(RLM wafer voters + clean $0 frontier arbitration) scored58/60 (97%) at 3.9 s median— tier-0 handled 83% of traffic at 98% precision (49/50 @ 3.0 s), the clean arbiter took the tail 9/10, frontier fraction 17% at $0, zero errors, zero local hardware. The local-voter convergence below remains the proof that on-box silicon slots in natively (and the best result when plan quota is scarce — its tier-0 was perfect).

Iteration 27 composed every measured law into one preset:competent-diverse voters(2× Cerebras gpt-oss-120b + gemma-4-31b +Qwen3-32B-AWQ running on the local RTX 5090) +clean plan-frontier arbitration(GPT-5.5 re-solves every disagreement from scratch — anchoring guard):

Metriccascade-convergence (AIME-60)Accuracy57/60 (95%) — breaks the 92±1.5% family ceilingTier-0 consensus precision39/39 — PERFECT (zero false consensus, first ever)Tier-1 (clean $0 frontier) on the hard tail18/21 (86%)Frontier fraction / cost35% of requests · $0 (ChatGPT plan)Median latency / errors14.1 s (tier-0: 6.6 s) · 0 errors**HMMT validation (iteration 29)**19/20 (95%) — matches GPT-5.5 solo; tier-0 8/9 @6.0s; clean arbitration 11/11; 0 errors **Why this one matters:**for the first time the number moved because of design, not noise — tier-0 became perfect BECAUSE the diversity law (independence × competence) fixed false consensus, and tier-1 lifted BECAUSE the anchoring law demanded clean arbitration. 95% vs GPT-5.5 solo’s 98%, built from Hugh’s own GPU + wafer inference + plan-covered frontier on a third of requests. This is the 4×96 GB box architecture, measured end-to-end with real local silicon in the pool.

**The completed frontier curve (iteration 28 closed it):**93% @ 5% frontier (cascade-wafer4) →95% @ 35% frontier (cascade-convergence — the optimum)→ 98% @ ~100% (unanimity-of-4 degenerated to frontier-solo: the thinking-heavy local voter often exhausted its 8000-token cap before emitting a candidate, making 4-of-4 unreachable — mechanism lesson: unanimity dials require every voter to reliably emit candidates; thinking voters need caps ≥ their reasoning budget). Pick your point on the curve per lane; all frontier usage rides the $0 plan.

6. Frontier / wafer / local mixes — HMMT Feb 2025

New batch (2026-07-04): can frontier+wafer+plan mixtures be “fast and smart”, and how far do locally-runnable-class stacks go? HMMT Feb 2025 (gradable 20-problem subset — integers/simple fractions), quorum on. GPT-5.5 rides theChatGPT planthrough the proxy (live hermes auth mounted, zero marginal cost); Opus/Fable still via OpenRouter (anthropic OAuth isn’t provisioned in hermes yet — one `hermes auth login anthropic` would unlock plan-Fable).

ConfigHMMT accAvg latencyReadingfable-solo20/2033 sFable saturates HMMT too — competition math can’t discriminate the frontier any morecouncil-fable (plan-GPT-5.5 + Opus refs → Fable)20/2051 sno lift possible at ceiling; no harm eitherwafer-fable (Cerebras refs → Fable)20/2037 swafer advice costs only +4 s and nothing in accuracygpt55-plan-solo19/2040 sthe value king: 95% HMMT at $0 marginal (plan)wafer-gpt55plan18/2047 swafer advice didn’t lift GPT-5.5 (−1, noise range)local-trio-glm52agg (v4flash + qwen27 + step refs → GLM-5.2)17/20 (85%)659 sbest local-class score — composition lifts again (+5 pp over best part), but 11 min/problemqwen27-solo (qwen3.6-27b)16/20 (80%)976 sthe local sleeper: 80% HMMT from a 27B — at 16 min of thinkingcere-moa15/20 (75%)25 sthe speed-quality frontier below GPT-5.5: 75% at 25 sstep35-solo / local-trio-stepagg8/20 · 9/20 *~500–590 s* retry at 900 s reproduced the failures: the OpenRouter provider returns empty/invalid responses on long generations (18/40 runs) — provider-broken, not model-refuted; completed-run accuracy ~70–80% Batch findings so far:(1) plan-integrated GPT-5.5 through the proxy is the single best value config for hard reasoning; (2) frontier councils can’t be evaluated on competition math any more — Fable saturates AIMEandHMMT; their test needs long-horizon agentic work; (3) the local tier repeats the composition story (+5 pp) but pays minutes of latency — the practical local sweet spot remains cere-moa-style wafer serving unless quality is paramount; (4) qwen3.6-27b is the strongest small local model we’ve measured.

7. moa:omni — the product, live

Every measured optimization assembled behind ONE model id (moa:auto): the 288 ms Cerebras classifier routes each request to its measured-best lane — checkable problems → consensus cascade (93% AIME @ 4.4 s median, 5% frontier), code → kimi solo with plan-frontier escalation on observed failure, chat → wafer MoA, trivia → wafer self-answer. Mixed-traffic showcase (10 prompts across all four classes):10/10 routed correctly, 2.6 s median latency— math answered by tier-0 consensus in 0.9–2.4 s, trivia in 0.7 s. Config shipped as the example in the branch.

**Verified cascade (iteration 19) — honest verdict:**adding wafer-generated python verification was NEUTRAL on AIME (92% vs 93%): the two WRONG verdicts correctly forced tier-2 escalations, but false consensus survived because a 31B verifier cannot independently check what 120B voters got wrong — verification pays exactly when the check is pure computation (the earlier +5 pp), not when it needs the missing insight. The honest frame: the wafer cascade ceilings at ~93% AIME, and above that you turn thefrontier dial(escalation rate) — 5% frontier → 93%, 100% frontier → 98% — rather than expecting free accuracy from tricks.

8. MoA pros & cons — updated for the cascade era

Measured pros

• Consensus is a near-perfect, near-free correctness signal: unanimous tier-0 53/53 cross-set; perfect 39/39 with a competent-diverse pool — the engine of the cascade.
• Fan-out lifts hard reasoning+4–7 pp over its own aggregator solo; composition beat every same-class solo (gpu4-composed 90% AIME).
• Escalation gives frontier parity at a fraction of frontier usage: SWE 20/25 at 18% $0-frontier turns; cascade-convergence 95%/95% (AIME/HMMT) at ~35%.
• Wafer slots collapse latency: classify 288 ms, tier-0 answers 1.4–6 s, fast-merge aggregation at zero accuracy cost.
• It learns(+8.3 pp for weak fleets, zero leakage) anddegrades gracefully(failed/dropped voters become notes, never failures).
• Local silicon slots in natively— a 5090-hosted voter made consensus perfect; the 4×96 GB box architecture, proven end-to-end.

Measured cons

• Advisory context contaminates precise work: −13 pp code editing, −23 pp draft-review, and it anchors even frontier arbiters (7/9 vs 98% solo) — hence solo lanes and clean_arbiter.
• **Wafer-cascade accuracy ceilings at ~92–95%**on AIME-class reasoning; the residue needs the frontier dial (verification is neutral when the check needs the missing insight).
• Fan-out latency kills time-budgeted agentic work(TB heavy: every failure an agent-timeout).
• Weak voters are noise; thinking voters break unanimity— diversity value = independence × competence, and candidates must actually be emitted (per-slot caps shipped, but 4-of-4 with a thinking 32B still never fired).
• **Token multiplier ~4–6×**vs cheap solos: MoA wins the cost argument only against frontier prices or when consensus short-circuits traffic.

9. Detailed results (aider / SWE-bench / Terminal-Bench / learning / multi-GPU)

Reasoning quality (40 verified exact-answer tasks)

ConfigAccLatencyTok/taskgpu4-composed (coder-next + gpt-oss + 80b-think → qwen3.5-122b)40/4066 s14 598cere-agg over heavy refs (gpt-oss merge)40/4061 s6 904cere-agg over heavy refs (gemma31 merge)40/4076 s9 399cere-gemma31 solo (wafer)39/401.1 s1 801all-Cerebras MoA38/406.8 s9 509tp2-class solo: deepseek-v4-flash38/4023 s2 336gpu1 solos: qwen3.5-122b / gpt-oss-120b37/40124 / 19 s8 189 / 1 199tp4-class solos: qwen3.5-397b / nemotron-550b35 / 24*383 / 32 s9 798 / 1 246

Code editing (aider polyglot, 30 Python, whole format)

Configpass@1pass@2s/caseRun costfrontier-opus-solo90.0%100%9.90\.95frontier\-gpt55\-solo73\.3%100%22\.71.71open-kimi-solo73.3%96.7%1731\.02open\-moa\-heavy \(3 refs → kimi\)60\.0%96\.7%347≈2.10open-moa-flash43.3%93.3%341≈0\.60**fable\-solo \(claude\-fable\-5\)**86\.7%100%13\.6≈1.90moa:auto v1 (coding→fan-out)63.3%96.7%527≈2\.10moa:auto v2 \(coding→kimi solo lane\)66\.7%96\.7%235≈1.10cascade (classifier→GLM-4.7/Fable)30.0%70.0%19.6≈0\.70**moa:auto \+ escalation lane \(LIVE, kimi→Fable on failure\)**83\.3%100%36\.63/30 exercises reached Fable**kimi→Fable escalation \(verification\-gated\)**100% composite \(pass@2\-style\)—~180≈1.50, −73% Fable calls Costs from measured tokens × live OpenRouter prices. The MoA pass@1 penalty (−13 pp vs its own aggregator) replicated on both combos → routing now supports solo lanes so coding traffic can bypass the fan-out.

Agentic benchmarks

BenchmarkConfigResultSWE-bench Lite (25 instances, mini-swe-agent, local docker eval)open-kimi-solo19/25 resolved (76%)open-moa-heavy17/23 resolved (2 instances wedged on fan-out latency) — confirms: no MoA gain on agentic codingfrontier-gpt55-solo20/25 resolved (80%)Terminal-Bench sample 2.0 (10 tasks, terminus-2, 2× timeout)frontier-gpt55-solo (paid / $0 plan)9/10 · 8/10 @ $0 in 13 minopen-moa-flash / open-kimi-solo /moa:auto6/10 each (auto: fewest timeouts of the MoA configs)open-moa-heavy5/10 — all failures were agent-timeouts (fan-out latency)SWE heavy/gpt5.5running Context: the official bash-only SWE-benchVerifiedleaderboard tops out around 76.8% (Claude 4.5 Opus) — different subset, so not directly comparable, but 76% from an open solo through our proxy is squarely competitive. Results below will be appended as chains finish.

Learning loop (evolve), paired held-out evals

CycleWith skillWithout (paired)Lift1 / 2 / 313 · 16 · 15 /2012 · 13 · 14 /20+1 · +3 · +1Aggregate44/60 (73.3%)39/60 (65.0%)+8.3 pp, flips 6:1 Zero leakage in all distilled skills (heuristics + per-model trust notes only). Residual misses are tool-shaped (exact big-number arithmetic) — see roadmap.

Multi-GPU throughput (RTX 5090 + 4090, vLLM, Qwen3-8B)

ConfigSingle-streamAggregate @165090 alone / 4090 alone98.2 / 58.4 tok/s1380 / 832 tok/sTP=2 over PCIe97.1 tok/s (no gain)1016 tok/s (2.2× worse than independent) Full 4×96 GB plan (hardware dossier, model fits, buy verdict):docs/plans/multi\-gpu\-moa\-plan\.mdin the branch.

10. Quality-improvement loop — every iteration & verdict

Hypothesize → test → report, one iteration at a time:

#HypothesisResultVerdict1Routing coding to a solo lane removes the MoA editing penaltymoa:auto v2: 66.7%@235s vs v1 63.3%@527s; 97% of turns routed correctly; kimi-parity within noiseconfirmed — v2 config shipped as the example2Self-MoA (best model ×3) beats mixed referencesAll configs 40/40 — kimi is at ceiling on the task set; solo costs 4× lessno signal at ceiling; retest on harder/agentic tasks3aClassifier-guessed difficulty can cascade cheap→Fable30% pass@1: “routine” work still exceeded the cheap lane’s editing abilityrefuted — don’t guess difficulty up-front3bVerification-gated escalation: cheap model first, Fable only on test failurekimi (22/30) + Fable on the 8 failures =30/30 composite with test feedback (8/8 fixed; 4/8 first-try) — full Fable pass@2 quality with Fable invoked on 27% of tasksconfirmed — the right cascade for verifiable work; 4 exercises beat both models first-try5Productize it: failure-gated escalation lane in the router (router\.escalation)Shipped + 6 unit tests: a sticky conversation whose latest tool/user feedback shows a failure marker re-routes once to the escalation preset; assistant text never triggers; stale failures don’t re-triggershipped (commit 1883a1263)6Live end-to-end: aider drives moa:auto with the escalation lane (kimi first, Fable on failure)83.3% pass@1 / 100% pass@2 @ 36.6 s/case — only 3 of 30 exercises ever reached Fable (90% never touched the frontier); routing trace: 26 classified→kimi, 3 escalated, escalated conversations stayed on Fableconfirmed in production shape10A python verifier tool lifts the acting model on hard reasoningHarness v1 (4-round cap) collapsed accuracy — a termination artifact (31/36 misses were empty answers, models still verifying). Harness v2 (8 rounds + forced final answer):kimi-solo 56/60 (93%) vs 88% baseline — +5 pp; cere-moa 52/60 (87%) vs 90% — no gain for wafer MoAconfirmed for strong thinking solos; artifact documented; tool costs +58% latency on kimi11Quorum straggler-dropping costs accuracymixed-heavy WITH quorum (grace 0.5):**60/60 (100%)**vs 55/60 (92%) baseline, similar latency (aggregator think-time dominates on AIME) — dropping the straggler (often the malformed-JSON reference) certainly costs nothing and may helpno accuracy cost — quorum stays on by recommendation for latency-sensitive lanes9Inverted MoA (draft→review→revise) rescues composition for code editingaider 30: kimi drafts + 2 cheap reviewers + revise =50% pass@1 @279svs kimi solo 73.3% @173srefuted — even review-only context degrades editing precision; the code lane stays solo + escalation7Capping reference advice cuts fan-out latency for freeheavy fan-out on 40 tasks: cap 1500→600 tokens = −17% latency, −15% tokens, same accuracy; cap 300 gains nothing more (provider think-time dominates)adopt 600 as default guidance; timeouts still need routing/escalation, not caps8Can a wafer-speed model be the cheap coding lane?Cerebras gemma-4-31b on aider: 2.2 s/case but 10% pass@1 (50% pass@2) — brilliant at reasoning (39/40), can’t do precise editing; GLM-4.7 similar (30%)refuted — kimi-class is the cheapest viable coding lane; wafer models own classify/self/reasoning lanes4Does escalation replicate on a hard agentic benchmark?SWE-bench Lite 25: kimi-solo resolved 19/25 (76%);Fable resolved 6/6 of the remainder → composite 25/25 (100%) with Fable driving only 24% of instancesconfirmed at scale — pattern holds on both benchmarks **How failure was detected — read before citing the escalation numbers.**The offline composites areoracle-detected upper bounds: the aider 30/30 used the benchmark’s gold tests to pick what to escalate, and the SWE-bench 25/25 used SWE-bench’shiddenevaluation tests — signals a runtime agent cannot see. Theliveescalation run is the real-world-shaped result: the router never detects failure itself; it pattern-matches failure evidence (FAILED / AssertionError / Traceback) in the conversation, and that evidence exists only because the client agent ran tests itself and fed the output back — which coding agents (aider auto-test, mini-swe-agent, CI loops) do as part of their normal loop. Boundary conditions: tasks with no verifiable signal never escalate; agent-visible tests can be weaker than gold tests (aider polyglot happens to ship its gold tests to the agent, which flatters the live number); a true live SWE composite would land between kimi’s 76% and the oracle 100%.

Loop takeaway:routing + verification-gated escalation is how you get frontier-class results at a fraction of the frontier calls— guess-free (escalate on measured failure, not predicted difficulty). This slots directly into agentic harnesses that already run tests (aider, SWE-bench, CI).

11. Roadmap — open experiments

The vision: a dockerized router/MoA of cheap open-weight models that matches frontier quality where it matters, improves from its own traffic, and runs anywhere (API slots today, a 4×96 GB box tomorrow). The gaps and the experiments that close them:

#Gap (measured)Experiment / improvementSuccess metric1MoA hurts precise code editing (−13 pp pass@1)Routed v2 config A/B(built, pending deploy): coding→kimi solo lane, general→all-Cerebras fast lane, hard reasoning→heavy fan-out, Cerebras classifier. Rerun aider onmoa:auto.moa:auto ≥ kimi solo’s 73% pass@1 at solo latency2Advisors distract editing; but review is where MoA caught errorsInverted MoA for code: aggregator writes solo first, references only review/veto the diff (one extra cheap pass).pass@1 > solo baseline on aider; ≤1.5× solo latency3Residual reasoning misses are tool-shaped (bigmul, digit sums)Verifier tool in the aggregator slot: let the proxy expose a sandboxed python-exec tool the aggregator may call before answering (the distilled skill alreadyasksfor verification).40-task set → 40/40 for nano/flash-class combos4Self-MoA literature says best-model self-ensemble ≥ mixed refsSelf-MoA bench: kimi 3-sample self-ensemble vs mixed heavy refs at equal token budget, same 40 tasks + aider subset.decide mixed-vs-self refs per route, data-backed5Learning proven only on exact-answer tasksAgentic learning cycles: aider exercises as train set (pass/fail = graded outcome, already wired into traces), evolve, eval on held-out exercises. Then per-route skills (traces already record routing).+pass@1 on held-out exercises after N cycles6Router classifies topic, not difficultyConfidence cascade: classifier also emits easy/hard; easy→self or fast lane, hard→heavy; escalate on low-confidence answers.−50% cost/latency at equal accuracy on mixed traffic7All slots are remote APIs todayLocal pilot of the box architecture: two vLLM models on 5090+4090 (e.g. a coder + gemma-class general) as custom providers behindmoa serve+ Cerebras classifier — the 4×96 GB stack in miniature.routed local endpoint ≥ open-solo API quality on the bench suite, $0 marginal8Sample sizes are small (10-task TB sample, 25 SWE instances)Scale the anchors: full Terminal-Bench 2.x (89 tasks) + SWE-bench Verified 50-instance slice on the winning configs only.leaderboard-comparable public numbers

12. How to replicate

# Code (public fork, branch feature/moa-openai-proxy)
git clone -b feature/moa-openai-proxy https://github.com/hughmadden/hermes-agent.git
cd hermes-agent

# Serve MoA as an OpenAI endpoint
mkdir moa-home   # write moa-home/config.yaml — presets + router (docs/moa-openai-endpoint.md)
export OPENROUTER_API_KEY=... CEREBRAS_API_KEY=...
docker compose -f docker-compose.moa.yml up -d
curl -s localhost:8646/v1/models | jq -r '.data[].id'

# Tests + benches (all dockerized; raw JSONs for every table in docs/plans/)
tests/integration/docker/run-moa-proxy-tests.sh
docker build -f tests/integration/docker/Dockerfile.moa-proxy -t hermes-moa-proxy-test .
D="docker run --rm -v $PWD/scripts:/app/scripts:ro -e OPENROUTER_API_KEY -e CEREBRAS_API_KEY hermes-moa-proxy-test"
$D python scripts/moa_router_bench.py --classifiers "custom:cerebras=gemma-4-31b"
$D python scripts/moa_learning_cycle.py --cycles 3 --config open-moa-nano --reset-learning
$D python scripts/moa_gpusim_bench.py
$D python scripts/moa_cerebras_bench.py

# Public harnesses against the endpoint (details: docs/plans/moa-public-bench-results-20260703.md)
# aider:    OPENAI_API_BASE=http://localhost:8646/v1  --model openai/moa:<preset>
# harbor:   harbor run -d [email protected] -a terminus-2 -m openai/moa:<preset> \
#             --ak api_base=http://localhost:8646/v1 --timeout-multiplier 2
# swebench: mini-extra swebench --model openai/moa:<preset> --subset lite --split test \
#             --slice 0:25 -c <default swebench.yaml> -c environment.timeout=300

15. 2× RTX 6000 Blackwell topology study — one big deployment vs two instances

Instrument: Qwen3-Next-80B-A3B-FP8 (262k native context, hybrid attention ≈2.3 GiB KV per 100k tokens), vLLM, prefix caching on, fp8 KV, workstation Blackwells with NO NVLink (PCIe only; TP=2 requires--disable-custom-all-reduce). No public benchmark of this comparison existed.

TopologySingle-stream decodeAggregate @16 streamsWarm/cold TTFT (60k prefix)180k session turn (warm)One instance, one GPU176 tok/s (TTFT 87 ms)729 tok/s2.95% (6.25 s → 0.18 s)0.38 sTwo independent instances175.7 + 177.3 tok/s simultaneously1,434 tok/s (16+16)0.66% per instanceunchangedTP=2 across both (PCIe)136 tok/s — 23% slower than ONE gpu708 tok/s (half of dual)1.88%0.40 s (cold 37.3 s) **Verdict:**for any model that fits a single 96 GB card, two independent instances dominate every axis — throughput, latency, and caching. TP=2 over PCIe on these cards is purely a FIT mechanism for weights >96 GB (tonight: GLM-4.7-IQ3 at ~165 GB), never a performance choice.**Agentic caching verdict:**GPU-resident prefix caching is near-total — a 180k-token session turn answers in 0.38 s warm (vs 35.3 s cold); long agent sessions are essentially free after the first turn, with zero system-RAM requirement. The 262k-context 80B endpoint now runs permanently on bench GPU 1 as a proxy provider.

14. Serving-cache economics — measured (agentic sessions)

An agent session re-sends a large stable prefix (system + history) with a small delta each turn. Whether that prefix is CACHED at the provider decides both latency and cost. Measured 2026-07-06 with a 30k-token synthetic agent prefix, 4 sequential calls per model (scripts/moa_cache_bench.py + moa_prefix_cache_probe.py):

Provider / modelWarm cached fractionPrompt-cost savingNotesdeepseek-v4-pro @OR99.5%~40%automatic, consistent; caches persist ~5 min across sessionsglm-5.2 @OR92%~40%fastest cold call (3.8 s)gpt-5.5 @OR99% reported0% billedcache-read happens upstream; OR cost fields never dropkimi-k2.6 @OR33% (1 in 3)78% when it hitsmulti-backend routing defeats per-instance cachesqwen3.6-27b @OR0%noneper-call provider switching; latency erraticCerebras gemma-4-31bwarm TTFT 33% of coldn/a (flat pricing)real prefix cacheCerebras gpt-oss-120bwarm TTFT 79% of coldn/amild; but cold prefill of 30k tokens is only ~1.5 s — wafer prefill speed IS the cache Design consequences for the proxy:(1) cache-critical agent lanes (long sessions, escalation tiers) should pin deepseek/glm-class providers — 40% real input-cost cuts at 92-99% hit rates; (2) OR usage carriesprompt_tokens_details.cached_tokens— never thecache_discountfield; (3) prompt layout matters: static content first, volatile fields (timestamps, ids) LAST or hit rates collapse; (4) the wafer voter tier needs no caching design at all — its prefill is faster cache-cold than most providers are cache-warm.

13. Tooling & execution tally — what ran each experiment

Experiment familyHarness / infraModels doing the workExact-answer benches (40-task, AIME, HMMT, dial/cascade runs)scripts/moa_*_bench.py in the hermes-moa-proxy-test container, driving the serve HTTP endpoint; datasets fetched live from the HuggingFace datasets-serverper config (section 1); frontier tiers on the ChatGPT plan; Cerebras paid key; local voters via vLLM on the RTX 5090Public benchmarksaider-benchmark image (own build), mini-swe-agent 2.4.4 + swebench-4.1.0 eval (docker-socket pattern), Harbor 0.16.1 + terminus-2 — fully containerized, zero host installsthe proxy’s presets; official harnesses for evaluationRLM wafer-agent bench (iter 30)scripts/moa_rlm_bench.py — harness-side agent loop, sandboxed python exec (-I/empty-env/tempdir/12 s), direct Cerebras API; built by one Sonnet builder + one Sonnet reviewer (no defects found)gemma-4-31b + gpt-oss-120b @cere; grading via the cascade’s normalize_candidateLearning loopscripts/moa_learning_cycle.py (traces → evolve → paired evals → leakage scan)subject open-moa-nano; distiller deepseek-v4-pro@ORFeature builds (cascade, judge gate, verified cascade)Workflow orchestration: parallel Sonnet builder agents + adversarial Sonnet reviewers off Fable-written specs (~1.4M Sonnet tokens, Claude plan); 6 confirmed review findings fixed by Fable before benchingSonnet 5 (build/review), Fable 5 (spec, integration, analysis)Orchestration & analysisthis Claude Code session (Fable 5): hypothesis selection, specs, defect fixes, result analysis, report upkeepFable 5 **Are we running on our own discoveries yet?**Partially — honestly tallied: the worker MoA endpoint (cere-moa + plan-GPT-5.5 lanes) served as an offload target for builder agents and as the blind grader in the judge eval, and every benchmark exercises the proxy itself. But the heavy thinking is still Fable + Sonnet on the Claude plans — the discoveries are the PRODUCT being built, not yet the build system. Queued: route bench analysis and doc drafting through moa:omni lanes and measure whether quality holds.

Budgets: OpenRouter ceiling**500**\(raised 2026\-07\-05\); spent ≈220 to date. Frontier tiers ride the ChatGPT plan at $0 and builds ride the Claude plan, so marginal experiment cost is mostly wafer pennies.

Generated autonomously by the Claude agent within Turq ·fork branch· report v24 — frontier curve complete: 93/95/98 at 5/35/100% frontier · 2026-07-03