TLDR on @deepseek_ai 0731 V4 Flash. It is comfortably the current SOTA for 190GB VRAM or unified memory based systems. Hits as hard or harder on quality scores as @UnslothAI’s 3 bit version of Qwen3.5-397B but runs about 3x as fast. Benchmarks below: https://github.com/Osmantic/MMBT-Messy-Model-Bench-Tests/tree/main/benchmarks…

Osmantic/MMBT-Messy-Model-Bench-Tests

Source: https://github.com/Osmantic/MMBT-Messy-Model-Bench-Tests

MMBT - Messy Model Bench Tests

This repository stores messy, real-world benchmark outputs from different hardware and LLM tests in my lab. It’s my messy research, and exists for my personal use but I’m making it public so that other people can use it too.

Five-minute answers

Operating point (read before quoting)

Most earlier agent-task benchmarks under benchmarks/ use Cyankiwi 4-bit AWQ quants on 2x RTX PRO 6000 Blackwell at 500 W cap. DeepSeek V4 Flash 0731 and Gemma 4 31B are explicit exceptions: DeepSeek uses official FP4 weights, FP8 KV, and a validated 1,048,576-token context; Gemma uses Google’s official QAT Q4_0 GGUF, Q8 KV, llama.cpp, and a validated native 262,144-token context per slot. Every entry README pins its own operating point. Other quants, VRAM tiers, hardware classes, and languages other than Python are not characterized unless an entry says otherwise. See COMPARISON.md section “What this benchmark doesn’t characterize” for the model-benchmark validity boundaries, and ROADMAP.md for what’s queued to fill those gaps.

Rig-characterisation studies under hardware-tests/ have their own operating-point scope. Start with hardware-tests/README.md before quoting hardware claims. In particular, hardware-tests/qwen3.6-q8-fleet-2026-05-17/ ranks four hardware classes on Q8_0 GGUF dense and MoE workloads under llama.cpp, with a Tower2 vLLM-FP8 appendix row for the MoE model the llama.cpp/CUDA path crashes on.

Layout

benchmarks/
  README.md                    agent-task benchmark landing page and navigation map
  gemma4-31b-q4/               complete Gemma campaign, N=3/N=10, strict audits, comparisons
  deepseek-v4-flash-0731/      cross-suite DeepSeek campaign, strict audits, and deployment evidence
  dreamserver-75-pr-audit/
    GPT-5.5/                   cloud, full audit
    Opus-4.7/                  cloud, full audit
    Qwen3.6-27B-AWQ/           local 30B-class, structurally complete + substantively partial
    Qwen3-Coder-Next-AWQ/      local MoE 80B/3B, no deliverable (failure-mode entry)
    findings-2026-04-27-local-models.md   cross-cutting writeup
  dreamserver-1-pr-audit/
    Qwen3-Coder-Next-AWQ/      local, single-PR deliverable (correct verdict, but variance-dominated — see entry README)
    Qwen3.6-27B-AWQ/           local, partial deliverable (excellent analysis, no verdict.md shipped)
    Qwen3.6-35B-A3B-AWQ/       local, floor failure (no artifacts produced)
  wallstreet-intern-test/
    GPT-5.5/                   cloud, full memo repo + board-of-advisors deck
    Opus-4.7/                  cloud, full memo repo
    Qwen3.6-27B-AWQ/           local, full memo repo (GTLB BUY, 1 of 3 runs shipped)
    Qwen3-Coder-Next-AWQ/      local, full memo repo (DOCU BUY, 1 of 3 runs shipped — verdict reliability caveat in README)
    Qwen3.6-35B-A3B-AWQ/       local, no usable deliverable (0 of 3 runs shipped, kept as failure-mode entry)
hardware-tests/
  README.md                       hardware-test landing page: coverage matrix, settled vs held questions
  vllm-power-sweep-2026-04-29/ rig characterisation: vLLM throughput vs GPU power cap, 28-cell sweep with raw CSVs + audit notes
  tower2-dual-qwen27-600w-2026-07-29/ 30 min dual-GPU 600 W thermal saturation run
  tower2-dual-qwen27-no-gap-2026-07-29/ adjacent-card 600/400 W transient and steady-state thermal runs
  qwen3.6-q8-fleet-2026-05-17/ cross-platform dense + MoE Q8 hardware comparison
  local-ai-hardware-valuation-2026-05-17/ recomputable buyer valuation worksheet

Benchmarks

For the benchmark landing page and per-folder navigation map, start with benchmarks/README.md.

Hardware tests

hardware-tests/ holds rig characterisation runs — power, throughput, and thermal sweeps on the lab hardware itself, separate from agent-task benchmarks. Start with hardware-tests/README.md: it makes the dense-vs-MoE coverage, backend exceptions, and “settled vs held” boundaries explicit.

At a glance

Two synthesis docs sit between this README and the per-entry detail:

• benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_VERIFIED_RESULTS.md — full Gemma result, native-256K deployment, N=3/N=10 quality, strict artifact audit, and direct Qwen3.6-27B comparison.
• benchmarks/deepseek-v4-flash-0731/DEEPSEEK_V4_FLASH_0731_VERIFIED_RESULTS.md — full verified DeepSeek result, including corrected-vs-raw grading, strict artifact audits, production validation, and caveats.
• COMPARISON.md — head-to-head decision doc for the three local model arms (Coder-Next vs 27B-thinking vs 27B-no-think). Organized by task class with cell-level evidence. Read this if your question is “which one should I use?”
• SCORECARD.md — single-table summary across all entries (spec compliance, factual accuracy, fabricated-claim count, tests run, wall, cost upper bound, failure mode, “when to use which” guide). Read this if your question is “what’s the full picture?”

Both link back to the per-entry artifacts they cite.

Reproducing the runs

The tooling/ folder is the reproduction pack — agent harness, sandbox Dockerfile, vLLM launch commands, all 12 microbench task prompts, input starters, ground truth, grader scripts, and batch-runner scripts. With everything there plus a CUDA-capable Linux box and a HuggingFace model, an external reader can rerun any of the local-model entries here.

• Replaying a published run: see tooling/REPRODUCING.md — receipt-driven walkthrough.
• Benchmarking a new local model: see tooling/ADDING-A-MODEL.md — end-to-end guide with a four-command friendly path (smoke_test.sh → run_microbench.sh → grade_microbench.sh → summarize.sh). Half-day to one-day operator time per new model.

How To Read A Model Entry

Start with the benchmark folder README, then open the model folder:

1. benchmarks/<benchmark>/README.md
2. benchmarks/<benchmark>/<model>/README.md
3. The model entry’s README for its artifact-specific read order, then the main deliverables such as report/ / prs/ or memo/ / model/.

For comparing multiple model entries within a benchmark, look for cross-cutting findings-*.md docs at the benchmark folder root (e.g. benchmarks/dreamserver-75-pr-audit/findings-2026-04-27-local-models.md).

A note on “Messy”

The “Messy” framing is intentional. Some model entries are clean audits with traceable line-by-line reasoning (Opus-4.7/ on the 75-PR task). Others are structurally-complete-but-substantively-partial scaffolds with a few hand-written reviews and 70+ template stubs (Qwen3.6-27B-AWQ/ on the same task). Others are deliberate failure-mode entries with no audit artifacts at all but with documented failure trajectories (Qwen3-Coder-Next-AWQ/ on the 75-PR task, Qwen3.6-35B-A3B-AWQ/ at N=1).

Before quoting any number from this repo, read KNOWN-LIMITATIONS.md. It consolidates the caveats that affect what claims this evidence can support — small N, cherry-picked successes, dirty harness git SHAs, hand-graded inputs without a formal rubric, hardware specificity, the gap in cloud-vs-local apples-to-apples grading. Useful evidence; not yet a leaderboard.

This repository is licensed under MIT. Third-party content (DreamServer code excerpts, SEC filings, cloud-LLM and local-model outputs, Cyankiwi quantizations) retains its original licensing — see NOTICE.

The repo keeps the failures because the kinds of failure are themselves the comparison data. A reader picking a model for their own work needs to know that “this model can’t complete this task” or “this model produces output shape without substance” — those are real properties of the model, not noise to filter out.

Current Entries

gemma4-31b-q4:

• Verified campaign entry — Gemma 4 31B QAT Q4_0 on two independent 500 W RTX PRO 6000 replicas, with native 262,144-token context per slot. Canonical N=10 is 89/120 raw and 99/120 corrected; extended strict result is 0/12. It beats Qwen3.6-27B on directly comparable bounded quality but not on batched serving.
• Completion audit — requirement-to-evidence handoff, including the pending production restore gate.

deepseek-v4-flash-0731:

• Verified campaign entry — DeepSeek V4 Flash 0731 on 2x RTX PRO 6000 at 500 W/GPU and 1,048,576 context. Canonical corrected result 35/36 (N=3); single-PR 2/3 expected verdicts with all three complete; investment workbooks 0/2 substantively valid; board deck shipped with material visual defects; frozen 75-PR strict result 0/3 (two scaffold-and-stop, one 815,279-token runaway-generation terminal failure).
• Completion audit — requirement-to-evidence handoff and artifact inventory.

dreamserver-75-pr-audit:

• GPT-5.5 — cloud, full audit (75 PRs, 34 merge / 40 revise / 1 reject)
• Claude Opus 4.7 (1M context) — cloud, full audit (51 clean MERGE / 14 categorized HOLDs)
• Qwen3.6-27B-AWQ — local, structurally complete (75/75 verdict files) but only 3 are real reviews; 72 are template stubs. Zero tests run.
• Qwen3-Coder-Next-AWQ — local, no audit deliverable across 5 attempts. Three distinct degenerate failure modes (loops, cyclic-name slop, stuck-in-research). Folder kept as failure-mode evidence.
• Cross-cutting findings doc — comparison writeup of the local-model entries against the cloud entries

dreamserver-1-pr-audit:

• Qwen3-Coder-Next-AWQ — local, single-PR deliverable, MERGE verdict (correct). Caveat in README: this is the cherry-picked correct run of three; other two gave REJECT (wrong, with fabricated technical issues).
• Qwen3.6-27B-AWQ — local, partial deliverable. Best analytical content of any local-model run on this PR. No verdict.md shipped (failure to follow spec); implicit MERGE in review.md.
• Qwen3.6-35B-A3B-AWQ — local, floor failure. Zero artifacts produced; model investigated for 28 iters then stopped without writing.

wallstreet-intern-test:

• GPT-5.5 — cloud, full memo repo + the follow-on board-of-advisors presentation in board-of-advisors-presentation/
• Claude Opus 4.7 (1M context) — cloud, full memo repo
• Qwen3.6-27B-AWQ — local, GitLab Inc. (GTLB) BUY recommendation. 1 of 3 attempts shipped (other 2: parser fault, 1-hour single-call timeout). 17 KB three-statement model, full audit trail.
• Qwen3-Coder-Next-AWQ — local, DocuSign (DOCU) BUY recommendation. 1 of 3 attempts shipped (other 2: scaffold-and-stop). 10.6 KB three-statement model. Verdict-reliability caveat in entry README — single-shot Coder-Next output can be confidently wrong with fabricated evidence (see PR-audit benchmark for documented examples).
• Qwen3.6-35B-A3B-AWQ — local, no usable deliverable. 0 of 3 attempts shipped. Folder kept as failure-mode evidence consistent with the model’s PR-audit floor failure.

microbench-2026-04-28:

• adversarial-hallucination/ — agent must distinguish 6 real bugs from 9 confident-but-wrong fabrications. Sharpest local-model superiority signal in the entire repo: 27B 3/3 PASS, Coder-Next 1/3 PASS with 2 confirmed-fabrications-as-real on the shipping run.
• market-research/ — 5-product enterprise password manager comparison + pricing math + cited sources. Inversion of the prior “both fail at internet research” expectation: 27B 3/3 STRUCTURAL_PASS (12-18 cites to 29-33 distinct URLs), Coder-Next 0/3 STRUCTURAL_FAIL.
• doc-synthesis/ — 1-page executive brief from 5 source documents, 700-word limit. Documents a 27B failure mode: 8/8 facts captured every run, but model can’t trim to length (765-775 words across N=3, two runs entered identical-call-loops on brief.md).
• findings.md — cross-cutting writeup spanning all 12 task families (3 published full + 9 summarized).

microbench-phase-b-2026-05-02:

• Qwen3-Coder-Next-AWQ — N=10 expansion across 4 differential cells. Headline: 0/10 STRUCTURAL_FAIL on p3_market (Wilson 95% [0%, 27.8%]) confirmed reproducible.
• Qwen3.6-27B-AWQ (thinking) — N=10 expansion. Word-trim loop on p3_doc bounded as a stable ~40% failure shape (4/10 wall_killed).
• Qwen3.6-27B-AWQ (no-think) — new third arm, full 12-family grid × N=10. 95.8% ship rate (Wilson 95% [90.5%, 98.2%]) — most reliable shipper of the three. Halves the p3_doc word-trim loop rate (4/10 → 2/10).
• findings.md — full per-cell breakdown with Wilson CIs, three identical-call-loop subclasses, cost-per-shipped-run analysis, “when to use which” updates.
• findings-pairwise-quality-three-model.md — hand-graded deliverable quality study; 27B-thinking and 27B-no-think substantively equivalent on shipped output.