I wrote up my own findings on model routing based on difficulty

quick note about myself: been an ai engineer at 4+ companies, computational physics background, currently building and experimenting, love to code, feel like society underestimates statistical rigor with llm applications. i am aware that the experiment i made should be more rigorous, but i don't have eternal funds. i spent 40 $s to run a small experiment on llamaindex's extractbench (the schema-guided document extraction benchmark from their recent paper): took the 36 government documents, customs forms, gsa price schedules and municipal audits, pulled the text with pypdf, and ran one-shot extraction with claude opus 5, qwen3.8-2.4t and qwen3.6-35b via openrouter, scored against the benchmark's gold standard with their cell-level f1 metric. two things surprised me. opus 5 one-shot on plain text scored ~0.94, roughly on par with the coding agents in the paper at a fraction of the cost per page. and qwen3.8 basically matched it (0.936) at a third of the price. also interesting: f1 correlated much more with document length (ρ ≈ −0.50) than with how many values had to be extracted (ρ ≈ −0.28), which fits nicely with the recent pre-inference routing paper (arxiv 2608.06607). plenty of caveats: n=36, short/medium documents only (i skipped the >50-page monsters), text-layer only so scanned forms were excluded, and my scorer is a reimplementation of the benchmark's, so absolute numbers may drift a bit from their leaderboard. full transparency: i directed the experiment, but claude wrote most of the code and parts of this text. here is the notebook, and even happier to hear where the methodology is wrong. i will be publishing lots more so you can also subscribe to the notebook if you wish.