@che_shr_cat: 1/ Parameter scale is a brute-force crutch. What if a 35B model could beat a 1,000B model simply by scaling its search …
Summary
Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.
View Cached Full Text
Cached at: 07/20/26, 09:29 AM
1/ Parameter scale is a brute-force crutch. What if a 35B model could beat a 1,000B model simply by scaling its search horizon at test time?
It is not magic. It is structured process feedback. Let us look under the hood.
2/ In “Scaling the Horizon, Not the Parameters”, Shanghai AI Lab introduces Agents-A1.
They prove that explicitly training a smaller model on structured execution traces can match or exceed trillion-parameter frontier models on complex tasks.
3/ The framework runs on a Knowledge-Action Graph (KAG).
Instead of treating trajectories as flat text, they formalize steps as verifiable 4-tuples: (Corpus, Action, Observation, Verifier).
Every action is written back to the graph only if it passes verification.
4/ Training uses a 3-stage recipe:
- Full-domain SFT
- Specialized domain RL teachers (using GRPO)
- Multi-Teacher On-Policy Distillation
This solves a massive problem: gradient conflicts when trying to mix tool-use, search, and instruction following.
5/ To merge these teachers, they use Salient Vocabulary Alignment (SVA).
Instead of distilling the entire vocabulary, the student only aligns with the domain-routed teacher on a compact top-k slice of valid tokens.
This keeps the training stable and prevents gradient clash.
6/ The results: Agents-A1 (35B MoE) hits 56.4 on SEAL-0, beating DeepSeek-V4 (55.0) and Kimi-K2.6 (50.5).
On FrontierScience-Olympiad, it scores 79.0, beating DeepSeek-V4-Pro’s 76.0.
We are seeing frontier-class capability from a 35B model.
7/ The catch? Serious latency.
Running an average sequence length of 45K tokens at test time is computationally brutal.
It also lags on MLE-bench (43.9 vs GPT-5.5’s 72.7), proving that long-term goal consistency for complex engineering is still highly challenging.
8/ The takeaway: Stop waiting for bigger models.
We should be focusing on structured process-level RL and test-time search. Unifying specialized teachers through on-policy distillation is a highly viable path for small, local deployments.
9/ Read my full technical breakdown here: https://arxiviq.substack.com/p/scaling-the-horizon-not-the-parameters…
Check out the original paper here: https://arxiv.org/abs/2606.30616
How are you structuring your agent trajectory data? Let’s discuss in the replies.
#MachineLearning
10/ Visual breakdown of the multi-teacher distillation process below. Sometimes a diagram is worth a trillion parameters.
Similar Articles
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Introduces Agents-A1, a 35B Mixture-of-Experts agentic model that achieves trillion-parameter-level performance through long-horizon trajectory scaling and a three-stage training approach including SFT, domain-level teachers, and multi-teacher distillation. The model outperforms or matches much larger models on long-horizon agent benchmarks.
A 4b model is now beating 30b ones at web research and the reason is not size
A 4 billion parameter open model from the Apodex family outperforms 30 billion parameter models on web research benchmarks, attributed to careful training data and self-verification techniques rather than raw scale, suggesting a more democratic trajectory for AI capability.
@che_shr_cat: 1/ A 5M-parameter model just beat frontier LLMs on hard logical puzzles at less than 1/100,000th of the inference cost.…
A 5M-parameter model outperforms frontier LLMs on hard logical puzzles at a fraction of the inference cost by using continuous latent space test-time compute.
Agents-A1-4B (Qwen3.7-4B ???) : Scaling the Horizon, Not the Parameters
Agents-A1-4B, a compact 4B-parameter model from InternScience, achieves state-of-the-art results across multiple agentic and reasoning benchmarks, outperforming larger models by scaling capabilities rather than parameters.
@j_golebiowski: A 1.7B parameter model beats GLM-5 (744B) on Schema Guided Dialogue — even when the training data is corrupted. That's …
A 1.7B parameter model surpasses 744B GLM-5 on Schema Guided Dialogue despite corrupted training data, showing 437× size efficiency.