@che_shr_cat: 1/ Parameter scale is a brute-force crutch. What if a 35B model could beat a 1,000B model simply by scaling its search …
Summary
Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.
View Cached Full Text
Cached at: 07/20/26, 09:29 AM
1/ Parameter scale is a brute-force crutch. What if a 35B model could beat a 1,000B model simply by scaling its search horizon at test time?
It is not magic. It is structured process feedback. Let us look under the hood.
2/ In “Scaling the Horizon, Not the Parameters”, Shanghai AI Lab introduces Agents-A1.
They prove that explicitly training a smaller model on structured execution traces can match or exceed trillion-parameter frontier models on complex tasks.
3/ The framework runs on a Knowledge-Action Graph (KAG).
Instead of treating trajectories as flat text, they formalize steps as verifiable 4-tuples: (Corpus, Action, Observation, Verifier).
Every action is written back to the graph only if it passes verification.
4/ Training uses a 3-stage recipe:
- Full-domain SFT
- Specialized domain RL teachers (using GRPO)
- Multi-Teacher On-Policy Distillation
This solves a massive problem: gradient conflicts when trying to mix tool-use, search, and instruction following.
5/ To merge these teachers, they use Salient Vocabulary Alignment (SVA).
Instead of distilling the entire vocabulary, the student only aligns with the domain-routed teacher on a compact top-k slice of valid tokens.
This keeps the training stable and prevents gradient clash.
6/ The results: Agents-A1 (35B MoE) hits 56.4 on SEAL-0, beating DeepSeek-V4 (55.0) and Kimi-K2.6 (50.5).
On FrontierScience-Olympiad, it scores 79.0, beating DeepSeek-V4-Pro’s 76.0.
We are seeing frontier-class capability from a 35B model.
7/ The catch? Serious latency.
Running an average sequence length of 45K tokens at test time is computationally brutal.
It also lags on MLE-bench (43.9 vs GPT-5.5’s 72.7), proving that long-term goal consistency for complex engineering is still highly challenging.
8/ The takeaway: Stop waiting for bigger models.
We should be focusing on structured process-level RL and test-time search. Unifying specialized teachers through on-policy distillation is a highly viable path for small, local deployments.
9/ Read my full technical breakdown here: https://arxiviq.substack.com/p/scaling-the-horizon-not-the-parameters…
Check out the original paper here: https://arxiv.org/abs/2606.30616
How are you structuring your agent trajectory data? Let’s discuss in the replies.
#MachineLearning
10/ Visual breakdown of the multi-teacher distillation process below. Sometimes a diagram is worth a trillion parameters.
Similar Articles
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Introduces Agents-A1, a 35B Mixture-of-Experts agentic model that achieves trillion-parameter-level performance through long-horizon trajectory scaling and a three-stage training approach including SFT, domain-level teachers, and multi-teacher distillation. The model outperforms or matches much larger models on long-horizon agent benchmarks.
@rohanpaul_ai: New Meta paper shows, small models may not be bad predictors of scale; they may just be getting under-tuned. Finds scal…
A new Meta paper reveals that small models can accurately predict scaling laws but require more extensive hyperparameter tuning. The study finds scaling laws emerge around 4M parameters and become clearer with proper tuning.
Deepseek, please explain to me how you make a 300B parameter model that is cheaper than a 9B parameter model by SO MUCH.
A Reddit user asks how DeepSeek can make a 300B parameter model cheaper than a 9B parameter model, sparking discussion on efficiency and cost.
A 4b model is now beating 30b ones at web research and the reason is not size
A 4 billion parameter open model from the Apodex family outperforms 30 billion parameter models on web research benchmarks, attributed to careful training data and self-verification techniques rather than raw scale, suggesting a more democratic trajectory for AI capability.
@che_shr_cat: 1/ A 5M-parameter model just beat frontier LLMs on hard logical puzzles at less than 1/100,000th of the inference cost.…
A 5M-parameter model outperforms frontier LLMs on hard logical puzzles at a fraction of the inference cost by using continuous latent space test-time compute.