Tag
This paper systematically studies context compaction in LLM agent scaffolds (OpenCode) across three frontier models and three long-horizon benchmarks, finding that current benchmarks are 'compaction-friendly' and that compaction silently drops task constraints and anti-cheating rules, with real user instructions being dropped about 6x more often than build specifications.
The paper introduces ASAP, a lightweight framework for evaluating agentic behavior in pretrained base models through interactive end-to-end tasks without converting them into chat agents, demonstrating discriminative performance across software-engineering and deep-research benchmarks.
Stanford researchers develop a Bayesian variance-decomposition framework using Generalizability theory to diagnose reliability in agent benchmarks, showing that adding more tasks cannot fix uncertainty caused by scaffold coverage and that pooling diverse benchmarks is a cheaper path to reliable rankings.
Introduces AgentSpec, a modular specification framework for systematically composing and analyzing embodied LLM agent scaffolds, revealing that performance depends on scaffold compatibility and interaction effects rather than isolated module strength.