Tag
A case study of an LLM coding agent implementing a multi-component data system, analyzing defects and evaluating retrieval strategies on HotpotQA, highlighting gaps in automated versus empirical testing.
QuoteBench reveals that execution-boundary parsing errors significantly reduce LLM coding agent success, and disclosing the boundary helps recover performance, showing that evaluation must account for deployment configuration rather than treating matched scores as intrinsic model properties.
CORVUS proposes a new trajectory architecture for LLM coding agents that decouples file-read actions from observations by maintaining a synchronized registry of relevant files, reducing input tokens by 9-50% and reasoning cycles by up to 37% while maintaining comparable pass rates on SWE-bench benchmarks.