Tag
Epoch AI introduces Benchmark Reviews to audit AI benchmarks, revealing flaws such as in DeepSWE v1.1 where the verifier discards changes without informing the model, leading to failures.
This article discusses how retries in AI systems, particularly with LLMs and agents, can exacerbate failures when underlying issues are not addressed, leading to repeated mistakes with increased cost and latency.
Observations on the shift from addressing AI hallucinations to the more pressing problem of production AI failures, emphasizing the need for system reliability, tracking decisions, and limiting blast radius in enterprise deployments.
The article argues that many failures of AI coding agents stem from vague specifications, not just model weaknesses. It suggests that writing clearer, more detailed work packets may be the next essential skill for developers using coding agents.
An opinion piece arguing that long context windows don't equate to memory and that agent failures are often mundane, like forgetting constraints or rereading files, emphasizing that reliability depends on context architecture decisions.
An archive called Agent Fail Museum documents recurring AI failure patterns and provides regression test drafts for submitted failures, aiming to prevent repeat incidents.