@garrytan: Modus built an open-source test of whether AI can do actual junior-auditor work, not just answer accounting questions. …
Summary
Modus Audit Labs released Financial Audit Bench, an open-source benchmark testing whether AI can perform real junior-auditor work rather than merely answering accounting questions. Garry Tan highlighted the release, arguing every job will eventually have similar eval benchmarks.
View Cached Full Text
Cached at: 10/02/26, 06:34 AM
Modus built an open-source test of whether AI can do actual junior-auditor work, not just answer accounting questions.
Every job in the world will end up having eval benchmarks like this.
Pranav Pillai (@PranavAPillai): Excited to announce the release of Financial Audit Bench - a seminal paper written by the @ModusAudit Labs team
Similar Articles
Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants
Mercor hired 12 licensed CPAs to complete simplified month-end close tasks from its APEX-Accounting benchmark and found that frontier AI models now score perfectly and are ~50x cheaper than the best junior accountant, having gone from below the human average to 100% accuracy in just 18 months.
@ApexAIHighlight: Most AI benchmarks test whether a model can give the right answer. @Accio_official is testing something far harder: Can…
CommerceAgentBench is a new benchmark with 107 real-world e-commerce tasks designed to test whether AI agents can actually complete jobs, moving beyond traditional answer-based AI benchmarks.
Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
This paper identifies five failure modes in perturbation-based benchmark-validity audits used for AI governance, demonstrating that implementation details can silently manufacture conclusions. It proposes a due-diligence gate to improve the reliability of evaluation evidence.
AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents
AgentAudit is an open, extensible framework for evaluating the full lifecycle of AI agents across capability, grounding, security, and behavioral dimensions, enabling precise failure attribution and highlighting trustworthiness differences among various language models.
AI is now being benchmarked on whether it can actually do laboratory science, not just answer questions: run experiments, handle equipment, read instruments, and recover from failures
AI systems are now being evaluated on their ability to perform practical laboratory tasks such as running experiments, handling equipment, reading instruments, and recovering from failures, marking a shift from theoretical to applied science.