Tag
FrontierChallenge is a cross-domain benchmark for evaluating end-to-end scientific workflows, revealing that frontier AI models achieve only 20% completion despite high partial scores, emphasizing the need for better evaluation of complete task delivery.
Rex is a statically typed, pure functional workflow language designed for scientific computing and data processing, featuring parallel execution, content-addressable storage, and Docker isolation.
This paper introduces AutoSupervision, a benchmark and method for verifying whether manuscript revisions actually address reviewer concerns using grounded evidence from peer-review records. Experiments on 56,000 Nature Communications articles show LLMs can summarize reviewer concerns but still struggle with evidence-based verification.
This paper introduces SocSci-Repro-Bench, a benchmark of 221 tasks to evaluate AI coding agents' ability to reproduce social science findings from original data and code. It finds that frontier agents like Claude Code and Codex can reproduce a large share of results, with Claude substantially outperforming Codex, and that results are not primarily driven by memorization.
This paper presents two agentic AI frameworks, DeepTS/DeepCollector and DeepScribe, that automate scientific workflows including time-series data curation and conversion of physics lectures into structured reports, using a hybrid local-cloud architecture with LLMs.
AgentCo-op is a retrieval-based synthesis framework for composing interoperable multi-agent workflows from reusable skills, tools, and external agents. It uses typed artifact handoffs and bounded self-guided local repair, achieving strong results on benchmarks and enabling collaborative discovery in open-world genomics tasks.
This paper proposes typed mediation, where language models orchestrate deterministic tools instead of generating analytical code, ensuring identical outputs across regenerations. Evaluated on photoluminescence analysis, the pattern achieves perfect reproducibility across multiple runs, unlike commercial foundation models, and has been deployed successfully in real instruments.