Tag
该论文构建了包含2882个LLM提出的ML实验想法的数据集，评估五个前沿大模型预测实验结果的准确性，发现最佳模型Claude Opus 5.5仅解释约一半方差（R²=0.49），且所有模型都系统性低估了变化幅度，表明当前LLM对ML想法的直觉有用但有限。
SciΠPivot introduces a 248-case benchmark for evaluating scientific judgment in autonomous research agents, built from six months of human corrections on 26 projects. Results show that frontier LLMs often fail to recognize human-corrected plans, with most performing below chance and peaking at 63.5% accuracy versus 73.4% for fresh human researchers.
一个实验性想法 Heisentools：让 agent 从真实观察到的工具调用中重写 mock.py，从而模拟任意服务，用于测试 agent 而避免速率限制、上下文漂移和污染数据库。已在真实 Slack 消息上验证。
SciΠvot 是一个基于 248 个人工纠正案例的基准，用于评估 LLM 智能体在科研自主研究关键时刻的科学判断力；结果显示前沿模型常偏好有缺陷的原始计划而非专家修正方案，整体表现低于人类研究者。
A paper showing an LLM agent rewriting the Sella molecular geometry optimizer to cut force-evaluation counts, producing AutoSella, a family of optimizers that reduces r2SCAN-3c DFT force calls to 40.2–77.2% of Sella's while matching energy reduction on held-out molecules.
The paper discovers "forking," a generalization failure in NanoGPT autoresearch where training and validation loss sharply diverge at epoch boundaries when models with an over-encoding n-gram memory branch are trained under data replay. It mechanistically explains the phenomenon and cautions about unintended side effects of techniques produced by autoresearch agents.
Agora is a system that uses Git as shared memory for AutoResearch agents, enabling collaborative and immutable recording of research contributions in an append-only directed acyclic graph.
The tweet identifies inference serving as a prime target for autoresearch, emphasizing end-to-end optimization with constraints on latency, quality, and throughput, covering various aspects in a unified search space and hinting at future developments.
A tweet from @_philschmid speculates that future AI agent harnesses will focus on coded extensions for automatic integration, with Pi leading the trend and DeepSeek as an extreme example, all driven by autoresearch and recursive self-improvements.
A developer shared their experience deploying and optimizing custom AI models for a customer over a week, using posttraining, inference tuning, and orchestration, and promoted services for high-spending AI workloads.
This paper identifies a scalability bottleneck in RL-trained automatic research agents—environment execution dominates training cost—and proposes World Model RL (WMRL) with online debiasing and inverse-variance denoising to replace real execution, achieving 3–4x training speedups and better generalization.
This paper identifies common failure modes in tree-search-based autoresearch agents applied to tabular datasets, such as repeated bug resolution, poor hyperparameter tuning, and ineffective exploration, and proposes targeted interventions like a global debug consultant and refined tree-search algorithms to recover wasted compute and improve performance without changing the underlying language model.
User iamai_omni praises Fable 5's self-correction ability, considering it comparable to ASI. Citing a recommendation from yibie, they point out that the Superpowers author had Fable 5 run an autoresearch loop, spending $165 to complete 25 experiments, increasing build speed by 50% and reducing token overhead by 60%, and documented failures and correction processes in detail.
The article describes an experiment using Claude Code to autonomously develop a file compression algorithm with constrained optimization, evaluating the viability of AI agents for unsupervised problem-solving.
Introspection, a new AI startup founded by ex-xAI engineers, introduces 'autoresearch' – a feedback loop system where agents maintain and improve themselves using signals, evals, and human input, moving beyond traditional agent harnesses.
A Twitter thread highlights emerging autoresearch labs that are building AI systems to automate the full research loop, from hypothesis to experimentation.
awesome-autoresearch is a curated list of automated research use cases. This update adds two entries: Cribl's production deployment and Colab TPU port.
Jim Fan announced that the team launched ENPIRE, bringing AutoResearch into the physical world for the first time, equipping 8 Codex agents with robots, GPUs, and tokens, enabling them to autonomously learn and collaborate on real hardware to complete tasks.
awesome-autoresearch is a curated list of autoresearch use cases. This new entry applies Karpathy's autoresearch pattern to everyday software engineering scenarios, providing concrete templates to lower the adoption barrier for engineers.
Researchers benchmarked 7 frontier models on autoresearch tasks. Fable-5 won overall, but the open model Kimi-K2.7-Code surpassed others on ML engineering tasks.