autoresearch

SciΠPivot introduces a 248-case benchmark for evaluating scientific judgment in autonomous research agents, built from six months of human corrections on 26 projects. Results show that frontier LLMs often fail to recognize human-corrected plans, with most performing below chance and peaking at 63.5% accuracy versus 73.4% for fresh human researchers.

一个实验性想法 Heisentools：让 agent 从真实观察到的工具调用中重写 mock.py，从而模拟任意服务，用于测试 agent 而避免速率限制、上下文漂移和污染数据库。已在真实 Slack 消息上验证。

A paper showing an LLM agent rewriting the Sella molecular geometry optimizer to cut force-evaluation counts, producing AutoSella, a family of optimizers that reduces r2SCAN-3c DFT force calls to 40.2–77.2% of Sella's while matching energy reduction on held-out molecules.

The paper discovers "forking," a generalization failure in NanoGPT autoresearch where training and validation loss sharply diverge at epoch boundaries when models with an over-encoding n-gram memory branch are trained under data replay. It mechanistically explains the phenomenon and cautions about unintended side effects of techniques produced by autoresearch agents.

Agora is a system that uses Git as shared memory for AutoResearch agents, enabling collaborative and immutable recording of research contributions in an append-only directed acyclic graph.

The tweet identifies inference serving as a prime target for autoresearch, emphasizing end-to-end optimization with constraints on latency, quality, and throughput, covering various aspects in a unified search space and hinting at future developments.

A tweet from @_philschmid speculates that future AI agent harnesses will focus on coded extensions for automatic integration, with Pi leading the trend and DeepSeek as an extreme example, all driven by autoresearch and recursive self-improvements.

A developer shared their experience deploying and optimizing custom AI models for a customer over a week, using posttraining, inference tuning, and orchestration, and promoted services for high-spending AI workloads.

This paper identifies a scalability bottleneck in RL-trained automatic research agents—environment execution dominates training cost—and proposes World Model RL (WMRL) with online debiasing and inverse-variance denoising to replace real execution, achieving 3–4x training speedups and better generalization.

This paper identifies common failure modes in tree-search-based autoresearch agents applied to tabular datasets, such as repeated bug resolution, poor hyperparameter tuning, and ineffective exploration, and proposes targeted interventions like a global debug consultant and refined tree-search algorithms to recover wasted compute and improve performance without changing the underlying language model.

User iamai_omni praises Fable 5's self-correction ability, considering it comparable to ASI. Citing a recommendation from yibie, they point out that the Superpowers author had Fable 5 run an autoresearch loop, spending $165 to complete 25 experiments, increasing build speed by 50% and reducing token overhead by 60%, and documented failures and correction processes in detail.

The article describes an experiment using Claude Code to autonomously develop a file compression algorithm with constrained optimization, evaluating the viability of AI agents for unsupervised problem-solving.

Introspection, a new AI startup founded by ex-xAI engineers, introduces 'autoresearch' – a feedback loop system where agents maintain and improve themselves using signals, evals, and human input, moving beyond traditional agent harnesses.

A Twitter thread highlights emerging autoresearch labs that are building AI systems to automate the full research loop, from hypothesis to experimentation.

awesome-autoresearch is a curated list of automated research use cases. This update adds two entries: Cribl's production deployment and Colab TPU port.

Jim Fan announced that the team launched ENPIRE, bringing AutoResearch into the physical world for the first time, equipping 8 Codex agents with robots, GPUs, and tokens, enabling them to autonomously learn and collaborate on real hardware to complete tasks.

awesome-autoresearch is a curated list of autoresearch use cases. This new entry applies Karpathy's autoresearch pattern to everyday software engineering scenarios, providing concrete templates to lower the adoption barrier for engineers.

Researchers benchmarked 7 frontier models on autoresearch tasks. Fable-5 won overall, but the open model Kimi-K2.7-Code surpassed others on ML engineering tasks.

Evo ported its autoresearch loop onto Anthropic's dynamic workflows in Claude Code, moving orchestration from in-context memory to deterministic JavaScript, solving long-horizon instruction adherence and enabling self-evolving workflows.

PostHog used an AI agent based on Karpathy's autoresearch to find a three-year-old bug in their ClickHouse query engine that prevented proper primary key usage for timestamp filters. Fixing it improved performance by 11% and reduced scanned granules by 62%.