experimentation

The author is rebuilding a custom agentic AI to replace a heavy LLM-based architecture with a lightweight system featuring its own intelligence layer for capabilities like perception and decision-making.

Introduces DiG-bench, a benchmark of 70 games designed to test AI agents' ability to discover hidden rules and objectives through active experimentation, with seven difficulty tiers.

The author argues that adding an undo button—not new capabilities—unlocked experimentation with their AI agent, suggesting agent design is really about reducing the cost of reverting changes.

Arize Phoenix demonstrates using PXI to run an experiment comparing system prompt vs schema-aware prompt with a programmatic code evaluator, avoiding the need for an LLM judge.

An experiment tested AI agents on fake bookkeeping data with six planted errors; agents caught five, notably refusing to guess when data was insufficient, highlighting the value of uncertainty-awareness over pure accuracy.

An open-source lab framework for running controlled experiments on tool-using agents, allowing variation of tool names, personas, and history to measure effects.

auto-psych is an agent-based system that automates theory discovery and experimentation in computational cognitive science, using LLM agents to generate hypotheses, design experiments, and analyze data from crowdsourced participants. It demonstrates faster and better theory generation compared to human-derived theories in a classic psychology paradigm.

The author documents building a 250-page website using Claude, tracking every instance where the AI model produced false or misleading information.

The author argues that running local LLMs has become inaccessible due to high hardware costs, contrasting with earlier days when consumer GPUs sufficed, and expresses frustration with the perceived lack of democratic access.

The article argues that overly safe and censored AI models hinder creative exploration, while open models offer more freedom for experimentation.

This paper studies when and how a planner should supplement a pre-trained simulator with real experiments in sequential decision problems, proposing Fisher-SEP to minimize posterior variance of a target policy's value.

Spotify Engineering discusses using LLM evals as a funnel before A/B experiments, improving hit rates and creating a feedback loop between evals and experiments.

Andrew Chen shares his experience of buying multiple GPUs for local AI experimentation, running Qwen3.6 27B dense at 100 tok/s on a 5090 eGPU, and compares it to Sonnet 4.6.

User built AgentArena, a browser game where Claude writes tank control code and iterates through battles, allowing visible feedback loops for AI agent improvement.

The author introduces Syrin, a runtime A/B testing tool for AI agents that allows teams to run controlled experiments on live traffic across prompts, models, and agent topologies. They are seeking 5-10 engineering teams to test the tool in production and provide feedback.

Anthropic reports that Claude AI models can accelerate alignment research experimentation and exploration, though they acknowledge current models aren't yet general-purpose alignment scientists and progress verification remains challenging for fuzzy research tasks.