I reimplemented Arbor (a research agent that grows a tree of hypotheses) on LangGraph — an agent that keeps its experiments instead of forgetting what failed

Most agent demos I see are a single LLM in a while-loop calling tools. I read a paper with a more interesting structure — a multi-agent setup that runs real experiments and *remembers what worked* — and rebuilt a small version of it to understand how it works. Sharing the reimplementation and a couple of agent-design lessons that stuck. The paper is Arbor: Toward Generalist Autonomous Research via Hypothesis-Tree Refinement (RUC Gaoling School of AI + Microsoft Research, arXiv 2606.11926). What Arbor does (the paper) Instead of one-shot attempts, a long-lived coordinator agent grows a tree of hypotheses : each node bundles a hypothesis, an executable artifact, the evidence from running it, and a distilled insight. Short-lived executor agents run the experiments and report back; the coordinator absorbs the results and refines. Crucially it only keeps changes that survive held-out evaluation, so it doesn't overfit to its own dev signal. The paper reports it beating Claude Code and Codex by ~2.5x average relative held-out gain across six optimization tasks on the same compute budget (their numbers — I did not reproduce the benchmarks). What I built. A much smaller reimplementation on LangGraph (I called it arbor-lg). I picked LangGraph specifically for two of its features: - StateGraph — I model the hypothesis tree as the graph state, a typed field (`tree: dict[str, Node]`) with an immutable merge reducer, rather than stuffing it into a message list. Nodes are written back as whole objects, so concurrent writes to different nodes compose. - SQLite checkpointer — the whole loop is durable, so a multi-hour run survives a crash and resumes from the last checkpoint. (Easy to demo: crash at cycle N, rerun, it picks up where it left off.) The loop is a cyclic graph: `observe → ideate → select → dispatch → backpropagate → decide`, with a conditional edge out of `decide` that loops back or ends. Two things I found genuinely interesting: 1. Invariants enforced by graph structure, not by discipline. Two rules I wanted to guarantee: (a) the held-out evaluator is called only in the merge-gate node, and (b) the executor never receives the full tree state — it runs as an isolated sub-agent that takes a compact input dict and returns a compact report. Because these are wired into the graph, test-set leakage and context bloat aren't things I have to remember to avoid — they're structurally impossible. 2. Results backpropagate as insights, not just scores. After experiments run, each node's lesson is distilled (via the LLM) and pushed up to its parent, so the next round of ideation is informed by everything already tried on that path. That "the tree learns from its own failures" part is what separates it from grid/random search. Concretely, on a Kaggle comp. I pointed it at Spaceship Titanic (a starter/tabular competition). Each hypothesis is a snippet of feature-engineering code the LLM writes — a full `engineer(df)` function that refines its parent's — which I execute, then train a fixed model and score with 5-fold CV. Failed code is pruned into a "lesson," not fatal. It climbed to a solid accuracy (~0.79–0.82 band); the goal was understanding the method, not topping a leaderboard. Training runs as a Kubeflow Pipelines job and everything is tracked in MLflow, with a live Streamlit dashboard showing the tree grow (video below). Honest caveats , since this is a learning reimplementation: - The code executor uses a whitelisted `__builtins__` + a banned-token screen. That's best-effort, not a real sandbox — the actual isolation boundary is the training subprocess/pod. Only run it against a model you trust. - I apply `engineer(df)` per-fold separately (so cross-row features don't leak across a fold), but that's not the gold-standard fit-on-train / transform-on-val transformer contract. It's a known simplification. - Spaceship Titanic is a toy competition. This is not a reproduction of the paper's benchmark results, and I'm not claiming their numbers. Curious whether anyone's used tree-search / hypothesis-refinement structures for goal-directed agent tasks outside ML — prompt optimization, code perf, config search, etc. If you've tried something similar (or see a hole in how I set up the merge gate / leakage handling), I'd genuinely like the feedback.