my fave question, talked about this coding agent Eval+Improvement loop infra + UX in my AIE talk yesterday!

biased but LangSmith is the best spot to Eval + continuously improve your coding agents, and we want to make it better so would love any feedback :) we eval all of our coding agents there –> supports Codex, Claude Code, OpenCode, Deep Agents, Pi, etc all into Tracing, sandbox infra for running evals, metrics + datasets for storing everything, and

imo the hardest parts of doing coding agent evals are:

1. having infra to easily store, update, run, and share coding agent evals

2. building a clear understanding of agent behavior & failure modes across all of the rollouts. you can use a coding agent + the langsmith-cli to look through all the data

or if teams want a managed, they can use LangSmith Engine to read every single trace from every eval, see what went wrong, prepare a report for you, and propose new changes directly in the code to fix issues

3. build new and better evals for your coding tasks. there’s many ways to do this, but we find that looking through existing failure modes from eval runs and prod is a really good grounded way to measure where agents lack today and turn that data into

we recently launched a LangSmith x @harborframework integration to double down on making it super easy for LangSmith users to improve their coding agents over time with built-in infra for running large-scale containerized evals so you can fully reproduce your tasks and read all of the traces https://langchain.com/blog/unified-stack-for-evaluating-agents…

i think most evals in the future will be shaped as environments and letting agents do real work in them, because agents are doing way more complex things and we need the eval shape to mirror how they work with us

if there’s any experience you’re looking for would love to chat

we have a ton of work underway on making each part of coding agent evaluation + improvement easier over time (as teams eval their coding agents for months and years)

as an aside –> evals are literally the training data for agents. the behaviors that we measure and reward in the evals directly get transferred as model/agent behavior as we hill-climb them. so i think there’s nothing more important than making it easy to build good evals over time :)

Harbor x LangChain: A Unified Stack for Evaluating Agents

Source: https://www.langchain.com/blog/unified-stack-for-evaluating-agents As agents increase in capabilities, evaluations have gotten more difficult.Agent harnesseslike Claude Code,Pi, andDeep Agentsnow give agents access to entire computers to read files, execute scripts, run code, and more. Every agent now needs to run in its own clean, reproducible environment for a giventask.

Evaluating long-running, stateful agents requires a new eval runner.Harborhas emerged as the industry leader in this space. In this blog, we first explain why everyone running agent evals should know what Harbor is and then show how to integrate Deep Agents, LangSmith Sandboxes, and LangSmith Experiments into Harbor.

We ultimately need to run agents in a real, reproducible, isolated environment, many times in parallel, with a deterministic check at the end.Harborsolves this problem and is now wired directly into Deep Agents, LangSmith Sandboxes, and LangSmith Observability.

How Harbor works

Harbor is aneval harness. You bring three things:

• Your agent
• Your dataset
• Your sandbox

Eachdatasethastasks, which consist of:

• An Environment (Dockerfile / Docker Compose YAML)
• An Instruction (Markdown)
• An Evaluation script (test.sh)

Compared to simpler LLM evaluation, there are two main differences:

• The environment where the agent is running in is very important - so important that it needs to be called out as part of the task! Simpler LLM evals don’t need an environment - they just call the LLM. Agents do!
• Judging the agent is done with a script. Oftentimes the agent produces other files or modifies state in some way. It’s not just enough to look at the agent’s final response - you need to look at the artifacts it creates along the way.

LangChain plugs into Harbor in three places. We integrate withDeep Agentsso any deep agent you build can run inside Harbor’s sandboxed environment. We integrate withLangSmith Sandboxesso Harbor can run each task in a LangSmith sandbox, giving each run its own clean machine. And we integrate withLangSmith Observability, the evaluation platform where you view results in detail: everyjoblands as adatasetand experiment with agent traces attached when the agent supports them.

Unifying LangChain agents with Harbor

You plug a custom agent into Harbor through its built-inlanggraphagent, selected with\-\-agent langgraph. It runs any LangGraph application including Deep Agents.

Harbor treatslanggraph\.jsonas a registry. It lists the dependencies your agent needs and maps a graph name to the function that builds it:

{
  "dependencies": [
    "deepagents>=0.6.10,<0.7.0",
    "langchain-fireworks>=1.3.1,<1.4.0"
  ],
  "graphs": {
    "deep_agent": "./agent.py:make_graph"
  }
}

Heredeep\_agentresolves tomake\_graphinagent\.py, which builds your Deep Agent and returns the compiled graph Harbor invokes:

from deepagents import create_deep_agent
from deepagents.backends import LocalShellBackend

def make_graph():
    return create_deep_agent(
        model="fireworks:accounts/fireworks/models/glm-5p2",
        backend=LocalShellBackend(),
    )

This is the only glue you write. Your agent stays your own code;make\_graphis just the entry point Harbor calls. By defaultcreate\_deep\_agentkeeps files in an in-memory virtual filesystem that never touches the sandbox, so pair it with aLocalShellBackendto give the agent real file and shell access to the environment Harbor runs it in.

For everytrial, Harbor copies this agent into that trial’s sandbox, installs thelanggraph\.jsondependencies into a fresh virtual environment there, and runs the graph inside the container. Each sandbox gets its own copy, so trials never share state and your agent runs in full isolation.

Side note:A graph can hardcode its model, but the entry can also be afactory functionthat Harbor calls with the run config. Harbor puts the model selected with\-\-modelinconfigurable\.model, so the factory above stays model-agnostic and hands whatever you pass on the command line straight tocreate\_deep\_agent.

from deepagents import create_deep_agent
from deepagents.backends import LocalShellBackend

def make_graph(config):
    return create_deep_agent(
        model=config["configurable"]["model"],
        backend=LocalShellBackend(),
    )

Unifying LangSmith sandboxes with Harbor

Running evals in cloud-based sandboxes lets youhorizontally scalefor much quicker feedback - hundreds oftrialsat once instead of one machine churning through them serially. And the sandbox is aconstrained execution environment, which is exactly what a long-running agent that touches its environment needs: a clean, isolated place to act without affecting anything outside it.

Everytrialruns in its own cloud sandbox. You bring theLangSmith Sandbox, selected with\-e langsmith, but the environment is pluggable. Harbor supports Daytona, Docker, Modal, and E2B too, all interchangeable behind the same\-eflag. Switching providers does not touch your agent, dataset, or verifier.

Atrialis the atomic unit of work: one run of your agent on onetask. Because agents are non-deterministic, you usually run each task more than oncen\_attemptsis how many times Harbor repeats every task and averages the scores so a single lucky or unlucky run does not define the result. Your wholejobis thereforen\_attempts × tasks: every task, runn\_attemptstimes, each repetition its own trial. Harbor orchestrates all of it.

For eachtrial, Harbor provisions a fresh sandbox and copies in everything that run needs: your agent code, thetask(cached on disk, then loaded into the sandbox VM), and whatever starting files the run begins from. It then runs the agent against the instruction, runs the verifier, and records the result. Harbor averages across trials into a single job result with the metrics you care about.

Unifying LangSmith Observability with Harbor

Theharbor\-langsmithintegration bringsfirst-class support for LangSmith tracinginto Harbor, plus logging todatasetsand experiments.

Enable it with a single flag,\-\-plugin langsmith. Harbor then records every job to LangSmith: it syncs the dataset, creates an experiment, and logs a run per trial with the verifier’s reward as feedback. If the agent under test supports LangSmith tracing, those traces attach directly to the experiment - so you get the full step-by-step trajectory alongside the score. If it does not trace, you still get the dataset, experiment, results, and feedback.

Under Datasets & Experiments we are able to view all of our active datasets that are being used.

An experiment is an entire run on a given dataset. To view the specific experiments and their respective scores and statistics for a given dataset, click into it.

We believe integrating traces into evals lets you further refine your evals, and in turn better understand and improve your agents. The score tells youwhethera trial passed; the trace tells youwhy.

The result: a full eval stack for agents

Put together, this is a complete stack for evaluating agents, where each layer does one job well:

• Harbor- the eval harness that orchestrates trials.
• Deep Agents- for building the agents under test.
• LangSmith sandboxes- the isolated cloud execution environment.
• LangSmith- the system of record for datasets, experiments, traces, and scores.

And the part you bring stays small:

• Your agent, with or without tracing.
• Your dataset, remote from a registry or local on disk.
• Your cloud sandbox— LangSmith, with\-e langsmith.
• Your UI view—\-\-plugin langsmith.

If you have a LangSmith account and a dataset, you can try the whole thing by installing Harbor with thelangsmithextra, which brings both the LangSmith sandbox environment and the eval plugin. Then set your LangSmith and model credentials, and turn on tracing so the agent’s traces attach to the experiment:

pip install "harbor[langsmith]"
export LANGSMITH_API_KEY="<LANGSMITH_API_KEY>"

export LANGSMITH_PROFILE=prod
export LANGSMITH_TRACING=true
export LANGSMITH_PROJECT=harbor-deepagents
export FIREWORKS_API_KEY="<FIREWORKS_API_KEY>"

harbor run \
  --agent langgraph \
  --model fireworks:accounts/fireworks/models/glm-5p2 \   # agent
  --ak project_path=./deep-agent --ak graph=deep_agent \  
  -d [email protected] \                                 # dataset of tasks
  -e langsmith \                                          # cloud environment
  --plugin langsmith

Read the Harbor integrations docsto get started. For more on running evals in Harbor, seeRun evals.

‍

Michael Thiessen (@MichaelThiessen): @Vtrivedy10 do you know of any eval platforms that work with coding agents?

Unless I’m blind, everything looks like it’s product-agent focused.

I need something that will work with coding agents on complex R&D tasks.

(currently building my own so we can properly eval our