Check out the video walkthrough to see how we tested a tool description change with Harbor and Phoenix https://youtu.be/OuGQ5zuWHuk?si=MH961XO8zbZrmGLM…

TL;DR: Nancy from Arize and Phoenix engineer Elizabeth share how to use the open-source framework Harbor to run offline benchmarks of AI agents and tools in sandboxes, and use the Arize Phoenix plugin to record, version, and compare scores, errors, and traces.

Your agent changed. Did things actually get better?

When you’re building an agent or the tools it uses, each individual change might look good on its own. But the harder question is whether a new model, prompt, tool, or environment actually improves the workflow—or simply moves the problem somewhere else.

This was the topic Nancy from Arize and Elizabeth, an engineer on the Phoenix team, discussed in the first Phoenix webinar in the AI Builder series: how to benchmark AI agents and tools using Harbor and Arize Phoenix.

What you’ll learn today

• Why benchmarking matters, and why it’s difficult
• What Harbor is: the open-source agent benchmarking tool Arize has been using internally
• How to use Harbor with Phoenix
• An end-to-end example showing how the Arize team uses it in practice

The problem: Why do we still need benchmarks?

Many people really do “just ship it and fix things when they break” these days. You certainly can do that. But when you inevitably start making changes, simple unit tests and online evaluations in production cannot answer these questions: Can you be sure it is running reliably and efficiently?

Offline benchmarks are the only way to answer these questions. They let you test agents before deployment, so you can iterate between versions faster and with greater confidence.

Unfortunately, this is harder than setting up online evaluations in production, because you need to:

• Run and compare different agent configurations on the same tasks, using the same tools and environment
• Reset state between runs
• Make conditions as realistic as possible; otherwise, the results will not generalize to production
• Because agents are inherently interactive, faithfully reproducing their behavior usually requires setting up a complete working environment, including a filesystem and various services for the agent to interact with

Arize’s firsthand lesson: Evaluating Pixie

Pixie is a coding agent built into Phoenix—an in-app assistant that helps users navigate the Phoenix product, analyze traces, design evaluations, and more.

The core agent is essentially a framework plus an LLM, which is exactly what they wanted to evaluate. But it needs to interact with many other components, most notably a real Phoenix server—a database full of trace data that Pixie is designed to interact with.

They initially tried to keep the evaluation simple by focusing only on Pixie’s main components, but that meant mocking the rest of the system. Setting up those mock components turned out to be anything but simple; it was actually harder than deploying a complete Phoenix server. Ultimately, they found it even harder to trust that this cobbled-together “Frankenstein” version of Pixie would behave like the version in production.

This problem isn’t unique to Pixie. Phoenix has also released an MCP server, a CLI, and skills for coding agents such as Claude Code to use with Phoenix. Testing them presents similar challenges—they still need an environment where the agent can work with a real Phoenix server and data.

What role does Harbor play here?

Harbor is an open-source framework and the framework behind the coding agent benchmark Terminal-Bench. But you don’t have to work at an AI lab to find it useful; it is valuable for smaller-scale benchmarks too.

The core problem it solves is: Define a set of tasks and success criteria in a sandbox environment, then run different agents on those tasks.

Harbor handles orchestration with your chosen sandbox provider, such as local Docker, Daytona, or any other option, while you retain complete control over the tasks, how the environment is built, and how results are verified.

For each trial, Harbor will:

1. Start a fresh sandbox
2. Run the agent on the task
3. Capture the agent’s steps as a Trajectory (Harbor’s trajectory format)
4. Run tests inside the sandbox after the agent finishes
5. Assign rewards based on the agent’s trajectory or the environment’s state

This lets you focus on designing good tasks and tests while Harbor handles the tedious infrastructure work.

What makes up a benchmark task?

Tasks are highly flexible, but they have these core components:

Instruction

This is the prompt or user query you want to use to test the agent. For Arize, it looks like a question a user might ask about data in Phoenix, such as:

“Which tool failed the most times in this project?”

Ideally, tasks should be based on real production traffic and usage; if you’re starting from scratch or need more coverage, you can also generate synthetic tasks.

Environment

This is usually a Dockerfile containing the dependencies or data your agent needs to work. For Arize, they actually start a Phoenix server already populated with trace data.

Tests

Any code you write to verify the agent’s solution. This usually produces a final binary reward—0 or 1, pass or fail—but you can write any number of additional rewards, using an LLM as a judge, deterministic code, or really anything at all.

One great feature is that you can run tests based on either the agent’s trajectory or the state of the environment.

Other settings

One important setting is: You can restrict the external network connections allowed in the sandbox, so your agent cannot cheat. “That’s pretty important these days.”

A Harbor run is defined by pointing to a set of tasks and one or more agents. You can also specify the sandbox provider, how many trials to run, and the Phoenix plugin to use.

How do you determine whether the agent got it right?

One great thing about Harbor tests is that you can write different tests for each task. If you have different criteria for different tasks, that is entirely possible.

Two examples from Arize’s benchmarks:

• Question tasks: “Which tool failed the most times in this project?” Because there is a clear correct answer, they use an LLM judge to determine whether the agent’s final response reasonably matches the reference answer.
• State-changing tasks: Ask the agent to label traces where users demonstrated some kind of difficulty. In this case, they check the Phoenix database directly to ensure those annotations were written correctly.

Across all tasks, they also capture efficiency metrics, such as the number of turns and tool calls in each trajectory. This is useful because agents can often reach the correct answer while taking an inefficient route.

A strong recommendation: Design tasks and verifiers independently of the implementation

You should try to measure outcomes without prescribing how the agent should get there. For example, their verifiers do not check whether a particular tool was called.

This has two benefits:

• The benchmark is more likely to remain valid after the agent changes
• It reduces the maintenance burden of the benchmark over time

This design also allows them to reuse the same set of tasks to evaluate all agent interfaces—including Pixie, MCP, and CLI.

What does the Phoenix plugin add?

Harbor can already run benchmarks and record results. The Phoenix plugin adds several things on top:

• Records results as Phoenix datasets and experiments: This is convenient if you’re already using Phoenix; even if you aren’t, the plugin provides a way to translate all Harbor concepts into Phoenix terminology.
• Automatically creates datasets for tasks and versions them when changes occur. This lets you distinguish changes to the agent from changes to the benchmark itself. Benchmark datasets are often “living documents” that grow and change over time, making version control very valuable.
• Each agent configuration gets its own experiment, making it easy to compare agents or versions.
• Repeated trials are recorded as repetitions within the experiment, allowing you to measure agent reliability.
• Verifier rewards become annotations, with a special infrastructure annotation added so you can distinguish “execution failures” from “agent behavior failures.”
• Converts agent trajectories (ATF) into OpenTelemetry traces, so you can easily inspect the agent’s steps in Phoenix and get timing, cost, and token information—for example, how often the LLM reads from or writes to the cache.

Finally, Phoenix makes it easy to share results and collaborate with your team or coding agents, with everything running on your self-hosted infrastructure.

End-to-end demo: A real run

Elizabeth showed what this looks like in Phoenix.

Benchmark setup: This particular benchmark uses trace data from a public dataset called Trail to prepopulate the Phoenix database. All tasks are questions about that project and its traces, so the name “Trail” appears throughout.

Run command: The run points to a YAML file—a more compact version of the CLI arguments—that specifies the appropriate tasks and the agents to run. This run uses Daytona, performs three repeated trials, and enables the Phoenix plugin.

Results in Phoenix:

• A dataset named Trail Benchmark containing 18 examples, each corresponding to a task in the Harbor benchmark
• A series of experiments run on that dataset, with a baseline containing five different agent configurations:
  • Claude Code with MCP
  • Claude Code with CLI
  • Codex with MCP
  • Codex with CLI
  • Phoenix chat agent

Key takeaways

1. Offline benchmarking is the only way to answer “Did this change actually improve the workflow?” But it is harder than online evaluation because you need a complete environment that is resettable, reproducible, and as realistic as possible.
2. Harbor handles sandbox orchestration, state resets, and trajectory capture; you control task design and verification logic.
3. Verifiers should focus on outcomes, rather than the path: Check state and final answers, without checking for specific tool calls. This keeps benchmarks valid as agents change and allows them to be reused across Pixie, MCP, and CLI.
4. The Phoenix plugin handles dataset versioning, experiment comparisons, reward annotations, distinguishing infrastructure failures, and converting trajectories into OpenTelemetry traces, making results easy to share and collaborate on.

Source: https://www.youtube.com/watch?v=OuGQ5zuWHuk