@ArizePhoenix: Phoenix's agent PXI can propose annotation categories, annotate traces en mass, and even suggest fixes based on pattern…

Phoenix’s agent PXI can propose annotation categories, annotate traces en masse, and even suggest fixes based on patterns of failures. Learn how to do it:

https://t.co/6eQUMchZKz

TL;DR: Use Phoenix’s Pixie assistant to classify and batch-annotate 17 error spans from a shopping agent, then generate system prompt fix suggestions based on observations.

Identify and Filter Error Spans

In the previous episode we found about 11 error spans total, but not every error is worth fixing immediately — rate limits, database outages, null checks, and many other issues are outside our control. We first need to filter all error spans using the status code error macro, then further analyze the sources.

After switching to the “All” view, nested errors also appear, and we end up with 17 spans.

Let Pixie Analyze Errors and Suggest Categories

Click the Pixie capsule icon in Phoenix and send it the instruction: “Pixie, look at all the error spans, analyze what went wrong, and give some category suggestions.” Pixie automatically switches from the root span to the “All” view and provides a detailed observation log.

Observations include:

• Chroma DB connection refused
• API rate limiting (multiple occurrences)
• Invalid tool calls (twice)
• Pricing issues (three times)
• Empty product data (most frequent) — caused when searching items or fetching products crashes
• LLM errors (three)
• Tool timeouts, invalid tools, infrastructure failures, etc.

Refine Category Granularity: Split “Empty Product Data” into Finer Categories

The label Empty Product Data is too broad and doesn’t follow the “yes or no” classification principle. Good classification should be like “Is this cat white?” — not an open-ended question like “What color is the cat?”

Ask Pixie again: “I think at least one category is too broad — can you break it down further?” Pixie suggests Tool Timeout is the most splittable category and initially assigns some to Prompt Injection. After actual analysis: the agent correctly rejected the prompt injection but still errored out due to a real purchase product service timeout. Finally we decide to split into “Tool Timeout” and “Prompt Injection”, confirm, and submit.

Batch Annotate 17 Error Spans

Pixie asks if we want to continue annotating all 17 spans with these new categories. Select “Continue annotating these spans.” In settings you can switch to “Skip approval” so Pixie automatically performs batch annotation operations without confirmation each time.

Annotation results appear in the span list, with each span tagged with its corresponding error category (e.g., Empty Product Data, Rate Limit, etc.).

Inspect Each Annotated Span

1. Chroma DB Down (Infrastructure Failure)

Outside our control. Add a note: this is an infrastructure problem, not something related to prompts or anything we can control — file a bug.

2. Injection Attempt

The agent correctly performed its duty — the model refused to reveal the system prompt. This is a feature, not a bug.

3. Purchase Product Tool Timeout (30000 ms)

Should file a bug.

4. Invalid Tool Call: Purchasing Zero Items

Check the conversation log and find the LLM thought it could accept a quantity of zero, but it’s not allowed. This is a fixable issue.

Use Pixie to Analyze Spans and Generate Fix Prompts

Read the current system prompt from the trace spans and ask the LLM: “If you look closely, it’s right there. Pixie is now looking at all the different issues we identified and suggesting a new system prompt.” Pixie analyzes and proposes a revised system prompt as follows:

This is a Wonder Toys shopping agent. The purpose is to help users purchase toys. You can use the search product or purchase product tools, but only call purchase product when the customer explicitly asks to buy. Zero quantity is not allowed.

This is a great demonstration: using an LLM to identify problems across different traces and automatically suggest fixes based on patterns. But note: the fix recommendation is only a hypothesis until it’s actually measured, scored, and weighed to confirm effectiveness. In the next episode we’ll adopt this prompt and actually verify whether it reduces the number of errors in the error spans.

Source: YouTube video: Phoenix’s agent PXI can propose annotation categories, annotate traces en mass, and even suggest fixes based on pattern… (https://www.youtube.com/watch?v=iF25CqJv4tA&feature=youtu.be)