@ArizePhoenix: Phoenix has an agent built into it now! PXI can help you find the crucial traces you should actually be reading. Short …

Phoenix has an agent built into it now! PXI can help you find the crucial traces you should actually be reading.

Short video on how to do use PXI in your daily flow: https://t.co/WRl3bYHQXi

TL;DR: This article introduces how to use Phoenix’s built-in Pixie smart assistant to quickly filter out silent failure traces where the agent span errors but the model reply is normal, so you can prioritize high-signal records.

Meet Pixie: Phoenix’s Built-in Assistant

Phoenix now comes with a built-in smart assistant, Pixie. Click the Pixie button in the bottom right corner to open it. The system will ask if you understand that the assistant uses an LLM and may sometimes make mistakes. You can choose to enable or disable sending trace information, and link those traces to your email to send to Arise’s Phoenix support team. I prefer to enable it, because sometimes there are bugs and this makes it easier for engineers to improve Phoenix (we release weekly updates). After confirming, you need to select a model. I chose Claude (built-in API key); Sonnet 4.6 should be sufficient.

Scenario: Finding Key Records Among a Large Number of Traces

Suppose your site has hundreds or even thousands of traces. Manually reading them all is difficult to determine which ones are important. Before improving the agent, you need to see what it actually did — read real examples, write down the problem in plain language, then decide whether to mark it as successful. The difficulty isn’t in reading itself (anyone can do it), but in identifying which traces are worth your time. Using Wonder Toys site as an example, inputs and outputs seem fine. How to first identify which ones are worth scoring? Use Pixie.

Voice Command: Find Silent Failures

I plan to use voice control. Say to the Pixie input box: “Find traces where the agent span errors but the model reply is normal.” This type of silent failure is very disappointing for users — the agent has a serious problem but replies as if nothing is wrong. Pixie starts searching for traces where the span errors but the LLM span completes with a status. It returns results: found 11 agent spans with errors, of which 8 have LLM spans that completed normally. Pixie even gives suggestions on error patterns and automatically filters the span table to show only agent spans with errors.

Analyzing Specific Examples

Example 1: Confirming Order with Quantity Zero

Problem: quantity needs to be greater than 1, but only 0 was received, so the purchase cannot be completed. However, looking at the LLM reply, it gives the customer an order ID and tells her the item is on its way. This is impossible because the product quantity in the order is 0. Nothing looks wrong in the reply: friendly tone, correct format, even an order ID. If you were reading customer conversation logs all day, you wouldn’t notice. But the span shows an error, yet the message indicates confirmation — this is a high-signal trace: inconsistency.

We add a note (different from a comment, like a draft written in the margin): “Told customer order confirmed, but purchase actually failed.” This is very useful when coming back to add annotations later.

Example 2: Chroma Connection Completely Failed

In this example, Chroma Connection completely failed, unable to search products. The customer is asking about board games. Instead of acknowledging the database connection failure, the LLM makes up some products the customer can buy. Luckily the customer didn’t place an order, so there’s no need to contact them to apologize. Although it didn’t hallucinate toys that don’t exist in the database, this kind of error is still worth attention.

Summary

Pixie found these spans in seconds, but reading and analysis still rely on us. Add notes to high-quality traces to understand why they failed and how they could be handled. The next episode will cover how Phoenix organizes structured annotations, but starting by recording where things go wrong is the right direction.

Source: https://youtu.be/5lUgdRFf4ZI