Shipping a Self-Improving Agent

The tech world is buzzing about self-improving agents. But there is still a real gap in rolling out self-improvement in enterprise agents. The core issue is that self-improvement is not one-size-fits-all. Rather, the requirements of the industry and the type of feedback you receive heavily impact how self-improvement is shipped in production.

At Pace, we deal with some of the most complex insurance tasks, involving lengthy documents with endless edge cases. Extracting fields and evaluating rules from those documents is a really hard problem, which is tasked to our extraction agent.

Some customers process thousands of documents per day, which begs the question: how can we improve the accuracy, latency, and cost of our extraction agent over time?

How does our extraction agent work?

An average extraction task at Pace involves complex documents and a large number of requested fields. At a high level, this is what our extraction workflow looks like:

There are a few moving parts here:

• Subagent delegation - a MapReduce style architecture to split work across subagents and merge their outputs at the end

• Model - the model that powers each extraction subagent

• Context - the descriptions that each subagent gets, which depend on the fields it’s extracting

With each of these parts, there are decisions made and questions that arise:

• How do we split up the extraction fields? Do we know if this is the optimal split of work?

• What model do we use? How do we route to the best model per extraction subtask?

• What context is necessary? How do we adjust context as we receive feedback?

Our thesis is clear:

For an extraction task, there exists an optimal subagent delegation, model selection, and context. As we do more tasks, we can uncover patterns and edge cases, allowing us to “descend” to that optimal agent.

Subagent Delegation

When we delegate work to different subagents, we want the optimal fanout.

What defines optimal here? When performing fanout, the main source of inefficiency is repeated work across sibling subagents.

Fanout is necessary because one agent can’t handle all the fields that our average task deals with. Stuffing all fields into one agent leads to exploding cost and latency, along with a reduction in accuracy.

So, the question is “how can we limit duplicated work, while still parallelizing among subagents?”

The answer lies in historical runs, but we need to construct the evidence.

The approach is simple: find where fields have been found historically, and construct a graph to represent which fields are found near each other.

So we can construct a graph where:

• Nodes = extraction fields

• Edges = our confidence that two fields share evidence

For each historical run, we can incrementally build this graph, resulting in a clear map of what fields should be grouped together:

The strength of these connections is determined by the confidence level that two fields would appear near each other. Once we calculate the relationships between fields, field groups can be efficiently created, ultimately reducing the amount of repetitive work done amongst subagents.

Not only does this help us group optimally, but we can also give the agent a “hint” as to where these fields likely appear in the documents because we have already determined that they share evidence.

By running this graph construction on a cadence, the entire extraction agent system takes steps in the right direction every single day. This “smarter fanout” has shown good evidence of decreasing document rereading among subagents, on average cutting cost by 19% and latency by 16%.

Model Routing

After deciding how to split work among subagents, we enter the model layer.

Imagine these objectives:

• Subagent A - extract someone’s personal information (i.e. first name, last name, email address, home address, DOB, etc.)

• Subagent B - extract the final coverage terms after accounting for the base policy and all modifying endorsements

Although these are subagents in the same task, they should not be treated equally. Routing each subtask to the appropriate model is key for minimizing latency and cost.

The core difficulty is that assessing task difficulty at runtime is extremely difficult.

One approach for model routing in the extraction context is to develop a heuristic (e.g. the token count of the field group descriptions) to route requests. However, this type of heuristic as a routing rule fails in some cases, causing a regression in accuracy. Accuracy is non-negotiable when processing insurance claims, so this approach fails.

How do we route? As we accumulate runs of an agent, we have a lot of information about which subagents receive harder or easier tasks. The nice part is that our subagent delegation is deterministic, so we can track the difficulty of tasks per subagent over time.

By learning from past traces and rerunning extractions with different models, we develop opinions on how to model route among subagents.

The main concern with model routing is degrading accuracy. In the insurance vertical, accuracy is the most important metric, so we only model route when we are extremely confident that there is no regression in accuracy. This conviction comes from rerunning extraction tasks with varying models to verify that accuracy is maintained.

Model routing has proven to be the best lever for reducing cost and latency. We have seen a 68% cost reduction on some agents with no accuracy regression.

Optimizing Agent Context

Insurance has endless edge cases. On the first pass, it’s nearly impossible to give the agent all the necessary context to cover all edge cases, which makes it even more important to self-improve.

How do we detect errors? One great thing about extractions is that the evaluation is pretty binary: it’s either correct or incorrect. So, we rely on feedback from agent owners and task auditors to detect errors.

How do we act on this feedback? When we get this explicit feedback, we diagnose the root cause and then propose a fix that is validated, relevant, and not overfit to a single error.

We handle this with a review agent and a proposer agent:

• Review Agent - Identifies the root cause by crawling through the trace. The review agent has context on common failure modes, the overall extraction pipeline, along with the failed task’s trace.

• Proposer Agent - Given the output of the review agent, the proposer agent decides whether this requires a context fix or a harness fix, resulting in a validated customer-facing proposal or a PR for the harness fix.

When proposing a context fix to a customer, not “slopping up” the field description is crucial. By contextualizing each error with previous proposals, customer tendencies, and the current state of the agent, we propose fixes that are understandable to customers.

Validation of proposed fixes is crucial. Evaluating the effect of a potential fix is fundamental to showing customers that this is truly a fix and won’t break the agent. Every potential fix is run against an eval set that is derived from previous runs of a task. Clearly showing agent owners that a proposed fix solves the original problem without introducing new ones is half the battle.

With this approach, we have seen a 75% decrease in the number of reported extraction errors per flow. Arguably the most important effect of this is the increased confidence from customers that our agents are self-healing.

Shipped in Production

Together, these improvements have made Pace’s extraction agent more accurate, faster, and cheaper in production:

These results are a snapshot of a self-improving system. Production runs will always surface edge cases and inefficiencies, so turning those issues into solutions is the key to earning customer trust.

In today’s world, self-improvement is not just a nice feature of an enterprise agent. It is core to the product.