built a local tool for narrowing down successful-but-wrong runs

Reddit r/AI_Agents Tools

Summary

The author built a local browser-based tool called Traser to help engineers diagnose runs that complete without errors but produce wrong outcomes by comparing execution steps and providing evidence-based investigation points.

hey peeps i built something i thought might help after talking to other engineers the field i got tired of hearing the same shit over and over the obvious pattern: run completes trace is there nothing obviously exploded the final outcome is still wrong and then somebody has to spend an afternoon manually diggoing through the execution trying to figure out which part is actually worth chasing first and yeah yeah i know the ideal answer is structure the system better up front. add invariants, eval each stage, verify state, make handoffs explicit, all that good stuff. some people absolutely do that it removes alot of this pain but from the conversations ive had plenty of people still end up manually comparing step inputs and outputs, diffing against a good run, tracing state backwards, or staring at the trace trying to figure out which difference actually matters so i said fuck it lets see if some of this can be narrowed down automatically stuff that kept comin up: previous step looks fine but the next step actually received different state tool call says success but the response body contains an error or rate limit wrong account/id gets carried forward while every step stays schema valid prompt or model input changes without the trace making it obvious retries eventually succeed but the execution path is now different the configured model/provider isnt necessarily the one that actually served the request a verification step gets skipped and everything downstream still completes so i built Traser u give it a suspicious run and optionally a trusted run. it normalizes the execution, aligns corresponding steps, compares inputs, prompts, tool args, outputs, state, retries, provenance and other recorded behavior, suppresses lower value diffs, then gives u a small set of evidence backed places worth checking theres no llm doing the diagnosis rn. the current engine is deterministic and the trace analysis runs locally in your browser ih and also deliberately not claiming root cause. if it finds an account id changed at step 4, thats evidence that step 4 might be worth investigating, not proof that step 4 caused the final outcome the part im trying to validate now is whether the ranking and alignment actually save people time on real runs or if its just a smarter looking diff this is obviously still “small potato“ stuff rn but have at it and let me know if u actually find it useful or not. if it gives u a dumb candidate, misses the important thing, or just doesnt save u any time id rather hear that
Original Article

Similar Articles

I'm tired of manually debugging traces

Reddit r/AI_Agents

A developer builds a debugging tool for AI agents that compares replays against reference runs to identify where behavior first drifted, expressing frustration with manual trace debugging.

Test-case Reducers Are Underappreciated Debugging Tools

Lobsters Hottest

This blog post explains the value of test-case reducers for debugging, detailing how they automate input reduction to isolate bugs, and explores advanced techniques like considering error frequency or instruction count.