The author accidentally developed a local retrieval tool for debugging in agentic AI software that efficiently narrows down code functions to find bugs, and is seeking more testing ideas to validate its effectiveness.
So, I was working on my agentic AI software. It is really good tbh so far, but ok, we are not going to talk about that in this post. So, during the testing, I found it was taking too many tokens to find the bugs. To fix it, I tried a lot of things like Aider's repo map, local LLMs, embedders, etc., but they all take a lot of time to build, + local LLMs and embedders can't run on very low-end laptops. I refused to use grep tools as a fallback while they are building, why? Idk I just don't like the idea. After that, I somehow got the idea for a new local tool (not gonna reveal the arch or code 🤫) and it is kind of working. So, it somehow decreases the pool of searchable functions to only 900 (max, can be less too) with a 100% presence rate of the bug in it. I am not using normal indexing or stuff just to clarify, but I can't reveal the exact method either. So I tested it in a few places—the 1st prototype version of it was on SWE-LITE 55 cases, out of which it had 52 in that 900 pool and caught 38 on the exact line while burning only 1k tokens just for a little assist (this excludes output tokens which is around 2k). (It takes the same time as running and finding bugs with tools currently being used in agentic AI since it's only a prototype right now, I never optimized it). But then I thought it might be because the LLM model (I'm testing it with Gemini 3.1 Pro and Opus 5, I keep switching since I'm broke) might've already been trained on SWE-LITE so it knows the bugs. So I tested it on my own L2D (Language to DST) project which is quite complex in itself. I injected 20 bugs in that project myself (logical bugs) and wrote vague prompts , for example - "It can't find the modules anymore when I start it", "It never clears memory and OOMs after 5 minutes", and "Stack traces are getting completely cut off when my code crashes". And the result blew my mind. It was 20/20 in the 900 pool of functions (total was around 18k something) and it gave 18/20 exact lines where the bugs were. ;-; Boi I was jumping around. But then I thought I only tested this on Python so far. And that freaked my mind out cz I forgot to code it for multiple langs, so I used Claude Code and made it support 90+ langs (ofc I told it how to make it) and then I tested it on the latest OpenMAIC repo so it didn't have trained data, and my own agentic AI source code (backup) which has 8 diff languages being used as of rn. (With vague prompts as much as the ones used in L2D, like "Regen is deleting all my work wtf" and "Zooming just crashes the whole page immediately") - Here were the results - "It hit 20/20 presence in the top 900 pool, got 16/20 exact func, and verified 13/20 exact mechanisms with ZERO false positive hallucinations (locally verified, this thing is still work in progress and i doubt it will get better, im thinking of removing it)." Then as a last test I ran my tool and Antigravity on the latest pagefind github repo. I injected 20 bugs with vague prompts. My tool took 12 mins (still in Python, I'll convert it to Rust if I think it really works) and Antigravity took 15 mins. Antigravity found 12/20 bugs correctly and 16/20 files correctly but located the wrong line of code. It took 6 million tokens (I mean most of it is cached so I'll count it as 1m fresh). My tool found 16/20 bugs, 20/20 presence in the 900 func pool, and took 20k tokens (all fresh) :) So my main ques was, where else should i test it to make sure yeah this arch really works and not just in few places.
The author is seeking 5-10 beta testers for Behave, an AI agent testing/evaluation tool that goes beyond simple answer checking to catch issues like hallucination, premature conclusions, unsafe advice, and failures to self-correct.
The author built Agent DevTools, a local debugger for AI agents that inspects prompts, memory, retrieval, and tool calls, with LangChain support and a free Groq demo.
A blog post benchmarks four approaches (Semgrep, GLM 5.1 with Strix, cloud SOTA with code review skill, and local AI with a custom harness) for finding a known LFI vulnerability in PHPIPAM, finding that the local AI harness with a tailored approach outperforms the others.
We released Trustabl Agent Analyzer, an open-source tool that scans AI agent repositories to find missing validation, retries, and error handling, generating a privacy-preserving local report.
The author finds local coding agents useful for small tasks but requires constant supervision to prevent errors and scope creep, describing an iterative workflow of small fixes, tests, and manual diffs.