built an agent where the LLM is structurally forbidden from writing the final output. looking for feedback + people willing to break it
Summary
The author describes an AI agent designed to reproduce production Python crashes using LangGraph, featuring a unique architecture where the LLM plans actions but deterministic Python functions generate the final test code to ensure reliability.
Similar Articles
Built an agent workstation where the environment does the structural reasoning so the LLM doesn't have to
Atlarix is a desktop environment that pre-parses codebases into a node/edge graph, allowing coding agents to navigate architecture via queries instead of reading raw text, which improves performance of smaller local models.
My agent quietly corrupted its own memory graph, and I am trying something.
The author describes a problem where an LLM agent's memory graph gets corrupted by incorrect edges, and proposes using a declared ontology to validate writes and traversals. A test on 120 deliberately broken traversals caught all errors.
Giving LLMs exec() power is a security nightmare. I built a open-source AST-based guardrail to stop malicious agent execution.
Introduces ast-guard, an open-source AST-based security tool that prevents malicious code execution from LLM-generated Python strings by parsing them into an abstract syntax tree and applying node-level whitelisting and context-aware safety checks.
Constraint Decay: The Fragility of LLM Agents in Back End Code Generation
This paper studies the fragility of LLM agents in backend code generation under structural constraints, finding a phenomenon they call 'constraint decay' where performance drops significantly as constraints accumulate.
Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness
This paper introduces layer-isolated evaluation for LLM agents, decomposing a production agent into architectural layers each tested with a deterministic, no-LLM harness. It demonstrates that per-slice baseline testing localizes regressions that aggregate metrics mask, validated by controlled regression injections across multiple tenants.