A practitioner describes building a Claude-based crypto news trading agent that skips almost everything, scores its refusals against a random baseline, and rewrites the prompt to force low-confidence actions for data collection.
I have been running a small agent for a few weeks that reads crypto news channels and decides whether an event is worth a paper or tiny live position. The model is Claude called through the CLI with a small set of market tools (order book depth, funding, recent price moves, past outcomes for the same coin). Most of the plumbing is boring code, and I think that part matters more than the prompt. What the code does before the model sees anything: keyword and ticker filters, liquidity checks, duplicate removal, and a cooldown per coin. Only about 1 to 2 percent of messages become candidates. After the model answers, a guard checks that the symbol it names actually belongs to the candidate, so a message cannot talk the model into trading something else. Every prompt and answer is stored in a hash chain so I can replay any decision later. The model behaves like a cautious analyst. In one week of 392 reviews it opened 8, put 4 on watch and skipped 380. That is fine by itself, but it made me ask whether the skips are right. So I score them: for every skipped candidate I measure the largest price move over the next 24 hours. A move of 10 percent or more followed about 33 percent of skips (176 of 534). For the same coins in random 24 hour windows it was about 17 percent. So skipped events do move about twice as often as a random window, in both directions, which does not mean I missed profit, but it tells me the skips are not all quiet. So I built a second agent with deliberately loose rules, running on a virtual account only. Same prompt and tools, plus an addendum saying to open on anything with a positive expected value and that a confidence of 0.30 is enough. Out of its first 23 reviews it opened none. The reasons it gave were sensible each time: reaction already priced in, weak catalyst, expected value around zero after costs. Which is exactly why it is a useless data collector, because it keeps answering the question "is this a good trade" when I asked "is there any plausible direction worth measuring". I changed the wording. Now the prompt says the account exists to collect statistics, that any plausible directional hypothesis counts (continuation, fade of an overreaction, mean reversion), that "expected value near zero" means enter with minimal confidence, and that skip is allowed only for a wrong ticker match, an order book that cannot be traded, missing price data or a duplicate of an open position. Costs are handled by requiring the take profit to clear fees and spread, not by refusing. I have not seen results from the new wording yet, so I will report back. Things I would pass on to anyone building something similar. Keep filtering in code and let the model see only candidates. Validate everything the model returns against the candidate it was given. Store prompts and answers so you can replay. Score the refusals, not just the actions, and compare them with a random baseline from the same coins, otherwise a number like "the agent skipped 380 events" says nothing. And be careful with the word "optional" in a prompt: if you tell a careful model to use judgment, it will use careful judgment. Question for people running agents that decide to act or not: how do you get a model to take low confidence actions on purpose, for data collection, without it rationalizing a skip every time? Different prompt framing, a separate agent, or forcing the action in code and letting the model only choose direction?
This article discusses the challenges of evaluating and monitoring LLM-based agents in production, covering offline evals, prompt engineering pitfalls, observability tools, review queues, labeling, clustering, topic classification, and cost-effective layering of human review, LLM-as-a-judge, and small classifiers.
The paper introduces Learning What to Skip (LW2S), a method that uses counterfactual credit assignment to optimize multi-agent LLM workflows by selectively skipping components, reducing token cost while maintaining or improving accuracy.
The author discusses challenges in testing LLM-based agent pipelines where outputs vary with similar prompts, advocating for property-based checks over exact matches to handle non-deterministic outputs.
A practitioner shares four hard guardrail patterns that enforce checks on AI agent tool calls themselves—rather than relying on prompt instructions—validated against two months of Claude Code history, showing that runtime argument validation, ordering constraints, worst-case budget reservation, and explicit refusal messages prevent agents from bypassing instructions.
The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.