We somehow ended up with three different versions of the same prompt in production

Reddit r/AI_Agents News

Summary

A developer describes chasing a model regression that turned out to be three different versions of the same prompt running in production due to hotfixes and incomplete merges, leading to adoption of a prompt management tool (OrqAI) for version visibility.

Spent two days chasing what i thought was a model regression. turned out we just had three different versions of the same prompt running. the weird part was only one team was complaining. everyone else said outputs looked normal. at first i ignored that because i figured if the model had changed it'd hit everyone eventually. rewrote the prompt a few different ways, reran evals, everything still passed. which just made it more confusing because if quality had actually dropped you'd expect the evals to start failing too. finally pulled traces from the team complaining and compared them to everyone else's. different prompt. not massively different either. just missing one instruction. dug a little further and found out someone had hotfixed staging months ago for that team's data. meant to be temporary. never made it back into the repo. meanwhile another small edit had gone into prod later for something completely unrelated. so we somehow ended up with three versions. repo. prod. staging. and the only people hitting the oldest one were the team that opened the support ticket. kind of embarrassing because nobody really did anything wrong. we just didnt have one place that answered the question "what prompt is actually live right now?" after that we stopped keeping prompts in the codebase. moved prompt management into OrqAI mainly because i got tired of playing git detective every time something looked off. at least now i can see exactly which version is deployed where instead of guessing which branch or environment someone edited six months ago. curious if anyone has a good way of catching prompt drift automatically. not model drift. prompt drift. feels like this should be one of those problems everybody solved years ago, but i dont think i've actually seen a clean approach.
Original Article

Similar Articles

How are teams handling prompt QA at scale?

Reddit r/AI_Agents

A practitioner at a company handling ~40k conversations/month describes the bottleneck of manual prompt QA and asks how teams are using automated systems to detect regressions and user frustration in production.

Same model, same prompt, 4 different agents

Reddit r/LocalLLaMA

Explores how different agent architectures yield varying outputs from the same underlying model and prompt, highlighting the impact of agent design on LLM behavior.

Same agent, same prompt, different runs. Which output do you ship?

Reddit r/AI_Agents

The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.