We somehow ended up with three different versions of the same prompt in production

Reddit r/AI_Agents News

Summary

A developer describes chasing a model regression that turned out to be three different versions of the same prompt running in production due to hotfixes and incomplete merges, leading to adoption of a prompt management tool (OrqAI) for version visibility.

Spent two days chasing what i thought was a model regression. turned out we just had three different versions of the same prompt running. the weird part was only one team was complaining. everyone else said outputs looked normal. at first i ignored that because i figured if the model had changed it'd hit everyone eventually. rewrote the prompt a few different ways, reran evals, everything still passed. which just made it more confusing because if quality had actually dropped you'd expect the evals to start failing too. finally pulled traces from the team complaining and compared them to everyone else's. different prompt. not massively different either. just missing one instruction. dug a little further and found out someone had hotfixed staging months ago for that team's data. meant to be temporary. never made it back into the repo. meanwhile another small edit had gone into prod later for something completely unrelated. so we somehow ended up with three versions. repo. prod. staging. and the only people hitting the oldest one were the team that opened the support ticket. kind of embarrassing because nobody really did anything wrong. we just didnt have one place that answered the question "what prompt is actually live right now?" after that we stopped keeping prompts in the codebase. moved prompt management into OrqAI mainly because i got tired of playing git detective every time something looked off. at least now i can see exactly which version is deployed where instead of guessing which branch or environment someone edited six months ago. curious if anyone has a good way of catching prompt drift automatically. not model drift. prompt drift. feels like this should be one of those problems everybody solved years ago, but i dont think i've actually seen a clean approach.
Original Article

Similar Articles