We somehow ended up with three different versions of the same prompt in production
Summary
A developer describes chasing a model regression that turned out to be three different versions of the same prompt running in production due to hotfixes and incomplete merges, leading to adoption of a prompt management tool (OrqAI) for version visibility.
Similar Articles
Anyone else lose track of which version of a prompt is the "good" one? I ended up building version control for prompts
The author describes building a version control system for AI prompts to solve the problem of tracking which prompt version is best.
We found four different versions of "the" system prompt and none of us could say which one was live
Researchers discovered four different versions of 'the' system prompt in a live AI model, highlighting confusion over which version is actually deployed.
How do you catch a prompt that silently breaks when the model version changes?
The article discusses techniques to identify prompt degradation upon model updates, such as using pinned evaluation cases and model testing matrices, and inquires about best practices for evaluating AI agents in production.
How are you letting non-engineer teammates edit prompts in production?
The author discusses challenges in allowing subject matter experts to edit prompts in production for AI agents in regulated domains, and shares a solution using a prompt editor with GitHub backend for version control.
the same-ish prompt gives you a different agent plan depending on the run and i still don't have a great answer for testing that
The author discusses challenges in testing LLM-based agent pipelines where outputs vary with similar prompts, advocating for property-based checks over exact matches to handle non-deterministic outputs.