@OpenAI: Simulated deployments also reduced evaluation awareness to levels close to real production traffic. We extended the met…
Summary
OpenAI discusses how simulated deployments reduce evaluation awareness to near real production levels, and extends the method to agentic deployments with stateful tools using tool simulators.
View Cached Full Text
Cached at: 06/16/26, 09:42 PM
Simulated deployments also reduced evaluation awareness to levels close to real production traffic.
We extended the method to agentic deployments with stateful tools, showing that tool simulators can produce realistic trajectories when given sufficient context and capabilities. https://t.co/8JMXApY8xe
Similar Articles
@OpenAI: Deployment Simulation works best with representative production data, which external evaluators often can’t access. In …
OpenAI explores whether public chat data (WildChat) can effectively predict real-world AI misalignments, finding that simulated deployment using public datasets provides surprisingly accurate predictions of failure rates despite data age gaps.
Predicting model behavior before release by simulating deployment
OpenAI introduces Deployment Simulation, a method to simulate future model deployments by replaying past conversations in a privacy-preserving manner with candidate models to predict real-world behavior and identify novel misalignment before release.
@OpenAI: We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle …
OpenAI reminds developers that eval results depend on API settings and harness design, recommending the Responses API, retaining reasoning, and using compaction for best performance.
Why agents that pass every eval still drift once they hit real production traffic
AI agents often drift in production after passing evals due to distribution shifts and upstream changes; continuous evaluation and real-time monitoring can mitigate this.
@OpenAI: Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benc…
OpenAI discusses the importance of evals (evaluations) for measuring and forecasting model progress, especially as benchmarks become saturated or gamed, featuring insights from Tejal Patwardhan and Andrew Mayne.