@OpenAI: We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle …
Summary
OpenAI reminds developers that eval results depend on API settings and harness design, recommending the Responses API, retaining reasoning, and using compaction for best performance.
Similar Articles
@OpenAI: Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benc…
OpenAI discusses the importance of evals (evaluations) for measuring and forecasting model progress, especially as benchmarks become saturated or gamed, featuring insights from Tejal Patwardhan and Andrew Mayne.
@OpenAI: As coding models improve, evals need to become harder, fairer, and more trustworthy. Better benchmarks help the field u…
OpenAI emphasizes the need for more rigorous and trustworthy evaluations for coding AI models to better measure real progress.
@levie: Almost all AI model and agent progress is downstream from evals. Open weights post training for specific domains comes …
Almost all AI model and agent progress depends on evaluations (evals). Understanding workflows and agent performance through evals will become a core enterprise competency for driving automation.
@_lamaahmad: We (@CedricWhitney, @SandhiniAgarwal, @EstherTetruas, @OliviaGWatkins2, @dgrobinson) wrote about nuances we’ve observed…
OpenAI researchers share lessons learned from working with third parties on frontier model evaluations, highlighting the importance of considering the evaluation harness and potential validity issues like reward hacking, contamination, and sandbagging.
@OpenAI: Simulated deployments also reduced evaluation awareness to levels close to real production traffic. We extended the met…
OpenAI discusses how simulated deployments reduce evaluation awareness to near real production levels, and extends the method to agentic deployments with stateful tools using tool simulators.