Tag
Meta 分享其 Catching JiTTests 方案在生产环境大规模部署的成果：利用 LLM 自动生成会失败的即时测试，在代码合入前捕获传统测试和 LLM 代码审查难以发现的 bug，并配套发布了详细论文。
The paper introduces TestPrism, a benchmark of 300 test tasks and 3000 candidate implementations, showing that single-reference evaluation overstates test quality (Joint Success Function only reaches 28% vs 59.67%). It also proposes TestHelix, improving Joint Success by 8.67-9.00 points via heterogeneous synthesis, peer cross-validation, and recursive self-improvement.
I'm impressed by your approach—using personal git history to create a tailored benchmark for evaluating local AI models on code tasks is both practical and insightful. It's particularly interesting to hear about early results from models like Flash Next and Swift performing unexpectedly well. I'd be curious to learn more about what specific aspects surprised you in their performance.
GPT-6 Astra in Codex is being used by Perplexity AI to test code end-to-end, with developers building test harnesses and mocking API responses for integration testing.
Perplexity uses GPT-6 Astra to automate end-to-end system management, including writing communications and monitoring production, reducing the need for frequent manual checks.