Tag
A 27B parameter AI scientist agent named Faraday surpasses Claude Opus 4.8 and GPT-5.5 on paper replication tasks by using a scalable reinforcement learning approach called Replica.
The author presents an off-label evaluation card for Qwen3.6-27B, covering quantization, reasoning mode effects, bias probes, and jailbreak resistance, and compares reasoning effects with Nemotron 3.5 Lightning, finding that thinking mode is net-negative for Qwen but positive for Nemotron.
This paper introduces a screen-and-confirm protocol to certify whether conditioning signals improve temporal point process models of customer return timing, finding that continuous-time decay is nearly sufficient and added conditioning is redundant or harmful.
The article discusses how Goodman's grue problem applies to AI agents in production: flawless performance on historical data doesn't guarantee correctness on future data, and more data can't resolve the fundamental ambiguity.
Curated links to recent reports on AI security incidents during model evaluations, including an OpenAI/Hugging Face incident, Anthropic's cybersecurity evals, and the UK AISI's report on unsanctioned agent behavior.
OpenAI's next major model, Astra, shows significant capability gains in agentic coding and cybersecurity, and is being treated as a 'critical' model for cybersecurity under the Preparedness Framework, with additional controls planned.
A study comparing 10 coding agent harnesses on SWE-bench Pro shows harness choice dramatically affects pass@1 and cost, revealing vendor harnesses are overfitted to large models and do not transfer to smaller ones.
A developer benchmarks 32 local models on fact extraction for agent memory, showing that F1 hides a critical failure mode: models with similar scores differ greatly in how often they invent facts on inputs that should output nothing. The article argues agent memory evaluation must include empty-output and retraction cases.
A developer describes running AI agents in a simulated restaurant world and finds that across different models and harnesses, the same three API-related mistakes dominate: guessing nonexistent endpoints, exceeding budgets, and acting on stale IDs. Shows persistent failure patterns despite fixing the system.
This paper analyzes why enterprise AI deployments stall in regulated firms, proposing a production bar that includes accuracy, reproducibility, groundedness, and detectability. It measures the human review burden across model and tool configurations, showing that confidence signals and source citation can cut review from 100% to 49% but self-verification adds latency without improving error tolerance.
Google announces GA of Agent and Model Evaluations in Gemini Enterprise Agent Platform, enabling consistent measurement and monitoring of AI agents in dev and production with pre-built metrics, adaptive rubrics, simulators, and online monitors.
This paper introduces a benchmark of 14,320 procedurally-generated prompts for evaluating LLMs on logical inference over probability operators like 'probably', 'might', and 'must'. Testing 29 models, the authors find systematic answer biases and show that only 9 exceed random chance.
Discusses how a 25% difference on ARC-AGI was due to harness setup, showing GPT-5.6 Sol scoring 38% with proper evaluation, and critiques naive benchmark reporting in the industry.
In the interview, Cognition President Russell Kaplan reviews the development of coding agent Devon, discusses model intelligence saturation and cost optimization, and introduces how the new evaluation Frontier Code measures code mergeability.
AI agents often drift in production after passing evals due to distribution shifts and upstream changes; continuous evaluation and real-time monitoring can mitigate this.
The article highlights the heated debate over whether a new AI model is actually worse than before, pointing out that the real problem lies in the lack of reliable evaluation methods.
Vercel releases DeepsecBench, a benchmark for evaluating AI models' ability to find cybersecurity vulnerabilities in application code, with findings that open-weight models are becoming more cost-effective for security scanning.
The article evaluates how different quantizations of the Qwen3.6-27B model affect output quality using KL divergence and top-1 token accuracy, as well as visual examples like SVG drawings.
This paper evaluates how small open-weight vision-language models (Qwen2-VL-2B and SmolVLM) handle realistic image degradations, finding that their verbalized confidence is unreliable while internal token probability provides much better error detection, though both fail under severe low-light conditions.
Discusses strategies to maintain the usefulness of local AI evaluations as models are frequently updated, focusing on adaptability and consistency.