reliability

A former employee looks back on the severe outage at Fubao Cloud (Alibaba Cloud) on November 12, 2023, and says the outage at Chrysanthemum Cloud (Huawei Cloud) on July 26, 2026 was more than a hundred times worse, involving millions of instances locked and stopped, underscoring the absence of circuit breakers and manual confirmation mechanisms at cloud providers.

A discussion of how ready AI agents are for real-world work, covering their current abilities and the key open questions around reliability, permissions, failures, and human oversight.

A practitioner asks about real-world experiences with AI agents reporting success while business outcomes are wrong, seeking operational feedback on manual checks and failure costs.

This paper evaluates the reliability of BERT-based QA models (RoBERTa, ALBERT, DistilBERT) under Monte Carlo Dropout and input paraphrasing, finding RoBERTa more consistent and validating MCD as a reliability metric.

This paper introduces REIN, an alignment framework that reduces hallucination in large reasoning models by training them to explicitly reflect before answering and to abstain when knowledge is insufficient. Experiments show consistent gains in selective accuracy and hallucination reduction across benchmarks.

This paper investigates the 'knowing-saying gap' in language models, showing that linear probes can detect corrupted context with near-perfect accuracy yet fail to predict final answer errors, with implications for deployment monitoring and intervention strategies.

This empirical study investigates how recurrent context compression affects long-horizon agent behavior, showing that compression can weaken recent interaction influence and cause instability. The authors introduce TRACE, a verifier-guided framework that improves compression reliability and performance on AppWorld.

A robotics practitioner explains that the bottleneck in robotics deployment isn't hardware or models but knowing whether deployments will keep working, focusing on feasibility, reliability, and economics.

John Regehr describes using tis-interpreter to search for undefined behavior in SQLite, finding bugs like dangling pointer uses and uninitialized reads that other tools missed.

A user reports that DeepSeek-V4-Flash-0731 is unreliable for non-coding office tasks like summarization and meeting notes, failing at concept extraction and speaker understanding despite strong benchmark scores, while Gemma-4-31B performs better.

This paper proposes a commutation theory for label-free reliability in vision-language figure reading, showing that consistency-based methods have a computable blind spot and introducing an Equivariance-Consistency Score enhanced by cyclic relabeling.

This paper introduces RENDEQ, a generator of render-equivalence sets for scientific figures, and measures how well model agreement across semantics-preserving re-renderings tracks correctness in open-weight VLMs. It finds that agreement certifies correctness only above a threshold and that fine-tuning on self-consensus can hurt accuracy.

This paper introduces PPDL, a probabilistic language for programming LLM-based flows that enables developers to quantify and propagate uncertainty throughout applications, with experimental and case studies on theorem proving.

OrchestraBench is a new benchmark that evaluates multi-agent orchestration frameworks on failure modes, recovery, and decomposition quality, using failure-injection and cascade-radius metrics to diagnose where and why pipelines fail.

A recent reliability report reveals that 51% of ~72,000 x402 endpoints are unreachable, only 67.8% of pay-to wallets have received payments, and $45.72M has settled across the ecosystem, highlighting the gap between listed and functional endpoints.

This paper introduces Relational Response Fields (RRF), a theoretical framework for determining when black-box LLM responses can be reliably recovered under corruption, establishing identifiability conditions and minimax bounds that separate response consistency from truth.

Introduces CARGO-VL, a group-relative optimization framework for vision-language models that improves handling of conflicting image-text evidence and unsupported-answer avoidance via counterfactual consistency and risk-constrained control, along with the XMC conflict training resource.

The article argues that many AI agent tools are just 1:1 API wrappers, pushing branching logic into the LLM and causing failures. The author recommends task-shaped tools like upsert_contact that encapsulate search/create/update logic in code, pass known context, validate inputs, and return structured errors.

Asks how developers budget agent retries to distinguish transient failures from persistent ones, and what signals best decide when to stop or retry in production agents.

Google reports that Gemini Robotics ER 2 detects precise stopping moments with high accuracy (91.3%) but is far less reliable at classifying overall task progress (57.4%), raising questions about whether completion detection should be a separate safety layer.