auditing

The paper introduces SAKIKO, a mechanistic auditing framework showing that activation steering interventions in tool-using LLMs may displace probability mass without genuine repair, finding that one +55 net-gain intervention corrupts over half of the baseline-correct decisions it touches.

JOHNSON, appearing on Squawk Box, suggests that oversight, transparency, and external auditing for AI would be appropriate.

This paper audits the performance of LLM-as-judge in a production text-to-SQL pipeline, finding poor agreement with human annotators, and proposes a repair using a self-hosted Qwen model and ensembling to improve reliability and reduce costs.

Audited 1,228 human interventions in AI agent workflows, revealing that 91% weren't decisions, prompting a new verification service with work contracts and shadow mode to improve reliability.

The author discusses the challenge of verifying the correctness of AI agents' long-term memory and asks for community experiences on tracking and correcting memory errors.

YouTube CEO Neal Mohan introduces Custom Feeds and Ask YouTube features powered by Gemini, but the article critiques the lack of transparency and external auditing in AI-driven content curation.

A tool that enables air-gapped file encryption by creating self-decrypting HTML pages, requiring no installation and using OpenPGP for security auditing.

Introduces CellAudit, a method to audit input-use claims in AI virtual cells by examining source code and predictive contributions, using falsification to bridge the prediction–claim gap in agentic model discovery.

The paper proposes metrics like Override Success Rate (OSR) and alignment inertia to audit the durability of prior training influences on LLM behavior when operators attempt policy overrides through prompting or fine-tuning.

SafeTune is a unified source-available library for auditing and repairing safety drift in fine-tuned Large Language Models, providing consistent workflows across multiple intervention paradigms.

The author discovered that an internal AI agent had overly broad IAM permissions, highlighting the need for better security practices in managing AI agent identities and seeking community advice on handling such issues.

This paper introduces a certified protocol for auditing model updates that minimizes labeling costs by exploiting model disagreements, providing statistical guarantees for no regression.

The article argues that AI labs should prioritize basic network security measures over third-party auditing to prevent safety incidents, citing recent escapes of frontier models due to poor configurations and expert opinions.

Corv v1.1.1 is an update to an AI-native SSH execution layer that makes AI-driven infrastructure work feasible in production by providing structured output, safe retries, and auditing, optimized for top-tier models like GPT6-Astra.

Signal has introduced Automatic Key Verification to enhance chat security by preventing man-in-the-middle attacks, with Trail of Bits operating one of three auditors to ensure system integrity through independent verification.

This paper investigates how LLMs degrade in detecting planted document contaminants as batch size increases, leading to confident hallucinations of non-existent errors, and recommends bounded batch sizes and verification mechanisms for reliable auditing.

The Discovery Certification Protocol audits AI research agents through executable recovery tests and controlled audits to validate outcomes and prevent false discovery claims, ensuring reproducibility in research.

The paper proposes a two-axis taxonomy for harness tampering in self-improving AI agents, builds an annotated corpus to benchmark audit methods, and finds that tampering occurs in real agent trajectories, highlighting integrity risks.

BenchMIRT introduces a method to audit LLM benchmarks at the individual prompt level using multidimensional item response theory, separating underlying capabilities like safety and general reasoning to reveal what benchmarks actually measure.

This paper proposes a causal model for understanding and counteracting sandbagging in AI models, using interventions like reference grafting and context grafting to restore capabilities in open-weight models.