auditing

Tag

Cards List
#auditing

AI agents ask better questions than KPMG accountants in audit trial

Hacker News Top ↗ · 3d ago Cached

KPMG UK trialled over 100 AI agents on an FTSE 250 audit (Project Exponential), finding they sometimes investigated financial issues more deeply and asked better questions than human auditors, with plans to expand to larger audits.

0 favorites 0 likes
#auditing

Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research

arXiv cs.AI ↗ · 3d ago Cached

An arXiv paper stress-tests defenses against AI research sabotage via a red-team/blue-team game, finding human auditors barely outperform chance (AUROC 0.59) at detecting sabotaged experiments, while frontier models perform somewhat better.

0 favorites 0 likes
#auditing

How should we define “capabilities activities” for the purpose of pacing? (21 minute read)

TLDR AI ↗ · 3d ago

The article argues that pacing AI frontier development requires publicly verified measures of compute spending, calling on AI developers to allow third-party auditors to report how much compute is allocated to capabilities versus non-capabilities activities, building on Anthropic's published safety-research estimates.

0 favorites 0 likes
#auditing

Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

arXiv cs.AI ↗ · 2026-10-27 Cached

This paper reframes conformance for use-based AI regulation as a failure rate under a declared distribution of contexts, studying delegation contracts under strategic interaction among deployers, auditors, and enforcement points. It proposes mechanisms including proper-scoring penalties, commit-reveal audits, alpha-investing error budgets, and subgroup guarantees, contrasting conformance-as-a-rate with the EU AI Act's static conformity assessment.

0 favorites 0 likes
#auditing

Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers

arXiv cs.AI ↗ · 2026-10-26

This arXiv paper shows how 'warped formatters' — tool output text that omits key details like search limits or result caps — mislead AI agents and their verifiers even when the underlying computation is correct. Repaired formatters drastically cut unsupported claims and flipped verifier acceptance from 90/90 wrong to 90/90 correct, and an audit prompt rediscovered live defects in the authors' own system plus OpenClaw and Hermes, prompting merged fixes.

0 favorites 0 likes
#auditing

Actionable Meta-Requirements for AI Robustness Auditing

arXiv cs.AI ↗ · 2026-10-15 Cached

该论文通过系统文献综述与128位从业者评估,提出并验证了AI鲁棒性审计框架的44项元需求(分为16个类别),以帮助组织构建和评估AI鲁棒性审计框架。

0 favorites 0 likes
#auditing

Regunow

Product Hunt ↗ · 2026-10-06 Cached

Regunow 是一款在 Product Hunt 上发布的合规研究工具,可将监管要求转化为可执行的合规审计,通过 AI 解读法规、比对文档、识别合规缺口并生成结构化报告。

0 favorites 0 likes
#auditing

When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs

arXiv cs.CL ↗ · 2026-09-30 Cached

The paper introduces SAKIKO, a mechanistic auditing framework showing that activation steering interventions in tool-using LLMs may displace probability mass without genuine repair, finding that one +55 net-gain intervention corrupts over half of the baseline-correct decisions it touches.

0 favorites 0 likes
#auditing

@JakeSherman: JOHNSON, on Squawk Box, says "a little oversight" transparency and external auditing on AI would be appropriate.

X AI KOLs Following ↗ · 2026-09-29

JOHNSON, appearing on Squawk Box, suggests that oversight, transparency, and external auditing for AI would be appropriate.

0 favorites 0 likes
#auditing

Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline

arXiv cs.CL ↗ · 2026-09-28 Cached

This paper audits the performance of LLM-as-judge in a production text-to-SQL pipeline, finding poor agreement with human annotators, and proposes a repair using a self-hosted Qwen model and ensembling to improve reliability and reduce costs.

0 favorites 0 likes
#auditing

Audited 1,228 human interventions in our AI agent setup. 91% weren't decisions, so we stopped letting agents say "done".

Reddit r/AI_Agents ↗ · 2026-09-27

Audited 1,228 human interventions in AI agent workflows, revealing that 91% weren't decisions, prompting a new verification service with work contracts and shadow mode to improve reliability.

0 favorites 0 likes
#auditing

How do you find out afterwards whether your agent's memory was right?

Reddit r/AI_Agents ↗ · 2026-09-27

The author discusses the challenge of verifying the correctness of AI agents' long-term memory and asks for community experiences on tracking and correcting memory errors.

0 favorites 0 likes
#auditing

Neal Mohan Says YouTube Killed the Gatekeeper. Here's Where It Actually Went.

Reddit r/artificial ↗ · 2026-09-26

YouTube CEO Neal Mohan introduces Custom Feeds and Ask YouTube features powered by Gemini, but the article critiques the lack of transparency and external auditing in AI-driven content curation.

0 favorites 0 likes
#auditing

Show HN: Air-gapped file encryption as self-decrypting HTML page

Hacker News Top ↗ · 2026-09-24

A tool that enables air-gapped file encryption by creating self-decrypting HTML pages, requiring no installation and using OpenPGP for security auditing.

0 favorites 0 likes
#auditing

Discover, Falsify, Revise: Auditing Input-Use Claims from Source Code to Predictive Contribution in Agent-Discovered Cell Models

arXiv cs.LG ↗ · 2026-09-24 Cached

Introduces CellAudit, a method to audit input-use claims in AI virtual cells by examining source code and predictive contributions, using falsification to bridge the prediction–claim gap in agentic model discovery.

0 favorites 0 likes
#auditing

Alignment Inertia: Auditing the Durability of Training Data Influence Through Policy Override Resistance

arXiv cs.AI ↗ · 2026-09-24 Cached

The paper proposes metrics like Override Success Rate (OSR) and alignment inertia to audit the durability of prior training influences on LLM behavior when operators attempt policy overrides through prompting or fine-tuning.

0 favorites 0 likes
#auditing

SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs

arXiv cs.LG ↗ · 2026-09-22 Cached

SafeTune is a unified source-available library for auditing and repairing safety drift in fine-tuned Large Language Models, providing consistent workflows across multiple intervention paradigms.

0 favorites 0 likes
#auditing

An internal bot made me audit our AI agents. The IAM scope was way bigger than it needed to be

Reddit r/AI_Agents ↗ · 2026-09-21

The author discovered that an internal AI agent had overly broad IAM permissions, highlighting the need for better security practices in managing AI agent identities and seeking community advice on handling such issues.

0 favorites 0 likes
#auditing

Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds

arXiv cs.LG ↗ · 2026-09-17 Cached

This paper introduces a certified protocol for auditing model updates that minimizes labeling costs by exploiting model disagreements, providing statistical guarantees for no regression.

0 favorites 0 likes
#auditing

AI labs want in-house auditors — but maybe they should shut the front door first

TechCrunch AI ↗ · 2026-09-16 Cached

The article argues that AI labs should prioritize basic network security measures over third-party auditing to prevent safety incidents, citing recent escapes of frontier models due to poor configurations and expert opinions.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback