How Anthropic enables self-service data analytics with Claude

Original article: How Anthropic enables self-service data analytics with Claude
Authors: Chen Chang, Clement Peng, Justin Leder, Johanne Jiao, Josh Cherry – all members of the Anthropic data science and data engineering team. The authors also thank Michael Segner for his contributions.
This is a complete Simplified Chinese translation of the original article.

As many data science and data engineering teams can attest, making business analytics self-service has historically been a chore.

Flattening and denormalizing data models to make them more approachable for less technical colleagues often leads to a proliferation of conflicting, overlapping views as the business scales – and does little to bridge the gap for employees who have no desire to learn SQL. Going the other route – building more walled-off standalone environments for users – often fails to address the long tail of business problems and leads to a flood of metrics and dashboards as teams silo their work.

The rise of large language models offers a new path to self-service analytics that bypasses these challenges. However, pointing Claude at a warehouse and letting the agent loose can create a false sense of precision.

The initial euphoria of being freed from ad hoc data requests gives way to worry when you realize this approach separates stakeholders from the underlying infrastructure, documentation, and expertise – the very things that previously guided them toward curated datasets.

At Anthropic, 95% of business analytics queries are automated by Claude, with an overall accuracy of about 95%. By handing off these often mechanical, repetitive tasks to Claude, our data science team can focus on more strategic work like causal modeling, forecasting, and machine learning.

After in-depth conversations with dozens of Anthropic’s top Claude Code users and seeing a variety of analytics agent design patterns, we’ve distilled best practices for other data teams working with LLMs. In this post, we’ll share tips and methods to maximize Claude’s ability to drive self-service business insights, including:

• Why analytics accuracy is a context and validation problem, not a code generation problem;
• Three failure modes that cause most errors;
• Our agent analytics stack built to address these errors;
• How we measure effectiveness; and
• A basic template we use for most skills (see appendix).

Data is not software

The generative power of LLMs is a double-edged sword: the same mechanisms that can devise creative solutions to complex problems can also hallucinate incorrect outputs. To fully understand the challenges analytics agents face, it’s helpful to contrast them with programming agents.

Programming has an open-ended solution space that rewards model creativity, while documentation and tests provide natural guardrails against hallucinations. In contrast, in analytics scenarios, there is often only one correct answer from a single correct source, and there is no deterministic way to prove its correctness.

For self-service agent-based business analytics, the complexity lies mainly in data ambiguity. The core problem ultimately boils down to whether we can map a user’s question to the specific, up-to-date entities in the data model and know the correct way to interact with them. If we can do that, the subsequent execution and SQL become trivial.

We identified three attributes of this problem that explain the vast majority of inaccurate answers:

• Concept ↔ entity ambiguity: With hundreds of feasible options in a data model (possibly from millions of fields), the agent cannot pick the correct field that best answers the user’s question. For example, when counting active users: which behaviors count as “active”? Should fraudulent users be included? What lookback window should be used?

• Data staleness: Data sources, business definitions, and schemas change constantly; assets and agent knowledge go out of date and begin returning subtly wrong answers.

• Retrieval failure: The correct information may indeed exist in the data model and be properly labeled, but given the vast search space, the agent cannot find it.

Our Agent Analytics Stack

At Anthropic, our primary defense against these three types of errors is our agent data stack. Each layer exists mainly to address one or more of these problems:

• Entity ambiguity: The data foundation and sources of truth narrow the space of trusted entities down to a single governed answer.
• Staleness: Maintenance and validation processes keep everything from decaying as the business changes.
• Retrieval failure: Skills ensure the agent reliably finds and correctly uses that answer.

In this section, we’ll explain how each layer is built.

Data Foundation

The most important part of ensuring an accurate analytics agent is a solid data foundation, which includes the data models, transformations, tests, and tables in the warehouse, as well as the metadata describing them. Standard data engineering and data quality practices like dimensional modeling, shift-left testing, freshness and completeness checks on critical pipelines still all apply (we won’t rehash them here).

What really changes is that the end users of your data model are no longer data experts (e.g., data scientists) but agents acting on behalf of users, whose data proficiency and understanding of underlying infrastructure vary widely. This shift brings a challenge: the results cannot require the user to verify their underlying correctness, simply because the end user doesn’t understand.

The data foundation layer mainly targets ambiguity: for example, if “revenue” resolves to a governed dataset rather than forty seemingly viable candidates, the problem largely disappears before the agent even starts searching. This is also the first line of defense against staleness, because the repository defining canonical models is naturally where they are forced to stay up to date.

We’ve seen several practices work particularly well:

• Create canonical datasets: By far the most common failure is the agent’s inability to map a concept (“revenue for product X”) to the one correct table, column, and metric definition, usually because multiple seemingly viable candidates exist with subtle implementation differences. The solution is fewer, more heavily governed logical models: carefully curate a small set of canonical, single-source-of-truth datasets that are clearly owned, ready to use, and easy to discover, then aggressively deprecate near-duplicate versions. Physical summary tables and caches still matter for cost and performance, but they should be mechanically derived from the canonical model, not sitting alongside it as alternatives. The goal: when an agent searches for a concept, it finds a single governed answer.

• Enforce your standards: We found that the foundation holds only when canonical models and metric definitions are enforced by tools (agents are structurally routed to them first, more on that later), by CI (changes that bypass them fail review), and by mandate (downstream teams either build on the governed layer or explain why not). Governance without enforcement quickly degrades back to the “multiple candidates” problem.

• Colocate artifacts: Our primary defense against ever-changing data models and business logic is colocation. Almost all data code (modeling, semantic layer, reference documentation, canonical dashboard definitions) lives in the same repository, with CI checks that protect integrity across layers. If a modeling change would break a downstream dashboard or invalidate a documented metric, CI flags it, and the fix ships in the same PR. (We’ll come back to this in the “Skills” section below.)

• Treat metadata as a first-class product: Programming agents perform well partly because codebases are readable: READMEs, type signatures, docstrings, etc. Your warehouse can be just as readable, but only if column and table descriptions, canonical metric definitions, granularity documentation, valid value ranges, lineage, ownership, and model layering are maintained with the same rigor as the transformations themselves. This isn’t a new insight, but good governance provides the critical context that helps agents select the right dataset.

Sources of Truth

If the data foundation is the warehouse itself, the sources of truth are the reference interfaces the agent consults to navigate that warehouse. This layer reduces concept↔entity ambiguity, turning a stakeholder’s question about “weekly active users” into a concrete, governed entity in the data model. Roughly ordered from highest to lowest trust:

• Semantic layer: Compiled metric and dimension definitions. If a question cleanly maps to a defined metric, the agent calls a function and gets a number – exactly the same number produced by every other surface in the company. Our agents are structurally instructed (via skill instructions) to prioritize using the semantic layer (see appendix). One idea we tried and dismissed: cold-starting the semantic layer by having an LLM auto-generate metric definitions from raw tables and query logs. It produced plausible definitions but encoded the very ambiguities we were trying to eliminate, and was net-negative on our evaluations compared to a smaller, human-curated layer. So our advice: use Claude to generate documentation, but keep definitions human-owned.

• Lineage and transformation graph: When the semantic layer can’t cover a question, lineage and table ranking (based on usage count) let the agent infer which upstream models feed a concept, which are deprecated, and which share the same granularity. This turns “I don’t know which metric to use” into “I know which governed model to aggregate from.” It’s also the backbone of freshness and provenance signals that we expose in online validation (discussed later).

• Query corpus: Historical SQL from dashboards, notebooks, and previous analyses. Intuitively, this should be valuable: it’s a record of every question that has been correctly answered. But in practice, we found that giving the agent raw retrieval access to thousands of historical queries improved accuracy by less than a percentage point (we’ll walk through that ablation in a later section). Unstructured retrieval can’t map a new question to the correct precedent. What works is distilling that corpus into structured, domain-organized reference documents and reusable analysis patterns described in skills. Treat query history as raw material for curation, not as a source of truth for the agent to read directly.

• Business context: This is the layer most teams skip, and the one we underestimated longest. An agent that doesn’t understand your business answers what the user asked, but misses what they really meant to ask. It won’t know that “the Q2 release” refers to a specific product, that two teams define the same term differently, or that a question is being asked because there’s a board meeting on Thursday. We ingest a company knowledge graph – indexed documents, roadmaps, decision logs, and our organizational structure – so the agent can resolve implicit references and ask better clarifying questions.

The common failure mode across all four is the same as for the data foundation: bad or stale documentation. Claude is exceptionally useful at bridging this gap (drafting column descriptions, proposing metric docs from query patterns, flagging undocumented models in CI), but curation and ownership remain human-managed.

In the next two sections, we discuss how to drive the cost of that “ownership” low enough that it actually happens.

Skills

If sources of truth are the agent’s declarative knowledge (what a metric means), then skills are its procedural knowledge: which sources to consult in what order, how to navigate ambiguous data, and what a finished analysis looks like.

In Claude Code, a skill is a markdown folder that an agent reads on demand. At Anthropic, the skills we’ve developed provide enormous value. Without skills, Claude’s ability to answer analytical questions accurately on our evaluations was no more than 21%. With skills, those numbers are consistently above 95%, often around 99% in certain areas. A skeleton for most of the skills we create is in the appendix.

Some best practices:

• Create pairwise skills: A knowledge skill acts as a thin top-level router that loads additional domain details on demand. It says: “Try the semantic layer first, but if it’s not covered, here are about 30 reference files for this domain describing relevant tables, columns, joins, and pitfalls.” This router is effectively our answer to retrieval failure: instead of letting the agent search a million-field warehouse, we narrow the space to a few dozen curated files before any query is written. The unbook skill encodes the process a senior analyst would follow: clarify the question, find sources (via the knowledge skill), run a query, then cycle the result through an adversarial review sub-agent. It also bundles a dozen reusable analysis patterns (retention curves, ratio decomposition, funnel analysis) so common requests don’t reinvent the wheel each time.

• Create qualified reference documentation: Write it for LLM retrieval from the start. Our reference documents describe tables (granularity, scope, exclusions), the mechanics of pitfalls (e.g., “exclude known free email domains, but retain custom domains like anthropic.com”), and explicit routing triggers (e.g., “if the question is about experiment lift… then don’t use it for raw event counts”), without writing prescriptive recipes that will go stale. Below is a skeleton we use for creating reference documents.

# [Domain] Table

## Quick Reference
### Business Context — [Explain what this domain means in plain language]
### Entity Granularity — [What one row represents]
### Standard Cleaning Filters — [Filters to apply to every query in this domain]

## Dimensions
- [How key dimensions are coded, and how the same concept is named differently across tables]

## Key Tables
### [table_name]
- ****Granularity****:[...] · ****Scope/Exclusions****:[...]
- ****Usage****:[When to use, when not to use, join keys, required filters]
[... one section per governed table …]

## Pitfalls
- [Error patterns a senior analyst would warn you about]

## Best Practices / Common Query Patterns
- [Default selects, standard split dimensions, mature patterns where the query itself is tricky]

## Cross-References
- [Neighboring domain docs covering adjacent questions]

• Treat skill maintenance as a first-class citizen: Skill documents describe a data model that changes daily, so without active maintenance, they become incorrect in weeks. We watched our offline accuracy drift from ~95% at launch to ~65% within a month before we treated it as an engineering problem. That means putting skill markdown files in the same repository as our transformation models, so a PR that changes a model is the same PR that updates the document describing it. A code review hook flags any report model change that doesn’t touch a skill file. Today, roughly 90% of our data model PRs include a skill change in the same diff. We also periodically prune scaffolding from skills as models improve and previously common failure modes no longer apply.

• Create a consistent, seamless experience across all surfaces: The same skill must give the same answer to a question asked in Slack, in an IDE, in a dashboard tool, or in a standalone agent session. We achieve this by ensuring there is only one canonical source (the data warehouse), and skill changes sync automatically. Once merged, skills sync to a plugin marketplace (for IDE users), to cloud storage blobs (for hosted apps that read single files), and are served directly as resources via MCP. We designed for portability from day one, avoiding hardcoded repository paths and interface-specific namespaces.

Validation

Finally, validation is the means by which you figure out which of the three failure modes is still leaking through.

Offline Evaluations

We often see a pattern: data teams build elaborate analytics environments but have no process for understanding their analytics agent’s accuracy.

One way to fill this gap is offline evaluations, which are simply “question / answer” pairs. You can think of offline evaluations like offline testing for an ML model: they don’t tell you how the online agent performs, but they do give you a rough sense – whether you have any critical blind spots.

At Anthropic we deploy two kinds of offline evaluations. Dashboard-based evaluations are auto-generated by Claude (then human-validated) and cover the most common stakeholder questions. Long-tail evaluations involve feeding Claude business context (roadmaps, table documentation) and having it generate plausible questions over the rest of that domain. We also continuously harvest every instance where a stakeholder corrects the agent in a discussion thread, because that correction is a candidate evaluation.

Other best practices include:

• Anchor reference answers so they don’t drift: An evaluation written against live data becomes obsolete the moment the underlying number changes. Pin each evaluation to a snapshot date, write it against a stable fact table, or have raters judge the agent’s query rather than the number it returns. Wire these evaluations into CI so a PR that touches a dependency re-runs the affected evaluations.

• Store results like telemetry, not like test logs: Each run lands in a warehouse table with skill version, git SHA, model ID, per-assertion pass/fail, token count, and wall-clock time. “Did that change actually help?” becomes a query, and you get a time series that catches slow regressions a single CI run misses.

• Set per-domain launch bars: A domain owner cannot announce the agent as available to their stakeholders until their slice of the evaluation set passes some threshold (we initially used ~90%). This forces reference documentation to be fixed before users see failures.

• Create the right number of evaluations: How many evaluations you should have depends on the complexity of the business domain and the underlying data model. Calibrate by tracking how well offline accuracy predicts online accuracy: we found diminishing returns beyond a few dozen per topic (e.g., “growth”), and this ceiling drops with each new generation of models.

• Offline evaluation accuracy should be ~100%; every correct answer should also hit your semantic layer (if you have one). Again, this level of accuracy does not tell you your system doesn’t produce wrong answers – it only says that, with reasonable evaluation coverage, there are no obvious gaps.

Ablation Techniques

Every structural decision about skills (which sources to expose, whether a sub-agent is worth its latency, whether to merge two skills into one) is made by pinning our offline evaluation set.

We change exactly one component at a time and compare pass rates. Each run takes about an hour and replaces a lot of argument. Methodology matters more than any single result:

• Design for null results: Our most useful ablation was a negative one. We gave the agent direct grep access to all our dashboards, transformations, and analyst notebook SQL (thousands of files). We then verified in transcripts that it actually read them before answering each time. Accuracy moved less than a percentage point in either direction. We then examined the obvious confounder: for questions it got wrong, was the answer actually in the corpus? About 80% of the time, yes. Did “answer exists” predict “now answered correctly”? No – the flip rate was flat. The information was there, the agent saw it, and it still didn’t use it. That one experiment told us our bottleneck wasn’t access to past work, but structure – mapping a question to the correct entity. That insight redirected several months of roadmap.

• Ablate at PR granularity: Every meaningful skill edit runs a before/after comparison on the relevant evaluation slice, and the delta is written into the PR description. This keeps “I improved documentation” honest and catches the surprisingly common scenario where a well-intentioned addition makes things worse.

• Keep a short list of “what didn’t work”: Two of ours: stacking more iterations of documentation refinement past a certain point (we hit three consecutive net-negative iterations: documentation got longer, not better), and swapping the adversarial reviewer to a cheaper model to reduce latency (it lost most of the accuracy gain without a real speedup). Negative results are cheap to record, and they prevent the next person from running the same experiment.

Online Validation

The final step is ensuring the actual online system performs as accurately as possible. Some measures we take:

• Adversarial review: We found that using a Claude skill to aggressively challenge all underlying assumptions behind a potential final answer improved accuracy by 6% within our evaluation set, at the cost of 32% more tokens and 72% higher latency.

• Provenance footer: Every answer includes a footer noting which source layer it came from (semantic layer > curated reference > raw table), how fresh the underlying data is, and who owns the model. It doesn’t make the answer more correct, but it helps consumers judge how much they can trust the response. A “raw table, freshness unknown” footer is a signal to double-check before forwarding, and one of our few mitigations against silent failures.

• Data quality checks: It’s possible the agent used the right field and the right method, but the data itself is wrong. Adding basic data quality checks to ensure referenced fields are current, complete, and anomaly-free is generally good hygiene.

• Passive monitoring: Two production signals we continuously track – the percentage of agent queries resolved via the semantic layer, and the percentage of answers containing corrective phrasing (“that’s the wrong table”, “you missed the fraud filter”). Both feed into a dashboard reviewed weekly alongside offline pass rates.

• Active harvesting of corrections: This is the closed-loop piece. A scheduled agent scans stakeholder channels every few hours for similar corrective phrasing, drafts a fix to the relevant reference document, and opens a PR @mentioning the domain owner. The fix path is deliberately boring – edit a markdown file, merge, auto-sync everywhere – so domain owners don’t spend much time on it. These same corrections backfill the offline evaluation set.

The failure mode none of this fully catches is the silent one: the answer is wrong but looks plausible and gets used without objection. Our mitigations are the provenance footer, explicit human sign-off on anything going to leadership, and a resident evaluation for each domain’s top KPI that runs a daily sanity check against a certified dashboard – though we don’t yet have a robust solution.

Getting Started

If you’re starting from scratch, a handful of canonical datasets, a few dozen offline evaluations, and a thin knowledge skill will capture most of the benefit; everything else in this post is what we added after those were already built.

We’ve shared many best practices, but not every one fits every data team. Align with your organization on a few principles that will influence your approach, by asking yourself:

• How important is a correct answer today, relative to tomorrow? AI models are improving rapidly. We often see companies build extensive infrastructure to compensate for model weaknesses that become irrelevant once those models improve. Knowing where the model is weak and waiting for model improvements to fill the gap is much cheaper, but may not match your company’s risk tolerance.

• How do you expect your business complexity to change over time? Some of the processes we discussed might be overkill – for example, if you produce little data, have only a few consumers of your outputs, or your data model is likely to stay simple.

• How technical is the target audience of your outputs? In other words, if you’re building this analytics system for data scientists who can identify when an answer is wrong, you may have a higher tolerance for errors than if your audience has no understanding of the underlying data model.

• How much are you willing to spend to improve accuracy? We found that certain processes like adversarial validation significantly improve accuracy, but often at the cost of higher latency and cost.

• What is your appetite for access control and internal data privacy? The more context an agent has, the stronger it tends to perform; however, broad data access conflicts with most companies’ governance posture. This determines whether you build one agent or many permission-constrained agents.

No matter which path you take, our biggest gains came from addressing each of the three failure modes: narrowing ambiguity to a single governed answer, making that answer easy to discover, and signaling when either goes stale.

Appendix

Skill File Skeleton

Below is the skeleton for our main warehouse skill: the structure of the real file, with internal details replaced by [square bracket placeholders]. It’s not meant to be copied verbatim; it’s meant to show the section types we found worth writing.

---
name: [warehouse-skill]
version: [x.y.z]
description: "If the user requests a query of [company]’s warehouse to resolve any [list of business domains] – then invoke this skill. Do not invoke for [adjacent engineering tasks] or questions unrelated to the warehouse."
---

# [Warehouse] Skill Instruction

## Description
Securely and efficiently query the single source of truth for [warehouse].
Referenced by other skills [list] for query execution guidance.

Play the role of a data analyst, providing strategic insights and data-driven recommendations, but actively seek guidance along the way.

****Out-of-scope decisions****: [product domains, etc.] → present data only, declare "the decision is with [owning team]", don’t take a stance, and don’t write code to fix them.

## Execute Query
Priority:
1. ****[Managed Connection]**** (if available): [query tool] / [schema tool]
2. ****[CLI Fallback]**** (if installed): [default project, fallback project]
3. ****Neither available** ** – ask the user to authenticate, then stop

---

# Semantic Layer (Mandatory First Step)

The governed semantic layer is the ****mandatory default path**** for every data question – the same numbers as [BI tool], joins/granularity/filters built in. Raw SQL via the reference documents below is a ****fallback****, only after the semantic layer path is proven unable to satisfy the request.

## Required Workflow
1. ****Load**** – [how to load the semantic layer in each runtime, with fallback]
2. ****Discover**** – Search metrics/dimensions by keyword; ****be sure to check segments**** (named canonical population filters – hand-crafting WHERE clauses for these is the number one error mode)
3. ****Compile + Run**** – Build spec → compile to SQL → execute
4. ****Fallback**** – Only when discovery finds no relevant metric or compilation fails → go to raw SQL via `references/*.md` (Part 3 below)

> ****Don’t give up early.**** Do not fall back to raw SQL for these reasons:
> - "[custom date filter / cohort]" → [covered by time dimension spec]
> - "[need a join]" → [metric layer already encapsulates its joins]
> - [3-4 more excuses the agent uses to skip the semantic layer, pre-rebutted]

### Date Windows and Timezones – Set Before Query
- ****as-of date vs rolling N days****: [respective conventions]
- ****"last week/month"**** → the previous *complete* calendar week/month, not rolling 7/30 days
- ****Default timezone****: [TZ]; [exceptions for certain report aggregations]
- ****Freshness lag****: [certain] tables settle late – anchor to MAX(date), not "yesterday"

---

# PART 1: Must-Know (Read First for Every Request)

## 🚀 Quick-Start Workflow
1. ****Check red flags first****: [restricted/PII requests, domain with gates, high-risk requests requiring extra validation]
2. ****Out of scope – escalate, don’t guess****: [permission requests, pipeline troubleshooting, stale dashboards, root cause assertions, product/pricing suggestions] → forward to [owning team], don’t answer
3. ****Clarify the request****: time period, cohort, what business decision it serves
4. ****Check if a dashboard already exists****: [dashboard catalog by domain]
5. ****Identify data sources****: [navigation map below; prefer governed/aggregated tables]
6. ****Execute analysis****: [required filters + adversarial review]
7. ****Deliver insights****: Show methodology, separate observations from interpretations

## 🏢 Business Context

### Entity Disambiguation (Always Clarify)
- ****"[Term A]" could mean****: [entity 1] or [entity 2] – must clarify which
- ****"[Term B]" could mean****: [entity 1] → [entity 2] → [entity 3] (one-to-many chain)
- ****"Users"****: [which identifier gives accurate counts, which inflate numbers]

### Business Terminology
- [Current product names vs deprecated aliases still appearing as frozen values in data layer – write with new names, filter with old names]
- [Key internal abbreviations]
- ****[Headline metric] calculation****: [per month / default window / leading indicator]
- ****Unfamiliar terms – search [internal docs], don’t guess****

### Data Integrity Requirements ⚠️
- ****Never****: Fabricate data/columns; make speculative assertions beyond what the data shows
- ****Always****: Use safe division; separate observations ("data shows X") from interpretations ("this suggests Y"); document limitations

---

# PART 2: How-To (Follow During Execution)

## 🔧 Technical Execution Guide
- [Managed connection tool and CLI invocation details]
- ****PII protection****: For restricted data, return SQL for the user to run themselves – do not return results

## 📊 Analytical Best Practices Guide
1. Clarify the ask before querying
2. Show your process (filters, inclusions/exclusions, freshness)
3. Clarify the denominator
4. Consider sample bias
5. Connect to business impact
6. ****Adversarial SQL Review (mandatory)**** – before giving a final answer, spawn a [sql-reviewer] sub-agent for each query; blocking findings must be fixed and re-reviewed; do not self-approve
7. ****Report with provenance**** – each answer ends with a footer:
   > ****Source:**** [semantic layer | governed table | raw exploration] · ****Confidence:**** [level] · ****Reviewed:**** [reviewer ✓, round N] · ****Freshness:**** [max date in data] · ****Ownership:**** [owning team]

---

# PART 3: Data References & Resources

## 📚 Knowledge Base Navigation
### [Domain A] → `references/[domain_a].md`
- ****Use for****: [which types of questions]
- ****Key tables****: [...]
- ****Dashboards****: `references/[domain_a]_dashboards.json`

### [Domain B] → `references/[domain_b].md`
- ****Use for****: [...]

[... one per business domain – a few dozen total …]

## ⚠️ Troubleshooting Guide

### When Information Is Missing
- [Table missing / access denied / documentation stale / unknown enum value → what to do]

### Field Naming Pitfalls
- Use `[field_x_v2]`, not `[field_x]`
- [Two similarly named tables report the same metric at different granularities – which one to use]
- [Of two seemingly viable sources, which is the canonical source for the headline metric]
- [... and a dozen more hard-won one-liners …]

This article was authored by Chen Chang, Clement Peng, Justin Leder, Johanne Jiao, and Josh Cherry, members of the Anthropic Data Science and Data Engineering team. The authors thank Michael Segner for his contributions.