Evaluate the query interface alongside the RCA model.

In Agent RCA Bench, six models investigated the same 14 incidents. Across 168 runs per interface, GreptimeDB’s SQL/PromQL interface produced 40% fewer wrong diagnoses and used ~48% fewer input tokens than our Prometheus/Loki/Tempo API wrappers.

These totals compare complete bundles of storage, query language, and tool design.

Interactive report: https://rca-bench.greptime.com Full writeup: https://greptime.com/blogs/2026-09-08-agent-rca-bench-interface-semantic-layer…

One database instead of Prometheus, Loki and Tempo — Agent RCA Bench

Source: https://rca-bench.greptime.com/

What we found

1. One database beat three backends on accuracy, tokens read, and cost.[confirmed]- Correct diagnoses: 130/168 against 105/168 on three backends. - Input tokens read: 199,823,586 against 381,843,662. - 6 of 6 models read fewer tokens on GreptimeDB; the case medians fall between 14% and 77%. - 5 of 5 priced models spent less on GreptimeDB than on three backends. gemini-3.8-flash has no complete cost estimate and is left out of the comparison. - 3 pre-specified endpoints passed Holm correction, all of them favouring GreptimeDB (claude-fable-5-1, gemini-3.8-flash, and qwen3.8-max-0902).
2. On service and dependency faults, the Semantic Graph was the most accurate interface.[descriptive]- Service and dependency faults: 112/120 with the layer, 106/120 without it, 85/120 on three backends. - No model returned fewer correct diagnoses with the layer on these faults; 4 of 6 returned more. - Focused retrieval, eligible results where the layer read back fewer rows: Discovery 35/35 and Graph-retrieval 11/12.
3. Every model read fewer tokens on GreptimeDB, and 5 of 6 were more accurate.[descriptive]- 6 of 6 models read fewer tokens on GreptimeDB. - 5 of 6 models produced more correct diagnoses. - 5 of 6 models reached a correct diagnosis in fewer tool calls.

The three interfaces compared

• Split: telemetry, prometheus-native-http-api, loki-native-http-api, tempo-native-http-api
• Raw: telemetry, ordinary-schema-metadata, read-only-sql, promql-without-metric-metadata
• Graph: telemetry, ordinary-schema-metadata, read-only-sql, promql, table-semantics, semantic-entities, semantic-relationships, usage-guidance, runtime-recovery-guidance, coverage-snapshot

Correct diagnoses by interface

• gpt-5.6-sol: Split 13/28, Raw 22/28, Graph 20/28
• deepseek-v4-pro: Split 13/28, Raw 18/28, Graph 18/28
• claude-fable-5-1: Split 27/28, Raw 26/28, Graph 23/28
• glm-5.3: Split 10/28, Raw 14/28, Graph 16/28
• gemini-3.8-flash: Split 22/28, Raw 25/28, Graph 24/28
• qwen3.8-max-0902: Split 20/28, Raw 25/28, Graph 23/28

Estimated end-to-end cost / 端到端估算成本

• Split: USD 183.95–187.39
• Raw: USD 98.37–100.50
• Graph: USD 106.88–108.97

Estimated end-to-end cost for all 6 models. The interval reflects missing cache detail.

全部 6 个模型的端到端成本估算；费用区间反映缓存明细缺失的影响。

Pre-specified questions

• storage shape: partially supported. 3 of 12 pre-specified endpoints survived Holm correction; corrected within separately frozen model cohorts: m = 8, m = 4; not pooled across all 12 endpoints
• semantic layer: not supported. No pre-specified endpoint of 12 survived Holm correction; corrected within separately frozen model cohorts: m = 8, m = 4; not pooled across all 12 endpoints
• model ranking: descriptive. Reported as a descriptive ranking; no significance is claimed

Among Graph runs, 168 of 168 runs that had it made at least one successful query_semantic_graph call, 369 calls in all. The two GreptimeDB interfaces issued 192 successful SQL JOIN calls across 92 runs, of which only 3 joined across signal kinds, in 3 runs. PromQL evaluation succeeded in 22 of 336 GreptimeDB runs and 160 of 168 three-backend runs.

696 completed runs across 6 models, 14 end-to-end cases and 8 micro cases. The full narrative report is inREPORT.md.

Measurement updated at 2026-09-08T03:50:20Z; report generated at 2026-09-08T04:51:03Z.

5 runs used a provider concurrency limit above the frozen value; per-run records are in the JSON. The measurement does not isolate the effect of this change.

Data correction: 168 Split cells were rerun and replaced; 336 Raw/Graph cells and all micro results were retained. The rerun corrected Tempo retention and repeated label names in query results. Split max_items now uses the original SQL max_rows guidance, with items as the returned unit; no aggregation advice was added to the main tool descriptions. Raw/Graph query_metrics parameter descriptions remain unchanged. Trace visibility and sample fidelity passed before and after every replacement investigation. Per-cell gates and superseded-result hashes are in the JSON. Raw/Graph retain their original PromQL encoding and ran at different times; this comparison does not isolate provider time effects or the individual corrections. A subsequent user authorization allowed temporary connection/provider failures to be retried at most three times per cell (four attempts total), with every failed attempt preserved. Incorrect diagnoses and exhausted budgets were not retried. 6 failed attempts were retried across 3 cells after SDK connection errors. Their additional observed cost was USD 28.9673; final replacement cells are already included in the investigation costs below. Requests without returned usage may have unobserved billed cost. The JSON retains the authorization, supervisor amendments, sanitized attempts and provenance hashes. Output-limited failures were retained without retry (gemini-3.8-flash / semantic-rca-transfer-014 / rep0); they remain scored failures and runner errors under the frozen runner contract. With queries and trajectories held fixed, offline reencoding with o200k_base and model-specific calibration estimates repeated-label overhead at 0.71% of retained Raw input and 0.21% of retained Graph input. These are estimates, not new provider usage measurements or effects on diagnosis accuracy. The retained overhead makes GreptimeDB appear more token-intensive; removing it widens the aggregate Split–Raw input gap by about 0.8%. Gemini requests shared a local 1,800,000-input-token/minute budget, including SDK retries. Input is estimated before sending and corrected from actual usage; elapsed time includes quota waits.

数据修正：重新执行并替换 168 个 Split 单元，保留 336 个 Raw/Graph 单元及全部 micro 结果。重跑修正了 Tempo 保留时间和查询结果中重复的标签名称。Split max_items 沿用原 SQL max_rows 的参数建议，返回单位改为 items；工具主描述未追加聚合建议。Raw/Graph 的 query_metrics 参数描述保持原样。每次替换调查前后均通过 trace 可见性和样本完整性检查，逐单元检查记录和被取代结果的哈希保存在 JSON 中。Raw/Graph 保留原始 PromQL 编码，运行时间也不同；本次比较没有单独检验 provider 随时间变化或各项修正的影响。后续用户授权允许对临时连接或 provider 错误每个单元最多重跑 3 次（总尝试最多 4 次），并保留每次失败记录；错误诊断和预算耗尽不重跑。3 个单元共发生 6 次失败，已按授权重跑，原因是 SDK 连接错误，额外观测费用为 USD 28.9673；最终替换单元的正常费用已计入下方调查成本。未返回用量的请求可能另有未观测到账单费用。JSON 保留授权、执行修订、脱敏失败尝试及来源哈希。输出受限的失败原样保留，未重跑（gemini-3.8-flash / semantic-rca-transfer-014 / rep0）；按冻结的运行器契约计为失败，并保留 runner error 标记。固定查询和轨迹，以 o200k_base 分词并按模型系数校准，重复标签名称的输入开销估算为保留 Raw 输入的 0.71%、Graph 输入的 0.21%。这些是离线估算，不是 provider 新用量测量，也不表示对诊断准确率的影响。保留该冗余使 GreptimeDB 显得更费 token，对 GreptimeDB 不利；扣除后，Split 与 Raw 的总输入差距扩大约 0.8%。Gemini 请求（含 SDK 重试）共用每分钟 1,800,000 输入 token 的本地额度，发送前估算、响应后按实际用量校正；总耗时包含等待额度的时间。