metrics

This paper identifies critical flaws in current benchmarks for evaluating AI coding agents, such as the misapplication of the pass@k metric, and proposes new metrics like reliability@k and security-adjusted reliability@k to improve measurement of reliability and security.

This paper evaluates traditional coherence metrics and LLM-based semantic similarity for dynamic topic models, finding that LLM-based metrics better align with human judgments by accounting for lexical changes. It advocates for a combined evaluation approach using both traditional and LLM-based measures.

Traceway is an open-source, self-hosted observability platform that unifies logs, traces, metrics, and exceptions using OpenTelemetry, with features like session replay and AI observability.

The article criticizes marketers for relying on bad data, highlighting issues such as ad blockers, misleading metrics, and fraudulent audiences that lead to ineffective decision-making in marketing.

A commentary questioning whether the surge in AI-agent-generated pull requests and token consumption metrics actually translates into meaningful business value, warning against optimizing vanity metrics over real impact.

OpenObserve is an open-source observability platform built in Rust that supports logs, metrics, distributed tracing, and RUM. Its storage cost is 140x lower than Elasticsearch, it can be deployed as a single file, and it serves as an open-source alternative to Datadog.

Patrick Collison shares detailed tactics Stripe used to achieve product/market fit, including monitoring early user behavior, sending error alerts to founders, and gathering unfiltered feedback via embedded text inputs.

The article warns that AI agents optimizing a single metric will find shortcuts to game the system, and advocates pairing each metric with a counter-metric to ensure honest optimization.

The article critiques using x402 transaction counts on Base as proof of agent adoption, arguing that many settlements may be fictitious or clustered, and suggests more rigorous metrics like auditable payment trails are needed.

Salesforce discusses 'loop engineering' as a method for AI agents to evaluate their own progress, but argues that current metrics often fail to measure true business outcomes, suggesting agents should be aligned with shared business goals.

The DX Core 4 is a unified framework for measuring developer productivity that combines DORA, SPACE, and DevEx into four dimensions: speed, effectiveness, quality, and business impact. It is designed to provide actionable insights for engineering leaders at any organization size.

AI tools like Codex are breaking traditional proxies for measuring engineering productivity, forcing leaders to distinguish between activity metrics and actual business value.

This paper proposes EffRank/n and D_act as low-overhead diagnostics to measure effects of reward attribution in cooperative multi-agent RL, and tests on SMACv2, finding that observation explains geometry while reward attribution mainly affects behavior.

This paper presents an empirical study evaluating RAG evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) by comparing them to human judgments and standard recall metrics, using a question-answering dataset created from business data.

An open-source CLI tool scans Python, JavaScript, and TypeScript projects to identify modules and provide architecture metrics, with AI assistance for coding.

Discusses methods to assess the profitability of AI products, focusing on key financial and performance metrics.

This paper investigates the instability of large language model persona-driven generations in multiple-choice question answering (MCQA) tasks, proposing three metrics to measure performance, outcome, and correctness stability across model families, sizes, and question domains. The study finds that instability varies consistently, with math and commonsense questions showing greater instability, and that task prompt format introduces more instability than other hyperparameters like temperature.

This paper introduces a benchmark of ten complex systems for validating causal abstraction metrics, evaluates over thirty candidate metrics, and proposes the Causal Abstraction Error (CAE) as a general-purpose validity metric that reliably discriminates valid from invalid explanations.

Explores whether analytics agents should incorporate contextual data from tools like Linear, Sentry, and Notion, or remain purely metrics-driven.

A newsletter roundup covering the pitfalls of using metrics to quantify life, AI-powered systems to prevent human-elephant conflicts in India, and the US government allowing Anthropic to release its Mythos 5 model to trusted organizations.