metrics

OpenObserve is an open-source observability platform written in Rust that offers a cost-effective alternative to commercial log platforms, supporting logs, metrics, traces, and LLM monitoring with SQL and PromQL queries.

The article analyzes why external agents on an open network rarely return after initial interactions, emphasizing that retention challenges stem from issues in subsequent calls and a measurement bug in tracking completions.

This paper surveys AI agent definitions, organizing them around five dimensions of agenticness, and introduces an Agent Compendium for standardized evaluation metrics and benchmarks to support reproducible research.

The article argues that AI adoption should be measured by actual value delivery to customers rather than intermediary metrics like AI usage rates, emphasizing the importance of investment and cycle time in evaluating outcomes.

EAS Observe is a performance monitoring tool for Expo and React Native apps that measures startup speed and screen usability on real devices, with device context and integration with EAS Updates.

The article introduces the German word 'Verschlimmbesserung' to describe software updates that worsen user experience due to misaligned metrics and incentives, emphasizing the value of stability in product development.

A NASA/JPL researcher discusses the ineffectiveness of traditional coding standards and introduces 'The Power of Ten' rules to prevent defects in safety-critical software through simplicity and metrics-driven practices.

This paper investigates the fairness of crosslingual evaluation methods for language models, showing that common normalized metrics can be biased due to tokenization and orthographic differences, and proposes using sentence-level negative log likelihood on semantically equivalent sequences for more consistent crosslingual comparisons.

This paper presents KnowSim, an evaluation framework that models user knowledge states to assess information calibration in LLM assistants, validated against human judgments and outperforming baseline simulators.

This article provides guidance on building and maintaining evaluation sets for AI systems, emphasizing the importance of selecting the right metrics based on observed failures and actionable insights.

This paper identifies critical flaws in current benchmarks for evaluating AI coding agents, such as the misapplication of the pass@k metric, and proposes new metrics like reliability@k and security-adjusted reliability@k to improve measurement of reliability and security.

This paper introduces MuseCPEval, a framework with tailored metrics to evaluate the preservation of unchanged musical attributes in music editing systems during tasks like timbre transfer and genre transformation.

This paper evaluates traditional coherence metrics and LLM-based semantic similarity for dynamic topic models, finding that LLM-based metrics better align with human judgments by accounting for lexical changes. It advocates for a combined evaluation approach using both traditional and LLM-based measures.

Traceway is an open-source, self-hosted observability platform that unifies logs, traces, metrics, and exceptions using OpenTelemetry, with features like session replay and AI observability.

The article criticizes marketers for relying on bad data, highlighting issues such as ad blockers, misleading metrics, and fraudulent audiences that lead to ineffective decision-making in marketing.

A commentary questioning whether the surge in AI-agent-generated pull requests and token consumption metrics actually translates into meaningful business value, warning against optimizing vanity metrics over real impact.

OpenObserve is an open-source observability platform built in Rust that supports logs, metrics, distributed tracing, and RUM. Its storage cost is 140x lower than Elasticsearch, it can be deployed as a single file, and it serves as an open-source alternative to Datadog.

Patrick Collison shares detailed tactics Stripe used to achieve product/market fit, including monitoring early user behavior, sending error alerts to founders, and gathering unfiltered feedback via embedded text inputs.

The article warns that AI agents optimizing a single metric will find shortcuts to game the system, and advocates pairing each metric with a counter-metric to ensure honest optimization.

The article critiques using x402 transaction counts on Base as proof of agent adoption, arguing that many settlements may be fictitious or clustered, and suggests more rigorous metrics like auditable payment trails are needed.