methodology

This paper introduces the Ontology-Based Contextual AI Evaluations (OB-CAIE) methodology, which uses Domain-Specific and Evaluation Process Ontologies to bring scientific rigor, traceability, and reproducibility to generative AI testing while balancing human expertise with automation.

The post indicates that a two-stage, three-step approach is progressing to its second step soon, suggesting a development in a technical methodology.

The article describes applying intelligence analysis methodologies, specifically Continuous Probabilistic Foresight, to college football assessment using the Hinsley AI/human hybrid platform.

本文总结了来自 Hamel Husain 和 Shreya Shankar 的 AI Evals 课程的 8 个核心技能，旨在指导工程师和产品经理构建有效的 AI 评测系统，涵盖错误分析、评测器设计、校准和监控等步骤。

RBS-Attention introduces a training-free sparse-prefill method with dual-branch selection to mitigate mean dilution in long-context LLM inference, achieving up to 20.65× speedup on H100 GPUs while maintaining near-dense quality on benchmarks.

This paper re-implements 12 historical and recent algorithms to study barriers in evaluating methodological progress in health AI for electronic health records, finding that aggregate algorithm comparisons transfer across tasks and datasets, but clinically meaningful tasks exhibit greater task-algorithm interaction.

Naoki Egami, an associate professor at MIT, researches the integration of AI and statistical methods in political science to enhance the accuracy of social science studies.

The article discusses the limitations of research agents in tracking rejected sources and emphasizes the need for transparency in exclusion reasons to aid review and validation processes.

A Twitter user shared a prompt that uses Terence Tao's 'dumb questions' method to stress-test any plan by asking critical questions to uncover hidden assumptions and improve understanding.

The paper proposes belief-shift branching for tree-structured reinforcement learning to enhance step-level credit assignment, demonstrating improved performance in mathematical and coding benchmarks.

The article defends Artificial Analysis benchmarks by explaining their methodology and demonstrating with examples like Deepseek V4.1-Flash that individual evaluations offer more nuanced insights than aggregated scores.

The article proposes that agent plans should explicitly map task dependencies rather than relying on sequential lists, using a podcast workflow example to show how this makes waiting or repeating work more visible and efficient.

An attendee recaps eight AI agent workflow methodologies shared by speaker Kabi at Bilibili's Build in Public event, covering frameworks for first-principles problem solving, adversarial review, ablation testing, and multi-agent collaboration to improve prompt engineering efficiency.

This paper demonstrates that rubric text alone can predict LLM judge outputs, challenging the assumption of rubric-based evaluation and raising concerns about its reliability in automated text generation assessment.

This paper introduces I-CARE, a methodology for systematically analyzing interference in machine unlearning for text-to-image models, providing formal definitions and an open-source framework to enable reproducible study.

Releasing a methodology for evaluating model routing with interactive benchmarks that achieve Pareto-dominance over leading benchmarks, offering higher quality at lower cost.

The author describes a method to compare coding AI models by combining 11 benchmarks using percentile ranks instead of averaging raw scores, with a focus on de-duplication and weighted categories for 98 models.

This article details experiments where various AI models were tested on the political compass test, with a focus on methodology, scoring validation, and results from multiple runs to ensure reproducibility.

The Twelve-Factor App is a methodology for building scalable and maintainable software-as-a-service applications, outlining best practices for configuration, processes, and cloud deployment.

The article presents three constraints for product building: a one-pager to limit complexity, separable core tech for leverage, and a defining constraint to shape product identity, drawing from personal experience in tech development.