Tag
This paper presents SCALE, a sequential cost-aware policy for hypothesis testing that uses AI judgments with selective human verification to minimize costs while controlling error rates, applicable in settings like software reliability assessment.
Discovery Foundation Models are proposed as general-purpose systems for enabling open-ended scientific discovery through iterative problem formulation, hypothesis testing, and evidence-based revision across dry and wet lab settings. The paper introduces a framework with capabilities like problem discovery and continual improvement, instantiated with systems like Zetema and GALILEO.
A developer built a custom AI memory system in Python that avoids hallucinations by using a rule-based architecture with hypothesis testing and consolidation phases, inspired by neuromodulation.
This paper proposes a novel framework using Weight of Evidence to evaluate feature importance methods in explainable AI, enhancing their alignment with prior knowledge and stability assessment.
This paper studies whether training logs from stochastic runs can improve the precision of model comparisons via arm-specific covariate adjustment, finding that simple adjustments can reduce uncertainty, though careful covariate selection is needed to avoid noise.
This paper analyzes sources of ranking uncertainty in LLM benchmarks like MMLU, proposing modifications to hypothesis tests for constructing rank confidence intervals, and shows that variability across subjects is substantial.
VeriEvol is a novel framework for scaling reinforcement learning in visual mathematical reasoning by ensuring reliable reward labels through a two-axis approach separating prompt difficulty from answer reliability, using evolutionary operators and hypothesis-testing verification. It achieves significant accuracy gains on a five-benchmark visual-math suite.
A researcher spent five days testing an alignment hypothesis across multiple AI systems, observing recurring themes like the value of uncertainty and collaboration over obedience, finding that ideas evolve through dialogue and criticism.
Arbor is an AI framework for autonomous scientific research that uses a coordinator, executors, and a persistent hypothesis tree to iteratively improve research outcomes across multiple domains, achieving strong results on six real research tasks.
FalsifyBench is a new evaluation framework for assessing inductive reasoning in LLMs, inspired by the Wason 2-4-6 task, where agents discover hidden semantic rules by proposing examples and receiving feedback. Evaluation of 12 LLMs shows reasoning models outperform instruction-tuned models, with negative testing (hypothesis falsification) being the key driver of success.
This paper introduces a margin-based confidence ranking method for LLM-as-a-judge systems, learning a dedicated estimator to ensure monotonicity between confidence and human-disagreement risk, with generalization guarantees and improved ranking accuracy across datasets.