statistical-inference

The paper introduces Conditional Poisson Sampling as a fixed-size drop-in replacement for Poisson Sampling in active statistical inference, showing it achieves no worse and often better asymptotic variance while eliminating random sample size issues, along with a released Python package.

A paper showing that running statistics over raw LLM judge scores inflates false positives, introducing PPI-based calibrated hypothesis tests (including the first PPI Wilcoxon/Mann-Whitney corrections) and the open-source Python package evalstats, which automatically selects statistically sound methods for small-sample AI evaluations.

This paper presents SCALE, a sequential cost-aware policy for hypothesis testing that uses AI judgments with selective human verification to minimize costs while controlling error rates, applicable in settings like software reliability assessment.

Introduces a conformalized split-sample framework for local model comparison, producing calibrated local best-model maps and finite-sample guarantees for declaring local superiority.

This paper introduces Boltzmann MapReduce, a partition-function reduce operation for forkable microVM sandboxes that uses inverse-variance pooling, showing theoretical consistency and robustness against malicious workers in distributed computing.

This paper investigates conditions for algorithmic machine learning (e.g., kNN, random forest) to achieve design-unbiased prediction and classification for finite populations, using probability sampling designs rather than assumed data models. It extends design-based inference from survey sampling to ML algorithms.

This paper argues for a sequential inference framework to enhance LLM trustworthiness by modeling interactions as dependent stochastic processes, ensuring validity under repeated use, and enabling online monitoring for behavioral shifts.

This paper identifies vulnerabilities in the AIVAT variance reduction technique when the heuristic value function is not fixed prior to evaluation, and shows how to propagate heuristic uncertainty to further reduce variance, achieving a 43% reduction in the number of samples needed for statistical conclusions.

This paper analyzes KV cache quantization schemes inspired by TurboQuant, using statistical inference and a new 6D error framework to evaluate quality measures like KL divergence and geometric error.

This paper introduces a statistical framework for adaptively auditing AI systems using Safe Anytime-Valid Inference (SAVI) to draw rigorous conclusions with limited data. It proposes a 'testing by betting' approach to validate model robustness while controlling type-I errors during adaptive sampling.