CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Summary
This paper introduces CheckMIABench, a benchmark for evaluating membership inference attacks on language models, using intermediate checkpoints to avoid distribution shifts. It evaluates attacks on Pythia and OLMo models and releases an open-source library.
View Cached Full Text
Cached at: 06/17/26, 05:38 AM
# CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models Source: [https://arxiv.org/abs/2606.17464](https://arxiv.org/abs/2606.17464) [View PDF](https://arxiv.org/pdf/2606.17464) > Abstract:Membership inference attacks \(MIAs\) are a canonical way to assess a machine learning model's privacy properties\. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques\. In particular, subtle distribution shifts between member and non\-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks\. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution\. Therefore, all open\-source models with intermediate checkpoints and public training data can be converted into MIA testbeds\. We apply our framework to a half\-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters\. To facilitate further privacy research, we open\-source a modular library for designing and implementing attacks in this setting:[this https URL](https://github.com/safr-ai-lab/pandora_llm)\. ## Submission history From: Jason Wang \[[view email](https://arxiv.org/show-email/07100efe/2606.17464)\] **\[v1\]**Tue, 16 Jun 2026 03:26:15 UTC \(100 KB\)
Similar Articles
Membership Inference Attacks on Discrete Diffusion Language Models
This paper studies membership inference attacks (MIA) on fine-tuned masked diffusion language models (MDLMs). It proposes a white-box attack using a 46-dimensional feature vector from the model's reconstruction loss at varying masking ratios, achieving high AUC scores and showing MDLMs are more vulnerable than previously thought.
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
This paper introduces EvalDetectBench, an open benchmark and pipeline for measuring evaluation awareness in frontier language models, addressing biases in existing methods to improve AI safety assessments.
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
MCBench is a new benchmark for assessing the safety of omnimodal large language models across vision, audio, and text modalities. It includes 1196 scenarios and finds current models struggle with cross-modal safety reasoning.
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Introduces Inspect India Evals, an open-source framework for evaluating LLMs in Indian linguistic and cultural contexts, with six benchmarks testing multilingual ability, bias, safety, and cultural knowledge. Tests on five models show Sarvam-M 24B and Gemma 2 27B lead.
JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models
This paper proposes JUMP, a single-pass membership inference attack for fine-tuned discrete diffusion language models that exploits their any-order and parallel decodability to improve detection accuracy with fewer queries.