CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Summary
This paper introduces CheckMIABench, a benchmark for evaluating membership inference attacks on language models, using intermediate checkpoints to avoid distribution shifts. It evaluates attacks on Pythia and OLMo models and releases an open-source library.
View Cached Full Text
Cached at: 06/17/26, 05:38 AM
# CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models Source: [https://arxiv.org/abs/2606.17464](https://arxiv.org/abs/2606.17464) [View PDF](https://arxiv.org/pdf/2606.17464) > Abstract:Membership inference attacks \(MIAs\) are a canonical way to assess a machine learning model's privacy properties\. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques\. In particular, subtle distribution shifts between member and non\-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks\. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution\. Therefore, all open\-source models with intermediate checkpoints and public training data can be converted into MIA testbeds\. We apply our framework to a half\-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters\. To facilitate further privacy research, we open\-source a modular library for designing and implementing attacks in this setting:[this https URL](https://github.com/safr-ai-lab/pandora_llm)\. ## Submission history From: Jason Wang \[[view email](https://arxiv.org/show-email/07100efe/2606.17464)\] **\[v1\]**Tue, 16 Jun 2026 03:26:15 UTC \(100 KB\)
Similar Articles
Membership Inference Attacks on Discrete Diffusion Language Models
This paper studies membership inference attacks (MIA) on fine-tuned masked diffusion language models (MDLMs). It proposes a white-box attack using a 46-dimensional feature vector from the model's reconstruction loss at varying masking ratios, achieving high AUC scores and showing MDLMs are more vulnerable than previously thought.
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
MCBench is a new benchmark for assessing the safety of omnimodal large language models across vision, audio, and text modalities. It includes 1196 scenarios and finds current models struggle with cross-modal safety reasoning.
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Researchers introduce MM-JudgeBias, a benchmark that exposes systematic compositional biases in multimodal large language models when used as automatic judges, testing 26 SOTA MLLMs across 1,800 samples.
Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
Artifact-Bench is a comprehensive benchmark that evaluates multimodal large language models on detecting and analyzing artifacts in AI-generated videos, revealing significant limitations and misalignment with human perception.
IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
IMCBench is a new benchmark for evaluating multimodal LLMs on image-grounded medical conversations, pairing clinical images with synthetic patient profiles. Evaluations across safety, accuracy, and uncertainty show that even strong models like Claude Opus 4.6 have safety issues, highlighting the need for multi-dimensional evaluation.