The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Summary
This paper analyzes the variance of FID scores across different training and sampling seeds, revealing significant reproducibility issues in image generation evaluation. It proposes a new evaluation protocol with error bars and per-cell optimal guidance tuning.
View Cached Full Text
Cached at: 06/20/26, 02:27 PM
Paper page - The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Source: https://huggingface.co/papers/2606.20536
Abstract
Analysis of FID variance across different training and sampling seeds reveals significant reproducibility issues in image generation evaluation, with retraining causing larger fluctuations than resampling, and recommends updated evaluation protocols with error bars and optimal guidance tuning.
TheFrechet Inception Distance(FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treatFIDas a random variable on a two-axis panel of training andgeneration seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed movesFID3.2x more (inInception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of theflow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding theFIDcoefficient of variation(CoV) inside a 1-2% band. (d) Per-cellclassifier-free-guidancetuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the sameFIDwith up to 2x less compute than an unlucky one. Based on these findings, we recommend a newFIDevaluation protocol: evaluate under per-cell optimal guidance, treat anyFIDgap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over severaltraining seedsrather than a singleFIDnumber.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.20536
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.20536 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.20536 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.20536 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Representation Fréchet Loss for Visual Generation
This paper introduces FD-loss, a method to optimize Fréchet Distance as a training objective for visual generation by decoupling population and batch sizes. It demonstrates that this approach improves generator quality and suggests FID may not always accurately reflect visual quality.
Position: Fairness Failure in Generative Models is an Evaluation Problem
This position paper argues that fairness failures in generative models are primarily due to evaluation problems and proposes Fairness Cards as a standardized reporting artifact to improve reproducibility and accountability.
MIND: Monge Inception Distance for Generative Models Evaluation
This paper introduces MIND (Monge Inception Distance), a new metric for evaluating generative models that is more sample-efficient, faster, and robust than the standard Fréchet Inception Distance (FID).
DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation
DMAD (Distribution Matching as Adversarial Distillation) recasts distribution matching as a classification problem, learning log-density ratios via discriminator heads to train few-step student models for fast visual generation without auxiliary score fitting. It achieves state-of-the-art results, including FID 1.04 on ImageNet-64x64 with one-step generation and VBench score 85.15 for four-step Wan2.1-T2V-14B.
The Null Is the Hard Part: Exact Tests for Memorization in Generative Models
This paper argues that memorization audits of generative models are unsound without proper null distributions, and supplies exact permutation-based nulls for whole models and single images that change the conclusions of benchmarks like MemBench and Stable Diffusion memorization tests.