Converge to Surprise: Evolutionary Self-supervised Image Clustering

# Converge to Surprise: Evolutionary Self-supervised Image Clustering
Source: [https://arxiv.org/html/2607.06887](https://arxiv.org/html/2607.06887)
Canlin Zhang Independent Researcher canlingrad@gmail\.comXiuwen Liu Department of Computer Science Florida State University xliu@fsu\.edu

###### Abstract

Most self\-supervised image clustering models, actually almost all deep learning approaches, are based on gradient descent: In order to calculate the loss, every optimization step requires a clearly defined target, whether a contrastive split, a masked patch or entity, an EMA\-teacher output, a pseudo\-label, or a differentiable information\-theoretic functional\. We propose a self\-supervised framework that drops this requirement for image clustering\. Without any prior knowledge, we have to assume that each pixel is i\.i\.d\. according to the Principle of Maximum Entropy\. Taking this as our null hypothesisℋ0\\mathcal\{H\}\_\{0\}, we define a ‘surprise score’ that measures how unlikely the model’s output representation would be underℋ0\\mathcal\{H\}\_\{0\}\. Maximizing the surprise score forces the deep learning model to rejectℋ0\\mathcal\{H\}\_\{0\}— equivalently, to discover non\-random feature from data\. Also, here is our fundamental assumption: a surprise score cannot, in general, be reduced to a per\-step loss\. Hence, we propose the ‘converge\-to\-surprise’ scheme to optimize our model: an evolution\-strategy \(ES\) outer loop, which directly maximizes the surprise score without needing its gradient, paired with a periodic gradient\-descent inner loop, which uses the surprising clusters already discovered by ES as surrogate targets\. On standard image benchmarks, our framework achieves new state\-of\-the\-art results in non\-parametric self\-supervised image clustering — the strictest deep\-clustering setting, in which the number of ground\-truth classes is not given to the model\.

00footnotetext:Pre\-print\. Code available at[https://github\.com/canlinzhang/converge\-to\-surprise](https://github.com/canlinzhang/converge-to-surprise)\.## 1Introduction

Self\-supervised image clustering aims at grouping unlabeled images into distinct, semantically meaningful categories without human intervention\[[11](https://arxiv.org/html/2607.06887#bib.bib11),[25](https://arxiv.org/html/2607.06887#bib.bib25)\]\. This is achieved by unifying self\-supervised representation learning and unsupervised clustering into a cohesive pipeline\[[7](https://arxiv.org/html/2607.06887#bib.bib7),[9](https://arxiv.org/html/2607.06887#bib.bib9)\]\.

A long line of work has approached this problem with different efforts:*Patch\-arrangement*methods aim at predicting structural corruptions: solving a jigsaw puzzle on permuted patches\[[37](https://arxiv.org/html/2607.06887#bib.bib37)\], predicting which rotation has been applied to the input\[[20](https://arxiv.org/html/2607.06887#bib.bib20)\], or predicting the relative position of two cropped patches\[[14](https://arxiv.org/html/2607.06887#bib.bib14)\]\.*Contrastive learning*pulls together two augmented views of the same image, and pushes apart views from different images\[[11](https://arxiv.org/html/2607.06887#bib.bib11),[24](https://arxiv.org/html/2607.06887#bib.bib24)\]\.*Masked autoencoders*reconstruct a randomly masked portion in the pixel space\[[25](https://arxiv.org/html/2607.06887#bib.bib25)\]\.*Deep clustering*methods alternatively train the deep network by current pseudo\-labels obtained from clustering, and re\-cluster the output features\[[7](https://arxiv.org/html/2607.06887#bib.bib7),[8](https://arxiv.org/html/2607.06887#bib.bib8)\]\.*Self\-distillation*methods train a student network to match the output of a teacher, whose weights are an exponential moving average of the student’s\[[21](https://arxiv.org/html/2607.06887#bib.bib21),[9](https://arxiv.org/html/2607.06887#bib.bib9)\]\.*Latent\-space prediction*methods, in the joint\-embedding\-predictive\-architecture family, predict the embedding of one part of an image from the embedding of another\[[2](https://arxiv.org/html/2607.06887#bib.bib2)\]\.*Information\-theoretic*methods optimize differentiable surrogates for mutual information between views or for de\-correlation across feature dimensions\[[49](https://arxiv.org/html/2607.06887#bib.bib49),[3](https://arxiv.org/html/2607.06887#bib.bib3),[26](https://arxiv.org/html/2607.06887#bib.bib26),[29](https://arxiv.org/html/2607.06887#bib.bib29)\]\.

All these methods share one commonness: they are gradient\-descent methods, or*loss\-based methods*\. Each optimization step requires a clearly defined target — a contrastive positive\-negative split, a masked patch in pixel or latent space, a surrogate output from an EMA\-updated teacher, a pseudo\-label from clustering or from spatial\-arrangement analysis, or a differentiable information\-theoretic functional — to serve as the loss function\.

Then, back propagation\[[41](https://arxiv.org/html/2607.06887#bib.bib41)\]makes it extraordinarily efficient to optimize the deep network when such a per\-step loss is present\. However, like every advantage has its cost, the disadvantage of gradient\-descent approach is that: The deep network will not be able to discover representations which cannot be reduced to a per\-step loss\.

This paper works on such a case\. Our self\-supervised learning framework aims at discovering*non\-randomness*from images\. Without any prior knowledge, the Principle of Maximum Entropy\[[22](https://arxiv.org/html/2607.06887#bib.bib22)\]forces the most conservative assumption — pixels are i\.i\.d\. random noise\. We take this assumption as our null hypothesisℋ0\\mathcal\{H\}\_\{0\}, and define a*surprise score*that measures how unlikely the model’s output representation would be underℋ0\\mathcal\{H\}\_\{0\}\. The higher the surprise score, the less plausibleℋ0\\mathcal\{H\}\_\{0\}becomes\. Thus, maximizing the surprise score forces the model to reject the null hypothesisℋ0\\mathcal\{H\}\_\{0\}\. Equivalently, this makes the model extract meaningful information, or*non\-randomness*, from images, which is finally used for clustering\.

Then, we propose another fundamental assumption of this paper:*a surprise score cannot, in general, be reduced to a per\-step loss*\. Although proving this assumption is beyond our scope, detailed analysis is provided\.

Accordingly, we propose*converge\-to\-surprise*to optimize the deep network when a per\-step loss is not present: We combine an evolution\-strategy \(ES\)\[[42](https://arxiv.org/html/2607.06887#bib.bib42)\]outer loop, which maximizes the surprise score without needing its gradient, with a periodic gradient\-descent inner loop, which uses the surprising clusters already discovered by ES as surrogate targets\. Experiments on several benchmark datasets show that our scheme achieves new state\-of\-the\-art results in non\-parametric self\-supervised image clustering — the strictest deep\-clustering scenario, in which case the number of ground\-truth classes is not given to the model\.

Here are our contributions:

1. 1\.We define a*surprise score*that measures how*surprising*, or how*non\-random*, the outputs of a deep network are, under the random noise null hypothesis\. We show in Section[3\.2](https://arxiv.org/html/2607.06887#S3.SS2)that our surprise score is naturally against representation collapse\. In addition, we propose a fundamental assumption: a surprise score cannot, in general, be reduced to a per\-step loss function\.
2. 2\.We propose*converge\-to\-surprise*, a hybrid optimization scheme that combines evolution strategy with gradient\-descent training to maximize the surprise score\.
3. 3\.On benchmark datasets, models trained from scratch using our framework achieve new state\-of\-the\-art results in non\-parametric self\-supervised image clustering – the strictest deep\-clustering setting\.

The remainder of the paper is organized as follows: Section[2](https://arxiv.org/html/2607.06887#S2)surveys related work in self\-supervised image clustering\. Section[3](https://arxiv.org/html/2607.06887#S3)describes the converge\-to\-surprise framework, including our complementary masking strategy, the surprise score, and the optimization scheme\. Section[4](https://arxiv.org/html/2607.06887#S4)reports experimental results, and Section[5](https://arxiv.org/html/2607.06887#S5)concludes\. Also, we insist people to read our discussions in appendix[A](https://arxiv.org/html/2607.06887#A1)\.

## 2Related Work

We organize related self\-supervised image clustering methods by what the network outputs and how that output becomes a cluster assignment at test time\. This split mirrors the experimental protocol of Section[4](https://arxiv.org/html/2607.06887#S4)\.

### 2\.1Self\-supervised embedding learning

Most self\-supervised image clustering methods produce a dense feature embedding inℝd\\mathbb\{R\}^\{d\}, which is further clustered or linearly probed at test time\. These span*contrastive*methods \(SimCLR\[[11](https://arxiv.org/html/2607.06887#bib.bib11)\], MoCo\[[24](https://arxiv.org/html/2607.06887#bib.bib24)\]\),*self\-distillation*\(BYOL\[[21](https://arxiv.org/html/2607.06887#bib.bib21)\], DINO\[[9](https://arxiv.org/html/2607.06887#bib.bib9)\]\),*latent\-space prediction*\(I\-JEPA\[[2](https://arxiv.org/html/2607.06887#bib.bib2)\]\),*masked autoencoders*\[[25](https://arxiv.org/html/2607.06887#bib.bib25)\],*information\-theoretic*objectives \(Barlow Twins\[[49](https://arxiv.org/html/2607.06887#bib.bib49)\], VICReg\[[3](https://arxiv.org/html/2607.06887#bib.bib3)\], Deep InfoMax\[[26](https://arxiv.org/html/2607.06887#bib.bib26)\]\), and*pseudo\-label clustering*\(DeepCluster\[[7](https://arxiv.org/html/2607.06887#bib.bib7)\], SwAV\[[8](https://arxiv.org/html/2607.06887#bib.bib8)\]\)\. They differ in the training signal but agree on the output: a continuous embedding, not a hard cluster assignment\.

These approaches first apply a deep encoder that maps images to a continuous embedding space\. Then, a shallow decoder \(usually a multi\-layer perceptron\[[39](https://arxiv.org/html/2607.06887#bib.bib39)\]\) maps the embedding to the output representation\. The decoder is usually abandoned after training\. In test time, classification is made based on nearest\-neighbor evaluation\[[36](https://arxiv.org/html/2607.06887#bib.bib36)\]or fine\-tuned linear projections using the embeddings\[[7](https://arxiv.org/html/2607.06887#bib.bib7)\]\. This is the most dominant yet mildest setting: The network is not required to produce*hard*representations of the image, only an embedding that captures semantic meaning\.

### 2\.2Parametric hard deep clustering

A second family trains the network to output a cluster index directly: each input is mapped to one ofKKdiscrete classes, withKK*specified in advance*\. At evaluation, predicted clusters are matched to ground\-truth labels by the Kuhn\-Munkres \(Hungarian\) linear\-assignment algorithm\[[30](https://arxiv.org/html/2607.06887#bib.bib30)\]\. DEC\[[47](https://arxiv.org/html/2607.06887#bib.bib47)\]sharpens a soft assignment overKKcentroids, and DAC\[[10](https://arxiv.org/html/2607.06887#bib.bib10)\]recasts clustering as pairwise same/different binary classification on image pairs\. Closest to our framework is IIC\[[29](https://arxiv.org/html/2607.06887#bib.bib29)\], which maximizes softmax mutual information \(MI\) between cluster predictions on two augmented views in a single end\-to\-end loss\. However, we use cluster co\-occurrence counts from disjoint views as the ‘surprise score’ in place of softmax MI\. Also, unlike IIC, our framework*does not*need to knowKKin ahead\.

### 2\.3Non\-parametric hard deep clustering

The third family does not fixKKin advance\. This is the strictest match to the truly unsupervised regime and the setting for which we report results\. DeepDPM\[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]adapts the clustering head dynamically via split/merge operations inspired by Dirichlet\-Process Gaussian\-Mixture Models, growing or shrinking the active component count during training\. UNSEEN\[[31](https://arxiv.org/html/2607.06887#bib.bib31)\]wraps deep\-clustering backbones \(DCN\[[48](https://arxiv.org/html/2607.06887#bib.bib48)\], DEC\[[47](https://arxiv.org/html/2607.06887#bib.bib47)\], DKM\[[17](https://arxiv.org/html/2607.06887#bib.bib17)\]\) in a ‘dying clusters’ mechanism: training starts from an upper boundKmaxK\_\{\\max\}, and unused clusters atrophy\. The deep Dirichlet Process Mixture \(DPM\) model of\[[32](https://arxiv.org/html/2607.06887#bib.bib32)\]combines a flow\-based generative network with Gibbs sampling over an infinite\-component DPM prior\. Two classical non\-deep clusterers also appear as comparators on top of learned features: moVB\[[27](https://arxiv.org/html/2607.06887#bib.bib27)\], a memorized online variational\-Bayes scheme for DPM inference; DBSCAN\[[16](https://arxiv.org/html/2607.06887#bib.bib16)\], density\-based non\-parametric clustering\.

## 3Main Theory

In this section, we introduce*converge\-to\-surprise*, a framework for non\-parametric self\-supervised image clustering\. We first introduce our complementary masking strategy, based on which the surprise score is calculated\. Then, we describe our hybrid optimization scheme combining evolution strategy with gradient\-descent training\.

### 3\.1Complementary masking strategy

Suppose we have a distribution𝐏\\mathbf\{P\}producing images of shape\(H,W,C\)\(H,W,C\)\. That is,𝐗∼𝐏\\mathbf\{X\}\\sim\\mathbf\{P\}with𝐗∈ℝH,W,C\\mathbf\{X\}\\in\\mathbb\{R\}^\{H,W,C\}\. In a self\-supervised learning scenario, we have no annotated samples or prior knowledge about𝐏\\mathbf\{P\}\. According to the Principle of Maximum Entropy\[[22](https://arxiv.org/html/2607.06887#bib.bib22)\], we have to assume that each pixel\(h,w\)\(h,w\)in𝐗\\mathbf\{X\}is independent of every other; In other words, we have to assume that𝐏\\mathbf\{P\}is the maximum\-entropy distribution overℝH,W,C\\mathbb\{R\}^\{H,W,C\}, producing totally random noise\. This is our null hypothesisℋ0\\mathcal\{H\}\_\{0\}\.

Then, given a sampled image𝐗∼𝐏\\mathbf\{X\}\\sim\\mathbf\{P\}with𝐗∈ℝH,W,C\\mathbf\{X\}\\in\\mathbb\{R\}^\{H,W,C\}, we partition itsH×WH\\times Wpixel grid into two disjoint subsets using a*chessboard pattern*\. Let

ℐ=\{\(h,w\):\(h\+w\)mod2=0\},𝒥=\{\(h,w\):\(h\+w\)mod2=1\}\\begin\{split\}\\mathcal\{I\}&=\\\{\(h,w\):\(h\+w\)\\bmod 2=0\\\},\\\\ \\mathcal\{J\}&=\\\{\(h,w\):\(h\+w\)\\bmod 2=1\\\}\\end\{split\}\(1\)denote the*white*and*black*chessboard positions, respectively\. Let𝐌\(ℐ\),𝐌\(𝒥\)∈\{0,1\}H,W\\mathbf\{M\}^\{\(\\mathcal\{I\}\)\},\\mathbf\{M\}^\{\(\\mathcal\{J\}\)\}\\in\\\{0,1\\\}^\{H,W\}be the corresponding binary masks, defined by𝐌h,w\(ℐ\)=𝟏​\[\(h\+w\)mod2=0\]\\mathbf\{M\}^\{\(\\mathcal\{I\}\)\}\_\{h,w\}=\\mathbf\{1\}\\\!\\left\[\(h\+w\)\\bmod 2=0\\right\]and𝐌\(𝒥\)=𝟏−𝐌\(ℐ\)\\mathbf\{M\}^\{\(\\mathcal\{J\}\)\}=\\mathbf\{1\}\-\\mathbf\{M\}^\{\(\\mathcal\{I\}\)\}, where𝟏​\[⋅\]\\mathbf\{1\}\[\\cdot\]is the indicator function\[[35](https://arxiv.org/html/2607.06887#bib.bib35)\]\. Applying these masks pixel\-wise to𝐗\\mathbf\{X\}, we obtain two complementary masked views of the same image:

𝐗\(i\)=𝐗⊙𝐌\(ℐ\),𝐗\(j\)=𝐗⊙𝐌\(𝒥\),\\mathbf\{X\}^\{\(i\)\}=\\mathbf\{X\}\\odot\\mathbf\{M\}^\{\(\\mathcal\{I\}\)\},\\quad\\mathbf\{X\}^\{\(j\)\}=\\mathbf\{X\}\\odot\\mathbf\{M\}^\{\(\\mathcal\{J\}\)\},\(2\)where⊙\\odotdenotes element\-wise multiplication\. Theii\-side view𝐗\(i\)\\mathbf\{X\}^\{\(i\)\}retains only the pixels at white chessboard positions and zeros out the rest, while thejj\-side view𝐗\(j\)\\mathbf\{X\}^\{\(j\)\}retains only those at black positions\. By construction, every pixel of𝐗\\mathbf\{X\}appears in exactly one of the two views, and the two views share no pixel in common\. Figure[1](https://arxiv.org/html/2607.06887#S3.F1)shows our masking strategy on an MNIST image\[[13](https://arxiv.org/html/2607.06887#bib.bib13)\]\.

![Refer to caption](https://arxiv.org/html/2607.06887v1/1.png)Figure 1:Chessboard masking:ii\-side can only view pixels in white chessboard positions;jj\-side can only view pixels in black chessboard positions\. The two views share no pixel in common\.As mentioned, the null hypothesisℋ0\\mathcal\{H\}\_\{0\}assumes pixels of𝐗\\mathbf\{X\}to be independent of one another\. Since𝐗\(i\)\\mathbf\{X\}^\{\(i\)\}and𝐗\(j\)\\mathbf\{X\}^\{\(j\)\}share no common pixel, they are themselves independent tensors and therefore share no mutual information\[[4](https://arxiv.org/html/2607.06887#bib.bib4)\]\. In addition, applying data augmentation\[[38](https://arxiv.org/html/2607.06887#bib.bib38)\]*independently*on𝐗\(i\)\\mathbf\{X\}^\{\(i\)\}and𝐗\(j\)\\mathbf\{X\}^\{\(j\)\}will not break this zero\-mutual\-information statement\. We use𝐗~\(i\)\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}and𝐗~\(j\)\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}to denote the augmented views\. Therefore, we have:

ℋ0⟹I​\(𝐗~\(i\);𝐗~\(j\)\)=0\.\\mathcal\{H\}\_\{0\}\\;\\Longrightarrow\\;I\\\!\\left\(\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}\\,;\\,\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}\\right\)=0\.\(3\)Here,I​\(⋅;⋅\)I\(\\cdot\\,;\\,\\cdot\)denotes the mutual\-information functional\.

If formula[3](https://arxiv.org/html/2607.06887#S3.E3)is rejected, then the null hypothesisℋ0\\mathcal\{H\}\_\{0\}cannot be true\. This will be our approach to rejectℋ0\\mathcal\{H\}\_\{0\}\.

### 3\.2Cluster co\-occurrence as a surprise score

Again, given an image𝐗∼𝐏\\mathbf\{X\}\\sim\\mathbf\{P\}, we obtain the two complementary masked and independently augmented views𝐗~\(i\)\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}and𝐗~\(j\)\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}\. Then, a deep learning modelfθ:ℝH×W×C→ℝKf\_\{\\theta\}:\\mathbb\{R\}^\{H\\times W\\times C\}\\to\\mathbb\{R\}^\{K\}assigns each view to one ofKKcandidate clusters, according to the argmax dimension in its output logits:

y^\(i\)=argmaxk\[fθ\(𝐗~\(i\)\)\]k,y^\(j\)=argmaxk\[fθ\(𝐗~\(j\)\)\]k\.\\hat\{y\}^\{\(i\)\}=\\arg\\max\_\{k\}\\bigl\[f\_\{\\theta\}\(\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}\)\\bigr\]\_\{k\},\\quad\\hat\{y\}^\{\(j\)\}=\\arg\\max\_\{k\}\\bigl\[f\_\{\\theta\}\(\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}\)\\bigr\]\_\{k\}\.\(4\)
Bothy^\(i\)\\hat\{y\}^\{\(i\)\}andy^\(j\)\\hat\{y\}^\{\(j\)\}take values in\{0,1,…,K−1\}\\\{0,1,\\ldots,K\-1\\\}\. Because eachy^\(⋅\)\\hat\{y\}^\{\(\\cdot\)\}is a deterministic function of the corresponding view, the data\-processing inequality\[[5](https://arxiv.org/html/2607.06887#bib.bib5)\]givesI​\(y^\(i\);y^\(j\)\)≤I​\(𝐗~\(i\);𝐗~\(j\)\)=0I\(\\hat\{y\}^\{\(i\)\};\\,\\hat\{y\}^\{\(j\)\}\)\\leq I\\\!\\left\(\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}\\,;\\,\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}\\right\)=0underℋ0\\mathcal\{H\}\_\{0\}\. That says, the cluster pseudo labels fromii\-side andjj\-side are statistically independent\.

Then, givenNNimages\{𝐗1,…,𝐗N\}\\\{\\mathbf\{X\}\_\{1\},\\dots,\\mathbf\{X\}\_\{N\}\\\}sampled i\.i\.d\. from𝐏\\mathbf\{P\}, we obtain their complementary masked and independently augmented views\{𝐗~1\(i\),…,𝐗~N\(i\)\}\\\{\\tilde\{\\mathbf\{X\}\}\_\{1\}^\{\(i\)\},\\dots,\\tilde\{\\mathbf\{X\}\}\_\{N\}^\{\(i\)\}\\\}and\{𝐗~1\(j\),…,𝐗~N\(j\)\}\\\{\\tilde\{\\mathbf\{X\}\}\_\{1\}^\{\(j\)\},\\dots,\\tilde\{\\mathbf\{X\}\}\_\{N\}^\{\(j\)\}\\\}, respectively\. We implement our modelfθf\_\{\\theta\}on each view and obtain the predicted cluster, leading to two integer sequences of lengthNN:

seq\(i\)=\(y^1\(i\),…,y^N\(i\)\),seq\(j\)=\(y^1\(j\),…,y^N\(j\)\)\.\\mathrm\{seq\}^\{\(i\)\}=\\bigl\(\\hat\{y\}\_\{1\}^\{\(i\)\},\\dots,\\hat\{y\}\_\{N\}^\{\(i\)\}\\bigr\),\\quad\\mathrm\{seq\}^\{\(j\)\}=\\bigl\(\\hat\{y\}\_\{1\}^\{\(j\)\},\\dots,\\hat\{y\}\_\{N\}^\{\(j\)\}\\bigr\)\.\(5\)
![Refer to caption](https://arxiv.org/html/2607.06887v1/2_new.png)Figure 2:Overview of our pipeline\. The deep network is implemented on complementary masked and independently augmented views\. Then, argmax is obtained from each output logit vector\. Finally, surprise score sums across over\-matching clusters\.In fact, we implicitly assume that the distributions of nearby pixels in an original image𝐗\\mathbf\{X\}are almost identical, although pixels are assumed to be i\.i\.d\. underℋ0\\mathcal\{H\}\_\{0\}\. Hence, after chessboard masking,𝐗\(i\)\\mathbf\{X\}^\{\(i\)\}and𝐗\(j\)\\mathbf\{X\}^\{\(j\)\}will have identical distribution; after independent augmentation*with the same hyper\-parameter*,𝐗~\(i\)\\tilde\{\\mathbf\{X\}\}^\{\(i\)\}and𝐗~\(j\)\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}will also have identical distribution\. Thus, although statistically independent,seq\(i\)\\mathrm\{seq\}^\{\(i\)\}andseq\(j\)\\mathrm\{seq\}^\{\(j\)\}will have identical distribution as well underℋ0\\mathcal\{H\}\_\{0\}\.

For each clusterk∈\{0,1,…,K−1\}k\\in\\\{0,1,\\dots,K\-1\\\}, letnk\(i\)n\_\{k\}^\{\(i\)\}be the number of timeskkappears inseq\(i\)\\mathrm\{seq\}^\{\(i\)\}, and definenk\(j\)n\_\{k\}^\{\(j\)\}analogously\. Then, the empirical marginal probability of clustering a view \(eitherii\-side orjj\-side\) to clusterkkis

pk=\(nk\(i\)\+nk\(j\)\)/\(2​N\)\.p\_\{k\}\\;=\\;\(n\_\{k\}^\{\(i\)\}\+n\_\{k\}^\{\(j\)\}\)\\ /\\ \(2N\)\.\(6\)
Again, sinceseq\(i\)\\mathrm\{seq\}^\{\(i\)\}andseq\(j\)\\mathrm\{seq\}^\{\(j\)\}are independent underℋ0\\mathcal\{H\}\_\{0\}, the*null*probability that*both*views of one image are predicted to the same clusterkk\(i\.e\.y^n\(i\)=y^n\(j\)=k\\hat\{y\}\_\{n\}^\{\(i\)\}=\\hat\{y\}\_\{n\}^\{\(j\)\}=kat any indexnn\) isqk=pk⋅pk=pk2q\_\{k\}\\;=\\;p\_\{k\}\\cdot p\_\{k\}\\;=\\;p\_\{k\}^\{\\,2\}\. Hence, underℋ0\\mathcal\{H\}\_\{0\}, the expected number of view matching at clusterkkover the whole batch isN⋅qkN\\cdot q\_\{k\}\.

On the other hand, we denote the*observed*number of view matching at clusterkkastkt\_\{k\}, which is the*actual*count of indicesn∈\{1,…,N\}n\\in\\\{1,\\dots,N\\\}for whichy^n\(i\)=y^n\(j\)=k\\hat\{y\}\_\{n\}^\{\(i\)\}=\\hat\{y\}\_\{n\}^\{\(j\)\}=k\.

Considering clusterkkin isolation,tkt\_\{k\}is then the number of successes inNNindependent Bernoulli trials, each with success probabilityqkq\_\{k\}\[[18](https://arxiv.org/html/2607.06887#bib.bib18)\]\. That is,

tk∼Binomial​\(N,qk\)under​ℋ0\.t\_\{k\}\\;\\sim\\;\\mathrm\{Binomial\}\(N,\\,q\_\{k\}\)\\quad\\text\{under \}\\mathcal\{H\}\_\{0\}\.\(7\)TheKKoutput dimensions are of course coupled: each view is predicted to exactly one cluster\. So, the countst1,…,tKt\_\{1\},\\dots,t\_\{K\}are correlated\. But this coupling has only a very weak effect on the distribution of any individualtkt\_\{k\}\. Treating eachtkt\_\{k\}on its own as binomial is the form we will use throughout\.

Then, we apply the binomial distribution formula\[[19](https://arxiv.org/html/2607.06887#bib.bib19)\]:

ℙℋ0​\[tk​or more matches at​k\]=∑s=tkN\(Ns\)​qks​\(1−qk\)N−s\.\\mathbb\{P\}\_\{\\mathcal\{H\}\_\{0\}\}\\\!\\left\[t\_\{k\}\\text\{ or more matches at \}k\\right\]=\\sum\_\{s=t\_\{k\}\}^\{N\}\\binom\{N\}\{s\}\\,q\_\{k\}^\{\\,s\}\\,\(1\-q\_\{k\}\)^\{N\-s\}\.\(8\)The sum on the right is exact but awkward to compute\. A standard Chernoff upper bound on the binomial upper tail\[[12](https://arxiv.org/html/2607.06887#bib.bib12)\]turns it into the compact inequality

ℙℋ0​\[tk​or more matches at​k\]≤exp⁡\(−N⋅D​\(q^k∥qk\)\),\\mathbb\{P\}\_\{\\mathcal\{H\}\_\{0\}\}\\\!\\left\[t\_\{k\}\\text\{ or more matches at \}k\\right\]\\leq\\exp\\bigl\(\-N\\\!\\cdot\\\!D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)\\bigr\),\(9\)whereD​\(q^k∥qk\)D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)is the binary Kullback–Leibler divergence\[[44](https://arxiv.org/html/2607.06887#bib.bib44)\]between the*observed*view\-matching probabilityq^k=tk/N\\hat\{q\}\_\{k\}=t\_\{k\}/Nand the*null*probabilityqk=pk2q\_\{k\}=p\_\{k\}^\{\\,2\}:

D​\(q^k∥qk\)=q^k​log⁡q^kqk\+\(1−q^k\)​log⁡1−q^k1−qk\.D\\\!\\left\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\\right\)\\;=\\;\\hat\{q\}\_\{k\}\\log\\frac\{\\hat\{q\}\_\{k\}\}\{q\_\{k\}\}\\;\+\\;\(1\-\\hat\{q\}\_\{k\}\)\\log\\frac\{1\-\\hat\{q\}\_\{k\}\}\{1\-q\_\{k\}\}\.\(10\)
After taking logarithm, formula[9](https://arxiv.org/html/2607.06887#S3.E9)becomes

−log⁡\(ℙℋ0​\[tk​or more matches at​k\]\)≥N​D​\(q^k∥qk\),\-\\log\(\\mathbb\{P\}\_\{\\mathcal\{H\}\_\{0\}\}\\\!\\left\[t\_\{k\}\\text\{ or more matches at \}k\\right\]\)\\;\\geq\\;N\\,D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\),\(11\)where the left side is the amount of information, or*level of surprise*, we have by observing the view\-matching result at clusterkk\[[43](https://arxiv.org/html/2607.06887#bib.bib43)\]\. So, the largerN⋅D​\(q^k∥qk\)N\\\!\\cdot\\\!D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)is, the more surprising and unlikely the result can be underℋ0\\mathcal\{H\}\_\{0\}\. RemovingNN, we callD​\(q^k∥qk\)D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)the*surprise score at clusterkk*\.

Formula[11](https://arxiv.org/html/2607.06887#S3.E11)shows the probability of observingtkt\_\{k\}or*more*view matching at clusterkkunder the null hypothesisℋ0\\mathcal\{H\}\_\{0\}\. So, we will ignore the surprise score ifq^k<qk\\hat\{q\}\_\{k\}<q\_\{k\}for any clusterkk\(*under\-matching*\)\. Then, summing across all*over\-matching*clusters, we define the*surprise score*

𝒮​\(θ\)=∑k:q^k\>qkD​\(q^k∥qk\)\.\\mathcal\{S\}\(\\theta\)\\;=\\;\\sum\\nolimits\_\{k\\,:\\,\\hat\{q\}\_\{k\}\>q\_\{k\}\}D\\\!\\left\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\\right\)\.\(12\)
𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)measures in total how strongly our model’s two\-view labeling rejectsℋ0\\mathcal\{H\}\_\{0\}, or how surprising the view\-matching results look like\. To achieve a large𝒮​\(θ\)\\mathcal\{S\}\(\\theta\), the model needs to discover reliable features shared by both views, which are essentially the meaningful information, or*non\-randomness*, or*surprise*, in data distribution𝐏\\mathbf\{P\}\. Hence, the optimization goal of our model is to maximize𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)\. We call this framework*converge\-to\-surprise*\. Figure[2](https://arxiv.org/html/2607.06887#S3.F2)briefly illustrates the pipeline of our framework\.

If the argmax of all logit vectors collapse to one fixed dimension, then it is easy to see that𝒮​\(θ\)=0\\mathcal\{S\}\(\\theta\)=0\. In contrast, if the pixels are really random noise, we can also have𝒮​\(θ\)≈0\\mathcal\{S\}\(\\theta\)\\approx 0since the two views will be independent\. Hence,*maximizing𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)naturally prevents representation collapse*\. Also, the argmax operation makes𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)irrelative toKK, enabling*non\-parametric*clustering\.

We admit that the argmax operation makes𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)non\-differentiable toθ\\theta\. This differs from the softmax mutual information maximization in IIC\[[29](https://arxiv.org/html/2607.06887#bib.bib29)\], which is differentiable end\-to\-end, yet*parametric*\(requiringK=K=the number of classes\)\. To some extent, one can say that non\-differentiability is our trade\-off to non\-parametric\.

More fundamentally, in one optimization step, any loss functionℒ​\(fθ​\(⋅\),y\)\\mathcal\{L\}\(f\_\{\\theta\}\(\\cdot\),y\)requires a clearly defined targetyytoward which gradient descent pushesfθ​\(⋅\)f\_\{\\theta\}\(\\cdot\)\. We have no suchyyoriginally in our approach: we do not know in advance which clusterkkeach masked image should be assigned to, nor which clusterkkwill eventually carry meaningful view\-matching result\. There is no per\-step target for gradient descent to chase\.

Moreover,𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)defined in this subsection is only one way to observe the output representation of a model\. In fact, there can be numerous types of output representations \(multiple logit vectors, dense output representations from a vision transformer\[[15](https://arxiv.org/html/2607.06887#bib.bib15)\], etc\); and there can be numerous ways to observe whether a model’s output representation rejects the random noise null hypothesis\. This leads to numerous ways to define a surprise score\. Thus, we propose another fundamental assumption in this paper:*a surprise score cannot, in general, be reduced to a per\-step loss*\.

Although proving this assumption is beyond our scope, here is our intuition: We have the ultimate goal as discovering meaningful information, or*non\-randomness*, or*surprise*, from the data distribution\. But there is no guarantee that we always have a clear target in every step\. We believe this is a more general learning scenario than what gradient decent is usually applied to\. Therefore, we essentially rely on evolution strategy to deal with this scenario\.

### 3\.3Optimization

#### Evolution strategy\.

We maximize the surprise score𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)with the evolution strategy \(ES\) described in\[[42](https://arxiv.org/html/2607.06887#bib.bib42)\], treating𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)as the fitness score of a black\-box optimization problem over the model’s flat parameter vectorθ∈ℝD\\theta\\in\\mathbb\{R\}^\{D\}\.

At each ES step, we sample a population ofmmperturbed parameter vectors around the currentθ\\theta\. To be specific, we split the population intom/2m/2mirrored pairs: For each pair, we drawϵi∼𝒩​\(0,σ2​ID\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\,\\sigma^\{2\}I\_\{D\}\), which forms two models with parameterθ\+ϵi\\theta\+\\epsilon\_\{i\}andθ−ϵi\\theta\-\\epsilon\_\{i\}, respectively\. Here,𝒩​\(0,σ2​ID\)\\mathcal\{N\}\(0,\\,\\sigma^\{2\}I\_\{D\}\)is theDD\-dim Gaussian distribution with varianceσ\\sigma\[[33](https://arxiv.org/html/2607.06887#bib.bib33)\]\.

GivenNNimages\{𝐗n\}n=1N∼𝐏\\\{\\mathbf\{X\}\_\{n\}\\\}\_\{n=1\}^\{N\}\\sim\\mathbf\{P\}, we obtain the complementary masked and independently augmented views\{𝐗~n\(i\)\}n=1N\\\{\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(i\)\}\\\}\_\{n=1\}^\{N\}and\{𝐗~n\(j\)\}n=1N\\\{\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(j\)\}\\\}\_\{n=1\}^\{N\}, which are shared across all models in allm/2m/2pairs\. We implement both models in each pair to obtain𝒮​\(θ\+ϵi\)\\mathcal\{S\}\(\\theta\+\\epsilon\_\{i\}\)and𝒮​\(θ−ϵi\)\\mathcal\{S\}\(\\theta\-\\epsilon\_\{i\}\)fori=1,⋯,m/2i=1,\\cdots,m/2\. Then, we rank\-shape\[[45](https://arxiv.org/html/2607.06887#bib.bib45)\]allmmscores into centered ranksri∈\[−12,12\]r\_\{i\}\\in\[\-\\tfrac\{1\}\{2\},\\tfrac\{1\}\{2\}\], and apply a weighted update toθ\\theta:

θ←\(1−η​λ\)​θ\+ηm​σ​∑i=1mri​ϵi\.\\theta\\;\\leftarrow\\;\(1\-\\eta\\lambda\)\\,\\theta\\;\+\\;\\frac\{\\eta\}\{m\\sigma\}\\,\\sum\\nolimits\_\{i=1\}^\{m\}r\_\{i\}\\,\\epsilon\_\{i\}\.\(13\)Here,η\\etais the learning rate andλ\\lambdais the weight decay rate\.

Mirrored pairs, rank\-shaping and weight\-decay are commonly used tricks in deep evolution strategies\[[45](https://arxiv.org/html/2607.06887#bib.bib45),[42](https://arxiv.org/html/2607.06887#bib.bib42),[6](https://arxiv.org/html/2607.06887#bib.bib6)\]\. More discussions can be found in\[[42](https://arxiv.org/html/2607.06887#bib.bib42)\]\.

#### Surrogate training using surprising results\.

ES on its own is sufficient to discover clusters with surprising view\-matching results\. But it gives equal credit to every output dimensionkkcontributing to𝒮​\(θ\)\\mathcal\{S\}\(\\theta\), including tiny, noisy micro\-clusters that share similar features with larger clusters\. So, we first applyT0T\_\{0\}epochs of pure ES optimization to warm\-up, during which clusters with surprising view\-matching results will emerge\.

Starting from epochT0\+1T\_\{0\}\+1, we applyT1,aT\_\{1,a\}epochs of gradient\-descent training, followed byT1,bT\_\{1,b\}epochs of ES optimization\. They form a complete period withT1,a\+T1,bT\_\{1,a\}\+T\_\{1,b\}epochs\. We implement multiple such periods\.

At the beginning of a gradient\-descent training epoch, we identify the set of*contributing positions*among the sampledNNoriginal images\{𝐗1,…,𝐗N\}\\\{\\mathbf\{X\}\_\{1\},\\dots,\\mathbf\{X\}\_\{N\}\\\}:

𝒞=\{n∈\{1,…,N\}:y^n\(i\)=y^n\(j\)=k​and​D​\(q^k∥qk\)≥τ\},\\mathcal\{C\}\\\!=\\\!\\Bigl\\\{\\\!n\\\!\\in\\\!\\\{1,\\dots,N\\\}\\\!:\\hat\{y\}\_\{n\}^\{\(i\)\}=\\hat\{y\}\_\{n\}^\{\(j\)\}=k\\;\\text\{and\}\\;D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)\\geq\\tau\\\!\\Bigr\\\},\(14\)wherey^n\(i\)=argmaxk\[fθ\(𝐗n\(i\)\)\]k\\hat\{y\}\_\{n\}^\{\(i\)\}=\\arg\\max\_\{k\}\[f\_\{\\theta\}\(\\mathbf\{X\}\_\{n\}^\{\(i\)\}\)\]\_\{k\}is the predicted cluster of the*un\-augmented*view𝐗n\(i\)\\mathbf\{X\}\_\{n\}^\{\(i\)\}of the original image𝐗n\\mathbf\{X\}\_\{n\},y^n\(j\)\\hat\{y\}\_\{n\}^\{\(j\)\}is obtained accordingly on the un\-augmented view𝐗n\(j\)\\mathbf\{X\}\_\{n\}^\{\(j\)\},D​\(q^k∥qk\)D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)is the surprise score at clusterkkdefined in formula[10](https://arxiv.org/html/2607.06887#S3.E10), andτ\\tauis a pre\-defined threshold on the per\-cluster surprise score\. To be specific, we regard clusterkkas*surprising*ifD​\(q^k∥qk\)≥τD\\\!\\left\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\\right\)\\geq\\tau\.

That says,nnis a contributing position when both views of𝐗n\\mathbf\{X\}\_\{n\}are predicted to the same cluster, and the view\-matching result of the predicted cluster is surprising enough underℋ0\\mathcal\{H\}\_\{0\}\. We obtain𝒞\\mathcal\{C\}on un\-augmented views so that the predicted clusters are relatively deterministic\.

Then, we group the*original images*at contributing positions by their predicted cluster:

𝒞k=\{𝐗n:n∈𝒞​and​y^n\(i\)=k\}\.\\mathcal\{C\}\_\{k\}=\\\{\\mathbf\{X\}\_\{n\}:n\\in\\mathcal\{C\}\\ \\text\{and\}\\ \\hat\{y\}\_\{n\}^\{\(i\)\}=k\\\}\.
Given the per\-cluster countnk=\|𝒞k\|n\_\{k\}=\|\\mathcal\{C\}\_\{k\}\|, letMMbe the median of\{nk\}D​\(q^k∥qk\)≥τ\\\{n\_\{k\}\\\}\_\{D\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\)\\geq\\tau\}\. Then, within each𝒞k\\mathcal\{C\}\_\{k\}, we selectmin⁡\(nk,M\)\\min\(n\_\{k\},\\,M\)images uniformly without replacement: large clusters are sub\-sampled down toMMimages, small clusters contribute all their images\. Combining all selected images, we obtain a training setℬ\\mathcal\{B\}with no single dominant cluster\.

For each selected image𝐗n∈ℬ\\mathbf\{X\}\_\{n\}\\in\\mathcal\{B\}, we use its predicted clustery^n\(i\)\\hat\{y\}\_\{n\}^\{\(i\)\}as the surrogate label, denoted asy^n\\hat\{y\}\_\{n\}\. We apply chessboard masking and independent augmentation*again*, to obtain*new*𝐗~n\(i\)\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(i\)\}and𝐗~n\(j\)\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(j\)\}from𝐗n\\mathbf\{X\}\_\{n\}\. Finally, the model is trained by gradient descent via cross\-entropy loss\[[34](https://arxiv.org/html/2607.06887#bib.bib34)\]:

ℒft​\(θ\)=∑n∈ℬ\[CE​\(fθ​\(𝐗~n\(i\)\),y^n\)\+CE​\(fθ​\(𝐗~n\(j\)\),y^n\)\]\.\\mathcal\{L\}\_\{\\text\{ft\}\}\(\\theta\)=\\sum\_\{n\\in\\mathcal\{B\}\}\\Bigl\[\\mathrm\{CE\}\\bigl\(f\_\{\\theta\}\(\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(i\)\}\),\\,\\hat\{y\}\_\{n\}\\bigr\)\+\\mathrm\{CE\}\\bigl\(f\_\{\\theta\}\(\\tilde\{\\mathbf\{X\}\}\_\{n\}^\{\(j\)\}\),\\,\\hat\{y\}\_\{n\}\\bigr\)\\Bigr\]\.\(15\)
In summary, when the optimization begins, we do not have a clear target \(or loss\) in each step\. So, we just let the model evolve toward the ultimate goal \(the surprise score\)\. After enough epochs, we may see some promising output representations, which in our case are the clusters with surprising view\-matching results\. Then, we train the network using these clusters as surrogate labels to further strengthen, or reinforce, the discovered promising representations\. Algorithm[1](https://arxiv.org/html/2607.06887#alg1)summarizes the full optimization procedure\.

Algorithm 1Converge to Surprise1:images

𝒳\\mathcal\{X\}; population

mm; noise

σ\\sigma; learning rate

η\\eta; weight decay

λ\\lambda; warm\-up

T0T\_\{0\}; training periods

T1,aT\_\{1,a\}and

T1,bT\_\{1,b\}; threshold

τ\\tau\.

2:initialize

θ\\theta
3:forepoch

e=1,2,…e=1,2,\\dotsdo

4:sample

NNimages; build complementary masked views and augment each independently

→𝐗~\(i\),𝐗~\(j\)\\rightarrow\\tilde\{\\mathbf\{X\}\}^\{\(i\)\},\\tilde\{\\mathbf\{X\}\}^\{\(j\)\}
5:// ES outer step \(explore\)

6:for

p=1p=1to

m/2m/2do

7:draw

ϵp∼𝒩​\(0,σ2​I\)\\epsilon\_\{p\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\)
8:score

θ±ϵp\\theta\\pm\\epsilon\_\{p\}using

𝒮​\(⋅\)\\mathcal\{S\}\(\\cdot\)\(Eq\.[12](https://arxiv.org/html/2607.06887#S3.E12)\)

9:endfor

10:shape the

mmscores into centered ranks

rpr\_\{p\}
11:

θ←\(1−η​λ\)​θ\+ηm​σ​∑prp​ϵp\\theta\\leftarrow\(1\-\\eta\\lambda\)\\,\\theta\+\\frac\{\\eta\}\{m\\sigma\}\\sum\_\{p\}r\_\{p\}\\,\\epsilon\_\{p\}⊳\\trianglerightEq\.[13](https://arxiv.org/html/2607.06887#S3.E13)

12:// gradient\-descent training \(consolidate\)

13:if

e\>T0e\>T\_\{0\}and

e%T1,b=0e\\mathbin\{\\%\}T\_\{1,b\}=0then

14:on*un\-augmented*views, take argmax labels and keep contributing positions

𝒞\\mathcal\{C\}\(Eq\.[14](https://arxiv.org/html/2607.06887#S3.E14)\)

15:median\-balance

𝒞\\mathcal\{C\}across clusters into

ℬ\\mathcal\{B\}
16:forepoch

e=1,2,…,T1,ae=1,2,\\dots,T\_\{1,a\}do

17:re\-augment both views; update

θ\\thetaby minimizing

ℒft​\(θ\)\\mathcal\{L\}\_\{\\text\{ft\}\}\(\\theta\)\(cross\-entropy to the surrogate labels\)

18:endfor

19:endif

20:endfor

21:return

θ\\theta

The gradient\-descent training uses surprising view\-matching results*already discovered by evolution strategy*as surrogate targets\. One cannot claim that the surprise score𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)can be reduced to a per\-step loss, because of this\.

## 4Experiments

We evaluate our converge\-to\-surprise framework in non\-parametric self\-supervised image clustering, the strictest image clustering setting in which the number of ground\-truth classes is*not*given to the model\. We report results on three standard image benchmarks and compare against the leading non\-parametric deep\-clustering methods, DeepDPM\[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]and UNSEEN\[[31](https://arxiv.org/html/2607.06887#bib.bib31)\]\.

### 4\.1Experimental setup

#### Network\.

Our modelfθf\_\{\\theta\}is a ResNet\-9\[[23](https://arxiv.org/html/2607.06887#bib.bib23)\]: two convolutional stems\[[1](https://arxiv.org/html/2607.06887#bib.bib1)\], two residual blocks\[[23](https://arxiv.org/html/2607.06887#bib.bib23)\], an adaptive average\-pooling layer, and a single linear head that returns a logit vector with dimensionK=64K=64\. The cluster assignment of a view is thearg⁡max\\arg\\maxover these logits\. The output isℓ2\\ell\_\{2\}\-normalized before the argmax\. We setK=64K=64, far above the ten ground\-truth classes of every dataset\. Since the model is never told the true number of classes,KKonly acts as an upper bound\. The number of*active*clusters is discovered during training\. The same architecture is used for all datasets, with a single input channel for an image view\.

#### Datasets\.

We use three handwritten\-digit and fashion\-item benchmarks, all with ten ground\-truth classes\.MNIST\[[13](https://arxiv.org/html/2607.06887#bib.bib13)\]contains 70k images of handwritten digits at28×2828\\times 28resolution \(10k for testing\)\.Fashion\-MNIST\[[46](https://arxiv.org/html/2607.06887#bib.bib46)\]matches MNIST in size and resolution but depicts ten clothing categories, and is substantially harder because several classes \(e\.g\. pullover/coat/shirt\) differ only in fine texture\.USPS\[[28](https://arxiv.org/html/2607.06887#bib.bib28)\]contains7,2917\{,\}291training and2,0072\{,\}007test handwritten digits at a native16×1616\\times 16resolution, and is notably class\-imbalanced\.

Table 1:Non\-parametric clustering results \(%, mean±\\pmstd\) on MNIST, Fashion\-MNIST, and USPS\. For each metric \(NMI, ARI, ACC\) higher is better, and the best mean per column is inbold\. Baselines are the values reported byRonen et al\. \[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]\(DBSCAN, moVB, DPM Sampler, DeepDPM; on pretrained\-autoencoder features\) andLeiber et al\. \[[31](https://arxiv.org/html/2607.06887#bib.bib31)\]\(UNSEEN variants; on an autoencoder backbone\)\. Our results are over five independent runs from fresh initializations, clustering directly from raw pixels\.Table 2:Inferred number of clustersK^\\hat\{K\}\(mean±\\pmstd\) on the three benchmarks; the ground\-truth value isK⋆=10K^\{\\star\}=10\. Baseline values are as reported byRonen et al\. \[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]\. With no prior knowledge, our discovered number of active clusters are close to the ground truth\.
#### Data augmentation\.

After the chessboard masking of Section[3\.1](https://arxiv.org/html/2607.06887#S3.SS1), the two complementary views of each image are augmented*independently*, preserving the zero\-mutual\-information property between two views underℋ0\\mathcal\{H\}\_\{0\}\. For MNIST and Fashion\-MNIST, augmentations include random rotation, random cropping, and brightness/contrast jitter\. Fashion\-MNIST additionally applies a random horizontal flip\. For USPS, we further introduce an anisotropic augmentation: We pad one randomly chosen axis \(either height or length\) with a black border\. Then, we resize back the padded image to introduce vertical or horizontal deformation\. This is motivated by the aspect\-ratio variation of the dominant digit\-0 class in USPS\. To accommodate this anisotropic augmentation, other augmentations are kept milder on the USPS dataset\. Full per\-dataset augmentation parameters are listed in Table[4](https://arxiv.org/html/2607.06887#A2.T4)in Appendix[B](https://arxiv.org/html/2607.06887#A2)\.

#### Optimization and protocol\.

We optimize𝒮​\(θ\)\\mathcal\{S\}\(\\theta\)with the ES outer loop under a population sizem=32m=32, interleaved with periodic gradient\-descent training using the discovered surprising clusters\. When identifying contributing positions according to formula[14](https://arxiv.org/html/2607.06887#S3.E14), we chooseτ=0\.005\\tau=0\.005\. For each dataset, we run*five*independent experiments from scratch and report the mean±\\pmstandard deviation\. At test time,*only*view\-i \(without augmentation\) from chessboard masking on each test image is passed through the network, and assigned to itsarg⁡max\\arg\\maxcluster\. The clustering results are then used in evaluations\.

#### Training schedule\.

On MNIST and Fashion\-MNIST, we use a*two\-stage*training schedule:20002000epochs of pure ES, followed by epochs20002000–30003000in which44gradient\-descent training epochs are run every2525ES epochs\. On USPS we use a*three\-stage*schedule:40004000epochs of pure ES; then a*weak*stage over epochs40004000–80008000, with22gradient\-descent training epochs every500500ES epochs; and finally a*strong*stage over epochs80008000–90009000, with44gradient\-descent training epochs every2525ES epochs \(identical to the other datasets\)\. USPS is run for many more epochs because it has far fewer training images \(7,2917\{,\}291vs\.70,00070\{,\}000\), so the model needs more epochs to evolve\.

#### Batch size\.

The surprise score is calculated over a batch ofNNimages at each ES step\. We useN=3000N=3000for MNIST/Fashion\-MNIST, andN=3650N=3650for USPS\. The latter is chosen so that the7,2917\{,\}291USPS training images are split into two nearly equal\-sized steps per epoch \(36503650and36413641\)\. This matters because each ES updating direction is estimated from the statistics of its batch alone\. Then, a last step with only a few images gives unreliable statistics, steering the evolution into a poorly\-estimated direction\. So, we chooseNNcarefully to avoid this\.

#### Metrics and baselines\.

Following the non\-parametric clustering protocol, we report three standard metrics: clustering accuracy \(ACC\) under the Kuhn–Munkres assignment, Normalized Mutual Information \(NMI\), and the Adjusted Rand Index \(ARI\)\. For all three, higher is better, with more details introduced in\[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]\. We additionally report the inferred number of active clustersK^\\hat\{K\}\(ground truthK⋆=10K^\{\\star\}=10\)\. As for baselines, we use the scores reported by DeepDPM\[[40](https://arxiv.org/html/2607.06887#bib.bib40)\]– including the performances of DeepDPM itself, performances of the classical non\-parametric clusterers DBSCAN\[[16](https://arxiv.org/html/2607.06887#bib.bib16)\], performances of a memorized variational DPM model called moVB\[[27](https://arxiv.org/html/2607.06887#bib.bib27)\], and performances of a DPM sampler\. According to DeepDPM\[[40](https://arxiv.org/html/2607.06887#bib.bib40)\],*all*these models are based on*pre\-trained*autoencoder features\. Also, performances of the three UNSEEN variants\[[31](https://arxiv.org/html/2607.06887#bib.bib31)\]\(UNSEEN\+DCN\[[48](https://arxiv.org/html/2607.06887#bib.bib48)\], UNSEEN\+DEC\[[47](https://arxiv.org/html/2607.06887#bib.bib47)\], and UNSEEN\+DKM\[[17](https://arxiv.org/html/2607.06887#bib.bib17)\]\), each of which is also based on a*pre\-trained*autoencoder backbone, are used as baselines\. In contrast to all these, our method is*trained from scratch*, with no pre\-trained or separately\-learned feature extractor\.

#### Computation and hardware\.

All our experiments are conducted using one NVIDIA H200 GPU\. We apply parallel computing within a single GPU: Given the ES population sizem=32m=32, we duplicate the network parameters for 16 times, perturb each of them individually, and implement them on 16 duplicated batches of image views\. So, one ES step requires32/16=232/16=2iterations, and in each iteration there are16/2=816/2=8mirrored pairs\. It takes around 5 hours to fully optimize a ResNet\-9 model on MNIST or FashionMNIST, and around 2 hours on USPS\.

Table 3:Ablation studies \(%, mean±\\pmstd over five runs\)\. Each row removes one component from the full model\. The anisotropic zoom\-out augmentation is USPS\-specific, so it is not applicable \(‘–’\) to MNIST and Fashion\-MNIST\.

### 4\.2Results

Table[1](https://arxiv.org/html/2607.06887#S4.T1)reports the comparison\. Our framework achieves state\-of\-the\-art performance on MNIST and USPS across all three metrics\. On Fashion\-MNIST, it obtains the best ACC and ARI scores, while remaining competitive on NMI\. The largest improvement occurs on USPS, where our model improves ACC from DeepDPM’s89%89\\%to94\.9%94\.9\\%, and lift NMI/ARI by roughly two and four points, respectively\. On Fashion\-MNIST, we reach64\.1%64\.1\\%ACC and52\.9%52\.9\\%ARI, both above the strongest baselines, while our NMI of65%65\\%is second to DeepDPM’s68%68\\%\. In addition, our model is trained*from scratch*, rather than from pre\-trained features\.

Crucially, these results are obtained*without*telling the model the number of classes\. As shown in Table[2](https://arxiv.org/html/2607.06887#S4.T2), the number of active clusters our method discovers stays close to the ground\-truth value of ten: exactly10\.010\.0on MNIST,10\.4±0\.510\.4\\pm 0\.5on USPS, and11\.6±0\.511\.6\\pm 0\.5on Fashion\-MNIST, comparable to the values inferred by DeepDPM \(10\.010\.0,9\.29\.2,10\.210\.2\)\. The remainingK−K^K\-\\hat\{K\}output dimensions die out during optimization and carry no test samples\. Appendix[D](https://arxiv.org/html/2607.06887#A4)shows how the number of active clusters changes across epochs\.

In summary, simply by maximizing the surprise score – with no per\-step loss, no pre\-trained model, and no prior knowledge – a deep network can naturally discover semantically meaningful*hard*representation, or*tokenized*representation, from raw images\. Moreover, the discovered representation is independent of the output tensor shape \(KKin our case\)\. Visualizations of the clustered images are exhibited in appendix[C](https://arxiv.org/html/2607.06887#A3)\.

### 4\.3Ablation studies

We ablate the three components that are most responsible for the above results: the independent two\-view augmentation, the inner\-loop gradient\-descent training, and the anisotropic \(zoom\-out\) augmentation used only on USPS\. Table[3](https://arxiv.org/html/2607.06887#S4.T3)reports the effect of removing each one\.

#### Augmentation is essential\.

If we skip augmentation and run ES directly on the complementary masked views, the model maximizes the surprise score by latching onto low\-level nuisance features that happen to be shared by the two views of an image – stroke thickness, brightness, and digit/item shape – rather than the semantic class\. Augmenting the two views*independently*breaks this shortcut: the two views of one image now differ in brightness, contrast, scale \(cropping\), and orientation \(rotation\)\. To assign both views to the same cluster, the network now must rely on augmentation\-invariant features, which are more likely the true underlying topology of the digits or items\.

#### Inner\-loop gradient\-descent training is important\.

After several thousand epochs of pure ES, the model consistently produces more clusters than the number of ground truth classes\. The gradient\-descent training then consolidates the clusters: when two clusters hold similar image views, the larger cluster provides the stronger pseudo\-label signal, which pulls across the image views from the smaller cluster, and finally absorbs the smaller cluster entirely\. Moreover, augmenting both views*again*before each gradient\-descent training step forces the network to map differently\-augmented views to the same cluster, which reinforces the surprising clusters discovered by ES\. In practice, the two\-view agreement rate rises from8080–85%85\\%after pure ES to nearly99%99\\%after a few training periods\.

However, we cannot say that gradient\-descent training is all we need: Without evolution epochs, there is no discovered surprising cluster at all for gradient\-descent training\.

#### Anisotropic augmentation is necessary for USPS\.

The USPS dataset is class\-imbalanced, with far more zeros and ones than other digits\. Also, its zeros vary widely in aspect ratio \(tall\-and\-thin versus short\-and\-wide\)\. The single\-axis zoom\-out randomly stretches or compresses each view along one axis, so that after independent augmentation, the two views of an image differ in width and height\. Aspect ratio is then no longer a salient feature the two views share, forcing the network to cluster views based on more essential, topology\-related structure of the digits\.

## 5Conclusion

We introduced*converge\-to\-surprise*, a self\-supervised learning framework without needing a per\-step loss\. Our null hypothesis states that pixels are i\.i\.d\. noise\. Then, we build two complementary masked views from one image, and define a*surprise score*to measure how strongly the network’s view\-clustering results reject the null hypothesis\. Assuming that a surprise score cannot be reduced to a per\-step loss function, we optimize our model combining evolution strategy with gradient descent\. Without any pre\-trained model or prior knowledge of the dataset, our framework attains state\-of\-the\-art performances on MNIST, Fashion\-MNIST, and USPS under non\-parametric self\-supervised image clustering setting\.

Once again, we encourage people to read our discussions provided in appendix[A](https://arxiv.org/html/2607.06887#A1)\.

## References

- Aloysius and Geetha \[2017\]Neena Aloysius and M Geetha\.A review on deep convolutional neural networks\.In*2017 international conference on communication and signal processing \(ICCSP\)*, pages 0588–0592\. IEEE, 2017\.
- Assran et al\. \[2023\]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas\.Self\-supervised learning from images with a joint\-embedding predictive architecture\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 15619–15629, 2023\.
- Bardes et al\. \[2021\]Adrien Bardes, Jean Ponce, and Yann LeCun\.Vicreg: Variance\-invariance\-covariance regularization for self\-supervised learning\.*arXiv preprint arXiv:2105\.04906*, 2021\.
- Batina et al\. \[2011\]L\. Batina, B\. Gierlichs, E\. Prouff, M\. Rivain, F\.\-X\. Standaert, and Veyrat\-Charvillon N\.Web text corpus for natural language processing\.*Journal of Cryptology*, 24\(2\):269–291, 2011\.
- Beaudry and Renner \[2011\]Normand J Beaudry and Renato Renner\.An intuitive proof of the data processing inequality\.*arXiv preprint arXiv:1107\.0740*, 2011\.
- Brockhoff et al\. \[2010\]Dimo Brockhoff, Anne Auger, Nikolaus Hansen, Dirk V Arnold, and Tim Hohm\.Mirrored sampling and sequential selection for evolution strategies\.In*International conference on parallel problem solving from nature*, pages 11–21\. Springer, 2010\.
- Caron et al\. \[2018\]Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze\.Deep clustering for unsupervised learning of visual features\.In*Proceedings of the European conference on computer vision \(ECCV\)*, pages 132–149, 2018\.
- Caron et al\. \[2020\]Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin\.Unsupervised learning of visual features by contrasting cluster assignments\.*Advances in neural information processing systems*, 33:9912–9924, 2020\.
- Caron et al\. \[2021\]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin\.Emerging properties in self\-supervised vision transformers\.In*Proceedings of the IEEE/CVF international conference on computer vision*, pages 9650–9660, 2021\.
- Chang et al\. \[2017\]Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan\.Deep adaptive image clustering\.In*Proceedings of the IEEE international conference on computer vision*, pages 5879–5887, 2017\.
- Chen et al\. \[2020\]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton\.A simple framework for contrastive learning of visual representations\.In*International conference on machine learning*, pages 1597–1607\. PMLR, 2020\.
- Cover \[1999\]Thomas M Cover\.*Elements of information theory*\.John Wiley & Sons, 1999\.
- Deng \[2012\]L\. Deng\.The mnist database of handwritten digit images for machine learning research \[best of the web\]\.*IEEE Signal Processing Magazine*, pages 141–142, 2012\.
- Doersch et al\. \[2015\]Carl Doersch, Abhinav Gupta, and Alexei A Efros\.Unsupervised visual representation learning by context prediction\.In*Proceedings of the IEEE international conference on computer vision*, pages 1422–1430, 2015\.
- Dosovitskiy et al\. \[2020\]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al\.An image is worth 16x16 words: Transformers for image recognition at scale\.*arXiv preprint arXiv:2010\.11929*, 2020\.
- Ester et al\. \[1996\]Martin Ester, Hans\-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al\.A density\-based algorithm for discovering clusters in large spatial databases with noise\.In*kdd*, pages 226–231, 1996\.
- Fard et al\. \[2020\]Maziar Moradi Fard, Thibaut Thonet, and Eric Gaussier\.Deep k\-means: Jointly clustering with k\-means and learning representations\.*Pattern Recognition Letters*, 138:185–192, 2020\.
- Feller \[1968\]W Feller\.An introduction to probability theory and its applications\.*Third edition\. New York: Wiley\. Theorem is in section VI\.3*, 1968\.
- Feller \[1991\]William Feller\.*An introduction to probability theory and its applications, Volume 2*\.John Wiley & Sons, 1991\.
- Gidaris et al\. \[2018\]Spyros Gidaris, Praveer Singh, and Nikos Komodakis\.Unsupervised representation learning by predicting image rotations\.*arXiv preprint arXiv:1803\.07728*, 2018\.
- Grill et al\. \[2020\]Jean\-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al\.Bootstrap your own latent\-a new approach to self\-supervised learning\.*Advances in neural information processing systems*, 33:21271–21284, 2020\.
- Guiasu and Shenitzer \[1985\]S\. Guiasu and A\. Shenitzer\.The principle of maximum entropy\.*The Mathematical Intelligencer\. doi\.org/10\.1007/BF03023004*, 7:42–48, 1985\.
- He et al\. \[2016\]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep residual learning for image recognition\.In*Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016\.
- He et al\. \[2020\]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick\.Momentum contrast for unsupervised visual representation learning\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9729–9738, 2020\.
- He et al\. \[2022\]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick\.Masked autoencoders are scalable vision learners\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 16000–16009, 2022\.
- Hjelm et al\. \[2018\]R Devon Hjelm, Alex Fedorov, Samuel Lavoie\-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio\.Learning deep representations by mutual information estimation and maximization\.*arXiv preprint arXiv:1808\.06670*, 2018\.
- Hughes and Sudderth \[2013\]Michael C Hughes and Erik Sudderth\.Memoized online variational inference for dirichlet process mixture models\.*Advances in neural information processing systems*, 26, 2013\.
- Hull \[2002\]Jonathan J\. Hull\.A database for handwritten text recognition research\.*IEEE Transactions on pattern analysis and machine intelligence*, 16\(5\):550–554, 2002\.
- Ji et al\. \[2019\]Xu Ji, Joao F Henriques, and Andrea Vedaldi\.Invariant information clustering for unsupervised image classification and segmentation\.In*Proceedings of the IEEE/CVF international conference on computer vision*, pages 9865–9874, 2019\.
- Kuhn \[1955\]Harold W Kuhn\.The hungarian method for the assignment problem\.*Naval research logistics quarterly*, 2\(1\-2\):83–97, 1955\.
- Leiber et al\. \[2024\]Collin Leiber, Niklas Strauß, Matthias Schubert, and Thomas Seidl\.Dying clusters is all you need\-deep clustering with an unknown number of clusters\.In*2024 IEEE International Conference on Data Mining Workshops \(ICDMW\)*, pages 726–733\. IEEE, 2024\.
- Li et al\. \[2022\]Naiqi Li, Wenjie Li, Yong Jiang, and Shu\-Tao Xia\.Deep dirichlet process mixture models\.In*Uncertainty in Artificial Intelligence*, pages 1138–1147\. PMLR, 2022\.
- MacKay et al\. \[1998\]David JC MacKay et al\.Introduction to gaussian processes\.*NATO ASI series F computer and systems sciences*, 168:133–166, 1998\.
- Mao et al\. \[2023\]Anqi Mao, Mehryar Mohri, and Yutao Zhong\.Cross\-entropy loss functions: Theoretical analysis and applications\.In*International conference on Machine learning*, pages 23803–23828\. pmlr, 2023\.
- McLean and Jernigan \[1991\]GF McLean and ME Jernigan\.Indicator functions for adaptive image processing\.*Journal of the Optical Society of America A*, 8\(1\):141–156, 1991\.
- Mucherino et al\. \[2009\]Antonio Mucherino, Petraq J Papajorgji, and Panos M Pardalos\.K\-nearest neighbor classification\.In*Data mining in agriculture*, pages 83–106\. Springer, 2009\.
- Noroozi and Favaro \[2016\]Mehdi Noroozi and Paolo Favaro\.Unsupervised learning of visual representations by solving jigsaw puzzles\.In*European conference on computer vision*, pages 69–84\. Springer, 2016\.
- Perez and Wang \[2017\]Luis Perez and Jason Wang\.The effectiveness of data augmentation in image classification using deep learning\.*arXiv preprint arXiv:1712\.04621*, 2017\.
- Popescu et al\. \[2009\]Marius\-Constantin Popescu, Valentina E Balas, Liliana Perescu\-Popescu, and Nikos Mastorakis\.Multilayer perceptron and neural networks\.*WSEAS transactions on circuits and systems*, 8\(7\):579–588, 2009\.
- Ronen et al\. \[2022\]Meitar Ronen, Shahaf E Finder, and Oren Freifeld\.Deepdpm: Deep clustering with an unknown number of clusters\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9861–9870, 2022\.
- Rumelhart et al\. \[1986\]D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams\.Learning internal representations by back\-propagating errors\.*Nature*, 323:533–536, 1986\.
- Salimans et al\. \[2017\]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever\.Evolution strategies as a scalable alternative to reinforcement learning\.*arXiv preprint arXiv:1703\.03864*, 2017\.
- Shannon \[1948\]Claude Elwood Shannon\.A mathematical theory of communication\.*The Bell system technical journal*, 27\(3\):379–423, 1948\.
- Shlens \[2014\]Jonathon Shlens\.Notes on kullback\-leibler divergence and likelihood\.*arXiv preprint arXiv:1404\.2000*, 2014\.
- Wierstra et al\. \[2014\]Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber\.Natural evolution strategies\.*The Journal of Machine Learning Research*, 15\(1\):949–980, 2014\.
- Xiao et al\. \[2017\]Han Xiao, Kashif Rasul, and Roland Vollgraf\.Fashion\-mnist: a novel image dataset for benchmarking machine learning algorithms\.*arXiv preprint arXiv:1708\.07747*, 2017\.
- Xie et al\. \[2016\]Junyuan Xie, Ross Girshick, and Ali Farhadi\.Unsupervised deep embedding for clustering analysis\.In*International conference on machine learning*, pages 478–487\. PMLR, 2016\.
- Yang et al\. \[2017\]Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong\.Towards k\-means\-friendly spaces: Simultaneous deep learning and clustering\.In*international conference on machine learning*, pages 3861–3870\. PMLR, 2017\.
- Zbontar et al\. \[2021\]Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny\.Barlow twins: Self\-supervised learning via redundancy reduction\.In*International conference on machine learning*, pages 12310–12320\. PMLR, 2021\.

## Appendix ADiscussions

The main paper presents converge\-to\-surprise as a novel self\-supervised learning framework\. Here, we step back and discuss the broader picture behind it\. We first argue that the null hypothesisℋ0\\mathcal\{H\}\_\{0\}need not stay fixed: learning can be viewed as a repeated cycle in which the model rejects the current null hypothesis, folds what it discovered into an updated one, and searches again \(Section[A\.1](https://arxiv.org/html/2607.06887#A1.SS1)\)\. Then, we further evaluate the surprise score from an information theory point of view, showing that learning means to build order out of chaos \(Section[A\.2](https://arxiv.org/html/2607.06887#A1.SS2)\)\.

These discussions are conceptual and supplemental\. They are not required for the results in the main paper\.

### A\.1Learning as iterative rejection of the null hypothesis

In the main paper, without any prior knowledge, we invoked the Principle of Maximum Entropy and assumed the data to be pure random noise\. In our case, this means that every pixel is i\.i\.d\. This gives us the null hypothesisℋ0\\mathcal\{H\}\_\{0\}\. The purpose of our learning framework is then to discover*non\-randomness*, or*surprise*, that rejectsℋ0\\mathcal\{H\}\_\{0\}\. However, this is only the starting point of a more general picture\.

Once the model has discovered some non\-randomness, we may fold that structure back into our description of the data\. Concretely, we update the data distribution to account for what we found, obtaining an improved hypothesisℋ1\\mathcal\{H\}\_\{1\}that is no longer pure noise: it already encodes the regularities discovered so far\. We may now treatℋ1\\mathcal\{H\}\_\{1\}as the*new*null hypothesis and repeat the process – further evolving and training the network to discover new surprise that rejectsℋ1\\mathcal\{H\}\_\{1\}\. If we succeed, we update the hypothesis again toℋ2\\mathcal\{H\}\_\{2\}, take it as the null, and continue:

ℋ0→ℋ1→ℋ2→⋯\\mathcal\{H\}\_\{0\}\\;\\rightarrow\\;\\mathcal\{H\}\_\{1\}\\;\\rightarrow\\;\\mathcal\{H\}\_\{2\}\\;\\rightarrow\\;\\cdots\(16\)Each round explains more of the data, and each new null hypothesis is harder to reject than the last\. The process terminates when no further surprise can be found, no matter what output representation we use or how we observe it\. At that point, we accept the current hypothesis and stop the learning process\.

In general, this refers to the meaning of ‘learning’, which we summarize as three principles:

1. 1\.Maximum\-entropy prior\.Without any prior knowledge, we must assume the data distribution to be totally random noise, following the Principle of Maximum Entropy\. This is our initial null hypothesis\.
2. 2\.Learning as discovering surprise\.To learn is to discover*non\-randomness*, or*surprise*, from the data distribution in order to reject the null hypothesis\.
3. 3\.Iterative updating\.We update the null hypothesis to incorporate what we have discovered\. Then, we seek new non\-randomness that can reject the updated hypothesis\. If, after enough updates, no representation and no way of observing can reveal further surprise, we accept the current hypothesis and stop learning\.

The converge\-to\-surprise framework in the main paper realizes a single round of this process\. Fully iterating through this process is a natural extension of our framework\.

### A\.2Orderliness: building order out of chaos

Now, we evaluate a surprise score from an information theory point of view\. Recall from Section[3\.2](https://arxiv.org/html/2607.06887#S3.SS2), we obtain the two cluster sequences from a batch ofNNimages:seq\(i\)=\(y^1\(i\),…,y^N\(i\)\)\\mathrm\{seq\}^\{\(i\)\}=\(\\hat\{y\}\_\{1\}^\{\(i\)\},\\dots,\\hat\{y\}\_\{N\}^\{\(i\)\}\)andseq\(j\)=\(y^1\(j\),…,y^N\(j\)\)\\mathrm\{seq\}^\{\(j\)\}=\(\\hat\{y\}\_\{1\}^\{\(j\)\},\\dots,\\hat\{y\}\_\{N\}^\{\(j\)\}\)\. Pairing the two views of each image gives the*cluster pair sequence*

Π=\(\(y^1\(i\),y^1\(j\)\),…,\(y^N\(i\),y^N\(j\)\)\),\\Pi\\;=\\;\\bigl\(\(\\hat\{y\}\_\{1\}^\{\(i\)\},\\hat\{y\}\_\{1\}^\{\(j\)\}\),\\,\\dots,\\,\(\\hat\{y\}\_\{N\}^\{\(i\)\},\\hat\{y\}\_\{N\}^\{\(j\)\}\)\\bigr\),\(17\)whose observed distribution over theK×KK\\times Kcluster pair space is

πa,b=1N​∑n=1N𝟏​\[y^n\(i\)=a​and​y^n\(j\)=b\]\.\\pi\_\{a,b\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[\\hat\{y\}\_\{n\}^\{\(i\)\}=a\\ \\text\{and\}\\ \\hat\{y\}\_\{n\}^\{\(j\)\}=b\\right\]\.\(18\)
Then, the*observed pair entropy*is

H^Π=−∑a,b:πa,b\>0πa,b​log⁡πa,b\.\\hat\{H\}\_\{\\Pi\}\\;=\\;\-\\\!\\\!\\sum\_\{a,b:\\,\\pi\_\{a,b\}\>0\}\\\!\\\!\\pi\_\{a,b\}\\,\\log\\pi\_\{a,b\}\.\(19\)
Supposena\(i\)n\_\{a\}^\{\(i\)\}is the number of times clusteraaappears inseq\(i\)\\mathrm\{seq\}^\{\(i\)\}, andnb\(j\)n\_\{b\}^\{\(j\)\}is the number of times clusterbbappears inseq\(j\)\\mathrm\{seq\}^\{\(j\)\}\. Then, it is easy to see that

pa\(i\)=∑bπa,b=na\(i\)/N,pb\(j\)=∑aπa,b=nb\(j\)/N\.p\_\{a\}^\{\(i\)\}=\\sum\_\{b\}\\pi\_\{a,b\}=n\_\{a\}^\{\(i\)\}\\ /\\ N,\\ \\ \\ p\_\{b\}^\{\(j\)\}=\\sum\_\{a\}\\pi\_\{a,b\}=n\_\{b\}^\{\(j\)\}\\ /\\ N\.\(20\)That is, unlike Section[3\.2](https://arxiv.org/html/2607.06887#S3.SS2), we observe the marginal probability distribution of a cluster within each sequence\.

Then, consider an*intra\-view shuffling*: we can permute the entries withinseq\(i\)\\mathrm\{seq\}^\{\(i\)\}and withinseq\(j\)\\mathrm\{seq\}^\{\(j\)\}independently, as many times as we like\. Butseq\(i\)\\mathrm\{seq\}^\{\(i\)\}andseq\(j\)\\mathrm\{seq\}^\{\(j\)\}do not exchange entries\. Such permutation leaves\{pk\(i\)\}k=0K−1\\\{p\_\{k\}^\{\(i\)\}\\\}\_\{k=0\}^\{K\-1\}and\{pk\(j\)\}k=0K−1\\\{p\_\{k\}^\{\(j\)\}\\\}\_\{k=0\}^\{K\-1\}unchanged, but scrambles the pairing and hence the cluster pair sequenceΠ\\Pi\.

Given the originalΠ\\Pi, suppose all pair sequences that we can possibly obtain after intra\-view shuffling forms the setℛΠ\\mathcal\{R\}\_\{\\Pi\}\. Then, we define the*shuffled\-maximal entropy*as the largest attainable value in the form:

HΠ⋆=maxΠ~∈ℛΠ⁡H^Π~\.H\_\{\\Pi\}^\{\\star\}=\\max\_\{\\tilde\{\\Pi\}\\in\\mathcal\{R\}\_\{\\Pi\}\}\\hat\{H\}\_\{\\tilde\{\\Pi\}\}\.\(21\)
It is easy to see thatH^Π≤HΠ⋆\\hat\{H\}\_\{\\Pi\}\\leq H\_\{\\Pi\}^\{\\star\}always holds true\. Also, by a routine analysis, we can get that

HΠ⋆=H​\(p\(i\)\)\+H​\(p\(j\)\),H​\(p\(⋅\)\)=−∑kpk\(⋅\)​log⁡pk\(⋅\);H\_\{\\Pi\}^\{\\star\}=H\\\!\\left\(p^\{\(i\)\}\\right\)\+H\\\!\\left\(p^\{\(j\)\}\\right\),\\ \\ H\(p^\{\(\\cdot\)\}\)=\-\\sum\_\{k\}p\_\{k\}^\{\(\\cdot\)\}\\log p\_\{k\}^\{\(\\cdot\)\};\(22\)and

HΠ⋆−H^Π=I​\(y^\(i\);y^\(j\)\)≥0\.H\_\{\\Pi\}^\{\\star\}\-\\hat\{H\}\_\{\\Pi\}\\;=\\;I\\\!\\left\(\\hat\{y\}^\{\(i\)\};\\hat\{y\}^\{\(j\)\}\\right\)\\;\\geq\\;0\.\(23\)Here,I​\(y^\(i\);y^\(j\)\)I\\\!\\left\(\\hat\{y\}^\{\(i\)\};\\hat\{y\}^\{\(j\)\}\\right\)is the mutual information between the original, un\-shuffledseq\(i\)\\mathrm\{seq\}^\{\(i\)\}andseq\(j\)\\mathrm\{seq\}^\{\(j\)\}, as described in Section[3\.2](https://arxiv.org/html/2607.06887#S3.SS2)\.

Finally, we define the*orderliness*\(or*neg\-entropy*\):

Ω=HΠ⋆\+λH^Π\+λ,\\Omega\\;=\\;\\frac\{H\_\{\\Pi\}^\{\\star\}\+\\lambda\}\{\\hat\{H\}\_\{\\Pi\}\+\\lambda\},\(24\)whereλ\>0\\lambda\>0is a small constant that keeps the ratio well\-defined in degenerated cases\. SinceH^Π≤HΠ⋆\\hat\{H\}\_\{\\Pi\}\\leq H\_\{\\Pi\}^\{\\star\}, we always haveΩ≥1\\Omega\\geq 1, with larger values indicating more order\.

Then, we describe two special cases:

1. 1\.*Fresh initialization:*Right after initializing the model,seq\(i\)\\mathrm\{seq\}^\{\(i\)\}andseq\(j\)\\mathrm\{seq\}^\{\(j\)\}are essentially independent\. So, the observed pairing is already close to its most disordered form\. Thus, we haveH^Π≈HΠ⋆\\hat\{H\}\_\{\\Pi\}\\approx H\_\{\\Pi\}^\{\\star\}andΩ≈1\\Omega\\approx 1\. This matches𝒮​\(θ\)≈0\\mathcal\{S\}\(\\theta\)\\approx 0: nothing surprising has been found\.
2. 2\.*Collapse:*If the network collapses so that every view is mapped to a single clusterk0k\_\{0\}, only the pair\(k0,k0\)\(k\_\{0\},k\_\{0\}\)is ever observed\. In this case, no shuffling can change the cluster pair sequenceΠ\\Pi\. Hence,HΠ⋆=H^Π=0H\_\{\\Pi\}^\{\\star\}=\\hat\{H\}\_\{\\Pi\}=0andΩ=λ/λ=1\\Omega=\\lambda/\\lambda=1always hold true\. In other words, collapse is not learning\.

Thus, genuine learning drivesΩ≫1\\Omega\\gg 1, which by Eq\.[24](https://arxiv.org/html/2607.06887#A1.E24)requires the observed pair entropyH^Π\\hat\{H\}\_\{\\Pi\}to be much smaller than the shuffled\-maximal entropyHΠ⋆H\_\{\\Pi\}^\{\\star\}\. The latter demands diverse marginals \(many clusters being observed\), i\.e\. high potential disorder; the former demands the actual pairing to be highly dependent \(both views of an image tend to agree\), i\.e\. strong order\. In other words, to learn is to keep the marginal diversity while imposing order on how these diversities are organized – to*build order out of chaos*\.

Combining this with Section[A\.1](https://arxiv.org/html/2607.06887#A1.SS1), we arrive at an intuitive picture of the whole process: learning repeatedly discovers order out of chaos and folds what being discovered into an ever\-stronger hypothesis, and continues until there is no more order left to discover\.

## Appendix BImplementation details

Table 4:Per\-dataset augmentation parameters, applied independently to each view after chessboard masking\. Rotation angle is drawn uniformly from\[−r,r\]\[\-r,r\], and being applied with a rotation probability\. Brightness/contrast are multiplicative jitters drawn from\[1−b,1\+b\]\[1\-b,\\,1\+b\]\. The square cropping size \(cropped edge length\) is sampled uniformly from the listed range and resized back toS×SS\\times S\. Also, edge lengthSSis among the listed range of cropping\. This means that although the cropping probability is set to 1, there is actually a chance of samplingSSitself and implementing no cropping\. Since USPS digits are relatively larger, the listed range of cropping is narrower\. Also, for USPS, we first obtain the masked views from the original16×1616\\times 16image\. Then, we upsample per view to32×3232\\times 32\. Additionally, we pad one random axis with22–66black pixels \(zoom\-out\)*independently*on each view, which is only applied to USPS\. A ‘–’ denotes a disabled augmentation\.Table[4](https://arxiv.org/html/2607.06887#A2.T4)lists the exact per\-dataset augmentation parameters referenced in Section[4\.1](https://arxiv.org/html/2607.06887#S4.SS1)\.

As mentioned in the main paper, after the chessboard masking, the two complementary views of each image are augmented*independently*, which preserves the zero\-mutual\-information property between two views underℋ0\\mathcal\{H\}\_\{0\}\. On MNIST and Fashion\-MNIST images, we apply random rotation, random square cropping, and brightness/contrast jitter\. On Fashion\-MNIST images, we additionally apply a horizontal flip with probability0\.50\.5\. Digits are not flipped, as a mirrored digit is not the same class\.

![Refer to caption](https://arxiv.org/html/2607.06887v1/MNIST_min50_col10_3_by_3.png)Figure 3:MNIST test\-set clustering\. Each panel shows nine random images of a single ground\-truth class assigned to a given cluster\. The ten active clusters map one\-to\-one to the ten digits, each at9797–99\.7%99\.7\\%purity\.![Refer to caption](https://arxiv.org/html/2607.06887v1/USPS_min50_col11_3_by_3.png)Figure 4:USPS test\-set clustering for a run that inferred1111clusters\. Clusters2323and5353both capture the digit0: cluster5353’s zeros are round and plain, while cluster2323’s are narrow, tall, and often carry an extra stroke – residual geometric variation the anisotropic augmentation did not fully absorb\.For USPS, we add a*single\-axis zoom\-out*after cropping, which pads one randomly chosen axis with a black border before resizing back, producing an*anisotropic*\(tall/short, wide/narrow\) deformation\. This is motivated by the imbalance of the USPS dataset: the digit\-0 class is dominant and exhibits far more aspect\-ratio variation than its MNIST counterpart\. The anisotropic augmentation de\-correlates this global shape so that same digits are less likely to be further split based on width or height\. To leave room for the anisotropic deformation, the remaining augmentations on USPS images are kept milder than on MNIST/Fashion\-MNIST images, as shown in Table[4](https://arxiv.org/html/2607.06887#A2.T4)\. Besides, USPS characters occupy a relatively larger fraction of the frame\. So, we mask at the native16×1616\\times 16resolution, upsample each view independently to32×3232\\times 32, and crop only down to a30×3030\\times 30window \(i\.e\., a milder cropping range:30∼3230\\sim 32\)\.

## Appendix CClustering visualization

To qualitatively inspect what the network discovered, we visualize the test\-set clustering of one trained model per dataset\. For each active clusterkkand each ground\-truth classpp, we draw a3×33\\times 3grid of randomly selected test images that belong to clusterkkand carry labelpp\. We only draw such a grid when at least5050such images exist \(rare classes in each cluster are ignored\)\. Each panel is captioned “Clusterkk, digitpp”, or “Clusterkk, itempp”\.

#### MNIST\.

Figure[3](https://arxiv.org/html/2607.06887#A2.F3)shows the MNIST clustering\. The optimized network produces a clean one\-to\-one map: each active cluster corresponds to a single ground\-truth class, and each class is captured by a single cluster\. This is confirmed by the cluster\-purity report, where every active cluster is dominated by one digit class at97%97\\%–99\.7%99\.7\\%purity:

```
Cluster 35 | n=1134 | 99.0% (dom=cls1)
Cluster 38 | n=1043 | 98.6% (dom=cls2)
Cluster 19 | n=1036 | 97.0% (dom=cls3)
Cluster 49 | n=1031 | 98.5% (dom=cls7)
Cluster 53 | n= 997 | 97.1% (dom=cls9)
Cluster 58 | n= 980 | 99.4% (dom=cls0)
Cluster 56 | n= 968 | 98.3% (dom=cls6)
Cluster 34 | n= 968 | 98.9% (dom=cls4)
Cluster 11 | n= 958 | 99.7% (dom=cls8)
Cluster 14 | n= 885 | 99.0% (dom=cls5)
```

Each of the ten digits is represented by exactly one high\-purity cluster, so Figure[3](https://arxiv.org/html/2607.06887#A2.F3)contains exactly ten panels\.

![Refer to caption](https://arxiv.org/html/2607.06887v1/FashionMNIST_min50_col7_3_by_3.png)Figure 5:Fashion\-MNIST test\-set clustering\. Beyond coarse categories, the network separates finer attributes: clusters4747vs\.5959split bags by the presence of a handle/strap, and clusters2424vs\.5353split upper\-body garments by texture \(patterned vs\. plain\) rather than contour\.
#### USPS\.

Next, Figure[4](https://arxiv.org/html/2607.06887#A2.F4)shows a deliberately*less\-than\-perfect*USPS run, in which the model settled on1111clusters instead of1010\. Its per\-cluster purity is still high, but two clusters share the same dominant class \(digit 0\), which is what pushes the count to1111:

```
Cluster 53 | n=313 | 97.1% (dom=cls0)
Cluster 52 | n=256 | 99.6% (dom=cls1)
Cluster 28 | n=199 | 95.0% (dom=cls2)
Cluster 38 | n=199 | 94.5% (dom=cls4)
Cluster 33 | n=188 | 93.1% (dom=cls9)
Cluster 15 | n=168 | 95.2% (dom=cls8)
Cluster 14 | n=166 | 97.6% (dom=cls3)
Cluster 31 | n=162 | 98.1% (dom=cls6)
Cluster 44 | n=157 | 95.5% (dom=cls5)
Cluster 12 | n=144 | 96.5% (dom=cls7)
Cluster 23 | n= 55 | 90.9% (dom=cls0)
```

Both cluster2323and cluster5353focus on the digit0: the zeros in cluster5353are relatively round and plain, whereas those in cluster2323are narrow, tall, and often carry an extra stroke beyond the bare loop\. This shows that the zoom\-out \(anisotropic\) augmentation largely works as intended, but does not perfectly remove every unwanted topological and geometric variation of the dominant digit\-0 class\. This leads to further split of the digit\-0 class\.

#### Fashion\-MNIST\.

Finally, Fashion\-MNIST is far more challenging for non\-parametric self\-supervised image clustering, since several classes differ only in fine texture rather than global contour\[[46](https://arxiv.org/html/2607.06887#bib.bib46)\]\. The per\-cluster purity is correspondingly lower and some clusters mix classes:

```
Cluster 53 | n=1303 | 31.1% (dom=cls6)
Cluster 23 | n=1165 | 82.1% (dom=cls1)
Cluster 48 | n=1164 | 63.4% (dom=cls3)
Cluster  2 | n=1136 | 81.4% (dom=cls7)
Cluster  7 | n=1112 | 86.2% (dom=cls9)
Cluster 55 | n=1014 | 76.4% (dom=cls0)
Cluster 24 | n= 979 | 46.7% (dom=cls2)
Cluster 31 | n= 770 | 99.2% (dom=cls5)
Cluster 47 | n= 524 | 96.8% (dom=cls8)
Cluster 59 | n= 476 | 90.1% (dom=cls8)
Cluster 42 | n= 357 | 40.9% (dom=cls2)
```

Even so, the visualization in Figure[5](https://arxiv.org/html/2607.06887#A3.F5)shows that the network discovers meaningful structure beyond the labels\. Clusters4747and5959both isolate*bags*, but along a finer distinction: cluster4747collects bags with a visible*handle/strap*, whereas cluster5959collects bags with no \(or only a very small, barely visible\) handle\. Clusters2424and5353both contain*upper\-body garments*, but separate them by*texture*rather than contour: the garments in cluster2424are more heavily textured/patterned, while those in cluster5353are comparatively plain\. In other words, despite using only a ResNet\-9, converge\-to\-surprise optimization enables the network to distinguish both contour and texture\.

Once again, although the per\-cluster purity on Fashion\-MNIST is lower than those on the other two datasets, we still achieve state\-of\-the\-art performances on Fashion\-MNIST\.

## Appendix DNumber of discovered clusters during training

![Refer to caption](https://arxiv.org/html/2607.06887v1/MNIST_num_clusters.png)Figure 6:Number of discovered \(surprising\) clusters vs\. training epoch on MNIST, for five independent runs\. The model over\-produces surprising clusters during pure ES stage\. Then, the number of surprise clusters are consolidated back to1010once gradient\-descent training begins \(two\-stage schedule, starts from epoch20002000\)\.![Refer to caption](https://arxiv.org/html/2607.06887v1/USPS_num_clusters.png)Figure 7:Number of discovered \(surprising\) clusters vs\. training epoch on USPS, for five independent runs \(three\-stage schedule\)\. Again, pure ES stage \(0–40004000\) over\-produces; weak training stage \(40004000–80008000\) yields the consolidation/re\-exploration sawtooth; the final strong gradient\-descent training stage settles the number of surprising clusters in each run to1010or1111\.![Refer to caption](https://arxiv.org/html/2607.06887v1/FashionMNIST_num_clusters.png)Figure 8:Number of discovered \(surprising\) clusters vs\. training epoch on Fashion\-MNIST, for five independent runs\. The curves behave similarly as in the experiments on MNIST, whereas the final number of surprising clusters is slightly above 10\. This is in line with the dataset’s harder, texture\-dominated structure\.Because the number of classes is never given to the model, it is instructive to watch how many clusters the model discovers as training proceeds\. Recall that we introducedD​\(q^k∥qk\)D\\\!\\left\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\\right\), the surprise score at clusterkk, via formula[10](https://arxiv.org/html/2607.06887#S3.E10)in Section[3\.2](https://arxiv.org/html/2607.06887#S3.SS2)\. Then, we regard clusterkkas*surprising*ifD​\(q^k∥qk\)≥τ=0\.005D\\\!\\left\(\\hat\{q\}\_\{k\}\\,\\\|\\,q\_\{k\}\\right\)\\geq\\tau=0\.005\. Figures[6](https://arxiv.org/html/2607.06887#A4.F6),[7](https://arxiv.org/html/2607.06887#A4.F7)and[8](https://arxiv.org/html/2607.06887#A4.F8)plot, for five independent runs per dataset, the number of surprising clusters after each ES epoch\.

We can see that during the pure\-ES stage, the model keeps discovering new candidate clusters, exceeding the number of ground truth classes\. We call this*over\-production*\. Once gradient\-descent training begins, each training round*consolidates*redundant clusters, pulling the count back down\. Then, the evolution strategy slightly rebuilds surprising clusters on the cleaner partition\. But finally, the evolution strategy and gradient\-descent training reach the balance\. No new surprising cluster is discovered in the last 200 epochs\. In general, results shown here coincide with our analysis in the ablation study[4\.3](https://arxiv.org/html/2607.06887#S4.SS3)\.

On MNIST \(Figure[6](https://arxiv.org/html/2607.06887#A4.F6), two\-stage schedule\), the over\-production peaks in the first20002000epochs, and then collapses cleanly to1010for all five runs\. On USPS \(Figure[7](https://arxiv.org/html/2607.06887#A4.F7), three\-stage schedule\) the effect is more pronounced: the pure\-ES stage \(0–40004000epochs\) over\-produces up to around2525clusters\. In the weak stage \(40004000–80008000epochs, with 2 training epochs every 500 ES epochs\), the ES optimization rebuilds back the number of surprising clusters after every gradient\-descent training epoch, creating the sawtooth curves\. Finally, the strong stage \(80008000–90009000epochs, with 4 training epochs every 25 ES epochs\) settles the number of surprising clusters to1010or1111\. Fashion\-MNIST \(Figure[8](https://arxiv.org/html/2607.06887#A4.F8)\) behaves similarly but stabilizes at a slightly larger count, reflecting its harder, texture\-dominated data structure\.

Once again, by converge\-to\-surprise, the model naturally discovers meaningful clusters from raw images without any prior knowledge\.