Verbalized Particle Posterior: Bayesian Inference over Natural Language Hypotheses

# Bayesian Inference over Natural Language Hypotheses
Source: [https://arxiv.org/html/2607.22961](https://arxiv.org/html/2607.22961)
## Verbalized Particle Posterior: Bayesian Inference over Natural Language Hypotheses

\\nameShikan Lian\\emailshikanlian@gmail\.com\\nameShibo Li\\emailshiboli@cs\.fsu\.edu \\addrDepartment of Computer Science, Florida State UniversityCorresponding author\.

###### Abstract

Verbalized Machine Learning \(VML\) parameterizes a model as a natural\-language prompt that an LLM evaluates asf​\(x;θ\)f\(x;\\theta\)\. The framework is interpretable, but it commits to a single hypothesis with no measure of uncertainty, and that hypothesis varies substantially across optimization runs on the same data\. We propose the Verbalized Particle Posterior \(VPP\), which treats verbalized learning as a Bayesian inference problem: maintain a population of natural\-language hypotheses as particles, update them with Metropolis–Hastings \(VPP\-MH\) or Sequential Monte Carlo \(VPP\-SMC\), and predict by Bayesian model averaging\. Both algorithms treat the LLM as a black box, requiring no access to logits or gradients\. A distinctive consequence follows\. In classical Bayesian learning, model selection sits outside the posterior; in VPP both model structure and parameters share a single language space, and the posterior ranges over both\. We evaluate VPP on regression, classification, and rule\-discovery benchmarks\. It improves over a single VML run on every benchmark and matches or exceeds an oracle\-best ensemble of independent VML runs on most, while eliminating the catastrophic single\-run failures that VML occasionally produces\. Because each particle is a human\-readable hypothesis, the posterior is itself something a reader can inspect, seeing in plain text which explanations the data supported and which it ruled out\.

## 1Introduction

Verbalized Machine Learning \(VML\)Xiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)parameterizes machine learning models entirely in natural language\. An LLM with a text promptθ\\thetaacts as a functionf​\(x;θ\)f\(x;\\theta\), whereθ∈Θlanguage\\theta\\in\\Theta\_\{\\text\{language\}\}is constrained to be human\-readable text, and a separate optimizer LLM iteratively refinesθ\\thetaby examining mini\-batch predictions against ground\-truth labels\. The framework is appealing for three reasons: it discovers the appropriate model class from data without the practitioner specifying linear, periodic, or piecewise; it produces hypotheses a human can read directly; and it accepts prior knowledge written in plain language\.

These properties come at a cost\. VML returns a single hypothesisθ∗\\theta^\{\*\}with no notion of uncertainty, and the optimizer’s stochasticity over a large discrete space means the returned hypothesis can vary by orders of magnitude across runs\. Across1515independent VML runs on linear regression in our experiments, test MSE ranges from1\.081\.08to48\.3048\.30, a44\.7×44\.7\\timesspread that gives the practitioner no way to tell whether a particular run was a good or bad draw\. The variance is not cosmetic: VML accepts every optimizer proposal without filtering, so a single poor revision can overwrite a good hypothesis with no path back, and individual runs on contains\-zero range from32%32\\%to100%100\\%accuracy on the same dataset\. And even when a run does fail, VML cannot signal it—there is no mechanism to flag inputs on which its hypothesis is unreliable\.

A posteriorp​\(θ∣𝒟\)p\(\\theta\\mid\\mathcal\{D\}\)is the standard remedy: rather than one hypothesis the data weakly supports, maintain a distribution over many\. Putting Bayes on VML is unusual on both sides\. The parameter spaceΘlanguage\\Theta\_\{\\text\{language\}\}is discrete, compositional, and admits no gradients or continuous neighborhoods, so most modern posterior\-inference machinery is unavailable\. The other side is more interesting\. In classical Bayesian learning the practitioner first chooses a model family—linear regression, Gaussian process, neural network—and only then performs posterior inference within that family; model selection sits outside the framework\. In VML, model structure and parameters live in the same language space, so a single posterior over verbalized hypotheses simultaneously selects the model and infers its parameters\. Classical MCMC has no natural way to express that target\.

We propose the*Verbalized Particle Posterior*\(VPP\), a particle\-based framework for Bayesian inference over verbalized hypotheses\. VPP maintainsKKnatural\-language hypotheses as particles, updates them via inference algorithms that treat the LLM as a black box, and predicts by Bayesian model averaging\. We instantiate VPP two ways\.VPP\-MHruns Metropolis–Hastings on each particle, using the optimizer LLM as an informed proposal and accepting each proposal with probability set by the batch likelihood ratio, so updates that would lower the likelihood are likely to be rejected—directly addressing the failure mode in which a single bad revision overwrites a good hypothesis\.VPP\-SMCadditionally maintains importance weights on the particles, reweights them by accumulated likelihood, and resamples when the effective sample size drops, concentrating the ensemble on well\-supported hypotheses while letting weak ones be replaced\. Because every particle is a complete verbal hypothesis, the resulting posterior is inspectable: a reader can see which hypotheses receive high weight and which the data has ruled out\.

The pattern\-discovery task*sum\-parity*—in which the model must discover from labeled examples that the label is11iff the digit sum is even—illustrates why posterior reweighting differs from extra compute\. VPP\-SMC reaches100%100\\%test accuracy on every seed\. With five times the budget, an oracle that picks the best of five independent VML runs also reaches100%100\\%; but the corresponding unweighted ensemble averages only67\.2%67\.2\\%, because incorrect runs outvote correct ones\. VPP\-SMC stays perfect because likelihood\-based reweighting concentrates weight on the rule the data supports\.

#### Contributions\.

\(i\)We formulate posterior inference over verbalized hypotheses and instantiate it with two black\-box algorithms, VPP\-MH and VPP\-SMC\.\(ii\)On seven benchmarks spanning regression, classification, and rule discovery, VPP improves over single\-run VML on every task and matches or exceeds an oracle\-best5×5\\timesensemble of independent VML runs on six of seven, reducing test MSE by up to24×24\\timeson polynomial regression\.\(iii\)VPP turns particle disagreement into an interpretable uncertainty signal, a capability standard VML lacks\.\(iv\)We compare MH and SMC in this setting and ablate the key design choices: the Metropolis–Hastings accept/reject step \(worth22\.222\.2pp on sum\-parity\) and the particle countKK\.

## 2Background

#### Verbalized Machine Learning\.

VMLXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)treats a text promptθ∈Θlanguage\\theta\\in\\Theta\_\{\\text\{language\}\}as the parameter of a model, with an LLM realizing the functionf​\(x;θ\)f\(x;\\theta\)\. A separate optimizer LLM refinesθ\\thetafrom mini\-batches of \(input, prediction, target\) triples to minimize the empirical loss

minθ∈Θlanguage⁡1N​∑n=1Nℓ​\(yn,f​\(xn;θ\)\),\\min\_\{\\theta\\in\\Theta\_\{\\text\{language\}\}\}\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\ell\\big\(y\_\{n\},\\,f\(x\_\{n\};\\theta\)\\big\),\(1\)whereℓ\\ellis squared error for regression or zero\-one loss for classification\. The initial promptθ0\\theta\_\{0\}is deliberately model\-class\-agnostic \(e\.g\., “You are designed to do regression”\), so the optimizer must induce both functional form and parameters from data\. Standard VML returns a single point estimateθ∗\\theta^\{\*\}—the prompt that survived the last optimizer step—and uses it directly for prediction, with no mechanism for combining or comparing the many alternative hypotheses the optimizer rejected along the way\.

#### Bayesian inference\.

Given data𝒟=\{\(xn,yn\)\}n=1N\\mathcal\{D\}=\\\{\(x\_\{n\},y\_\{n\}\)\\\}\_\{n=1\}^\{N\}and parameterθ\\theta, Bayesian inference replaces the point estimate with the posterior

p​\(θ∣𝒟\)=p​\(𝒟∣θ\)​p​\(θ\)p​\(𝒟\)∝p​\(𝒟∣θ\)​p​\(θ\),p\(\\theta\\mid\\mathcal\{D\}\)\\;=\\;\\frac\{p\(\\mathcal\{D\}\\mid\\theta\)\\,p\(\\theta\)\}\{p\(\\mathcal\{D\}\)\}\\;\\propto\\;p\(\\mathcal\{D\}\\mid\\theta\)\\,p\(\\theta\),\(2\)which assigns a probability to everyθ\\thetain proportion to how well it explains the data, scaled by the priorp​\(θ\)p\(\\theta\)\. Predictions at a newx∗x^\{\*\}are obtained by averaging the model over the posterior,

p​\(y∗∣x∗,𝒟\)=∫p​\(y∗∣x∗,θ\)​p​\(θ∣𝒟\)​𝑑θ,p\(y^\{\*\}\\mid x^\{\*\},\\mathcal\{D\}\)\\;=\\;\\int p\(y^\{\*\}\\mid x^\{\*\},\\theta\)\\,p\(\\theta\\mid\\mathcal\{D\}\)\\,d\\theta,\(3\)so the prediction reflects the spread of plausible explanations rather than a single committed one\. The integral is intractable for any nontrivial model\. Two families of approximations dominate: variational inferenceBlei et al\. \([2017](https://arxiv.org/html/2607.22961#bib.bib1)\); Kingma and Welling \([2014](https://arxiv.org/html/2607.22961#bib.bib16)\)fits a tractable parametric family top​\(θ∣𝒟\)p\(\\theta\\mid\\mathcal\{D\}\)by minimizing a divergence, and Markov chain Monte Carlo \(MCMC\) generates samples whose empirical distribution converges top​\(θ∣𝒟\)p\(\\theta\\mid\\mathcal\{D\}\)\.

#### Why MH and SMC for VML\.

The VML parameter spaceΘlanguage\\Theta\_\{\\text\{language\}\}is unfriendly to most posterior\-inference machinery: it is discrete, so reparameterization\-based variational families do not apply; it is variable\-dimensional, since verbal hypotheses can be of any length; and it admits no useful gradient or continuous neighborhood, ruling out gradient\-based methods such as Hamiltonian Monte CarloNeal \([2011](https://arxiv.org/html/2607.22961#bib.bib22)\)and its adaptive successorsHoffman and Gelman \([2014](https://arxiv.org/html/2607.22961#bib.bib14)\)\. Among gradient\-free options, Metropolis–Hastings is a natural fit: it requires only pointwise likelihood evaluation and a proposal distribution, both available in VML—the LLM evaluates predictions for the likelihood, and the optimizer LLM generates proposals\. Sequential Monte Carlo lifts MH to a population of particles with reweighting and resampling, which both escapes the single\-chain bottleneck and explores multimodal posteriors more effectively than a single MH chain\. We build VPP from these two algorithms\.

#### Metropolis–Hastings \(MH\)\.

MHMetropolis et al\. \([1953](https://arxiv.org/html/2607.22961#bib.bib20)\); Hastings \([1970](https://arxiv.org/html/2607.22961#bib.bib13)\)constructs a Markov chain whose stationary distribution isp​\(θ∣𝒟\)p\(\\theta\\mid\\mathcal\{D\}\)\. At each step a candidateθ′\\theta^\{\\prime\}is drawn from a proposalq​\(θ′∣θ\)q\(\\theta^\{\\prime\}\\mid\\theta\)and accepted with probability

α=min⁡\(1,p​\(𝒟∣θ′\)​p​\(θ′\)p​\(𝒟∣θ\)​p​\(θ\)⋅q​\(θ∣θ′\)q​\(θ′∣θ\)\);\\alpha\\;=\\;\\min\\\!\\left\(1,\\,\\frac\{p\(\\mathcal\{D\}\\mid\\theta^\{\\prime\}\)\\,p\(\\theta^\{\\prime\}\)\}\{p\(\\mathcal\{D\}\\mid\\theta\)\\,p\(\\theta\)\}\\cdot\\frac\{q\(\\theta\\mid\\theta^\{\\prime\}\)\}\{q\(\\theta^\{\\prime\}\\mid\\theta\)\}\\right\);\(4\)otherwise the chain stays atθ\\theta\. The acceptance ratio enforces detailed balance, which guarantees that the chain converges to the target posterior under mild regularity conditions, regardless of initialization\. Section[3\.2](https://arxiv.org/html/2607.22961#S3.SS2)discusses an approximation to the proposal\-density ratio in the verbalized setting\.

#### Sequential Monte Carlo \(SMC\)\.

SMCDoucet et al\. \([2001](https://arxiv.org/html/2607.22961#bib.bib7)\); Del Moral et al\. \([2006](https://arxiv.org/html/2607.22961#bib.bib6)\)approximates the posterior with a weighted particle set\{\(θ\(k\),wk\)\}k=1K\\\{\(\\theta^\{\(k\)\},w\_\{k\}\)\\\}\_\{k=1\}^\{K\}that is propagated through a sequence of intermediate distributions bridging prior and posterior\. A common bridge is likelihood tempering,pt​\(θ\)∝p​\(𝒟∣θ\)βt​p​\(θ\)p\_\{t\}\(\\theta\)\\propto p\(\\mathcal\{D\}\\mid\\theta\)^\{\\beta\_\{t\}\}\\,p\(\\theta\)with0=β0<β1<⋯<βT=10=\\beta\_\{0\}<\\beta\_\{1\}<\\cdots<\\beta\_\{T\}=1: early stages stay close to the prior so particles remain diverse, while later stages sharpen toward the true posterior\. At each stage the algorithm performs three operations\.*Reweighting*multiplies each particle’s weight by the incremental likelihood, concentrating probability mass on hypotheses the data favors\.*Resampling*, triggered when the effective sample sizeESS=1/∑kwk2\\mathrm\{ESS\}=1/\\sum\_\{k\}w\_\{k\}^\{2\}drops below a threshold, replaces low\-weight particles with copies of high\-weight ones to prevent the population from collapsing onto a few surviving hypotheses\.*Mutation*then diversifies the resampled particles via a Markov kernel that leavesptp\_\{t\}invariant, restoring exploration after the resampling step’s loss of diversity\.

## 3Method

### 3\.1Posterior Inference over Verbalized Hypotheses

We approximate the posteriorp​\(θ\|𝒟\)∝p​\(𝒟\|θ\)​p​\(θ\)p\(\\theta\|\\mathcal\{D\}\)\\propto p\(\\mathcal\{D\}\|\\theta\)\\,p\(\\theta\)over the verbalized hypothesis spaceΘlanguage\\Theta\_\{\\text\{language\}\}with a particle set\{\(θ\(k\),wk\)\}k=1K\\\{\(\\theta^\{\(k\)\},w\_\{k\}\)\\\}\_\{k=1\}^\{K\}, where eachθ\(k\)\\theta^\{\(k\)\}is a complete natural\-language hypothesis andwkw\_\{k\}is its weight\. We specify the likelihoodp​\(𝒟\|θ\)p\(\\mathcal\{D\}\|\\theta\)and the priorp​\(θ\)p\(\\theta\)in turn\.

#### Likelihood\.

Because VPP treats the LLM as a black box, we define the likelihood externally by comparing the LLM’s predictions to ground\-truth labels, with no access to token\-level probabilities\. For regression on a data batch\{\(xm,ym\)\}m=1M\\\{\(x\_\{m\},y\_\{m\}\)\\\}\_\{m=1\}^\{M\}:

log⁡p​\(\{ym\}\|\{xm\},θ\)=−12​τ​∑m=1M\(ym−f​\(xm;θ\)\)2,\\log p\(\\\{y\_\{m\}\\\}\|\\\{x\_\{m\}\\\},\\theta\)=\-\\frac\{1\}\{2\\tau\}\\sum\_\{m=1\}^\{M\}\\big\(y\_\{m\}\-f\(x\_\{m\};\\theta\)\\big\)^\{2\},\(5\)whereτ\>0\\tau\>0controls the sensitivity to prediction errors \(we useτ=1\\tau=1\)\. For classification, we use a smoothed zero\-one likelihood:

log⁡p​\(\{ym\}\|\{xm\},θ\)=∑m=1M\[𝟙​\[ym=y^m\]​log⁡\(1−ϵ\)\+𝟙​\[ym≠y^m\]​log⁡ϵ\],\\log p\(\\\{y\_\{m\}\\\}\|\\\{x\_\{m\}\\\},\\theta\)=\\sum\_\{m=1\}^\{M\}\\big\[\\mathbb\{1\}\[y\_\{m\}=\\hat\{y\}\_\{m\}\]\\log\(1\{\-\}\\epsilon\)\+\\mathbb\{1\}\[y\_\{m\}\\neq\\hat\{y\}\_\{m\}\]\\log\\epsilon\\big\],\(6\)wherey^m=f​\(xm;θ\)\\hat\{y\}\_\{m\}=f\(x\_\{m\};\\theta\)andϵ\>0\\epsilon\>0is a small smoothing constant \(we useϵ=0\.05\\epsilon=0\.05\) that prevents infinite log\-likelihood ratios from single misclassifications\. Because the likelihood is computed from predictions alone, any LLM that produces predictions can be plugged into VPP, including API\-only models without logit access\.

#### Prior\.

The priorp​\(θ\)p\(\\theta\)is the LLM’s generative distribution over text conditioned on a model\-class\-agnostic seed prompt \(e\.g\., “You are designed to do regression”\)\. Pointwise evaluation ofp​\(θ\)p\(\\theta\)would require marginalizing over the LLM’s token distributions for an entire natural\-language hypothesis and is intractable; throughout VPP we therefore only*sample*from the prior\. Concretely, theKKinitial particlesθ0\(1\),…,θ0\(K\)\\theta\_\{0\}^\{\(1\)\},\\ldots,\\theta\_\{0\}^\{\(K\)\}are produced by running the optimizer LLMKKtimes on a shared seed prompt and seed batch, each call at a different sampling temperature:θ0\(k\)←fopt​\(θinit,seed batch\)\\theta\_\{0\}^\{\(k\)\}\\leftarrow f\_\{\\text\{opt\}\}\(\\theta\_\{\\text\{init\}\},\\text\{seed batch\}\)fork=1,…,Kk=1,\\ldots,K; we treat these temperature\-diverse optimizer outputs as informed samples from the prior\. The LLM’s pretraining distribution thus acts as an implicit, informative prior that we never score pointwise\. The consequence of this design decision for the Metropolis–Hastings acceptance ratio is discussed in Section[3\.2](https://arxiv.org/html/2607.22961#S3.SS2)\.

#### Prediction via Bayesian Model Averaging\.

Given the particle ensemble, prediction at a new inputx∗x^\{\*\}marginalizes over the approximate posterior\. For regression,

y^​\(x∗\)=∑k=1Kwk​f​\(x∗;θ\(k\)\),\\hat\{y\}\(x^\{\*\}\)=\\sum\_\{k=1\}^\{K\}w\_\{k\}\\,f\(x^\{\*\};\\theta^\{\(k\)\}\),\(7\)and for classification we use the analogous weighted majority vote,

y^​\(x∗\)=arg⁡maxc​∑k=1Kwk​1​\[f​\(x∗;θ\(k\)\)=c\],\\hat\{y\}\(x^\{\*\}\)=\\arg\\max\_\{c\}\\sum\_\{k=1\}^\{K\}w\_\{k\}\\,\\mathbb\{1\}\\\!\\left\[f\(x^\{\*\};\\theta^\{\(k\)\}\)=c\\right\],\(8\)with ties broken arbitrarily\. Weights are uniform \(wk=1/Kw\_\{k\}=1/K\) in VPP\-MH and the normalized tempered importance weights \(Section[3\.3](https://arxiv.org/html/2607.22961#S3.SS3)\) in VPP\-SMC\. Predictive uncertainty is quantified by particle disagreement \(Section[5\.3](https://arxiv.org/html/2607.22961#S5.SS3)\)\.

### 3\.2VPP\-MH: Metropolis\-Hastings in Language Space

The key design question for applying MH to verbalized hypotheses is the choice of proposal\. InΘlanguage\\Theta\_\{\\text\{language\}\}we use the optimizer LLM itself: given the current hypothesisθ\(k\)\\theta^\{\(k\)\}and a mini\-batch of predictions and targets, the optimizer generates a refinedθ~\(k\)=fopt​\(θ\(k\),batch\)\\tilde\{\\theta\}^\{\(k\)\}=f\_\{\\text\{opt\}\}\(\\theta^\{\(k\)\},\\text\{batch\}\)\. This is an informed proposal far more sample\-efficient than random perturbation in a combinatorial text space\.

At each training step, for each particleθ\(k\)\\theta^\{\(k\)\}, VPP\-MH performs: \(1\) a*forward pass*, evaluatingf​\(xm;θ\(k\)\)f\(x\_\{m\};\\theta^\{\(k\)\}\)on the current mini\-batch; \(2\) a*proposal step*, where the optimizer LLM generates a candidateθ~\(k\)\\tilde\{\\theta\}^\{\(k\)\}conditioned on the current hypothesis and batch\-level errors; and \(3\) an*accept/reject step*, where the candidate is accepted with probability

α=min⁡\(1,p​\(𝒟batch\|θ~\(k\)\)p​\(𝒟batch\|θ\(k\)\)\)\.\\alpha=\\min\\\!\\left\(1,\\,\\frac\{p\(\\mathcal\{D\}\_\{\\text\{batch\}\}\|\\tilde\{\\theta\}^\{\(k\)\}\)\}\{p\(\\mathcal\{D\}\_\{\\text\{batch\}\}\|\\theta^\{\(k\)\}\)\}\\right\)\.\(9\)If rejected, the particle retains its current hypothesis\. The accept/reject step is what distinguishes VPP\-MH from running VMLKKtimes in parallel: a particle never degrades once it has found a strong hypothesis, since a poor optimizer proposal is rejected\. Standard VML has no such filter; the ablation in Section[5\.4](https://arxiv.org/html/2607.22961#S5.SS4)confirms this matters empirically\.

The acceptance ratio in Eq\.[9](https://arxiv.org/html/2607.22961#S3.E9)contains only the likelihood ratio, dropping both the prior ratiop​\(θ′\)/p​\(θ\)p\(\\theta^\{\\prime\}\)/p\(\\theta\)and the proposal ratioq​\(θ\|θ′\)/q​\(θ′\|θ\)q\(\\theta\|\\theta^\{\\prime\}\)/q\(\\theta^\{\\prime\}\|\\theta\)that appear in the general MH form \(Eq\.[4](https://arxiv.org/html/2607.22961#S2.E4)\)\. Both omissions are forced by the same constraint: we cannot pointwise evaluate the LLM’s generative density, whether it is scoring an unconditional hypothesis \(the prior\) or a conditional refinement \(the proposal\)\. These approximations break exact detailed balance, so VPP\-MH is best understood as a pragmatic MCMC\-like algorithm targeting a distribution close to, but not exactly, the posterior\. Empirically, retaining the accept/reject decision under this simplified ratio substantially outperforms always\-accepting—the property we actually need for robustness\. Per training step, VPP\-MH issues3​K3KLLM calls—one forward pass, one optimizer proposal, and one proposal evaluation per particle; particle weights remain uniform and ESS monitoring is not used\. The full algorithm is in Algorithm[1](https://arxiv.org/html/2607.22961#alg1)\.

Algorithm 1VPP\-MH: Verbalized Particle Posterior via Metropolis\-Hastings0:Training data

𝒟\\mathcal\{D\}, particles

KK, epochs

TT, batch size

MM
1:Initialize

θ0\(k\)←fopt​\(θinit,seed batch\)\\theta\_\{0\}^\{\(k\)\}\\leftarrow f\_\{\\text\{opt\}\}\(\\theta\_\{\\text\{init\}\},\\text\{seed batch\}\)for

k=1,…,Kk=1,\\ldots,Kwith varying temperature

2:for

t=1,…,Tt=1,\\ldots,Tdo

3:Sample mini\-batch

\{\(xm,ym\)\}m=1M\\\{\(x\_\{m\},y\_\{m\}\)\\\}\_\{m=1\}^\{M\}
4:for

k=1,…,Kk=1,\\ldots,Kdo

5:

y^m←f​\(xm;θt−1\(k\)\)\\hat\{y\}\_\{m\}\\leftarrow f\(x\_\{m\};\\theta\_\{t\-1\}^\{\(k\)\}\)for all

mm// Forward pass

6:

θ~\(k\)←fopt​\(θt−1\(k\),\{xm,y^m,ym\}\)\\tilde\{\\theta\}^\{\(k\)\}\\leftarrow f\_\{\\text\{opt\}\}\(\\theta\_\{t\-1\}^\{\(k\)\},\\\{x\_\{m\},\\hat\{y\}\_\{m\},y\_\{m\}\\\}\)// Propose

7:

y^m′←f​\(xm;θ~\(k\)\)\\hat\{y\}^\{\\prime\}\_\{m\}\\leftarrow f\(x\_\{m\};\\tilde\{\\theta\}^\{\(k\)\}\)for all

mm// Evaluate proposal

8:Compute

α\\alphavia Eq\.[9](https://arxiv.org/html/2607.22961#S3.E9)

9:

θt\(k\)←θ~\(k\)\\theta\_\{t\}^\{\(k\)\}\\leftarrow\\tilde\{\\theta\}^\{\(k\)\}w\.p\.

α\\alpha, else

θt\(k\)←θt−1\(k\)\\theta\_\{t\}^\{\(k\)\}\\leftarrow\\theta\_\{t\-1\}^\{\(k\)\}// Accept/reject

10:endfor

11:endfor

12:Predict:

y^​\(x∗\)=1K​∑k=1Kf​\(x∗;θT\(k\)\)\\hat\{y\}\(x^\{\*\}\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}f\(x^\{\*\};\\theta\_\{T\}^\{\(k\)\}\)// BMA

### 3\.3VPP\-SMC: Sequential Monte Carlo in Language Space

Like VPP\-MH, VPP\-SMC approximates the posteriorp​\(θ\|𝒟\)p\(\\theta\|\\mathcal\{D\}\)—the tempering schedule terminates atβT=1\\beta\_\{T\}=1—but addresses two practical limitations of VPP\-MH in the verbalized setting\. First, as particles improve and the current batch likelihood becomes high, the acceptance rate decays—proposals that would lower the likelihood are accepted with vanishing probability, while proposals that would improve on a near\-perfect hypothesis are themselves rare\. This can cause particles to stagnate at local optima\. Second, all particles receive equal weight1/K1/Kin BMA regardless of their quality, meaning a few poor particles can degrade the ensemble prediction\.

VPP\-SMC addresses both issues using the standard SMC cycle \(Section[2](https://arxiv.org/html/2607.22961#S2)\)—reweight, resample, mutate—which concentrates the ensemble on well\-supported hypotheses while replacing weak ones\. Applying SMC to the verbalized hypothesis space requires three modifications\.

#### Tempered importance weights\.

Reweighting particles by their raw likelihood is numerically fragile: the log\-likelihoods from Eqs\.[5](https://arxiv.org/html/2607.22961#S3.E5)–[6](https://arxiv.org/html/2607.22961#S3.E6)can span orders of magnitude across particles, causing a single hypothesis to dominate and collapsing the effective population to size one\. We therefore use tempered weights,

wk∝exp⁡\(log⁡p​\(𝒟buffer\|θ\(k\)\)βt\),w\_\{k\}\\propto\\exp\\\!\\left\(\\frac\{\\log p\(\\mathcal\{D\}\_\{\\text\{buffer\}\}\|\\theta^\{\(k\)\}\)\}\{\\beta\_\{t\}\}\\right\),\(10\)withβt\\beta\_\{t\}annealed linearly fromβ0=10\\beta\_\{0\}=10\(flat weights, encouraging exploration\) toβT=1\\beta\_\{T\}=1\(sharp weights concentrated on the best\-supported hypotheses\) over training\. Eq\.[10](https://arxiv.org/html/2607.22961#S3.E10)is a pragmatic reweighting scheme rather than a formal SMC sampler over annealed posteriorspt​\(θ\)∝p​\(𝒟\|θ\)βt​p​\(θ\)p\_\{t\}\(\\theta\)\\propto p\(\\mathcal\{D\}\|\\theta\)^\{\\beta\_\{t\}\}p\(\\theta\): we do not maintain incremental weight ratios between intermediate distributions\.

#### Cumulative evaluation buffer\.

Per\-batch likelihood estimates are noisy: a hypothesis may score perfectly on the current batch but poorly on previous ones, causing oscillatory reweighting\. We stabilise the evaluation by computing likelihoods on a rolling buffer of the most recentB=40B=40data points \(four batches\) rather than the current batch alone, at the cost of a modest increase in forward\-pass calls\.

#### Adaptive mutation\.

After resampling, duplicated particles must be diversified to avoid population collapse\. We use the optimizer LLM as the mutation kernel and adapt its sampling temperature to particle quality, measured by the same buffer log\-likelihood used for reweighting \(equivalently: average loss for regression, average accuracy for classification, computed over the buffer\)\. Particles above the median receive conservative refinement: the optimizer is called with low sampling temperature, encouraging minor parameter adjustments within the current functional form\. Particles below the median receive exploratory mutation: higher sampling temperature encourages structural changes such as switching from a linear to a piecewise rule\.

Resampling is triggered whenESS=1/∑kwk2\\mathrm\{ESS\}=1/\\sum\_\{k\}w\_\{k\}^\{2\}falls belowK/2K/2, using systematic resamplingDoucet et al\. \([2001](https://arxiv.org/html/2607.22961#bib.bib7)\)\. Algorithm[2](https://arxiv.org/html/2607.22961#alg2)in Appendix[B](https://arxiv.org/html/2607.22961#S2a)gives the full procedure\.

## 4Related Work

#### Verbalized machine learning\.

VMLXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)introduced the framework of using natural language as the model parameter space, with LLMs serving as both learner and optimizer\. The VML paper draws an informal analogy between optimizer\-temperature sampling and Bayesian neural networks \(Appendix I ofXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)\) but does not implement or evaluate a Bayesian extension\. We formalize this connection with MCMC and SMC algorithms over verbalized hypotheses, computing explicit posterior weights from a black\-box likelihood\.

#### Bayesian methods for LLM\-based prediction\.

VPP sits in a longer tradition of approximate Bayesian deep learning: variational treatments of network weightsBlundell et al\. \([2015](https://arxiv.org/html/2607.22961#bib.bib2)\); Kingma and Welling \([2014](https://arxiv.org/html/2607.22961#bib.bib16)\), dropout\-based posterior approximationsGal and Ghahramani \([2016](https://arxiv.org/html/2607.22961#bib.bib10)\), ensemble approachesLakshminarayanan et al\. \([2017](https://arxiv.org/html/2607.22961#bib.bib18)\), and the broader probabilistic perspectiveWilson and Izmailov \([2020](https://arxiv.org/html/2607.22961#bib.bib34)\), with Bayesian optimizationSnoek et al\. \([2012](https://arxiv.org/html/2607.22961#bib.bib29)\)as an orthogonal use of the same toolkit\. We differ from all of these in placing the posterior on verbal model hypotheses\. More recent LLM\-Bayesian works operate on different objects than VPP: BayesAgentHuang et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib15)\)reasons over task\-specific latent variables in hand\-designed graphs; LLM ProcessesRequeima et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib26)\)treat an LLM as a stochastic process for numerical prediction; conformal language modelingQuach et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib25)\)gives distribution\-free coverage over generations but needs token logits\. Closest in mechanism is BC\-LLMFeng et al\. \([2026](https://arxiv.org/html/2607.22961#bib.bib8)\), which uses Metropolis–Hastings with LLM proposals to search over interpretable concept sets for classification\. BC\-LLM searches over feature subsets within a fixed linear classifier, whereas VPP performs inference over complete verbal hypotheses that jointly specify both model structure and parameters\.

#### Prompt and hypothesis search\.

VML can be viewed as a special case of prompt optimization: OPROYang et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib36)\)uses LLMs to iteratively optimize prompts via in\-context examples; APEZhou et al\. \([2023](https://arxiv.org/html/2607.22961#bib.bib41)\)performs best\-of\-NNsampling; ProTeGiPryzant et al\. \([2023](https://arxiv.org/html/2607.22961#bib.bib23)\)applies gradient\-inspired text editing; TextGradYuksekgonul et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib39)\)formalises a text\-based analog of automatic differentiation; and EvoPromptGuo et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib12)\)uses evolutionary search over prompt candidates\. These methods search for a single best prompt without maintaining a posterior or performing data\-driven reweighting\. In the inductive\-reasoning literature, Hypothesis SearchWang et al\. \([2024b](https://arxiv.org/html/2607.22961#bib.bib31)\)and iterative hypothesis refinementQiu et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib24)\)generate and filter natural\-language hypotheses on the ARC benchmarkChollet \([2019](https://arxiv.org/html/2607.22961#bib.bib4)\), but neither maintains a posterior or performs Bayesian model averaging\. These build on the chain\-of\-thought tradition of eliciting verbal reasoning from LLMsWei et al\. \([2022](https://arxiv.org/html/2607.22961#bib.bib33)\); Kojima et al\. \([2022](https://arxiv.org/html/2607.22961#bib.bib17)\), and connect to per\-instance aggregation \(Self\-ConsistencyWang et al\. \([2023](https://arxiv.org/html/2607.22961#bib.bib32)\), Tree of ThoughtsYao et al\. \([2023a](https://arxiv.org/html/2607.22961#bib.bib37)\)\) and verbal self\-improvement of agents that interleave reasoning and acting \(ReActYao et al\. \([2023b](https://arxiv.org/html/2607.22961#bib.bib38)\), ReflexionShinn et al\. \([2023](https://arxiv.org/html/2607.22961#bib.bib27)\), VoyagerWang et al\. \([2024a](https://arxiv.org/html/2607.22961#bib.bib30)\), STaRZelikman et al\. \([2022](https://arxiv.org/html/2607.22961#bib.bib40)\)\)\. None of these maintain a posterior over a learned model that can be queried on new inputs\.

#### Equation discovery and symbolic regression\.

A separate line of work uses LLMs to propose mathematical expressions or code: LLM\-SRShojaee et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib28)\), LaSRGrayeli et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib11)\), In\-Context Symbolic RegressionMerler et al\. \([2024](https://arxiv.org/html/2607.22961#bib.bib19)\), and Codex\-style code generatorsChen et al\. \([2021](https://arxiv.org/html/2607.22961#bib.bib3)\); classical symbolic regression with deep priorsCranmer et al\. \([2020](https://arxiv.org/html/2607.22961#bib.bib5)\)likewise targets equations\. These methods restrict the hypothesis form\. VPP performs Bayesian inference over arbitrary natural\-language hypotheses—including verbal classification rules no equation\-discovery method can directly express\.

#### MCMC in discrete text spaces\.

CGMHMiao et al\. \([2019](https://arxiv.org/html/2607.22961#bib.bib21)\)uses Metropolis–Hastings for constrained sentence generation, and Quality\-Aware DecodingFernandes et al\. \([2022](https://arxiv.org/html/2607.22961#bib.bib9)\)applies MCMC\-style reranking to machine translation\. Both target text generation as the end product\. VPP differs in what the posterior is over: each particle is itself a model that will be queried on new inputs, not a piece of generated text\.

## 5Experiments

### 5\.1Setup

The evaluation is organized around four questions: \(i\) does VPP reduce the run\-to\-run variance of single\-run VML; \(ii\) does it eliminate the catastrophic single\-run failures that VML occasionally produces; \(iii\) does the particle ensemble yield a useful uncertainty signal; and \(iv\) which of VPP\-MH and VPP\-SMC should a practitioner choose? We evaluate VPP on seven benchmarks across three task types, all matching or extending the evaluation in the VML paperXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)\. The three*regression*tasks fit a linear function \(y=3​x\+4\+ϵy=3x\+4\+\\epsilon\), a polynomial \(y=3​x2\+x\+2\+ϵy=3x^\{2\}\+x\+2\+\\epsilon\), and a sinusoid \(y=sin⁡\(x\)\+2\+0\.01​ϵy=\\sin\(x\)\+2\+0\.01\\epsilon\), with inputs drawn uniformly over task\-specific ranges\. The two*classification*tasks are linearly separable two\-blobs and nonlinearly separable two\-circles\. The two*pattern\-discovery*tasks classify44\-integer sequences by hidden rules the model must discover from data: sum parity \(class11if the digit sum is even\) and contains\-zero \(class11if any element is zero\)\. The three task types exercise qualitatively different capabilities: regression tests numerical prediction under LLM arithmetic limitations, classification tests verbal rule learning and decision\-boundary discovery, and pattern discovery tests the ability to identify latent structure from purely symbolic input\.

We compare four methods\.VMLis a single run of the original algorithmXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)\.VML×\\times5runs five independent VML instances and is reported two ways: \(*ens*\) unweighted majority vote across the five final hypotheses for classification, or unweighted mean for regression; and \(*best*\) the oracle best\-of\-55, selecting the run with the lowest test loss\.VPP\-MHusesK=10K\{=\}10particles updated by Metropolis–Hastings \(Algorithm[1](https://arxiv.org/html/2607.22961#alg1)\)\.VPP\-SMCalso usesK=10K\{=\}10particles, with tempering scheduleβ:10→1\\beta:10\\to 1, evaluation bufferB=40B\{=\}40, and ESS resampling thresholdK/2K/2\.

To ensure fair comparison, we follow the VML experimental protocol exactly:100100training points and a6060\-point test set drawn from the same generator, batch size1010,22epochs \(2020optimization steps\), and Llama\-3\-70B\-Instruct served via vLLM\. The published VML prompt templates are used unchanged for both the learner and optimizer\. The initial hypothesisθ0\\theta\_\{0\}is model\-class\-agnostic for all tasks except two\-circles, where we include the prior “The decision boundary is a circle” to match the VML paper\. All runs are repeated over three random seeds; we report mean and standard deviation across seeds\.

### 5\.2Main Results

Table 1:Main results across all seven benchmarks\. Mean \(and standard deviation across three seeds\) of test MSE \(↓\\downarrow\) for regression and test accuracy \(↑\\uparrow\) for classification and pattern discovery\. Bold marks the best result per task\.Table[1](https://arxiv.org/html/2607.22961#S5.T1)reports mean performance across seeds\. VPP matches or exceeds single\-run VML on all seven benchmarks, and is best or tied\-best on six of the seven tasks even when the stronger VML×\\times5 baselines are included\. The single exception is two\-blobs, where VML×\\times5 reaches99\.2%99\.2\\%while both VPP variants reach98\.3%98\.3\\%\.

#### Regression: VPP reduces MSE by up to24×24\\times\.

VPP wins on all three regression tasks\. For linear regression, VPP\-MH achieves MSE1\.001\.00compared to single\-run VML’s8\.688\.68and the VML×\\times5 ensemble’s4\.544\.54\. On polynomial regression, VPP\-SMC reaches3\.103\.10—a24×24\\timesreduction relative to single\-run VML \(74\.7774\.77\), still better than the oracle VML×\\times5 best\-of\-55\(4\.444\.44\)\. On sine, VPP\-SMC reaches0\.400\.40, ahead of both VPP\-MH \(0\.930\.93\) and VML×\\times5 best\-of\-55\(0\.780\.78\)\.

#### Classification: VPP eliminates residual failures\.

On two\-circles, both VPP methods reach perfect mean accuracy \(100%100\\%\), improving over single\-run VML \(97\.5%97\.5\\%\) and the VML×\\times5 ensemble \(98\.3%98\.3\\%\); the remaining VML errors come from occasional verbal hypotheses that fail outright, which VPP eliminates\. On the easier two\-blobs task all methods are near saturation: VML×\\times5 reaches99\.2%99\.2\\%and both VPP variants reach98\.3%98\.3\\%, far above single\-run VML’s80\.0%80\.0\\%\. This is the only benchmark where VML×\\times5 slightly exceeds VPP; when nearly any reasonable hypothesis suffices, uniform ensembling is already competitive\.

#### Pattern discovery: VPP\-SMC reaches100%100\\%on both symbolic tasks\.

On sum parity, VPP\-SMC reaches100%100\\%accuracy on all three seeds, matching the oracle VML×\\times5 best\-of\-55while outperforming single\-run VML \(87\.2%87\.2\\%\), the VML×\\times5 ensemble \(67\.2%67\.2\\%\), and VPP\-MH \(90\.6%90\.6\\%\)\. The contrast between best\-of\-55and the unweighted ensemble is the cleanest demonstration of why posterior reweighting matters: multiple VML runs can contain the correct rule, yet naive averaging still fails when incorrect rules receive equal vote\. On contains\-zero, both VPP methods reach100%100\\%, while single\-run VML averages only78\.3%78\.3\\%and the VML×\\times5 ensemble reaches91\.1%91\.1\\%\.

### 5\.3Analysis

#### MH versus SMC\.

The two algorithms are complementary \(Table[2](https://arxiv.org/html/2607.22961#S5.T2)\)\. VPP\-MH wins on linear regression \(1\.001\.00vs\.2\.432\.43\), where the accept/reject mechanism preserves a strong hypothesis once found\. VPP\-SMC wins on polynomial, sine, and sum parity, and ties VPP\-MH on two\-blobs, two\-circles, and contains\-zero \(SMC:33wins, MH:11,33ties\)\. Because SMC is never far from MH and dominates the harder nonlinear and symbolic tasks, we recommend VPP\-SMC as a default\.

Table 2:VPP\-MH vs\. VPP\-SMC head\-to\-head on the seven benchmarks \(means from Table[1](https://arxiv.org/html/2607.22961#S5.T1)\)\.
#### Uncertainty quantification\.

VPP provides uncertainty estimates through particle disagreement, measured as the fraction of test points where the particle majority is not unanimous\. On tasks where VPP\-SMC reaches perfect accuracy—two\-circles, sum parity, and contains\-zero—the particles agree unanimously on every test point across all three seeds, reporting zero uncertain predictions\. Under VPP\-MH, uncertainty is much higher on sum parity \(the three seeds yield2929,3030, and33uncertain points out of6060\) and nonzero on contains\-zero in two of three seeds \(1414and22uncertain points\), reflecting the coexistence of correct and incorrect hypotheses in the unweighted MH ensemble\. Standard VML offers no such “I don’t know” signal\.

#### Robustness across runs\.

On a representative dataset \(Table[3](https://arxiv.org/html/2607.22961#S5.T3)\), five independent VML runs span MSE14\.514\.5–48\.348\.3on linear regression and accuracy61\.7%61\.7\\%–100%100\\%on contains\-zero, while both VPP variants land within0\.760\.76MSE of zero on linear and at100%100\\%on contains\-zero\. The particle ensemble prevents weak members from dominating the BMA prediction; a single VML run has no such buffer\.

Table 3:Robustness on a representative dataset: five independent VML runs vs\. VPP on the same data\.

### 5\.4Ablations

We isolate two design choices in VPP: the role of the Metropolis–Hastings accept/reject step, and the effect of the particle countKK\.

#### The accept/reject step is what makes VPP\-MH work\.

A natural question is whether VPP\-MH’s gains come from runningKKparticles in parallel, or from the accept/reject step that prevents a particle from regressing\. We compare standard VPP\-MH against an ablated variant \(*always\-accept*\) that retains theKK\-particle ensemble and BMA prediction but unconditionally accepts every optimizer proposal—making it equivalent to runningKKindependent VML chains and averaging\. Table[4](https://arxiv.org/html/2607.22961#S5.T4)shows the result\. Removing the rejection step drops sum\-parity accuracy by22\.222\.2pp \(97\.2%→75\.0%97\.2\\%\\to 75\.0\\%\) and more than doubles linear\-regression MSE \(0\.88→2\.090\.88\\to 2\.09\)\. Two\-circles is saturated at∼100%\\sim\\\!100\\%in both variants\. The accept/reject step is therefore the operative ingredient that distinguishes VPP\-MH from naive parallel VML\.

Table 4:Ablation: with vs\. without the Metropolis–Hastings accept/reject step\. Both variants useK=10K\{=\}10particles and BMA prediction;*always\-accept*unconditionally accepts every optimizer proposal\. Lower is better for linear regression \(MSE\), higher for the rest \(accuracy\)\.
#### Effect of particle countKK\.

SweepingK∈\{1,3,5,10\}K\\in\\\{1,3,5,10\\\}for VPP\-MH on three representative tasks \(full table in Appendix[A](https://arxiv.org/html/2607.22961#S1a)\): linear\-regression MSE monotonically improves from4\.534\.53atK=1K\{=\}1to0\.660\.66atK=10K\{=\}10\(7×7\\timesreduction\); two\-circles is saturated at∼98%\\sim\\\!98\\%across all settings; sum parity improves from64%64\\%atK=1K\{=\}1up to85%85\\%atK=5K\{=\}5and dips back to76%76\\%atK=10K\{=\}10\(within one standard deviation\), indicating mild particle degeneracy on harder symbolic tasks under our default schedule\. We useK=10K\{=\}10as a conservative default; an adaptive per\-task scheme is left for future work\.

### 5\.5Compute

VPP requires3​K3KLLM calls per step \(forward, propose, proposal\-eval per particle\) versus22for single VML, with comparable cost for VPP\-SMC\. On an H100 with Llama\-3\-70B\-Instruct via vLLM, mean wall\-clock ratios over single VML are14\.7×14\.7\\times\(VPP\-MH\) and15\.8×15\.8\\times\(VPP\-SMC\),1010–19×19\\timesdepending on task, and≈3×\\approx\\\!3\\timesover the budget\-matched VML×\\times5 baseline \(Appendix[E](https://arxiv.org/html/2607.22961#S5a)\)\. The overhead is justified when test\-error reduction or eliminating catastrophic single\-run failures is consequential\.

## 6Conclusion and Limitations

VPP applies approximate Bayesian inference to natural\-language hypotheses via two black\-box algorithms \(VPP\-MH, VPP\-SMC\), yielding uncertainty quantification, robustness, and prediction gains over single\-hypothesis VML; the MH accept/reject step is the operative ingredient, and because each particle is human\-readable the posterior is itself inspectable\. In our benchmark suite, VPP\-SMC reaches perfect mean accuracy on three of the four classification and rule\-discovery tasks where single\-run VML stalls, and reduces polynomial\-regression error by more than an order of magnitude; per\-particle inspection lets a reader see both the rule the data ruled in and the alternatives the data ruled out\.

*Limitations*: VPP inherits VML’s arithmetic limitation on regression and the MH approximations flagged in Section[3\.2](https://arxiv.org/html/2607.22961#S3.SS2), which together leave the method best suited to classification and rule discovery rather than precise numerical prediction; experiments use a single LLM on small synthetic tasks, with broader transfer to other LLMs and to large real\-world datasets left open\. TheKK\-ablation \(Appendix[A](https://arxiv.org/html/2607.22961#S1a)\) also reveals mild particle degeneracy atK=10K\{=\}10on harder symbolic tasks, suggesting an adaptiveKKand tempering schedule as a pragmatic next step\.

## References

- Blei et al\. \(2017\)David M\. Blei, Alp Kucukelbir, and Jon D\. McAuliffe\.Variational inference: A review for statisticians\.*Journal of the American Statistical Association*, 112\(518\):859–877, 2017\.
- Blundell et al\. \(2015\)Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra\.Weight uncertainty in neural network\.In*Proceedings of the 32nd International Conference on Machine Learning*, volume 37, pages 1613–1622\. PMLR, 2015\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert\-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N\. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- Chollet \(2019\)Franccois Chollet\.On the measure of intelligence\.*arXiv preprint arXiv:1911\.01547*, 2019\.
- Cranmer et al\. \(2020\)Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho\.Discovering symbolic models from deep learning with inductive biases\.In*Advances in Neural Information Processing Systems*, volume 33, pages 17429–17442\. Curran Associates, Inc\., 2020\.
- Del Moral et al\. \(2006\)Pierre Del Moral, Arnaud Doucet, and Ajay Jasra\.Sequential Monte Carlo samplers\.*Journal of the Royal Statistical Society: Series B \(Statistical Methodology\)*, 68\(3\):411–436, 2006\.
- Doucet et al\. \(2001\)Arnaud Doucet, Nando de Freitas, and Neil Gordon, editors\.*Sequential Monte Carlo Methods in Practice*\.Springer, 2001\.
- Feng et al\. \(2026\)Jean Feng, Avni Kothari, Lucas Zier, Chandan Singh, and Yan Shuo Tan\.Bayesian concept bottleneck models with LLM priors\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2026\.URL[https://openreview\.net/forum?id=oXSkzIXgbk](https://openreview.net/forum?id=oXSkzIXgbk)\.
- Fernandes et al\. \(2022\)Patrick Fernandes, António Farinhas, Ricardo Rei, José G\. C\. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins\.Quality\-aware decoding for neural machine translation\.In*Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1396–1412\. Association for Computational Linguistics, 2022\.
- Gal and Ghahramani \(2016\)Yarin Gal and Zoubin Ghahramani\.Dropout as a Bayesian approximation: Representing model uncertainty in deep learning\.In*Proceedings of The 33rd International Conference on Machine Learning*, volume 48, pages 1050–1059\. PMLR, 2016\.
- Grayeli et al\. \(2024\)Arya Grayeli, Atharva Sehgal, Omar Costilla\-Reyes, Miles Cranmer, and Swarat Chaudhuri\.Symbolic regression with a learned concept library\.In*Advances in Neural Information Processing Systems*, volume 37, pages 44678–44709\. Curran Associates, Inc\., 2024\.
- Guo et al\. \(2024\)Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang\.Connecting large language models with evolutionary algorithms yields powerful prompt optimizers\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=ZG3RaNIsO8](https://openreview.net/forum?id=ZG3RaNIsO8)\.
- Hastings \(1970\)W\. K\. Hastings\.Monte Carlo sampling methods using Markov chains and their applications\.*Biometrika*, 57\(1\):97–109, 1970\.
- Hoffman and Gelman \(2014\)Matthew D\. Hoffman and Andrew Gelman\.The No\-U\-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo\.*Journal of Machine Learning Research*, 15\(47\):1593–1623, 2014\.
- Huang et al\. \(2025\)Hengguan Huang, Xing Shen, Guang\-Yuan Hao, Songtao Wang, Lingfa Meng, Dianbo Liu, David Alejandro Duchene, Hao Wang, and Samir Bhatt\.BayesAgent: Bayesian agentic reasoning under uncertainty via verbalized probabilistic graphical modeling\.In*Proceedings of the 40th AAAI Conference on Artificial Intelligence \(AAAI\-26\)*, volume 40, pages 21939–21947, 2025\.
- Kingma and Welling \(2014\)Diederik P\. Kingma and Max Welling\.Auto\-encoding variational Bayes\.In*International Conference on Learning Representations \(ICLR\)*, 2014\.URL[https://arxiv\.org/abs/1312\.6114](https://arxiv.org/abs/1312.6114)\.
- Kojima et al\. \(2022\)Takeshi Kojima, Shixiang \(Shane\) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa\.Large language models are zero\-shot reasoners\.In*Advances in Neural Information Processing Systems*, volume 35, pages 22199–22213\. Curran Associates, Inc\., 2022\.
- Lakshminarayanan et al\. \(2017\)Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell\.Simple and scalable predictive uncertainty estimation using deep ensembles\.In*Advances in Neural Information Processing Systems*, volume 30\. Curran Associates, Inc\., 2017\.
- Merler et al\. \(2024\)Matteo Merler, Katsiaryna Haitsiukevich, Nicola Dainese, and Pekka Marttinen\.In\-context symbolic regression: Leveraging large language models for function discovery\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\)*, pages 427–444, 2024\.
- Metropolis et al\. \(1953\)Nicholas Metropolis, Arianna W\. Rosenbluth, Marshall N\. Rosenbluth, Augusta H\. Teller, and Edward Teller\.Equation of state calculations by fast computing machines\.*The Journal of Chemical Physics*, 21\(6\):1087–1092, 1953\.
- Miao et al\. \(2019\)Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li\.CGMH: Constrained sentence generation by Metropolis\-Hastings sampling\.In*Proceedings of the Thirty\-Third AAAI Conference on Artificial Intelligence \(AAAI\-19\)*, volume 33, pages 6834–6842, 2019\.
- Neal \(2011\)Radford M\. Neal\.MCMC using Hamiltonian dynamics\.In*Handbook of Markov Chain Monte Carlo*, pages 113–162\. Chapman and Hall/CRC, 2011\.
- Pryzant et al\. \(2023\)Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng\.Automatic prompt optimization with “gradient descent” and beam search\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7957–7968\. Association for Computational Linguistics, 2023\.
- Qiu et al\. \(2024\)Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren\.Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=bNt7oajl2a](https://openreview.net/forum?id=bNt7oajl2a)\.
- Quach et al\. \(2024\)Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S\. Jaakkola, and Regina Barzilay\.Conformal language modeling\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=pzUhfQ74c5](https://openreview.net/forum?id=pzUhfQ74c5)\.
- Requeima et al\. \(2024\)James Requeima, John Bronskill, Dami Choi, Richard E\. Turner, and David Duvenaud\.LLM processes: Numerical predictive distributions conditioned on natural language\.In*Advances in Neural Information Processing Systems*, volume 37, pages 109609–109671\. Curran Associates Inc\., 2024\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 36, pages 8634–8652\. Curran Associates, Inc\., 2023\.
- Shojaee et al\. \(2025\)Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K\. Reddy\.LLM\-SR: Scientific equation discovery via programming with large language models\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=m2nmp8P5in](https://openreview.net/forum?id=m2nmp8P5in)\.
- Snoek et al\. \(2012\)Jasper Snoek, Hugo Larochelle, and Ryan P\. Adams\.Practical Bayesian optimization of machine learning algorithms\.In*Advances in Neural Information Processing Systems*, volume 25\. Curran Associates, Inc\., 2012\.
- Wang et al\. \(2024a\)Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\.Voyager: An open\-ended embodied agent with large language models\.*Transactions on Machine Learning Research*, 2024a\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=ehfRiF0R3a](https://openreview.net/forum?id=ehfRiF0R3a)\.
- Wang et al\. \(2024b\)Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman\.Hypothesis search: Inductive reasoning with language models\.In*The Twelfth International Conference on Learning Representations*, 2024b\.URL[https://openreview\.net/forum?id=G7UtIGQmjm](https://openreview.net/forum?id=G7UtIGQmjm)\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\.Self\-consistency improves chain of thought reasoning in language models\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=1PL1NIMMrw](https://openreview.net/forum?id=1PL1NIMMrw)\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou\.Chain\-of\-thought prompting elicits reasoning in large language models\.In*Advances in Neural Information Processing Systems*, volume 35, pages 24824–24837\. Curran Associates, Inc\., 2022\.
- Wilson and Izmailov \(2020\)Andrew G Wilson and Pavel Izmailov\.Bayesian deep learning and a probabilistic perspective of generalization\.In*Advances in Neural Information Processing Systems*, volume 33, pages 4697–4708\. Curran Associates, Inc\., 2020\.
- Xiao et al\. \(2025\)Tim Z\. Xiao, Robert Bamler, Bernhard Schölkopf, and Weiyang Liu\.Verbalized machine learning: Revisiting machine learning with language models\.*Transactions on Machine Learning Research*, 2025\.URL[https://openreview\.net/forum?id=k3Ab6RuJE9](https://openreview.net/forum?id=k3Ab6RuJE9)\.
- Yang et al\. \(2024\)Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen\.Large language models as optimizers\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=Bb4VGOWELI](https://openreview.net/forum?id=Bb4VGOWELI)\.
- Yao et al\. \(2023a\)Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L\. Griffiths, Yuan Cao, and Karthik Narasimhan\.Tree of thoughts: Deliberate problem solving with large language models\.In*Advances in Neural Information Processing Systems*, volume 36, pages 11809–11822\. Curran Associates, Inc\., 2023a\.
- Yao et al\. \(2023b\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\.ReAct: Synergizing reasoning and acting in language models\.In*The Eleventh International Conference on Learning Representations*, 2023b\.URL[https://openreview\.net/forum?id=WE\_vluYUL\-X](https://openreview.net/forum?id=WE_vluYUL-X)\.
- Yuksekgonul et al\. \(2024\)Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou\.TextGrad: Automatic “differentiation” via text\.*arXiv preprint arXiv:2406\.07496*, 2024\.
- Zelikman et al\. \(2022\)Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D\. Goodman\.STaR: Bootstrapping reasoning with reasoning\.In*Advances in Neural Information Processing Systems*, volume 35, pages 15476–15488\. Curran Associates, Inc\., 2022\.
- Zhou et al\. \(2023\)Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba\.Large language models are human\-level prompt engineers\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=92gvk82DE\-](https://openreview.net/forum?id=92gvk82DE-)\.

## AParticle\-Count Ablation

Section[5\.4](https://arxiv.org/html/2607.22961#S5.SS4)reports the headline numbers for theK∈\{1,3,5,10\}K\\in\\\{1,3,5,10\\\}sweep on three representative tasks\. Table[5](https://arxiv.org/html/2607.22961#S1.T5)gives the full ablation grid \(mean and standard deviation across three seeds\) for completeness\. The two qualitative trends discussed in the main text are visible: linear\-regression MSE improves monotonically withKK, and sum parity exhibits a non\-monotonic pattern with a peak atK=5K\{=\}5\. Two\-circles remains near saturation for every particle count\.

Table 5:Ablation over particle countKKfor VPP\-MH \(mean±\\pmstd across three seeds\)\. Summarised in Section[5\.4](https://arxiv.org/html/2607.22961#S5.SS4)\.
## BVPP\-SMC Pseudocode

Algorithm[2](https://arxiv.org/html/2607.22961#alg2)gives the complete VPP\-SMC procedure described prosaically in Section[3\.3](https://arxiv.org/html/2607.22961#S3.SS3)\. The structure parallels Algorithm[1](https://arxiv.org/html/2607.22961#alg1)\(VPP\-MH\) with three additions: a rolling buffer that accumulates the lastBBdata points \(line 3\), a tempered importance\-weight reweighting step \(line 7\), and an ESS\-triggered systematic resample \(line 8\)\. The mutation step on line 13 reuses the same optimizer LLM as VPP\-MH but with a temperature that depends on the particle’s buffer log\-likelihood relative to the population median \(line 12\), implementing the adaptive\-mutation strategy of Section[3\.3](https://arxiv.org/html/2607.22961#S3.SS3)\.

Algorithm 2VPP\-SMC: Verbalized Particle Posterior via Sequential Monte Carlo0:Training data

𝒟\\mathcal\{D\}, particles

KK, epochs

TT, batch size

MM, buffer size

BB, tempering schedule

\{βt\}t=0T\\\{\\beta\_\{t\}\\\}\_\{t=0\}^\{T\}, ESS threshold

K/2K/2
1:Initialize

θ0\(k\)←fopt​\(θinit,seed batch\)\\theta\_\{0\}^\{\(k\)\}\\leftarrow f\_\{\\text\{opt\}\}\(\\theta\_\{\\text\{init\}\},\\text\{seed batch\}\)for

k=1,…,Kk=1,\\ldots,Kwith varying temperature; set

w0\(k\)←1/Kw\_\{0\}^\{\(k\)\}\\leftarrow 1/K; initialize empty buffer

𝒟buffer\\mathcal\{D\}\_\{\\text\{buffer\}\}
2:for

t=1,…,Tt=1,\\ldots,Tdo

3:Sample mini\-batch

\{\(xm,ym\)\}m=1M\\\{\(x\_\{m\},y\_\{m\}\)\\\}\_\{m=1\}^\{M\}; append to

𝒟buffer\\mathcal\{D\}\_\{\\text\{buffer\}\}, keep the last

BBpoints

4:for

k=1,…,Kk=1,\\ldots,Kdo

5:Evaluate

log⁡p​\(𝒟buffer\|θt−1\(k\)\)\\log p\(\\mathcal\{D\}\_\{\\text\{buffer\}\}\|\\theta\_\{t\-1\}^\{\(k\)\}\)// Buffer likelihood

6:endfor

7:Compute tempered weights

wt\(k\)w\_\{t\}^\{\(k\)\}via Eq\.[10](https://arxiv.org/html/2607.22961#S3.E10)with temperature

βt\\beta\_\{t\}; normalize

8:if

ESS​\(wt\)<K/2\\mathrm\{ESS\}\(w\_\{t\}\)<K/2then

9:Systematic\-resample the particles according to

wt\(k\)w\_\{t\}^\{\(k\)\}; reset

wt\(k\)←1/Kw\_\{t\}^\{\(k\)\}\\leftarrow 1/K
10:endif

11:for

k=1,…,Kk=1,\\ldots,Kdo

12:Set optimizer temperature based on

θt−1\(k\)\\theta\_\{t\-1\}^\{\(k\)\}’s buffer log\-likelihood \(above/below median\)// Adaptive mutation

13:

θt\(k\)←fopt​\(θt−1\(k\),\{xm,y^m,ym\}\)\\theta\_\{t\}^\{\(k\)\}\\leftarrow f\_\{\\text\{opt\}\}\(\\theta\_\{t\-1\}^\{\(k\)\},\\\{x\_\{m\},\\hat\{y\}\_\{m\},y\_\{m\}\\\}\)
14:endfor

15:endfor

16:Predict:

y^​\(x∗\)=∑k=1KwT\(k\)​f​\(x∗;θT\(k\)\)\\hat\{y\}\(x^\{\*\}\)=\\sum\_\{k=1\}^\{K\}w\_\{T\}^\{\(k\)\}f\(x^\{\*\};\\theta\_\{T\}^\{\(k\)\}\)\(regression\) or weighted majority vote \(classification\)// BMA

## CPrompt Templates

We use the published VML prompt templates fromXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)without modification, both for the learner LLM \(ff\) and the optimizer LLM \(foptf\_\{\\text\{opt\}\}\)\. The learner template instructs the LLM to act as a functionf​\(x;θ\)f\(x;\\theta\), prompted byθ\\thetaand queried on inputxx; the optimizer template feeds the current hypothesis, a mini\-batch of inputs and targets, and the learner’s predictions, and asks for a revised hypothesis\. We refer the reader to Appendix B ofXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)for the verbatim text\.

For the SMC mutation kernel \(Section[3\.3](https://arxiv.org/html/2607.22961#S3.SS3)\), we vary the optimizer’s sampling temperature based on each particle’s buffer log\-likelihood: particles above the median receive temperature0\.30\.3\(conservative refinement, encouraging local edits to the existing hypothesis\), and particles below the median receive temperature1\.01\.0\(exploratory mutation, allowing structural rewrites such as switching from a linear to a piecewise rule\)\. All other prompting choices are inherited from VML\.

## DHyperparameters

Table[6](https://arxiv.org/html/2607.22961#S4.T6)lists every hyperparameter used in the main experiments\. The values forKK,τ\\tau,ϵ\\epsilon,BB, the SMC tempering schedule, and the ESS threshold are also stated where they first appear in Sections[3](https://arxiv.org/html/2607.22961#S3)–[5](https://arxiv.org/html/2607.22961#S5); the table consolidates them for reproducibility\. Choices not informed by ablation \(e\.g\.τ=1\\tau\{=\}1,ϵ=0\.05\\epsilon\{=\}0\.05\) match the defaults fromXiao et al\. \([2025](https://arxiv.org/html/2607.22961#bib.bib35)\)\.

Table 6:Full hyperparameter list\.
## EPer\-Task Wall\-Clock Time

Section[5\.5](https://arxiv.org/html/2607.22961#S5.SS5)reports the mean wall\-clock ratios over single VML; Table[7](https://arxiv.org/html/2607.22961#S5.T7)gives the per\-task breakdown that those summary numbers were computed from\. All measurements are wall\-clock seconds for a complete training run \(100100data points,2020optimization steps,K=10K\{=\}10for VPP variants,55independent runs for VML×\\times5\), averaged over three seeds\.

Table 7:Wall\-clock time per run \(mean across three seeds, in seconds\) on a single H100 with Llama\-3\-70B\-Instruct via vLLM\. Summarised in Section[5\.5](https://arxiv.org/html/2607.22961#S5.SS5)\.The per\-task ratios are reasonably uniform across the seven tasks, ranging from10×10\\times\(sine\) to18×18\\times\(two\-circles\) for VPP\-MH versus single VML, and from11×11\\timesto19×19\\timesfor VPP\-SMC\. The variance is dominated by tokenisation and prompt\-length differences across tasks rather than by VPP\-specific overhead\.

## FFull Per\-Seed Results

Table[8](https://arxiv.org/html/2607.22961#S6.T8)expands Table[1](https://arxiv.org/html/2607.22961#S5.T1)of the main paper to per\-seed values, with one row per \(task, seed\) and one column per method\. Reading down the rows for a given task gives the variance across seeds; reading across the columns gives the variance across methods\. The standard deviations reported in Table[1](https://arxiv.org/html/2607.22961#S5.T1)are the spreads of the three seed\-rows for each task\.

A few patterns are visible only at the per\-seed granularity:

- •Single\-VML failures cluster on specific seeds\.On contains\-zero, one seed produces a36\.7%36\.7\\%run while the other two produce100%100\\%and98\.3%98\.3\\%\. The same dataset is therefore solvable; the failure is in the optimization trajectory, not in the data\.
- •VPP variants are stable on classification and rule\-discovery tasks\.On contains\-zero and two\-circles, both VPP\-MH and VPP\-SMC report100%100\\%on every seed; VPP\-SMC additionally reaches100%100\\%on every seed of sum parity\. Individual VML runs on the same datasets vary widely\.
- •VML×\\times5 ensemble is sensitive to vote distribution\.On sum parity, the unweighted ensemble produces80%80\\%,61\.7%61\.7\\%,60%60\\%across seeds while the oracle best\-of\-55produces100%100\\%on all three: the correct rule is present in the population but is outvoted whenever a majority of runs converge to a different \(incorrect\) hypothesis\.

Table 8:Per\-seed test metrics for all77tasks×\\times55methods×\\times33seeds\. Seeds are anonymised as Run 1/2/3\.
## GVerbatim Particle Examples

The interpretability claim of Section[3](https://arxiv.org/html/2607.22961#S3)—each particle is a complete, human\-readable hypothesis—is best illustrated by quoting the actual text VPP recovers\. The examples below are top\-weighted particles from VPP\-SMC \(K=10K\{=\}10\) on three tasks where the method achieves100%100\\%test accuracy\. The text is reproduced verbatim from the saved particle files; only mathematical notation has been re\-typeset\.

#### Sum parity \(test accuracy100%100\\%, top\-particle weight0\.1110\.111\)\.

> “The input is a sequence of44integers\. You need to output the class label: an integer in the set\{0, 1\}\. The class label is11if the sum of the integers in the sequence is even, and0otherwise\.”

The posterior collapses to44unique paraphrases of this rule across1010particles, each with the same likelihood; the BMA prediction inherits perfect accuracy from any of them\. The four paraphrases differ only in surface form \(whether “sum is even” is restated as “digit sum is even,” whether the variable name appears, etc\.\)\.

#### Two circles \(test accuracy100%100\\%, top\-particle weight0\.1250\.125\)\.

> “You are designed to do binary classification\. The input is a 2D point\(x1,x2\)\(x\_\{1\},x\_\{2\}\); you need to output the class label, i\.e\., an integer in the set\{0, 1\}\. The decision boundary is a circle centered at the origin\(0,0\)\(0,0\)with a radius of approximately0\.6150\.615\. If the distance from the origin to the input point, calculated asx12\+x22\\sqrt\{x\_\{1\}^\{2\}\+x\_\{2\}^\{2\}\}, is less than or equal to the radius, output11; otherwise, output0\.”

The numerical radius the particle settles on \(0\.6150\.615here,0\.530\.53in another high\-weight particle\) lies inside the true decision boundary; the ensemble averages out the residual uncertainty over the exact radius\. Across the1010surviving particles, the radius estimates span0\.450\.45–0\.690\.69, all of which classify the test set perfectly because the geometry of the task is generous to slight under\-estimates of the radius\.

#### Contains zero \(test accuracy100%100\\%, top\-particle weight0\.1110\.111\)\.

> “The model takes a sequence of44integers as input and outputs a class label:11if the sequence contains at least one0;0if the sequence does not contain any0s\.”

The hypothesis converges quickly \(within the first few SMC steps\) and remains stable across mutations\.

#### Failure\-mode comparison\.

On the same dataset \(contains\-zero\), single\-run VML ends with the following hypothesis at36\.7%36\.7\\%test accuracy:

> “… The model takes a sequence of44integers as input and outputs a class label, which is11if the sequence contains exactly two identical integers that are not adjacent and are separated by at least one different integer, and0otherwise\.”

This is a structurally plausible but completely wrong rule—it is the kind of pattern a careful human might propose after looking at a small number of examples and over\-fitting to the surface structure of those examples\. VML has no mechanism to recover from such a hypothesis once the optimizer commits to it: every subsequent revision starts from this incorrect rule and either preserves the error or, occasionally, drifts in unrelated directions\. VPP\-MH and VPP\-SMC on the same dataset both reach100%100\\%accuracy because at least one particle finds the correct rule, and the accept/reject step \(VPP\-MH\) or likelihood reweighting \(VPP\-SMC\) prevents the wrong rule from dominating the BMA prediction\.

## HSequential Monte Carlo Dynamics

VPP\-SMC’s training trajectory is non\-monotonic: rather than smoothly converging to a single posterior mode, the effective sample size oscillates as particles are reweighted, resampled, and mutated\. Table[9](https://arxiv.org/html/2607.22961#S8.T9)reports the per\-step ESS on six tasks for a single representative seed, with the thresholdK/2=5K/2=5marking when resampling is triggered\.

Two patterns emerge\. On classification and pattern\-discovery tasks, the ESS rapidly stabilises near the maximum value ofK=10K=10once the population has identified the correct rule \(sum parity from step77onward, contains\-zero from step55\)\. On regression tasks, the ESS oscillates throughout training because no single arithmetic hypothesis dominates: the population continues to mix between roughly equally good arithmetic refinements until the end\. The minimum ESS is reached at different steps for different tasks \(sum parity at step33, contains\-zero at step44, polynomial regression at the final step\), which the resampling\-on\-threshold mechanism handles automatically by triggering systematic resampling whenever the threshold is crossed\.

Table 9:Effective sample size of VPP\-SMC over the2020training steps for six tasks \(single representative seed\)\. The ESS threshold for resampling isK/2=5K/2=5\.#### Diversity at convergence\.

The number of unique verbal hypotheses surviving in the final particle set varies with task difficulty and across seeds\. On sum parity, the population converges to44–99unique paraphrases of the correct rule \(typically a small number on simpler seeds, more on harder ones\); on contains\-zero,55–88unique hypotheses survive; on the regression tasks,88–1010unique hypotheses are present at the end, reflecting a more multimodal posterior over arithmetic rules of comparable likelihood\. Diversity collapse is therefore not necessary for high accuracy: the posterior reaches the right answer either by concentrating on a single rule \(for tasks with a clean symbolic structure\) or by averaging over an ensemble of equivalently good ones \(for tasks where the LLM’s arithmetic limitations make exact convergence impossible\)\.

## IPer\-Task Variance of Single VML Runs

Table[3](https://arxiv.org/html/2607.22961#S5.T3)in the main paper showed the five individual VML runs that constitute VML×\\times5 on a representative dataset for three tasks\. Table[10](https://arxiv.org/html/2607.22961#S9.T10)extends this to all seven tasks and all three seeds, exposing the run\-to\-run variance across the full benchmark suite\. The pattern is consistent: regression tasks exhibit large per\-run variance \(factor of∼10\\sim\\\!10–30×30\\timesbetween best and worst individual VML run on the same dataset\), and classification and rule\-discovery tasks show frequent catastrophic failures even when most runs succeed \(e\.g\., a0%0\\%\-accuracy individual VML run on sum parity, multiple runs at≤50%\\leq 50\\%for sum parity on another seed, and two runs at32%32\\%for contains\-zero\)\. Across all7×3=217\\times 3=21task\-seed combinations, the unweighted VML×\\times5 ensemble strictly underperforms the oracle best\-of\-55on1111and ties on another77; only on three regression cases does prediction\-averaging beat the single best run\. VPP\-MH and VPP\-SMC \(Table[8](https://arxiv.org/html/2607.22961#S6.T8)\) consistently land at the upper end of the spread\.

Table 10:Individual test metrics of the55independent VML runs that constitute VML×\\times5, for all77tasks×\\times33seeds \(anonymised as Run 1/2/3\)\. Format:55comma\-separated values per row\. Lower is better for MSE \(regression\); higher for accuracy \(classification, pattern discovery\)\.