Recipes for Steering and Scaling LLMs via Sampling

# Recipes for Steering and Scaling LLMs via Sampling
Source: [https://arxiv.org/html/2608.26120](https://arxiv.org/html/2608.26120)
Jiajun He1 jh2383@cam\.ac\.uk&Zongyu Guo2 zongyuguo@microsoft\.comJosé Miguel Hernández\-Lobato1 jmh233@cam\.ac\.uk&Yuanqi Du3,4 yuanqidu@microsoft\.com

###### Abstract

Large Language Models \(LLMs\) are probabilistic models, typically defined by an autoregressive factorization\. While recent work has begun to study richer target distributions beyond the base model, the sampling strategies remain highly inefficient\. In this paper, we present a flexible and theoretically grounded framework for steering and scaling autoregressive LLMs with sampling\. Within this framework, we describe two algorithms—one based on Sequential Monte Carlo \(SMC\) and one based on Replica Exchange \(RE\)—that steer generation toward*powering*,*product*or*tilting*of the base model distribution\. We illustrate this framework through scaling the generation quality of LLMs without external supervision or reward models\. Experimental results demonstrate our methods scale more favorably than Best\-of\-N and standard MCMC baselines\. Overall, this paper offers a systematic recipe for probabilistic inference with LLMs via sampling\.

## 1 Introduction

Large language models \(LLMs\) have achieved remarkable success across language understanding, reasoning, code generation, and scientific problem solving\(Wei et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib34); Achiam et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib1); Romera\-Paredes et al\.,[2024](https://arxiv.org/html/2608.26120#bib.bib27); Shojaee et al\.,[2024](https://arxiv.org/html/2608.26120#bib.bib28); Du et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib7)\)\. At their core, LLMs are probabilistic models: they define distributions over sequences through an autoregressive factorization, where each token is sampled conditionally on the previously generated prefix\.

Respecting the probabilistic nature of the LLMs, massive progress have been made to improve their inference process, from faster inference with speculative decoding to fine\-tuned model inference with reinforcement learning and controlled decoding for targeted generation\(Leviathan et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib18); Rafailov et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib25); Mudgal et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib23)\)\. In addition to asymptotically accurate inference algorithms, numerous methods have simply leveraged the stochasticity of LLMs to scale their inference accuracy with more compute, including evolutionary algorithm, Monte Carlo tree search, Best\-of\-N decoding, confidence guidance and more\(Wang et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib33),[2025](https://arxiv.org/html/2608.26120#bib.bib32); Hao et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib9); Fu et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib8)\)\.

Alternatively, the fundamental mechanism of LLMs has been studied from the probabilistic inference perspective\. One notable example is the sharpening mechanism observed that fine\-tuning progressively sharpens the base distribution from a pre\-trained LLM\(Huang et al\.,[2024](https://arxiv.org/html/2608.26120#bib.bib13); Wu et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib35); Liu et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib21)\)\. Recently, Markov chain Monte Carlo \(MCMC\) methods have also been developed to sample from the sharpened base distribution\(Karan and Du,[2025](https://arxiv.org/html/2608.26120#bib.bib14)\)\. Furthermore, the coverage of the base LLM has also been increasingly studied and found to be crucial for the success of post\-training procedure\(Chen et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib2)\)\.

In this paper, we systematically study sampling targets and algorithms in LLM scaling and steering\. We first introduce a class of flexible target distributions, including powered, product and tilted base distributions, to sample\. Then, we formulate the autoregressive inference process as a time\-dependent transport that bridges the initial empty sentence to the distribution of sampled complete sentences\. We establish a generalized Bayes’ rule behind LLMs\. From there, we introduce principled sequential Monte Carlo and replica exchange algorithms for efficient inference\. We demonstrate the flexibility of the framework by enhancing the capability of the base LLM by purely altering its inference process without any external reward on mathematical reasoning problems\.

## 2 Background

### 2\.1 Autoregressive Large Language Models

Let𝒱\\mathcal\{V\}be the vocabulary set and𝒱~:=𝒱∪\{∅\}\\widetilde\{\\mathcal\{V\}\}:=\\mathcal\{V\}\\cup\\\{\\varnothing\\\}\. Define an extended state space𝒳N\\mathcal\{X\}^\{N\}for variable\-length sequences of lengthl≤Nl\\leq Npadded with empty tokens𝒳N:=\{x0:N∈𝒱~N:∃l∈\{0,…,N\},s\.t\.​x0:l∈𝒱l,xl\+1:N=∅\}\\mathcal\{X\}^\{N\}:=\\left\\\{x\_\{0:N\}\\in\\widetilde\{\\mathcal\{V\}\}^\{N\}:\\exists\\,l\\in\\\{0,\.\.\.,N\\\},\\text\{s\.t\.\}\\,x\_\{0:l\}\\in\\mathcal\{V\}^\{l\},x\_\{l\+1:N\}=\\varnothing\\right\\\}\. An autoregressive LLM parameterizes a factorized joint distribution over discrete tokens

p​\(x0:N\)=p​\(x0\)​∏i=1Np​\(xi\|x<i\)\\displaystyle p\(x\_\{0:N\}\)=p\(x\_\{0\}\)\\prod\_\{i=1\}^\{N\}p\(x\_\{i\}\|x\_\{<i\}\)\(1\)
wherexi∈𝒱~x\_\{i\}\\in\\widetilde\{\\mathcal\{V\}\}with the following absorbing constraint:

p​\(xi=∅\|x1:i−1\)=1,when​xi−1=∅\\displaystyle p\(x\_\{i\}=\\varnothing\|x\_\{1:i\-1\}\)=1,\\qquad\\text\{when\}\\;\\;x\_\{i\-1\}=\\varnothing\(2\)Sampling from LLMs is straightforward: one first drawsx0∼p​\(x0\)x\_\{0\}\\sim p\(x\_\{0\}\), and then sequentially samplesxix\_\{i\}conditional on all previous prefix tokens until the empty token is sampled\.

### 2\.2 Sampling From Intractable Target Distributions

Assume we want to draw samples from some intractable target distributionπ\\pi\.*What should we do?*

There are two commonly used strategies\. One strategy aims to find a tractable distributionqqthat coversπ\\pi\. We then draw samples fromqqand select those that look plausible underπ\\pi\. Another strategy aims to exploreπ\\pistep by step: starting from an initial guess, we repeatedly propose a small move in a new direction\. If the direction looks promising, we take the step; otherwise, we stay where we are\. Two widely used algorithms for these two strategies are importance sampling and Markov Chain Monte Carlo \(MCMC\), which we briefly explain below:

#### Importance Sampling

To sample fromπ\\pi, we first drawNNsamples from the tractable proposal distributionqq:x\(1\),…,x\(N\)∼q\.x^\{\(1\)\},\\ldots,x^\{\(N\)\}\\sim q\.Then, we calculate the*importance weight*

w​\(x\(n\)\)=π​\(x\(n\)\)q​\(x\(n\)\)\.\\displaystyle w\(x^\{\(n\)\}\)=\\frac\{\\pi\(x^\{\(n\)\}\)\}\{q\(x^\{\(n\)\}\)\}\.\(3\)Samples with larger weights are more plausible underπ\\pi, while samples with smaller weights are less useful\. We can therefore either reweight the samples or resample them according to the weights

w¯​\(x\(n\)\)=w​\(x\(n\)\)∑m=1Nw​\(x\(m\)\)\.\\displaystyle\\bar\{w\}\(x^\{\(n\)\}\)=\\frac\{w\(x^\{\(n\)\}\)\}\{\\sum\_\{m=1\}^\{N\}w\(x^\{\(m\)\}\)\}\.\(4\)This gives an approximate weighted sample from the target distributionπ\\pi\. However, importance sampling can fail whenqqdoes not coverπ\\piwell, because most samples may receive negligible weight\.

#### Markov Chain Monte Carlo

MCMC takes a different approach\. Instead of drawing independent samples from a proposalqq, it constructs a Markov chain

x0→x1→⋯→xN\\displaystyle x\_\{0\}\\to x\_\{1\}\\to\\cdots\\to x\_\{N\}\(5\)whose stationary distribution isπ\\pi\. Starting from an initial pointxtx\_\{t\}, we propose a local movext→x′x\_\{t\}\\to x^\{\\prime\}with the proposal conditional distribution with densityq​\(x′\|xt\)q\(x^\{\\prime\}\|x\_\{t\}\)\. If the proposed point is more plausible underπ\\pi, we tend to accept it; otherwise, we may reject it and remain atxtx\_\{t\}\. For example, in the Metropolis\-Hastings algorithm, the move is accepted with probability

a​\(xt,x′\)=min⁡\{1,π​\(x′\)​q​\(xt\|x′\)π​\(xt\)​q​\(x′\|xt\)\}\.\\displaystyle a\(x\_\{t\},x^\{\\prime\}\)=\\min\\left\\\{1,\\frac\{\\pi\(x^\{\\prime\}\)q\(x\_\{t\}\|x^\{\\prime\}\)\}\{\\pi\(x\_\{t\}\)q\(x^\{\\prime\}\|x\_\{t\}\)\}\\right\\\}\.\(6\)If accepted, we setxt\+1=x′x\_\{t\+1\}=x^\{\\prime\}; otherwise, we setxt\+1=xtx\_\{t\+1\}=x\_\{t\}\. In this way, MCMC gradually explores the high\-probability regions ofπ\\pi\. Unlike importance sampling, it does not require a proposalqqthat globally covers the target, but it can mix slowly if moving between important regions ofπ\\piis difficult\. For example, when the target distributionπ\\piis multimodal, MCMC can become inefficient, as the Markov chain can get stuck in a local mode and cannot explore the entire support well\.

### 2\.3 Accelerated Sampling with Intermediate Distributions

IS and MCMC give us two prototypes for how to draw samples from intractable distributions\. However, as we discussed, both approaches can face challenges when the target is multimodal or when the proposal is less calibrated toward the target\.*In what way can we improve these approaches?*

The common idea is*annealing*\. In other words, we choose some simple distribution to start with, and create a sequence of intermediate distributions, going from the simple distribution to the more complicated target distribution\. Building on this idea, IS and MCMC develop their annealing variants: sequential Monte Carlo and Replica Exchange\.

Letπ0\\pi\_\{0\}be a tractable prior distribution, andπN\\pi\_\{N\}be our complicated target\. We introduce a sequence of intermediate distributions between them, given by the annealing pathπi∝π0\(1−i/N\)​πN\(i/N\)\\pi\_\{i\}\\propto\\pi\_\{0\}^\{\(1\-i/N\)\}\\pi\_\{N\}^\{\(i/N\)\}\.

#### Sequential Monte Carlo

Sequential Monte Carlo \(SMC\) approximates the sequence\{πi\}i=0N\\\{\\pi\_\{i\}\\\}\_\{i=0\}^\{N\}with a population of weighted particles\. Given particles approximately distributed according toπi−1\\pi\_\{i\-1\}, SMC propagates them with a forward kernelxi∼Fi\(⋅∣xi−1\)x\_\{i\}\\sim F\_\{i\}\(\\cdot\\mid x\_\{i\-1\}\), and corrects the mismatch between the proposal and the next target in the annealing path using an incremental importance weight\. Introducing a backward kernel,Bi−1​\(xi−1∣xi\)B\_\{i\-1\}\(x\_\{i\-1\}\\mid x\_\{i\}\), the incremental weight is

Gi​\(xi−1,xi\)=πi​\(xi\)πi−1​\(xi−1\)​Bi−1​\(xi−1∣xi\)Fi​\(xi∣xi−1\)\.\\displaystyle G\_\{i\}\(x\_\{i\-1\},x\_\{i\}\)=\\frac\{\\pi\_\{i\}\(x\_\{i\}\)\}\{\\pi\_\{i\-1\}\(x\_\{i\-1\}\)\}\\frac\{B\_\{i\-1\}\(x\_\{i\-1\}\\mid x\_\{i\}\)\}\{F\_\{i\}\(x\_\{i\}\\mid x\_\{i\-1\}\)\}\.\(7\)
Thus the particle weight is updated bywi=wi−1​Gi​\(xi−1,xi\)w\_\{i\}=w\_\{i\-1\}G\_\{i\}\(x\_\{i\-1\},x\_\{i\}\)\. The forward kernel is used as a proposal for moving samples forward along the path, while the backward kernel specifies the reverse process used to derive weights that correct the samples\.

#### Replica Exchange

Replica Exchange \(RE\) designs a Markov Chain on the joint space for the joint distribution

π0×π1×⋯×πN,\\displaystyle\\pi\_\{0\}\\times\\pi\_\{1\}\\times\\cdots\\times\\pi\_\{N\},\(8\)with one replica per intermediate distributionπi\\pi\_\{i\}\. A local Markov kernel is used to update each replica under its own targetπi\\pi\_\{i\}\. In addition, neighboring replicas are occasionally swapped, known as communication steps\. For a proposed swap between levelsiiandi−1i\-1, the Metropolis\-Hastings acceptance probability is

Ai,i−1=min⁡\{1,πi​\(xi−1\)πi​\(xi\)​πi−1​\(xi\)πi−1​\(xi−1\)\}\.\\displaystyle A\_\{i,i\-1\}=\\min\\left\\\{1,\\frac\{\\pi\_\{i\}\(x\_\{i\-1\}\)\}\{\\pi\_\{i\}\(x\_\{i\}\)\}\\frac\{\\pi\_\{i\-1\}\(x\_\{i\}\)\}\{\\pi\_\{i\-1\}\(x\_\{i\-1\}\)\}\\right\\\}\.\(9\)

#### Accelerated RE

We can also introduce a forwardFFand backward kernelBBfor RE\(Zhang et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib38)\)\. Starting fromxix\_\{i\}at the chain corresponding toπi\\pi\_\{i\}andyi−1y\_\{i\-1\}atπi−1\\pi\_\{i\-1\}, we propose a “swap" by

xi−1∼Bi−1​\(xi−1\|xi\),yi∼Fi​\(yi\|yi−1\)\.\\displaystyle x\_\{i\-1\}\\sim B\_\{i\-1\}\(x\_\{i\-1\}\|x\_\{i\}\),\\quad y\_\{i\}\\sim F\_\{i\}\(y\_\{i\}\|y\_\{i\-1\}\)\.\(10\)Then, the acceptance rate is

Ai,i−1=min⁡\{1,πi​\(yi\)​Bi−1​\(yi−1\|yi\)πi−1​\(yi−1\)​Fi​\(yi\|yi−1\)​πi−1​\(xi−1\)​Fi​\(xi\|xi−1\)πi​\(xi\)​Bi−1​\(xi−1\|xi\)\}\.\\displaystyle A\_\{i,i\-1\}=\\min\\left\\\{1,\\frac\{\\pi\_\{i\}\(y\_\{i\}\)B\_\{i\-1\}\(y\_\{i\-1\}\|y\_\{i\}\)\}\{\\pi\_\{i\-1\}\(y\_\{i\-1\}\)F\_\{i\}\(y\_\{i\}\|y\_\{i\-1\}\)\}\\frac\{\\pi\_\{i\-1\}\(x\_\{i\-1\}\)F\_\{i\}\(x\_\{i\}\|x\_\{i\-1\}\)\}\{\\pi\_\{i\}\(x\_\{i\}\)B\_\{i\-1\}\(x\_\{i\-1\}\|x\_\{i\}\)\}\\right\\\}\.\(11\)When kernels are chosen or learned well, this new variation of RE will have better performance \(i\.e\., a higher acceptance rate between different distributions\) than the standard version\.

## 3 Methods

In this section, we establish our methodology\. We first present the perspective of considering an LLM as a time\-dependent transport\. Next, we move to a flexible sampling target for LLM inference\. Furthermore, we establish principled SMC and RE algorithms to sample from the flexible targets with practical implementation considerations\.

### 3\.1 LLM as a Time\-dependent Transport

An LLM defines a forward Markov transition kernelp​\(xi\|xi−1\)p\(x\_\{i\}\|x\_\{i\-1\}\), as discussed in[Section˜2\.1](https://arxiv.org/html/2608.26120#S2.SS1)\. We can then define a forward transition kernel for the transport

Fi​\(x0:i∣x0:i−1\)=p​\(xi∣x<i\)\.F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)=p\(x\_\{i\}\\mid x\_\{<i\}\)\.\(12\)We can also obtain a backward kernel from Bayes’ rule:

Bi−1​\(x0:i−1∣x0:i\)=pi−1​\(x0:i−1\)​Fi​\(x0:i∣x0:i−1\)pi​\(x0:i\)=1\.B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)=\\frac\{p\_\{i\-1\}\(x\_\{0:i\-1\}\)F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}\{p\_\{i\}\(x\_\{0:i\}\)\}=1\.\(13\)Therefore, for the base autoregressive model, the backward transport is deterministic: it simply removes the last generated token\. The nontrivial part of autoregressive generation lies in the forward kernel, which appends a token according top​\(xi∣x<i\)p\(x\_\{i\}\\mid x\_\{<i\}\)\.

After defining the transition kernel, we are able to run SMC or the accelerated RE to draw samples from some intractable distribution\. In the next section, we discuss the target distributions we consider\.

### 3\.2 Sampling from Flexible Target Distributions

#### Flexible target distributions

Inspired by recent works in controlling diffusion models\(Wu et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib36); Skreta et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib30); Singhal et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib29); He et al\.,[2025a](https://arxiv.org/html/2608.26120#bib.bib10)\), we consider steering the distribution towards a new distributionq​\(x0:N\)q\(x\_\{0:N\}\)\. Some examples of this new distribution include \(1\) the “power" version of original distributionπ​\(x0:N\)∝pβ​\(x0:N\)\\pi\(x\_\{0:N\}\)\\propto p^\{\\beta\}\(x\_\{0:N\}\), as considered byKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\); \(2\) the tilted distributionπ​\(x0:N\)∝p​\(x0:N\)​exp⁡\(r​\(x0:N\)\)\\pi\(x\_\{0:N\}\)\\propto p\(x\_\{0:N\}\)\\exp\(r\(x\_\{0:N\}\)\)with a reward, likelihood or verifier functionrr, which was also considered byZhao et al\. \([2024](https://arxiv.org/html/2608.26120#bib.bib39)\); \(3\) the product or quotient of distributions of two \(or more\) LLMsπ​\(x0:N\)∝p​\(x0:N\)​p′​\(x0:N\)\\pi\(x\_\{0:N\}\)\\propto p\(x\_\{0:N\}\)p^\{\\prime\}\(x\_\{0:N\}\); \(4\) any combination of the previous examples\. In summary, we consider the following task:

Examples of Steering ObjectivesWe consider steering an auto\-regressive modelp​\(x0:N\)=p​\(x0\)​∏i=1Np​\(xi∣x<i\)p\(x\_\{0:N\}\)=p\(x\_\{0\}\)\\prod\_\{i=1\}^\{N\}p\(x\_\{i\}\\mid x\_\{<i\}\)towards a target distributionπ​\(x0:N\)\\pi\(x\_\{0:N\}\)of the following forms:1\.Powered distribution:π​\(x0:N\)∝p​\(x0:N\)β\\displaystyle\\pi\(x\_\{0:N\}\)\\propto p\(x\_\{0:N\}\)^\{\\beta\}\(14\)2\.Tilted \(reward\-conditioned\) distribution:π​\(x0:N\)∝p​\(x0:N\)​er​\(x0:N\)\\displaystyle\\pi\(x\_\{0:N\}\)\\propto p\(x\_\{0:N\}\)e^\{r\(x\_\{0:N\}\)\}\(15\)3\.Product/quotient of experts \(multiple LLMs\):π​\(x0:N\)∝p​\(x0:N\)​p′​\(x0:N\)​or​p​\(x0:N\)/p′​\(x0:N\)\\displaystyle\\pi\(x\_\{0:N\}\)\\propto p\(x\_\{0:N\}\)\\,p^\{\\prime\}\(x\_\{0:N\}\)\\text\{ \\ \\ \\ or \\ \\ \\ \}p\(x\_\{0:N\}\)/p^\{\\prime\}\(x\_\{0:N\}\)\(16\)4\.Compositionsof the above objectives\.

Now,*how can we obtain sample\(s\)x0:N∼π​\(x0:N\)x\_\{0:N\}\\sim\\pi\(x\_\{0:N\}\)?*This is a standard sampling task\. Similar to what was discussed byKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\), while the base LLM’s distribution,p​\(x0:N\)=p​\(x0\)​∏i=1Np​\(xi\|x<i\)p\(x\_\{0:N\}\)=p\(x\_\{0\}\)\\prod\_\{i=1\}^\{N\}p\(x\_\{i\}\|x\_\{<i\}\), is factorized autoregressively, the corresponding target as defined above does not typically have a similar factorization structure\. Because of this,Karan and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\)introduced an MCMC algorithm to obtain samples from the power distribution\. However, as discussed in[Section˜2\.3](https://arxiv.org/html/2608.26120#S2.SS3), directly running MCMC can be less effective when the target distribution is complicated\. On the other hand, the LLM itself already defines a natural “annealing" in the form of different resulting sequence lengths\. For example, we can define a sequence of annealing distributionsπ​\(x0:i\)∝p​\(x0:i\)β\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)^\{\\beta\}for power,π​\(x0:i\)∝p​\(x0:i\)​er​\(x0:i\)\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)e^\{r\(x\_\{0:i\}\)\}for tilting, etc\. Leveraging this property, we introduce two algorithms based on SMC and RE for more efficient sampling\.

#### Sampling with Sequential Monte Carlo

Following the particle reweighting for general SMC algorithms in[Section˜2\.3](https://arxiv.org/html/2608.26120#S2.SS3), we define the intermediate target distributions and we can obtain the corresponding incremental weights:

Powered distribution:π​\(x0:i\)∝p​\(x0:i\)β,\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)^\{\\beta\},andG​\(x0:i,x0:i−1\)=p​\(x0:i\)βp​\(x0:i−1\)β​Bi−1​\(x0:i−1∣x0:i\)Fi​\(x0:i∣x0:i−1\)\.G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{0:i\}\)^\{\\beta\}\}\{p\(x\_\{0:i\-1\}\)^\{\\beta\}\}\\frac\{B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)\}\{F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}\.Tilted \(reward\-conditioned\) distribution:π​\(x0:i\)∝p​\(x0:i\)​er​\(x0:i\)\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)e^\{r\(x\_\{0:i\}\)\}, andG​\(x0:i,x0:i−1\)=p​\(x0:i\)​er​\(x0:i\)p​\(x0:i−1\)​er​\(x0:i−1\)​Bi−1​\(x0:i−1∣x0:i\)Fi​\(x0:i∣x0:i−1\)\.G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{0:i\}\)e^\{r\(x\_\{0:i\}\)\}\}\{p\(x\_\{0:i\-1\}\)e^\{r\(x\_\{0:i\-1\}\)\}\}\\frac\{B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)\}\{F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}\.Product/quotient of experts \(multiple LLMs\):π​\(x0:i\)∝p​\(x0:i\)​p′​\(x0:i\),\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)\\,p^\{\\prime\}\(x\_\{0:i\}\),andG​\(x0:i,x0:i−1\)=p​\(x0:i\)​p′​\(x0:i\)p​\(x0:i−1\)​p′​\(x0:i−1\)​Bi−1​\(x0:i−1∣x0:i\)Fi​\(x0:i∣x0:i−1\)\.G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{0:i\}\)p^\{\\prime\}\(x\_\{0:i\}\)\}\{p\(x\_\{0:i\-1\}\)p^\{\\prime\}\(x\_\{0:i\-1\}\)\}\\frac\{B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)\}\{F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}\.Let’s take a closer look at these distributions taking the power case as an example:G​\(x0:i,x0:i−1\)=p​\(x0:i\)βp​\(x0:i−1\)β​Bi−1​\(x0:i−1∣x0:i\)Fi​\(x0:i∣x0:i−1\)G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{0:i\}\)^\{\\beta\}\}\{p\(x\_\{0:i\-1\}\)^\{\\beta\}\}\\frac\{B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)\}\{F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}\. This formula indicates that, every time we sample the next token, we need to evaluate the entire sequence likelihoodp​\(x0:i\)p\(x\_\{0:i\}\), which can be prohibitive in LLM sampling\.

Fortunately, recall that the base distribution is factorized as an AR model, hence, we have

p​\(x0:i\)βp​\(x0:i−1\)β=p​\(xi\|x<i\)β\\displaystyle\\frac\{p\(x\_\{0:i\}\)^\{\\beta\}\}\{p\(x\_\{0:i\-1\}\)^\{\\beta\}\}=p\(x\_\{i\}\|x\_\{<i\}\)^\{\\beta\}\(17\)Similarly, using also an LLM as the proposal, with[Equation˜13](https://arxiv.org/html/2608.26120#S3.E13), we have

Bi−1​\(x0:i−1∣x0:i\)Fi​\(x0:i∣x0:i−1\)=qi−1​\(x0:i−1\)qi​\(x0:i\)=1q​\(xi\|x<i\)\\displaystyle\\frac\{B\_\{i\-1\}\(x\_\{0:i\-1\}\\mid x\_\{0:i\}\)\}\{F\_\{i\}\(x\_\{0:i\}\\mid x\_\{0:i\-1\}\)\}=\\frac\{q\_\{i\-1\}\(x\_\{0:i\-1\}\)\}\{q\_\{i\}\(x\_\{0:i\}\)\}=\\frac\{1\}\{q\(x\_\{i\}\|x\_\{<i\}\)\}\(18\)Here we useqqto denote that the proposal LLM, which can be different from our target LLM\. This allows us to obtain the following SMC weights:

Sequential Monte Carlo WeightsAssume we have a sequencex0:i−1x\_\{0:i\-1\}, and we propose with an proposal LLMqqbyx0:i∼Fi​\(x0:i\|x0:i−1\)\.x\_\{0:i\}\\sim F\_\{i\}\(x\_\{0:i\}\|x\_\{0:i\-1\}\)\.The incremental SMC weight is given by1\.Powered distribution:π​\(x0:i\)∝p​\(x0:i\)β,G​\(x0:i,x0:i−1\)=p​\(xi\|x<i\)βq​\(xi\|x<i\)\\displaystyle\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)^\{\\beta\},\\hskip 18\.49988ptG\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{i\}\|x\_\{<i\}\)^\{\\beta\}\}\{q\(x\_\{i\}\|x\_\{<i\}\)\}\(19\)2\.Tilted \(reward\-conditioned\) distribution:π​\(x0:i\)∝p​\(x0:i\)​er​\(x0:i\),G​\(x0:i,x0:i−1\)=p​\(xi\|x<i\)​er​\(x0:i\)q​\(xi\|x<i\)​er​\(x0:i−1\)\.\\displaystyle\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)e^\{r\(x\_\{0:i\}\)\},\\qquad G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{i\}\|x\_\{<i\}\)e^\{r\(x\_\{0:i\}\)\}\}\{q\(x\_\{i\}\|x\_\{<i\}\)e^\{r\(x\_\{0:i\-1\}\)\}\}\.\(20\)3\.Product/quotient of experts \(multiple LLMs\):π​\(x0:i\)∝p​\(x0:i\)​p′​\(x0:i\),G​\(x0:i,x0:i−1\)=p​\(xi\|x<i\)​p′​\(xi\|x<i\)q​\(xi\|x<i\)\.\\displaystyle\\pi\(x\_\{0:i\}\)\\propto p\(x\_\{0:i\}\)\\,p^\{\\prime\}\(x\_\{0:i\}\),\\qquad G\(x\_\{0:i\},x\_\{0:i\-1\}\)=\\frac\{p\(x\_\{i\}\|x\_\{<i\}\)p^\{\\prime\}\(x\_\{i\}\|x\_\{<i\}\)\}\{q\(x\_\{i\}\|x\_\{<i\}\)\}\.\(21\)

#### Sampling with Replica Exchange

Similarly, we can also derive the Replica Exchange rate with target base LLMppand proposal LLMqq, as stated below:

Replica Exchange Swap RatiosAssume we have sequencesx0:i−1x\_\{0:i\-1\}andy0:iy\_\{0:i\}, and we propose to “swap" them using the proposal LLMqq:y0:i−1∼Bi−1​\(y0:i−1\|y0:i\),x0:i∼Fi​\(x0:i\|x0:i−1\)\.\\displaystyle y\_\{0:i\-1\}\\sim B\_\{i\-1\}\(y\_\{0:i\-1\}\|y\_\{0:i\}\),\\qquad x\_\{0:i\}\\sim F\_\{i\}\(x\_\{0:i\}\|x\_\{0:i\-1\}\)\.\(22\)The acceptance rate isA=min⁡\{1,α\}A=\\min\\left\\\{1,\\alpha\\right\\\}, withα\\alphadefined according to the following cases:1\.Powered distributionπ∝pβ\\pi\\propto p^\{\\beta\}:α=p​\(xi\|<i\)βp​\(yi\|<i\)β​q​\(yi\|<i\)q​\(xi\|<i\)\.\\displaystyle\\alpha=\\frac\{p\(x\_\{i\|<i\}\)^\{\\beta\}\}\{p\(y\_\{i\|<i\}\)^\{\\beta\}\}\\frac\{q\(y\_\{i\|<i\}\)\}\{q\(x\_\{i\|<i\}\)\}\.\(23\)2\.Tilted \(reward\-conditioned\) distributionπ∝p​er\\pi\\propto pe^\{r\}:α=er​\(x0:i\)−r​\(x0:i−1\)er​\(y0:i\)−r​\(y0:i−1\)​p​\(xi\|<i\)p​\(yi\|<i\)​q​\(yi\|<i\)q​\(xi\|<i\)\.\\displaystyle\\alpha=\\frac\{e^\{r\(x\_\{0:i\}\)\-r\(x\_\{0:i\-1\}\)\}\}\{e^\{r\(y\_\{0:i\}\)\-r\(y\_\{0:i\-1\}\)\}\}\\frac\{p\(x\_\{i\|<i\}\)\}\{p\(y\_\{i\|<i\}\)\}\\frac\{q\(y\_\{i\|<i\}\)\}\{q\(x\_\{i\|<i\}\)\}\.\(24\)3\.Product of experts \(multiple LLMs\)π∝p​p′\\pi\\propto pp^\{\\prime\}:α=p​\(xi\|<i\)p​\(yi\|<i\)​p′​\(xi\|<i\)p′​\(yi\|<i\)​q​\(yi\|<i\)q​\(xi\|<i\)\.\\displaystyle\\alpha=\\frac\{p\(x\_\{i\|<i\}\)\}\{p\(y\_\{i\|<i\}\)\}\\frac\{p^\{\\prime\}\(x\_\{i\|<i\}\)\}\{p^\{\\prime\}\(y\_\{i\|<i\}\)\}\\frac\{q\(y\_\{i\|<i\}\)\}\{q\(x\_\{i\|<i\}\)\}\.\(25\)

*We also highlight that the RE algorithm we proposed is not a direct application of standard replica exchange but belongs to a new family of RE methods\.*In standard RE, one constructs a ladder of distributions by annealing a difficult target into simpler ones, with all replicas living in the same state space\. Recent path\-space extensions, such asZhang et al\. \([2025](https://arxiv.org/html/2608.26120#bib.bib38)\), lift RE from endpoint distributions to path measures, but still require defining a ladder of path distributions over a common path space\.

By contrast, our construction applies RE directly to the autoregressive generation*process*\. The replicas evolve through chunks of tokens, and these replicas differ in length and hence are not in the same space\. This gives a different and more flexible form of replica exchange, naturally suited to variable\-length and autoregressive state spaces\.

#### SMC and RE in Chunks

Although we present the SMC weight and RE swap ratio for a single incremental step fromi−1i\-1toii, in practice, both algorithms can be applied in “chunks", corresponding to a contiguous sequence of tokens\. Given a chunk sizeBB, the algorithm moves between chunk states that differ byBBautoregressive steps \(e\.g\., fromi−Bi\-Btoii\)\. The weight or swap ratio for the whole chunk is then obtained by multiplying the corresponding single\-step quantities within that chunk\. We summarize the algorithm for SMC and RE in[Figure˜1](https://arxiv.org/html/2608.26120#S3.F1)\.

Algorithm 1SMC1:proposal LLM

qq, particles

KK, length

NN, chunk size

BB
2:Let

nm=m​Bn\_\{m\}=mB, with

M=N/BM=N/B
3:Initialize

x0:n0\(k\)∼q​\(x0:n0\)x\_\{0:n\_\{0\}\}^\{\(k\)\}\\sim q\(x\_\{0:n\_\{0\}\}\),

w\(k\)←1w^\{\(k\)\}\\leftarrow 1
4:for

m=1,…,Mm=1,\\ldots,Mdo

5:for

k=1,…,Kk=1,\\ldots,Kdo

6:\# extend each sample one chunk

7:for

j=nm−1\+1,…,nmj=n\_\{m\-1\}\+1,\\ldots,n\_\{m\}do

8:Sample

xj\(k\)∼q\(⋅∣x<j\(k\)\)x\_\{j\}^\{\(k\)\}\\sim q\(\\cdot\\mid x\_\{<j\}^\{\(k\)\}\)
9:endfor

10:

x0:nm\(k\)=\(x0:nm−1\(k\),xnm−1\+1:nm\(k\)\)x\_\{0:n\_\{m\}\}^\{\(k\)\}=\(x\_\{0:n\_\{m\-1\}\}^\{\(k\)\},x\_\{n\_\{m\-1\}\+1:n\_\{m\}\}^\{\(k\)\}\)
11:

w\(k\)←w\(k\)​Gm​\(x0:nm\(k\),x0:nm−1\(k\)\)w^\{\(k\)\}\\leftarrow w^\{\(k\)\}G\_\{m\}\(x\_\{0:n\_\{m\}\}^\{\(k\)\},x\_\{0:n\_\{m\-1\}\}^\{\(k\)\}\)
12:endfor

13:Normalize

\{w\(k\)\}k=1K\\\{w^\{\(k\)\}\\\}\_\{k=1\}^\{K\}
14:Resample particles

15:Reset

w\(k\)←1/Kw^\{\(k\)\}\\leftarrow 1/K
16:endfor

17:return

\{x0:N\(k\),w\(k\)\}k=1K\\\{x\_\{0:N\}^\{\(k\)\},w^\{\(k\)\}\\\}\_\{k=1\}^\{K\}

Algorithm 2RE1:proposal LLM

qq, iterations

KK, length

NN, chunk size

BB
2:Let

nm=m​Bn\_\{m\}=mB, with

M=N/BM=N/B
3:Initialize initial guess

zm∼q​\(x0:nm\)z\_\{m\}\\sim q\(x\_\{0:n\_\{m\}\}\)
4:for

s=1,…,Ks=1,\\ldots,Kdo

5:for

m=1,3,5​…m=1,3,5\\ldotsor

2,4,6​…2,4,6\\ldotsdo

6:Let

x0:nm−1=zm−1x\_\{0:n\_\{m\-1\}\}=z\_\{m\-1\},

y0:nm=zmy\_\{0:n\_\{m\}\}=z\_\{m\}
7:\# remove one chunk fromy0:nm−1y\_\{0:n\_\{m\-1\}\}

8:Set

y~0:nm−1=y0:nm−1\\tilde\{y\}\_\{0:n\_\{m\-1\}\}=y\_\{0:n\_\{m\-1\}\}
9:\# extend one chunk fromx0:nm−1x\_\{0:n\_\{m\-1\}\}

10:for

j=nm−1\+1,…,nmj=n\_\{m\-1\}\+1,\\ldots,n\_\{m\}do

11:Sample

x~j∼q\(⋅∣x~<j\)\\tilde\{x\}\_\{j\}\\sim q\(\\cdot\\mid\\tilde\{x\}\_\{<j\}\)
12:endfor

13:Set

x~0:nm=\(x0:nm−1,x~nm−1\+1:nm\)\\tilde\{x\}\_\{0:n\_\{m\}\}=\(x\_\{0:n\_\{m\-1\}\},\\tilde\{x\}\_\{n\_\{m\-1\}\+1:n\_\{m\}\}\)
14:

\(zm−1,zm\)←\(y~0:nm−1,x~0:nm\)\(z\_\{m\-1\},z\_\{m\}\)\\leftarrow\(\\tilde\{y\}\_\{0:n\_\{m\-1\}\},\\tilde\{x\}\_\{0:n\_\{m\}\}\)with probability

Am=min⁡\{1,αm\}A\_\{m\}=\\min\\\{1,\\alpha\_\{m\}\\\}
15:endfor

16:endfor

17:returnfinal\-level replica

zMz\_\{M\}

Figure 1:Sampling algorithms for flexible LLM inference targets\. HereGmG\_\{m\}is the chunk\-level incremental SMC weight from lengthnm−1n\_\{m\-1\}tonmn\_\{m\}, andαm\\alpha\_\{m\}is the chunk\-level Metropolis\-Hastings ratio\. For RE, swaps are performed in alternating adjacent pairs: whenssis odd, we swap between\(1,2\),\(3,4\),…\(1,2\),\(3,4\),\.\.\., and whenssis even, we swap between\(2,3\),\(4,5\),…\(2,3\),\(4,5\),\.\.\.\. Therefore, the budget for RE and SMC is the same when the RE iteration is twice the number of SMC particles\.
#### EOS Tokens

Finally, we note that LLM generations are typically variable\-length\. For SMC, EOS tokens can be handled straightforwardly: once a particle reaches EOS, we stop extending that particle and keep it in the final answer pool, while the remaining particles continue to evolve\.

For RE, we maintain the convention that each complete sequence ends with an EOS token\. During initialization \(initial guess\), we generate until reaching EOS token, and divide them into chuncks, corresponding to each intermediate target\. During sampling, if a newly proposed chunk contains an EOS token and is accepted, we truncate the sequence at that EOS token and discard all subsequent chunks\. Conversely, if the current final chunk does not contain an EOS token, we keep extending the sequence with additional chunks until an EOS token is reached\.

## 4 Related Works

### 4\.1 Steering LLMs at Inference\-time

Since the emergence of general\-purpose LLMs, a substantial body of work has studied how to elicit, control, and improve their behavior at inference time\. Early methods focused on prompting, leveraging the conversational interface to induce intermediate reasoning or structured decompositions\(Kojima et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib15); Wei et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib34)\)\.Yang et al\. \([2024](https://arxiv.org/html/2608.26120#bib.bib37)\)proposed to further optimize the text prompt to control the text output\. In parallel, mechanistic and representation\-centric approaches aimed for more direct control by mapping and intervening the internal representations of LLMs\(Li et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib19); Cunningham et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib4); Kong et al\.,[2024b](https://arxiv.org/html/2608.26120#bib.bib17)\)\. Another branch of work integrates LLM inference with implicit or explicit search and external feedback\(Valmeekam et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib31); Madaan et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib22)\)\. Among them, evolutionary algorithms demonstrated strong performances in searching mathematical programs, molecular structures and symbolic equations\(Romera\-Paredes et al\.,[2024](https://arxiv.org/html/2608.26120#bib.bib27); Novikov et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib24); Wang et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib32); Shojaee et al\.,[2024](https://arxiv.org/html/2608.26120#bib.bib28)\)\. The most relevant work to our setting is sampling\-based approaches:Mudgal et al\. \([2023](https://arxiv.org/html/2608.26120#bib.bib23)\)approximated the optimal value function to control the sampling process;Zhao et al\. \([2024](https://arxiv.org/html/2608.26120#bib.bib39)\)applied twisted sequential Monte Carlo to steer LLMs; recently,Karan and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\)proposed to enhance LLM reasoning via sampling from a tempered distribution \(power\) using MCMC\.

### 4\.2 Inference\-time Control with Sampling in Diffusion Models

Inference\-time control has also been extensively studied in other generative models, especially diffusion models\. A common strategy is to modify the generation process using guidance so that samples satisfy additional constraints or exhibit desired properties\(Dhariwal and Nichol,[2021](https://arxiv.org/html/2608.26120#bib.bib5); Ho and Salimans,[2022](https://arxiv.org/html/2608.26120#bib.bib12); Chung et al\.,[2022](https://arxiv.org/html/2608.26120#bib.bib3); Kong et al\.,[2024a](https://arxiv.org/html/2608.26120#bib.bib16)\)\. While such heuristic guidance methods are often effective, they can introduce bias and may fail under strong constraints\. This has motivated more principled sampling\-based approaches, including annealed importance sampling, sequential Monte Carlo, Feynman\-Kac\-based correction, and replica\-exchange\-style methods\(Wu et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib36); Skreta et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib30); Singhal et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib29); Du et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib6); He et al\.,[2025b](https://arxiv.org/html/2608.26120#bib.bib11)\)\. These methods use annealing intermediate distributions to progressively steer generation\. Our work brings this sampling\-based test\-time control perspective to autoregressive LLM generation, where the generation trajectory is discrete, variable\-length, and naturally factorized token by token\.

## 5 Experiments

We evaluate whether the proposed sampling algorithms can improve LLMs at inference time\.

#### Benchmarks and models\.

FollowingKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\), we evaluate on MATH500\(Lightman et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib20)\)and GPQA\-Diamond\(Rein et al\.,[2023](https://arxiv.org/html/2608.26120#bib.bib26)\)\. We use the same data split and answer format asKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\)\. For MATH500, we evaluate both Qwen2\.5\-Math\-7B and Qwen2\.5\-7B\. For GPQA\-Diamond, we evaluate Qwen2\.5\-32B\-Instruct, since we found that the 7B base model used inKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\)does not reliably follow the required answer format, leading to unreliable correctness estimates\. All generations use chain\-of\-thought prompting\. For evaluation, we parse the last boxed expression from each generated solution and compare it against the ground\-truth answer\.

#### Sampling setup\.

Unless otherwise stated, we sample with maximum generation length30723072, proposal temperature0\.250\.25, and1616chunks\. Thus, each chunk contains at most192192newly generated tokens\. The proposal distribution is the low\-temperature autoregressive sampler, while the target distribution is a powered and optionally entropy\-tilted version of the base model distribution\. In particular, the power parameter isα=1/0\.25=4\\alpha=1/0\.25=4, and we additionally consider an intrinsic “confidence" tilt based on the model entropy\.

Note that we do not tilt with the likelihood as the confidence, as this would lead to the same formulation as powering\. Instead, we use the model’s predictive entropy as an intrinsic confidence signal\. More precisely, for a generated chunkxa:bx\_\{a:b\}, we tilt with

log⁡γ​\(xa:b\)=−λ​H¯​\(xa:b\)\.\\displaystyle\\operatorname\{log\}\\gamma\(x\_\{a:b\}\)=\-\\lambda\\bar\{H\}\(x\_\{a:b\}\)\.\(26\)whereH¯​\(xa:b\)\\bar\{H\}\(x\_\{a:b\}\)is the average next\-token entropy over the chunk:

H¯​\(xa:b\)=1b−a\+1​∑j=abHj,where​Hj=−∑v∈𝒱p​\(v\|x<j\)​log⁡p​\(v\|x<j\)\.\\displaystyle\\bar\{H\}\(x\_\{a:b\}\)=\\frac\{1\}\{b\-a\+1\}\\sum\_\{j=a\}^\{b\}H\_\{j\},\\text\{ where \}H\_\{j\}=\-\\sum\_\{v\\in\\mathcal\{V\}\}p\(v\|x\_\{<j\}\)\\operatorname\{log\}p\(v\|x\_\{<j\}\)\.\(27\)Equivalently, this upweights chunks where the base model has lower predictive entropy, i\.e\., higher confidence\. We sweepλ∈\{0,100,500,1000\}\\lambda\\in\\\{0,100,500,1000\\\}\. Importantly, this tilt uses only the base model’s own predictive uncertainty and does not rely on an external verifier, similar to\(Fu et al\.,[2025](https://arxiv.org/html/2608.26120#bib.bib8)\)\.

#### Research questions\.

In the following, we consider five questions: \(1\) do SMC and RE improve over the autoregressive base sampler and naive low\-temperature sampling? \(2\) do they provide better inference\-time scaling than the MCMC sampler ofKaran and Du \([2025](https://arxiv.org/html/2608.26120#bib.bib14)\)? \(3\) can the model performance be improved without any external verifier using both powering and tilting? \(4\) are these sampling methods better than simple Best\-of\-NN? and \(5\) since SMC and RE produce a population of samples rather than only a single selected output, can their sample diversity bring us more benefit?

![Refer to caption](https://arxiv.org/html/2608.26120v1/fig/results.png)Figure 2:SMC, RE, MCMC, BON, BON with block, Naive Tempering on MATH500 with Qwen\-2\.5\-Math\-7B\.
![Refer to caption](https://arxiv.org/html/2608.26120v1/fig/results2.png)Figure 3:SMC, RE, MCMC, BON, BON with block, Naive Tempering on MATH500 with Qwen\-2\.5\-7B\.

#### SMC and RE improve over naive autoregressive sampling\.

As shown in[Figures˜3](https://arxiv.org/html/2608.26120#S5.F3)and[3](https://arxiv.org/html/2608.26120#S5.F3), both SMC and RE substantially outperform naive low\-temperature sampling on MATH500\. This follows the same argument as\(Karan and Du,[2025](https://arxiv.org/html/2608.26120#bib.bib14)\), where both SMC and RE can be viewed as a global “search", while the standard low\-temperature sampling is greedy\.

#### SMC and RE scale more favorably than MCMC\.

We then compare our proposed framework against the recent MCMC\-based sampling method\(Karan and Du,[2025](https://arxiv.org/html/2608.26120#bib.bib14)\)\. More precisely, they consider powering the LLM output with MCMC\. From[Figures˜3](https://arxiv.org/html/2608.26120#S5.F3)and[3](https://arxiv.org/html/2608.26120#S5.F3), we can see that SMC and RE all present better scaling properties compared to MCMC\. The reason is that MCMC revises the trajectory by randomly removing and refilling\. This causes a dilemma: if the sentence needs to be largely revised, we expect to refill a larger chunk\. However, when the chunk need to refill is larger, the acceptance rate typically drops drastically\. On the other hand, both SMC and RE are based on*annealing*through a sequence of intermediate targets\. These targets serve as “checkpoints", making the proposal being used more efficient\.

#### Powering and entropy tilting improve performance without an external verifier\.

Beyond power, we now verify if the proposed entropy tilting could also improve the performance without an extra verifier\. In[Table˜1](https://arxiv.org/html/2608.26120#S5.T1), we investigate this on both MATH500 and GPQA\-diamond datasets\. We also sweep with different power and different tilting strengths\. For both SMC and RE, both of powering and tilting improve the accuracy of the base model, and combining them further improves the results\. This suggests that the model’s own predictive entropy is a useful intrinsic confidence signal\.

At the same time, the effect of entropy tilting is not monotone\. Very large tilt strengths can slightly hurt performance\. A possible reason is that a strong tilt makes the target and proposal too different, which decreases the acceptance rate, especially for RE\. This is also consistent with the observation in[Table˜1\(a\)](https://arxiv.org/html/2608.26120#S5.T1.st1), where RE tends to drop for larger strength while SMC tends to plateau\.

#### Best\-of\-NNis still a strong baseline\.

We also compare the sampling method with BON\. We consider two different BON setups: for the simplest BON, we directly finish the generation and then select the best outcome according to the likelihood; for the second case, which we refer to as BON block\-wise, we use the same chunk idea as SMC and RE\. We select the best candidate after finishing each chunk, and then use this as the prefix for the next chunk\. The results in[Figures˜3](https://arxiv.org/html/2608.26120#S5.F3)and[3](https://arxiv.org/html/2608.26120#S5.F3)show that Best\-of\-NNis a strong baseline\. It not only outperforms MCMC, but also outperforms RE and on par with SMC\. However, we note that SMC and RE aim to draw samples from the entire distribution, while BON is searching for the best one\. Therefore, when we care about sample diversity, SMC or RE can be effective yet simple choices, while BON typically returns a single sample\.

#### Population\-based sampling gives additional pass@kkbenefits\.

Following the discussion above, we next ask whether the sample population produced by SMC and RE is useful beyond selecting a single final answer\. To evaluate this, we report pass@kkaccuracy in[Figure˜4](https://arxiv.org/html/2608.26120#S5.F4)\. For a fair comparison, we run SMC with1515particles and for RE, we run3030swap iterations\. This gives the same sampling budget to both methods, and both correspond to at most1515final\-level candidate sequences\. We then evaluate whether the correct answer appears among thekkgenerated samples\. For RE, we simply take the firstkksamples for evaluation, while for SMC, we randomly selectkkfrom the final answer pool\.

The pass@kkcurves show that both SMC and RE benefit from having multiple samples\. This confirms that the generated populations are not merely redundant copies of a single solution but contain useful alternative reasoning traces\. However, the two methods behave differently\. SMC gives stronger first\-sample performance, but at the same time, the pass@kkcurve tends to improve only mildly askkincreases, reflecting that the samples are mostly collapsed\. RE exhibits the opposite behavior\. Although its first sample can be less competitive, its pass@kkaccuracy continues to increase more substantially withkkand in the end outperform SMC\. This suggests that the RE population preserves more diverse candidates, which also aligns with the findings by recent work in diffusion control\(He et al\.,[2025b](https://arxiv.org/html/2608.26120#bib.bib11)\)\. Thus, SMC is especially effective when one wants a single strong answer, while RE is particularly useful when multiple candidate solutions can be exploited\.

ModelMethod01005001000MathmodelRE0\.7460\.7560\.7740\.769SMC0\.7700\.7770\.7720\.771Naive Temp0\.685BasemodelRE0\.6790\.7040\.7010\.693SMC0\.7020\.7110\.7170\.720Naive Temp0\.633

\(a\)MATH500 accuracy under different tilting strengths\.
Tilt strength = 0Tilt strength = 500PowerRESMCRESMC0\.250\.4640\.4860\.4970\.4640\.500\.4910\.4580\.5030\.5050\.750\.4840\.4730\.5120\.5021\.000\.4830\.4830\.4970\.477

\(b\)GPQA results on Qwen2\.5\-Instruct\-32B under different power values and tilt strengths\.

Table 1:Results on MATH500 and GPQA\. We run both SMC and RE for the same budget \(15 samples in SMC, or 30 iterations in RE\)\.![Refer to caption](https://arxiv.org/html/2608.26120v1/fig/passk1.png)\(a\)Qwen\-7B\-Math
![Refer to caption](https://arxiv.org/html/2608.26120v1/fig/passk2.png)\(b\)Qwen\-7B

Figure 4:Pass@K results for SMC and RE\.

## 6 Conclusion

We presented a probabilistic sampling framework for steering and scaling autoregressive LLMs at inference time\. By viewing autoregressive generation as a time\-dependent transport, we derived flexible target distributions, including powered, tilted, and product\-form objectives, together with practical SMC and RE algorithms for sampling from them\. Empirically, we showed that these methods improve over naive low\-temperature sampling and scale more favorably than standard MCMC on reasoning benchmarks\. We also found that intrinsic entropy tilting can further improve performance without using an external verifier or reward model\.

Our results highlight complementary strengths of the two samplers\. SMC is particularly effective for obtaining a strong single answer, while RE better preserves sample diversity and gives stronger pass@kkgains\. Although Best\-of\-NNremains a strong search baseline, SMC and RE provide a principled way to sample from the full target distribution rather than only selecting one high\-scoring output\. We hope this work serves as a step toward a broader view of LLM as probabilistic inference over autoregressive trajectories and can inspire future work in LLM scaling and sampling\.

#### Limitations\.

Our method improves inference\-time scaling through better sampling, but it also introduces additional computational overhead compared with standard decoding\. Its effectiveness depends on the quality of the underlying model, and a poor base model may get saturated quickly\. Additionally, in most tasks of LLMs, the exact form of the target distribution may not be important\. In such cases, as we have demonstrated, BON might be a simple alternative with strong performance\.

## Acknowledgment

JH acknowledges support from the University of Cambridge Harding Distinguished Postgraduate Scholar Programme\. JMHL acknowledges funding from AI Hub in Generative Models, under grant EP/Y028805/1\. YD acknowledges support from Cornell University\.

## References

- Achiam et al\. \[2023\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, et al\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- Chen et al\. \[2025\]F\. Chen, A\. Huang, N\. Golowich, S\. Malladi, A\. Block, J\. T\. Ash, A\. Krishnamurthy, and D\. J\. Foster\.The coverage principle: How pre\-training enables post\-training\.*arXiv preprint arXiv:2510\.15020*, 2025\.
- Chung et al\. \[2022\]H\. Chung, J\. Kim, M\. T\. Mccann, M\. L\. Klasky, and J\. C\. Ye\.Diffusion posterior sampling for general noisy inverse problems\.*arXiv preprint arXiv:2209\.14687*, 2022\.
- Cunningham et al\. \[2023\]H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey\.Sparse autoencoders find highly interpretable features in language models\.*arXiv preprint arXiv:2309\.08600*, 2023\.
- Dhariwal and Nichol \[2021\]P\. Dhariwal and A\. Nichol\.Diffusion models beat gans on image synthesis\.*Advances in neural information processing systems*, 34:8780–8794, 2021\.
- Du et al\. \[2023\]Y\. Du, C\. Durkan, R\. Strudel, J\. B\. Tenenbaum, S\. Dieleman, R\. Fergus, J\. Sohl\-Dickstein, A\. Doucet, and W\. S\. Grathwohl\.Reduce, reuse, recycle: Compositional generation with energy\-based diffusion models and mcmc\.In*International conference on machine learning*, pages 8489–8510\. PMLR, 2023\.
- Du et al\. \[2025\]Y\. Du, B\. Yu, T\. Liu, T\. Shen, J\. Chen, J\. G\. Rittig, K\. Sun, Y\. Zhang, Z\. Song, B\. Zhou, et al\.Accelerating scientific discovery with autonomous goal\-evolving agents\.*arXiv preprint arXiv:2512\.21782*, 2025\.
- Fu et al\. \[2025\]Y\. Fu, X\. Wang, Y\. Tian, and J\. Zhao\.Deep think with confidence\.*arXiv preprint arXiv:2508\.15260*, 2025\.
- Hao et al\. \[2023\]S\. Hao, Y\. Gu, H\. Ma, J\. Hong, Z\. Wang, D\. Wang, and Z\. Hu\.Reasoning with language model is planning with world model\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 8154–8173, 2023\.
- He et al\. \[2025a\]J\. He, J\. M\. Hernández\-Lobato, Y\. Du, and F\. Vargas\.Rne: plug\-and\-play diffusion inference\-time control and energy\-based training\.*arXiv preprint arXiv:2506\.05668*, 2025a\.
- He et al\. \[2025b\]J\. He, P\. Jeha, P\. Potaptchik, L\. Zhang, J\. M\. Hernández\-Lobato, Y\. Du, S\. Syed, and F\. Vargas\.Crepe: Controlling diffusion with replica exchange\.*arXiv preprint arXiv:2509\.23265*, 2025b\.
- Ho and Salimans \[2022\]J\. Ho and T\. Salimans\.Classifier\-free diffusion guidance\.*arXiv preprint arXiv:2207\.12598*, 2022\.
- Huang et al\. \[2024\]A\. Huang, A\. Block, D\. J\. Foster, D\. Rohatgi, C\. Zhang, M\. Simchowitz, J\. T\. Ash, and A\. Krishnamurthy\.Self\-improvement in language models: The sharpening mechanism\.*arXiv preprint arXiv:2412\.01951*, 2024\.
- Karan and Du \[2025\]A\. Karan and Y\. Du\.Reasoning with sampling: Your base model is smarter than you think\.*arXiv preprint arXiv:2510\.14901*, 2025\.
- Kojima et al\. \[2022\]T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa\.Large language models are zero\-shot reasoners\.*Advances in neural information processing systems*, 35:22199–22213, 2022\.
- Kong et al\. \[2024a\]L\. Kong, Y\. Du, W\. Mu, K\. Neklyudov, V\. De Bortoli, D\. Wu, H\. Wang, A\. Ferber, Y\.\-A\. Ma, C\. P\. Gomes, et al\.Diffusion models as constrained samplers for optimization with unknown constraints\.*arXiv preprint arXiv:2402\.18012*, 2024a\.
- Kong et al\. \[2024b\]L\. Kong, H\. Wang, W\. Mu, Y\. Du, Y\. Zhuang, Y\. Zhou, Y\. Song, R\. Zhang, K\. Wang, and C\. Zhang\.Aligning large language models with representation editing: A control perspective\.*Advances in Neural Information Processing Systems*, 37:37356–37384, 2024b\.
- Leviathan et al\. \[2023\]Y\. Leviathan, M\. Kalman, and Y\. Matias\.Fast inference from transformers via speculative decoding\.In*International Conference on Machine Learning*, pages 19274–19286\. PMLR, 2023\.
- Li et al\. \[2023\]K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg\.Inference\-time intervention: Eliciting truthful answers from a language model\.*Advances in Neural Information Processing Systems*, 36:41451–41530, 2023\.
- Lightman et al\. \[2023\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\.Let’s verify step by step\.*arXiv preprint arXiv:2305\.20050*, 2023\.
- Liu et al\. \[2025\]Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin\.Understanding r1\-zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- Madaan et al\. \[2023\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, et al\.Self\-refine: Iterative refinement with self\-feedback\.*Advances in Neural Information Processing Systems*, 36:46534–46594, 2023\.
- Mudgal et al\. \[2023\]S\. Mudgal, J\. Lee, H\. Ganapathy, Y\. Li, T\. Wang, Y\. Huang, Z\. Chen, H\.\-T\. Cheng, M\. Collins, T\. Strohman, et al\.Controlled decoding from language models\.*arXiv preprint arXiv:2310\.17022*, 2023\.
- Novikov et al\. \[2025\]A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\.\-S\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian, et al\.Alphaevolve: A coding agent for scientific and algorithmic discovery\.*arXiv preprint arXiv:2506\.13131*, 2025\.
- Rafailov et al\. \[2023\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\.Direct preference optimization: Your language model is secretly a reward model\.*Advances in neural information processing systems*, 36:53728–53741, 2023\.
- Rein et al\. \[2023\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\.Gpqa: A graduate\-level google\-proof q&a benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.
- Romera\-Paredes et al\. \[2024\]B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi, et al\.Mathematical discoveries from program search with large language models\.*Nature*, 625\(7995\):468–475, 2024\.
- Shojaee et al\. \[2024\]P\. Shojaee, K\. Meidani, S\. Gupta, A\. B\. Farimani, and C\. K\. Reddy\.Llm\-sr: Scientific equation discovery via programming with large language models\.*arXiv preprint arXiv:2404\.18400*, 2024\.
- Singhal et al\. \[2025\]R\. Singhal, Z\. Horvitz, R\. Teehan, M\. Ren, Z\. Yu, K\. McKeown, and R\. Ranganath\.A general framework for inference\-time scaling and steering of diffusion models\.*arXiv preprint arXiv:2501\.06848*, 2025\.
- Skreta et al\. \[2025\]M\. Skreta, T\. Akhound\-Sadegh, V\. Ohanesian, R\. Bondesan, A\. Aspuru\-Guzik, A\. Doucet, R\. Brekelmans, A\. Tong, and K\. Neklyudov\.Feynman\-kac correctors in diffusion: Annealing, guidance, and product of experts\.*arXiv preprint arXiv:2503\.02819*, 2025\.
- Valmeekam et al\. \[2022\]K\. Valmeekam, A\. Olmo, S\. Sreedharan, and S\. Kambhampati\.Large language models still can’t plan \(a benchmark for llms on planning and reasoning about change\)\.In*NeurIPS 2022 Foundation Models for Decision Making Workshop*, 2022\.
- Wang et al\. \[2025\]H\. Wang, M\. Skreta, C\. T\. Ser, W\. Gao, L\. Kong, F\. Strieth\-Kalthoff, C\. Duan, Y\. Zhuang, Y\. Yu, Y\. Zhu, et al\.Efficient evolutionary search over chemical space with large language models\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- Wang et al\. \[2022\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\.Self\-consistency improves chain of thought reasoning in language models\.*arXiv preprint arXiv:2203\.11171*, 2022\.
- Wei et al\. \[2022\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou, et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.*Advances in neural information processing systems*, 35:24824–24837, 2022\.
- Wu et al\. \[2025\]F\. Wu, W\. Xuan, X\. Lu, M\. Liu, Y\. Dong, Z\. Harchaoui, and Y\. Choi\.The invisible leash: Why rlvr may or may not escape its origin\.*arXiv preprint arXiv:2507\.14843*, 2025\.
- Wu et al\. \[2023\]L\. Wu, B\. Trippe, C\. Naesseth, D\. Blei, and J\. P\. Cunningham\.Practical and asymptotically exact conditional sampling in diffusion models\.*Advances in Neural Information Processing Systems*, 36:31372–31403, 2023\.
- Yang et al\. \[2024\]C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen\.Large language models as optimizers\.In*The Twelfth International Conference on Learning Representations*, 2024\.
- Zhang et al\. \[2025\]L\. Zhang, P\. Potaptchik, J\. He, Y\. Du, A\. Doucet, F\. Vargas, H\.\-D\. Dau, and S\. Syed\.Accelerated parallel tempering via neural transports\.*arXiv preprint arXiv:2502\.10328*, 2025\.
- Zhao et al\. \[2024\]S\. Zhao, R\. Brekelmans, A\. Makhzani, and R\. Grosse\.Probabilistic inference in language models via twisted sequential monte carlo\.*arXiv preprint arXiv:2404\.17546*, 2024\.

## Appendix AExperimental Hyperparameters

We use the same MATH500 and GPQA\-Diamond datasets asKaran and Du \[[2025](https://arxiv.org/html/2608.26120#bib.bib14)\]\. For MATH500, we evaluate Qwen2\.5\-Math\-7B and Qwen2\.5\-7B\. For GPQA\-Diamond, we evaluate Qwen2\.5\-32B\-Instruct\. All experiments use chain\-of\-thought prompting\.

We use maximum generation lengthTmax=3072T\_\{\\max\}=3072, proposal temperatureτ=0\.25\\tau=0\.25, and1616chunks\. Thus, each chunk contains at most3072/16=1923072/16=192newly generated tokens\.

## Appendix BBroader Impact

This paper studies inference\-time scaling for LLMs through improved sampling methods\. Like many techniques that increase the capability of LLMs, our work may have broader societal implications\. However, these implications are not specific to our method but are instead associated with the development and deployment of LLMs more generally\. We advocate for the responsible and appropriate use of LLMs, with careful consideration of potential risks and impacts\.

## Appendix CDeclaration of Experiments compute resources

We run our results on NVIDIA A100 40G GPUs and NVIDIA A100 80G GPUs\. Each question is fast, while collecting the results for all datasets and all methods can take several days to weeks\.

## Appendix DDeclaration of LLM usage

We used LLMs to assist with coding, manuscript polishing, and proofreading\. We also study relatively lightweight LLMs \(Qwen\-7B/32B\) in this paper\.

## Appendix ELicenses for existing assets

GPQA and MATH500 are both released with MIT license\.