将预训练LLMs重新利用为高保真连续文本自编码器

# Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders
Source: [https://arxiv.org/html/2609.27248](https://arxiv.org/html/2609.27248)
Unnat JainAlexander C\. BergAffiliation:University of California, IrvineAffiliation:\{arkanatp, unnatj, bergac\}@uci\.edu

###### Abstract

Next\-token prediction has enabled highly fluent autoregressive language models, but it represents global structure only indirectly through sequential factorization\. In contrast, high\-fidelity autoencoders have become a standard primitive in image generation, enabling generative models to operate over continuous latent spaces; text lacks a comparably faithful continuous representation\. We propose LLMAE, a method for repurposing a pretrained decoder\-only language model as a continuous text autoencoder by exposing an intermediate fixed\-length latent bottleneck within its internal activations\. Instantiated with a parameter\-efficient 270M Gemma 3 model, LLMAE uses structured attention masks, LoRA adaptation, and KL regularization to learn an autoencoding interface that leverages the generative prior of the original LLM\. We train LLMAE to reconstruct text sequences up to 1024 tokens, significantly improving on this task to achieve near\-perfect reconstruction\. Furthermore, we demonstrate the downstream utility of this representation by training a latent text diffusion model for detailed image captioning using the learned LLMAE autoencoder\. By mapping text into a fixed\-length continuous latent space, our approach provides an effective substrate for downstream adaptation while benefiting from the fluency of the original LLM\.

## 1Introduction

Autoregressive LLMs dominate text generation, demonstrating remarkable efficacy and fluency, in part by capitalizing on the inherently sequential nature of language\. However, this strict token\-by\-token factorization necessitates local decoding decisions, fundamentally constraining the model’s capacity for global foresight and planning[Yao et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib5)\. Furthermore, downstream adaptation remains expensive and non\-trivial[Hu et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib26), as traditional fine\-tuning often induces “catastrophic forgetting”[Kotha et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib10)of the model’s original generative prior[Holtzman et al\. \(2019\)](https://arxiv.org/html/2609.27248#bib.bib11)\. Non\-autoregressive \(NAR\) generation has been studied to circumvent this sequential bottleneck, yet these methods often struggle to model word\-order dependencies explicitly, leading to text that lacks the grammatical correctness and fluency characteristic of autoregressive models[Li et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib9)\. A prominent framework for NAR generation is text diffusion, however, token\-level diffusion remains non\-trivial, requiring specific pre\-training objectives and carefully designed training/sampling procedures[Lou et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib29);[Dieleman et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib42)\.

Drawing inspiration from the visual domain, where high\-fidelity autoencoders are a standard primitive[Rombach et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib48), we propose LLMAE \(LLM AutoEncoder\) to learn a faithful text autoencoder\. By mapping text sequences into fixed\-length continuous representations, we provide a compelling alternative to the autoregressive paradigm for downstream adaptation, mitigating the costs and forgetting inherent in token\-level LLM fine\-tuning\. LLMAE partitions a single lightweight LLM into an integrated encoder and decoder \(Fig\.[1\(c\)](https://arxiv.org/html/2609.27248#S1.F1.sf3)\) by utilizing structured attention masks\. Specifically, we demonstrate that the internal architecture of LLMs can be effectively leveraged for autoencoding, where activations at an intermediate transformer layer aggregate abstract information to serve as latent encodings, while subsequent layers function as a fluent decoder\. This enables high\-fidelity reconstruction while benefiting from the fluency of the original model\. We further add a single\-layer, identity\-initialized additive codec over the latent sequence, which learns a smoother latent interface under KL regularization while also improving the reconstruction quality\. We instantiate LLMAE using a parameter\-efficient 270M\-parameter Gemma 3[Team et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib49)model, and our finetuning recipe utilizing LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib26)requires only 10\.7M trainable parameters\.

Our main contributions in this paper are the following:

- •LLMAE Framework:We introduce a novel architecture for repurposing pre\-trained LLMs as text autoencoders by establishing a fixed\-length continuous latent bottleneck through structured attention masks\.
- •Reconstruction Analysis:Through comprehensive experimentation, we demonstrate near\-perfect reconstruction on text sequences up to 1024 tokens long, significantly outperforming previous text autoencoders\. We further provide detailed ablation studies for our contributions and confirm the intermediate “entropy valley”[Skean et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib27)as the optimal depth for reconstruction\.
- •Downstream Generative Utility:We validate the effectiveness of the autoencoder by training a latent text diffusion model for detailed image captioning\. Operating on frozen LLMAE latents, our lightweight diffusion model \(111M trainable parameters\) generates fluent, high\-quality captions, outperforming larger autoregressive baselines while narrowing the performance gap with frontier VLMs\. These results establish LLMAE as an effective substrate for multimodal integration and complex downstream tasks\.

\(a\)Soft Prompting\(b\)Token Compressor\(c\)LLMAE \(Ours\)
Figure 1:Related LLM\-based autoencoding architecturesa\) Trainable “soft prompts” as latent tokens\. b\) LLM token embeddings compressed as latent tokens\. c\) Our method utilizes attention masks to derive latent tokens from an intermediate layer\.
## 2Related Work

We discuss lines of work most related to this paper\. These include challenges for diffusion modeling for text, inspiration from context compression in LLMs, analysis of what is encoded in different layers of LLMs, and alternate text autoencoding architectures\.

Text Diffusion\.Diffusion for text is in part inspired by the success of diffusion[Ho et al\. \(2020\)](https://arxiv.org/html/2609.27248#bib.bib24)for images where it often operates on compressed continuous representations obtained from pre\-trained autoencoders[Rombach et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib48)\. Recent work shows potential advantages for more semantically grounded encoders[Zheng et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib47)\.

Text diffusion can work via discrete noising processes on tokens,[Shi et al\. \(2024\)](https://arxiv.org/html/2609.27248#bib.bib46);[Nie et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib45);[Von Rütte et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib44);[Lou et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib29), by Gaussian noise on continuous token embeddings[Li et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib43);[Dieleman et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib42);[Shabalin et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib41), or by Gaussian noise on latent embeddings that encode multi\-token or long\-form text[Lovelace et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib30);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib40);[Lovelace et al\. \(2024\)](https://arxiv.org/html/2609.27248#bib.bib38);[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)\. Diffused latent embeddings are commonly decoded autoregressively for fluent output, but most such work has only targeted short texts\. Challenges include achieving low latent\-to\-text conversion error and smoothness in the latent distribution as highlighted in[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib40);[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)\. Learning latent representations that can effectively encode long passages and work for diffusion is still an open challenge[Arriola et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib39)that we address in this paper\.

Context Compression\.Our approach that fine\-tunes an LLM with a bottleneck in an intermediate layer as the basis for a latent representation shares commonalities with research in text compression that adapts pre\-trained LLMs to condense long text prompts into compact representations\. Early methods demonstrated that learnable continuous tokens, whether only at the input layer \(Soft Prompting[Lester et al\. \(2021\)](https://arxiv.org/html/2609.27248#bib.bib36)\) or also integrated into intermediate layers \(Prefix Tuning[Li and Liang \(2021\)](https://arxiv.org/html/2609.27248#bib.bib35)\), can compress context and steer model behavior\. This concept evolved into structural techniques that cache LLM internal activations for compression \(e\.g\., Gist Tokens[Mu et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib34)\)\. The standard transformer attention mask is modified so that query tokens can only attend as far back as the cached activations for the Gist tokens, forming a bottleneck that effectively represents context but has trouble scaling to longer sequences without architecture modification[Petrov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib33)\. The In\-Context Autoencoder \(ICAE\)[Ge et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib32)fine\-tunes a 7B\-parameter LLM to compress context into soft prompts that are decoded by a frozen LLM for reconstruction and task completion\. Our approach shows that using activations of an intermediate layer provides better reconstruction quality and is achievable using a single, lighter weight, 270M\-parameter LLM\.

LLM Representations\.Research exploring the characteristics of different layers in Transformer LLMs suggests the existence of an “entropy valley,”[Skean et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib27)where autoregressive models exhibit a distinct dip in entropy and a peak in representational accuracy at intermediate layers\. Our ablation studies over different choices of layers for the bottleneck used in LLMAE corroborate this \(and other work[Lad et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib28)\) and provide empirical evidence that deep intermediate layers represent the information needed to serve as a good bottleneck in an accessible manner\.

Text Autoencoder Architectures\.Approaches in related recent literature on text autoencoder architecture generally fall into two categories: those using “soft prompts”[Ge et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib32);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib40), shown in Fig\.[1\(a\)](https://arxiv.org/html/2609.27248#S1.F1.sf1)and those compressing LLM output token embeddings into latents for use in diffusion[Lovelace et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib30);[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37), shown in Fig\.[1\(b\)](https://arxiv.org/html/2609.27248#S1.F1.sf2)\.

The “soft prompt” approaches \(Fig\.[1\(a\)](https://arxiv.org/html/2609.27248#S1.F1.sf1)\) can use a variety of models for encoder and decoder\. The In\-Context Autoencoder \(ICAE\)[Ge et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib32)fine\-tunes an LLM with LoRA as an encoder, but uses a fixed LLM as the decoder\. PLANNER[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib40)takes the first k activations from the final layer of a fine\-tuned BERT encoder as the latent and uses an autoregressive GPT\-2 decoder to decode back to text\.

The other category of approach \(Fig\.[1\(b\)](https://arxiv.org/html/2609.27248#S1.F1.sf2)\) compresses LLM output into latents for diffusion\. Latent Diffusion for Language Generation \(LD4LG\)[Lovelace et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib30)repurposes pretrained encoder\-decoder backbones \(such as BART\) by utilizing a Perceiver[Jaegle et al\. \(2021\)](https://arxiv.org/html/2609.27248#bib.bib7)style query\-based compression mechanism to gather information into a fixed\-length latent sequence\. These compressed tokens are then used to condition the autoregressive decoder for short\-form text generation\. COSMOS[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)follows a related approach, taking semantically rich hidden representations from a frozen BERT encoder and utilizing a Perceiver Resampler architecture[Alayrac et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib8)to compress the variable\-length sequence into a fixed\-size latent matrix\. This fixed\-size matrix is later expanded by a symmetric decompressor back into corresponding BERT tokens, which are then linearly projected into vocabulary probabilities for token prediction\.

Our approach \(Fig\.[1\(c\)](https://arxiv.org/html/2609.27248#S1.F1.sf3)\) partitions the same \(fine\-tuned\) LLM for both encoding and decoding, simplifying modeling compared to soft\-prompting and using the same LLM to perform compression instead of adding on a separate compressor and decoder\.

## 3Method

Figure 2:Architectural bottleneck via attention maskingWe propose a layer\-specific attention masking strategy to facilitate an autoencoding flow within an autoregressive decoder\-only backbone\. Color\-coding indicates causal boundaries: tokens of the same color share a unified causal segment, where attention is restricted to the corresponding segment in the preceding layer\. Within the Encoder Block in blue \(0<l≤D0<l\\leq D\), Embed tokens aggregate information by attending to the Input tokens\. Within the Decoder Block in orange \(l\>Dl\>D\), target tokens are decoded by attending exclusively to the resulting latent tokens \(ziz\_\{i\}\) and the autoregressively decoded target tokens\.Our LLMAE approach works by using the activations\{zi\}1​…​K\\\{z\_\{i\}\\\}\_\{1\\ldots K\}from one intermediate layer of an autoregressive LLM for a sequence ofKKlearnable “Embed” tokens as the basis for a latent encoding of text\. The LLM is fine\-tuned as an autoencoder using LoRA treating those activations as a bottleneck\. This is achieved by modifying the standard attention mask so that the decoder—the part of the LLM downstream from the bottleneck—does not attend to anything before the bottleneck\. This modeling approach is illustrated in Fig\.[2](https://arxiv.org/html/2609.27248#S3.F2)\. The rest of this section goes into detail of the model, describes the addition of a single layer transformer as an additive refinement, lays out the training objective functions, and describes training details\.

##### Model Details\.

We partition the depthLLof the model into two functional blocks: an Encoder Block \(0≤l≤D0\\leq l\\leq D\) and a Decoder Block \(D<l≤L−1D<l\\leq L\-1\)\. At the transition layerDD, the activations corresponding to theKKsequence positions where the learnable Embed tokenseewere placed define the latent representation𝐳∈ℝK×d\\mathbf\{z\}\\in\\mathbb\{R\}^\{K\\times d\}\. Therefore,𝐳\\mathbf\{z\}represents the transformation of the original Embed tokens through the depth of the Encoder Block\. At layerD\+1D\+1within the subsequent Decoder Block, the masking constraints are relaxed to allow the Output tokens to attend to the resulting latent states𝐳\\mathbf\{z\}while maintaining a strict mask against the original Input sequence\. Since the Output tokens always remain isolated from the Input tokens, this configuration ensures that the reconstruction process is channeled through the latent bottleneck\. For more details, we specify the attention mask modification in Table[6](https://arxiv.org/html/2609.27248#A2.T6)in Appendix[B](https://arxiv.org/html/2609.27248#A2)\. Formally, this partitioned flow can be expressed as:

𝐳=Encoder\(𝐱\);𝐱^=Decoder\(𝐳,𝐱^<i\)\\mathbf\{z\}=\\text\{Encoder\}\(\\mathbf\{x\}\);\\text\{ \}\\hat\{\\mathbf\{x\}\}=\\text\{Decoder\}\(\\mathbf\{z\},\\hat\{\\mathbf\{x\}\}\_\{<i\}\)For encoding and downstream modeling,𝐳\\mathbf\{z\}can be cached as the latent representation of the input\. At inference time for decoding, we inject the latent𝐳\\mathbf\{z\}as input to layerD\+1D\+1corresponding to the Embed tokens, resulting in an inference\-time complexity comparable to standard LLM decoding\.

##### Additive Codec Refinement\.

To address the flexibility constraints of low\-rank adaptation \(LoRA\), we include an additive Transformer Codec that refines the latent activations\. The codec utilizes single\-layer Transformer blocks \(𝒯e​n​c,𝒯d​e​c\\mathcal\{T\}\_\{enc\},\\mathcal\{T\}\_\{dec\}\) with per\-dimension learnable scalar multipliers \(α∈ℝD\\alpha\\in\\mathbb\{R\}^\{D\}\) initialized to zero at the start of training\. The refinement is computed as:

𝐳c​o​d​e​c=𝐳\+αe​n​c⊙𝒯e​n​c​\(𝐳\);𝐳^=𝐳c​o​d​e​c\+αd​e​c⊙𝒯d​e​c​\(𝐳c​o​d​e​c\)\\mathbf\{z\}\_\{codec\}=\\mathbf\{z\}\+\\mathbf\{\\alpha\}\_\{enc\}\\odot\\mathcal\{T\}\_\{enc\}\(\\mathbf\{z\}\);\\quad\\hat\{\\mathbf\{z\}\}=\\mathbf\{z\}\_\{codec\}\+\\alpha\_\{dec\}\\odot\\mathcal\{T\}\_\{dec\}\(\\mathbf\{z\}\_\{codec\}\)The multipliers are shared across allKKtokens but learned per hidden dimensionDD\. We enforce the replacement𝐳←𝐳^\\mathbf\{z\}\\leftarrow\\hat\{\\mathbf\{z\}\}to ensure the decoder conditions on the refined latent, stashing the detached𝐳c​o​d​e​c\\mathbf\{z\}\_\{codec\}as the definitive latent representation for KL regularization during training and downstream inference\. Because the codec is additive by design, we finetune it jointly over the LLMAE backbone\.

### 3\.1Training Objectives

The reconstruction task is formulated by constructing a composite sequence𝐬\\mathbf\{s\}that concatenates the input source, the introduced Embed tokens, and the reconstruction target\. Let𝐱=\{x1,…,xN\}\\mathbf\{x\}=\\\{x\_\{1\},\\dots,x\_\{N\}\\\}denote the input sequence and𝐞=\{e1,…,eK\}\\mathbf\{e\}=\\\{e\_\{1\},\\dots,e\_\{K\}\\\}represent the learnable Embed tokens\. Shorter sequences are padded to reach theNNtoken limit\. The full training sequence is constructed as:

𝐬=\[𝐱e​n​c\]\|\[𝐞\]\|\[𝐱d​e​c\]\\mathbf\{s\}=\[\\mathbf\{x\}\_\{enc\}\]\\parallel\[\\mathbf\{e\}\]\\parallel\[\\mathbf\{x\}\_\{dec\}\]where𝐱e​n​c\\mathbf\{x\}\_\{enc\}and𝐱d​e​c\\mathbf\{x\}\_\{dec\}are identical sequences of tokensxi∈𝒱x\_\{i\}\\in\\mathcal\{V\}\. Training follows the Next\-Token Prediction \(NTP\) paradigm\. The total loss is a weighted sum:

ℒ=ℒL​L​M\+λK​L−z​ℒK​L−z\+λK​L−r​e​f​ℒK​L−r​e​f\\mathcal\{L\}=\\mathcal\{L\}\_\{LLM\}\+\\lambda\_\{KL\-z\}\\mathcal\{L\}\_\{KL\-z\}\+\\lambda\_\{KL\-ref\}\\mathcal\{L\}\_\{KL\-ref\}
whereℒL​L​M\\mathcal\{L\}\_\{LLM\}is the standard NTP cross\-entropy loss computed exclusively over the finalNNpositions corresponding to𝐱d​e​c\\mathbf\{x\}\_\{dec\}\. Gradients flow back through the decoding layers into the latentzzand through the entirety of the encoding block and the Embed tokens\. We also include two forms of KL regularization objectives\. First, we penalize the divergence between the continuous latent representation𝐳\\mathbf\{z\}and a standard normal prior𝒩⁡\(0,𝐈\)\\mathcal\{N\}\(0,\\mathbf\{I\}\)\. This penalty exerts a bottleneck pressure to regularize the latent representations into a smooth manifold that is more amenable to downstream generative modeling\. The lossℒK​L−z\\mathcal\{L\}\_\{KL\-z\}is then calculated as the analytical KL divergence \(seeEq\.[1](https://arxiv.org/html/2609.27248#A3.E1)in Appendix[C](https://arxiv.org/html/2609.27248#A3)\) by fitting a moment\-matching Gaussianqk​d=𝒩⁡\(μ^k​d,σ^k​d2\)q\_\{kd\}=\\mathcal\{N\}\(\\hat\{\\mu\}\_\{kd\},\\hat\{\\sigma\}^\{2\}\_\{kd\}\)where the meanμ^k​d\\hat\{\\mu\}\_\{kd\}and varianceσ^k​d2\\hat\{\\sigma\}\_\{kd\}^\{2\}are estimated across batch samples for each of the latent coordinates\. To further encourage linguistic fluency, we anchor the model to the initial, frozen, Gemma model referenceπr​e​f\\pi\_\{ref\}\. This penalty measures the KL divergenceKL\(πθ∥πr​e​f\)KL\(\\pi\_\{\\theta\}\\parallel\\pi\_\{ref\}\), calculated as the KL divergence summed over the entire vocabulary\-wide softmax distribution at each active output position, anchoring the LLMAE decoder to the Gemma checkpoint\. The reference softmax is computed via a plain LLM forward pass on the target output tokens, performed entirely without encoding input or Embed tokens\.

![Refer to caption](https://arxiv.org/html/2609.27248v1/captioning_diffusion_training.png)\(a\)Training
![Refer to caption](https://arxiv.org/html/2609.27248v1/captioning_diffusion_inference.png)\(b\)Inference

Figure 3:Latent Diffusion for CaptioningA lightweight denoising score network conditioned on frozen SigLIP image features through cross\-attention\. LLMAE remains frozen throughout\.
### 3\.2Training Details

We instantiate LLMAE withGemma\-3\-270Mas the pre\-trained LLM backbone, and employN=1024N=1024input token length andK=256K=256Embed token length\. We train on a curated 1\.4 million samples corpus by using two text data sources: 1M samples of Pile\-uncopyrighted[Biderman et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib20), and 400K samples of C4\-realnewslike[Raffel et al\. \(2020\)](https://arxiv.org/html/2609.27248#bib.bib23)\. To ensure the fixed\-length latent token bottleneck generalizes across variable sequence lengths, we implement a stratified collection algorithm for each source\. Data is partitioned into 41 character\-length buckets to achieve a nearly uniform length distribution, preventing the model from favoring the medium\-length modes typical of natural language\. We also found it crucial to adopt a curriculum learning schedule, where samples are sorted by ascending length to stabilize early convergence before confronting longer reconstruction\. We trained LLMAE using 8 NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, where the full training took≈22\\approx 22hours\.

## 4Experiments

Our experiments focus on three core objectives: reconstruction fidelity, representational smoothness, and downstream generative utility\. We first evaluate LLMAE’s autoencoding performance against existing baselines \(Sec\.[4\.1\.3](https://arxiv.org/html/2609.27248#S4.SS1.SSS3)\), while conducting systematic ablations on the latent bottleneck \(Sec\.[4\.1\.4](https://arxiv.org/html/2609.27248#S4.SS1.SSS4)\)\. We then analyze the distribution of the latent space \(Sec\.[4\.2](https://arxiv.org/html/2609.27248#S4.SS2)\), evaluating the impact of KL regularization and the additive codec\. Finally, we demonstrate the representation’s utility \(Sec\.[4\.3](https://arxiv.org/html/2609.27248#S4.SS3)\) by training a latent diffusion model for detailed image captioning\.

### 4\.1Reconstruction Quality

#### 4\.1\.1Baselines

##### ICAE\.

We use the publicly released ICAE[Ge et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib32)v2checkpoint, which fine\-tunes the 7B LLMMistral\-7B\-Instruct\-v0\.2[Jiang et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib4)at a compression ratio of4×4\\times\. We use the Stage 1 \(reconstruction\-only\) pretrained checkpoint, prior to any instruction fine\-tuning for context compression\.

##### COSMOS\.

We use the publicly released COSMOS[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)autoencoder checkpoint that encodes sequences up to 512 tokens to 512 latent tokens \(no compression\)\. COSMOS was not trained to support inputs beyond 512 tokens\.

#### 4\.1\.2Evaluation Datasets

We evaluate on three held\-out test sets of samples drawn from different text domains to assess generalization across writing styles and content types\. Since the COSMOS baseline[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)is limited to a 512\-token length, we report results on two test samples for each source in Table[1](https://arxiv.org/html/2609.27248#S4.T1): a medium\-length sample \(up to 512 tokens\) and a long\-length sample \(up to 1024 tokens\)\.

##### C4\-RealNewsLike \(stratified\)\.

We sample from the C4[Raffel et al\. \(2020\)](https://arxiv.org/html/2609.27248#bib.bib23)corpus with theRealNewsLikefilter for news\-like domains\. To ensure a robust evaluation across varying context lengths, we perform stratified sampling, partitioned into 41 character\-length buckets\. Unless specified otherwise, we use this source with stratified sampling up to 1024 tokens to measure reconstruction quality in this paper\.

##### OpenWebText\.

OpenWebText[Gokaslan et al\. \(2019\)](https://arxiv.org/html/2609.27248#bib.bib21)is an open\-source reproduction of the WebText dataset used to train GPT\-2\. We sample from the specific held\-out test split shared by COSMOS[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37), which allows us to evaluate our model on the distribution used to train the COSMOS autoencoder checkpoint\.

##### CreationMMBench\.

CreationMMBench[Fang et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib22)offers a test benchmark of image\-grounded long\-form text generation tasks spanning four categories: Literary Writing, Common Functional Writing, Professional Functional Writing, and Creative Multimodal Understanding\. We evaluate on thereference\_answer\_by\_gpt4ofield from the benchmark dataset, generated by GPT\-4o\.

Table 1:LLMAE exhibits near\-perfect reconstruction across different domains and sequence lengths\. We report BLEU\-4 \(↑\\uparrow\)[Papineni et al\. \(2002\)](https://arxiv.org/html/2609.27248#bib.bib3), Perplexity \(↓\\downarrow\)[Radford et al\. \(2019\)](https://arxiv.org/html/2609.27248#bib.bib1), and BERTScore F1 \(↑\\uparrow\)[Zhang et al\. \(2019\)](https://arxiv.org/html/2609.27248#bib.bib2)\. We report on two test samples for each source, up to 512 tokens \(upper\), and up to 1024 tokens \(lower\)\.

#### 4\.1\.3Reconstruction Performance

We demonstrate near\-perfect reconstruction in Table[1](https://arxiv.org/html/2609.27248#S4.T1), maintaining BLEU\-4 scores near or exceeding0\.990\.99, significantly outperforming both ICAE and COSMOS\. We also achieve high BERT scores, and PPL metrics \(computed with GPT\-2\-Large\) close to the source text\. We observe that ICAE’s reconstruction quality degrades as sequence length increases, with its BLEU score dropping from0\.8310\.831to0\.7100\.710on the C4\-News\-Stratified split\. This demonstrates that LLMAE provides a more robust bottleneck than the soft\-memory compression used in ICAE, despite being an order of magnitude more parameter\-efficient \(278\.8M vs\. 7\.46B parameters\)\. Notably, we also attempted to train an ICAE\-style autoencoder using the lightweight Gemma backbone used in LLMAE, but were unable to achieve high\-quality reconstruction \(Appendix[E](https://arxiv.org/html/2609.27248#A5)\), suggesting that simply applying soft\-prompt compression to smaller models is insufficient for high quality reconstruction\. Finally, our ablation "w/o Codec" demonstrates that while the core architecture is highly capable, the 9\.8M parameter additive refinement was essential for closing the remaining gap to perfect reconstruction\.

Table 2:Quality w\.r\.t\. latent layer depth\.
Table 3:Quality w\.r\.t\. number of latent tokens\.

#### 4\.1\.4Latent Bottleneck Ablations

We systematically vary the intermediate latent depthDDand the number of latent tokensKKto measure their impact on reconstruction quality\. As shown in Table[3](https://arxiv.org/html/2609.27248#S4.T3), we observe a clear “entropy valley” effect within the 18\-layer Gemma backbone\. The reconstruction performance peaks at a deep intermediate layer \(11\), whereas shallow layers \(layers 2, 5, 8\) and late stage layers \(layer 14\) are not effective\. We also evaluate the effect of the bottleneck widthKKin Table[3](https://arxiv.org/html/2609.27248#S4.T3)\. We observe optimal reconstruction atK=256K=256tokens \(4×4\\timescompression ratio\)\. Notably, increasing the latent capacity toK=512K=512or10241024leads to a slight degradation in performance\. We hypothesizeK=256K=256matches the natural granularity of the LLM’s internal representation for the 1024\-token context window at the11t​h11^\{th\}transformer layer; while lower values \(e\.g\.,K=64K=64\) suffer from information collapse\.

### 4\.2Latent Space Distribution

Table 4:Latent space distribution statistics and reconstruction quality\.We evaluate the distribution statistics of the latent spaceZ∈ℝN×K×DZ\\in\\mathbb\{R\}^\{N\\times K\\times D\}, whereN=500N=500is the number of test samples,K=256K=256is the latent token length, andD=640D=640is the latent dimension\. We report the latent distributional statistics in Table[4](https://arxiv.org/html/2609.27248#S4.T4)by computing global empirical moments mean, standard deviation, and maximum absolute value, pooled across alln,t,dn,t,dindices\. We observe that default training activations exhibit highly irregular distributions; however, we normalize these statistics effectively through the integration of KL regularization and the codec layer\. This stabilization of the latent manifold occurs alongside a concurrent improvement in reconstruction quality\.

### 4\.3Latent Diffusion for Image Captioning

Table 5:Detailed image captioning\. We report length \(in words\), VLM Judge Score[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib31), CapArena\-Auto[Cheng et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib15), RefCLIPScore[Hessel et al\. \(2021\)](https://arxiv.org/html/2609.27248#bib.bib14)and Perplexity\. Our latent diffusion model comprises 819M total parameters, with the 111M\-parameter score network being the only trainable component for this task\.![Refer to caption](https://arxiv.org/html/2609.27248v1/images/test_02641.jpg)VLM Judge Score:6/6Caption:A full shot captures a wooden bench positioned amidst a natural, forest\-like setting\. The wooden bench, the focal point of the image, prominently features a curved back, armrests, and vertical slats that enhance its classic design\. It rests on a rectangular stone platform, flanked by two stout, light\-colored bases\. The ground surrounding the bench or platform is covered in small, light gray gravel, creating a neutral and textured surface\. The background consists of dense foliage, predominantly leafless trees and bushes, indicating a natural and serene environment\. These branches reach towards the sky, lending a sense of depth and enclosure to the scene\.The lighting appears soft and diffused, casting gentle shadows across the scene, suggesting an overcast day\. The dominant colors are muted greens, greys, and a muted touch of somber tones\. The texture varies from the smooth, weathered wood of the bench to the rough stone and the smooth bark of the trees and branches\.

There are no people or animals visible in the frame\. The overall mood is peaceful, evoking a sense of quiet contemplation within the natural surroundings\. The color palette is dominated by shades of brown, beige, and grey, which contribute to the earthy and muted aesthetic of the scene\.

Figure 4:Latent Diffusion with LLMAEQualitative demonstration of detailed captioning ability\.We demonstrate downstream utility on detailed image captioning, a multimodal task that remains largely unexplored via latent diffusion despite the paradigm’s dominance in text\-to\-image synthesis[Rombach et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib48)\. By keeping both the visual and linguistic representations frozen \(Fig\.[3](https://arxiv.org/html/2609.27248#S3.F3)\) and training only a lightweight 111M\-parameter diffusion model, we enable an efficient training pipeline where cached latents minimize redundant computation across epochs\. By operating on a global latent representation, we can utilize a standard denoising score matching objective in the normalized latent space:

L=𝔼z0,t,ϵ​\[\|z^0​\(zt,t,I\)−z0\|2\]L=\\mathbb\{E\}\_\{z\_\{0\},t,\\epsilon\}\\left\[\|\\hat\{z\}\_\{0\}\(z\_\{t\},t,I\)\-z\_\{0\}\|^\{2\}\\right\]whereIIrepresents the image conditioning\. The score network is a 12\-layer transformer, directly adapted from[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37)\. Further architectural details of the score network are provided in Appendix[F](https://arxiv.org/html/2609.27248#A6)\. Image conditioning is provided by a frozen SigLIP[Zhai et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib25)encoder\. The diffusion model operates on the LLMAE continuous latents, which are subsequently decoded into text by the frozen LLMAE decoder during inference\. A linear layer projects the 729 SigLIP patch tokens from their 1152\-dimensional embedding space to the 640\-dimensional score network width\. Image conditioning is facilitated via cross\-attention between the score network intermediate layers and these projected visual features\.

The current landscape of detailed image captioning is dominated by autoregressive VLMs\. To achieve high descriptive fidelity, these models are typically trained on massive image\-text datasets for modality alignment\. A prominent example is the ShareGPT4V\-PT dataset, which consists of 1\.2 million detailed captions generated by Share\-Captioner[Chen et al\. \(2024\)](https://arxiv.org/html/2609.27248#bib.bib19), a specialized captioning model\. Such datasets are now commonly included in the training pipelines of frontier VLMs\. As an alternative recipe, the Vision\-Language\-Vision \(VLV\)[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib31)framework builds an image encoder in the continuous language space, by distilling information from a pre\-trained text\-to\-image diffusion model\. The image encoding is decoded back into text by a fine\-tuned LLM\. They finetune the image encoder on 40M LAION\-Aesthetic[Schuhmann et al\. \(2022\)](https://arxiv.org/html/2609.27248#bib.bib13)images and finetune the captioning model on the publicly available VLV\-6M dataset: a 6 million subset of LAION\-Aesthetic with captions generated by Gemini\-2\-Flash\.

We utilize the VLV\-6M dataset to train our latent diffusion model from scratch\. Notably, while both Share\-Captioner and VLV also involve fine\-tuning the underlying LLM backbone for the captioning task, our approach maintains a completely frozen LLMAE autoencoder as well as a frozen SigLIP image encoder\. This enables a lightweight training process that achieves high\-quality captioning purely by sampling in LLMAE’s native latent space\.

To assess the captioning performance, we evaluate our model on theCapArena\-Autobenchmark[Cheng et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib15)\. CapArena is a human\-preference benchmark for detailed captioning, consisting of thousands of pairwise "battles" to establish a preference\-based Elo ranking\. CapArena\-Auto serves as an automated proxy for this protocol; it utilizes a fixed set of 600 test images from the DOCCI dataset[Onoe et al\. \(2024\)](https://arxiv.org/html/2609.27248#bib.bib12)and employs GPT\-4o as a pairwise judge\. This automated protocol was shown to achieve a 94\.3% correlation with human model rankings while significantly reducing evaluation overhead[Cheng et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib15)\.

We report the captioning performance in Table[5](https://arxiv.org/html/2609.27248#S4.T5), where we include CapArena\-Auto and three additional metrics to capture distinct measurements on the same 600\-image test set used by CapArena:

- leftmargin=\*,noitemsepCapArena\-Auto Score:The mean normalized win rate against 3 reference VLM captioners\.
- leftmargin=\*,noitemsepVLM Judge \(0–6\):We adopt the system prompt with a 7\-point rubric exactly from[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib31)\. This system comprehensively evaluates the coverage of image elements, spatial layout consistency, and the absence of hallucinations\. We use GPT\-4o as the VLM judge \(snapshotgpt\-4o\-2024\-08\-06\)\.
- leftmargin=\*,noitemsepRefCLIPScore:The harmonic mean of CLIP similarities between \(caption, image\) and \(caption, reference\) using OpenCLIP ViT\-B/32, capturing the generated caption’s semantic alignment[Hessel et al\. \(2021\)](https://arxiv.org/html/2609.27248#bib.bib14)\.
- leftmargin=\*,noitemsepPerplexity \(PPL\):The perplexity of generated captions to measure the linguistic fluency\.

In addition to VLV and Share\-Captioner, we evaluate against three representative models from the CapArena leaderboard: Gemini\-1\.5\-Pro is the best performer, LLaVA\-1\.5\-7B as a poor performer, and Qwen2\-VL\-2B as relatively lightweight VLM\. Our lightweight latent diffusion model achieves a competitive VLM Judge score of 3\.44, outperforming both the 7\.1B LLaVA\-1\.5 \(2\.89\) and the specialized Share\-Captioner \(3\.34\)\. Generative reliability is high, with none of captions marked as non\-informative \(score 0\) and only 12\.5% receiving a score of 1; the remaining 87\.5% achieve a score of 2 or higher, which designates a caption as "informative" with valid semantic grounding under the VLV rubric\. Substituting our autoencoder with COSMOS[Meshchaninov et al\. \(2025\)](https://arxiv.org/html/2609.27248#bib.bib37), while maintaining the identical diffusion architecture and parameters, which itself is adapted from their original work, results in a significant performance degradation to a 2\.24 VLM score and 48\.5 PPL\. This confirms that the high\-fidelity LLMAE autoencoder is better suited for this task\. A granular analysis of the failure modes \(Appendix[A](https://arxiv.org/html/2609.27248#A1)\) reveals that the model demonstrates high fidelity in describing aesthetic scenes and atmospheric nuances, but exhibits limited grounding in symbolic tasks such as OCR\.

## 5Limitations

Despite the strong performance and efficiency of LLMAE, some limitations remain\. First, our empirical study is based on the Gemma 270M backbone for LLMAE; future work is needed to explore other LLM families\. Second, because LLMAE repurposes a pretrained LLM as an autoregressive decoder, it inherits the sequential latency of text generation\. While the encoding process is efficient, decoding long outputs remains subject to the computational downsides of autoregressive sampling\. We note that because the masking protocol respects causal ordering in each layer, the framework is fully compatible with standard KV caching[Pope et al\. \(2023\)](https://arxiv.org/html/2609.27248#bib.bib6)\. Third, our autoencoder training is currently limited to 1024\-token text, that roughly translates to 4K characters\. We did not perform any experiments for longer sequence lenghts\. Finally, we observe a fundamental constraint involving the balance of autoencoding fidelity, compression, and latent smoothness\. By prioritizing high\-resolution, near\-lossless reconstruction, LLMAE creates a dense latent manifold that may require a strong conditional signal for generative modeling\. Future research may help address some of these concerns without sacrificing the reconstruction quality\.

## 6Conclusion

We introduced LLMAE, a framework for repurposing pre\-trained decoder\-only language models as high\-fidelity continuous text autoencoders\. By establishing a fixed\-length latent bottleneck through structured attention masks and lightweight, parameter\-efficient fine\-tuning, we demonstrated that standard LLM architectures can be adapted to represent long\-form text \(up to 1024 tokens\) with near\-perfect reconstruction\. The empirical confirmation of the “entropy valley” at deep intermediate layers provides a principled basis for extracting dense latents for tetxt reconstruction\.

Furthermore, our results in latent text diffusion for detailed image captioning validate the downstream utility of this representation\. Ultimately, LLMAE offers a parameter\-efficient text autoencoder for grounding continuous generative models in the rich priors of LLMs, enabling flexible architectures for multimodal tasks and contributing to more structurally aware, globally consistent text generation\.

## References

- \[1\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.Advances in neural information processing systems36,pp\. 11809–11822\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1)\.
- \[2\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1),[§1](https://arxiv.org/html/2609.27248#S1.p2.1)\.
- \[3\]S\. Kotha, J\. M\. Springer, and A\. Raghunathan\(2023\)Understanding catastrophic forgetting in language models via implicit inference\.arXiv preprint arXiv:2309\.10105\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1)\.
- \[4\]A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi\(2019\)The curious case of neural text degeneration\.arXiv preprint arXiv:1904\.09751\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1)\.
- \[5\]Y\. Li, K\. Zhou, W\. X\. Zhao, and J\. Wen\(2023\)Diffusion models for non\-autoregressive text generation: a survey\.arXiv preprint arXiv:2303\.06574\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1)\.
- \[6\]A\. Lou, C\. Meng, and S\. Ermon\(2023\)Discrete diffusion modeling by estimating the ratios of the data distribution\.arXiv preprint arXiv:2310\.16834\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1),[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[7\]S\. Dieleman, L\. Sartran, A\. Roshannai, N\. Savinov, Y\. Ganin, P\. H\. Richemond, A\. Doucet, R\. Strudel, C\. Dyer, C\. Durkan,et al\.\(2022\)Continuous diffusion for categorical data\.arXiv preprint arXiv:2211\.15089\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p1.1),[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[8\]R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer\(2022\)High\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 10684–10695\.Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p2.1),[§2](https://arxiv.org/html/2609.27248#S2.p2.1),[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p1.1)\.
- \[9\]G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2609.27248#S1.p2.1)\.
- \[10\]O\. Skean, M\. R\. Arefin, D\. Zhao, N\. N\. Patel, J\. Naghiyev, Y\. Lecun, and R\. Shwartz\-Ziv\(2025\)Layer by layer: uncovering hidden representations in language models\.InInternational Conference on Machine Learning,pp\. 55854–55875\.Cited by:[2nd item](https://arxiv.org/html/2609.27248#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2609.27248#S2.p5.1)\.
- \[11\]J\. Ho, A\. Jain, and P\. Abbeel\(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p2.1)\.
- \[12\]B\. Zheng, N\. Ma, S\. Tong, and S\. Xie\(2025\)Diffusion transformers with representation autoencoders\.arXiv preprint arXiv:2510\.11690\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p2.1)\.
- \[13\]J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. Titsias\(2024\)Simplified and generalized masked diffusion for discrete data\.Advances in neural information processing systems37,pp\. 103131–103167\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[14\]S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li\(2025\)Large language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[15\]D\. Von Rütte, J\. Fluri, Y\. Ding, A\. Orvieto, B\. Schölkopf, and T\. Hofmann\(2025\)Generalized interpolating discrete diffusion\.arXiv preprint arXiv:2503\.04482\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[16\]X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto\(2022\)Diffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[17\]A\. Shabalin, V\. Meshchaninov, E\. Chimbulatov, V\. Lapikov, R\. Kim, G\. Bartosh, D\. Molchanov, S\. Markov, and D\. Vetrov\(2025\)Tencdm: understanding the properties of the diffusion model in the space of language model encodings\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25110–25118\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[18\]J\. Lovelace, V\. Kishore, C\. Wan, E\. Shekhtman, and K\. Q\. Weinberger\(2023\)Latent diffusion for language generation\.Advances in Neural Information Processing Systems36,pp\. 56998–57025\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1),[§2](https://arxiv.org/html/2609.27248#S2.p6.1),[§2](https://arxiv.org/html/2609.27248#S2.p8.1)\.
- \[19\]Y\. Zhang, J\. Gu, Z\. Wu, S\. Zhai, J\. Susskind, and N\. Jaitly\(2023\)Planner: generating diversified paragraph via latent language diffusion model\.Advances in Neural Information Processing Systems36,pp\. 80178–80190\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1),[§2](https://arxiv.org/html/2609.27248#S2.p6.1),[§2](https://arxiv.org/html/2609.27248#S2.p7.1)\.
- \[20\]J\. Lovelace, V\. Kishore, Y\. Chen, and K\. Weinberger\(2024\)Diffusion guided language modeling\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 14936–14952\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[21\]V\. Meshchaninov, E\. Chimbulatov, A\. Shabalin, A\. Abramov, and D\. Vetrov\(2025\)Cosmos: compressed and smooth latent space for text diffusion modeling\.arXiv preprint arXiv:2506\.21170\.Cited by:[§F\.2](https://arxiv.org/html/2609.27248#A6.SS2.p1.1),[§2](https://arxiv.org/html/2609.27248#S2.p3.1),[§2](https://arxiv.org/html/2609.27248#S2.p6.1),[§2](https://arxiv.org/html/2609.27248#S2.p8.1),[§4\.1\.1](https://arxiv.org/html/2609.27248#S4.SS1.SSS1.Px2.p1.1),[§4\.1\.2](https://arxiv.org/html/2609.27248#S4.SS1.SSS2.Px2.p1.1),[§4\.1\.2](https://arxiv.org/html/2609.27248#S4.SS1.SSS2.p1.1),[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p1.2),[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p6.1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.5.5.1.1),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.9.1.1)\.
- \[22\]M\. Arriola, A\. Gokaslan, J\. T\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. S\. Sahoo, and V\. Kuleshov\(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.arXiv preprint arXiv:2503\.09573\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p3.1)\.
- \[23\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 3045–3059\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p4.1)\.
- \[24\]X\. L\. Li and P\. Liang\(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4582–4597\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p4.1)\.
- \[25\]J\. Mu, X\. Li, and N\. Goodman\(2023\)Learning to compress prompts with gist tokens\.Advances in Neural Information Processing Systems36,pp\. 19327–19352\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p4.1)\.
- \[26\]A\. Petrov, M\. Sandler, A\. Zhmoginov, N\. Miller, and M\. Vladymyrov\(2025\)Long context in\-context compression by getting to the gist of gisting\.arXiv preprint arXiv:2504\.08934\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p4.1)\.
- \[27\]T\. Ge, J\. Hu, L\. Wang, X\. Wang, S\. Chen, and F\. Wei\(2023\)In\-context autoencoder for context compression in a large language model\.arXiv preprint arXiv:2307\.06945\.Cited by:[Appendix E](https://arxiv.org/html/2609.27248#A5.p1.1),[§2](https://arxiv.org/html/2609.27248#S2.p4.1),[§2](https://arxiv.org/html/2609.27248#S2.p6.1),[§2](https://arxiv.org/html/2609.27248#S2.p7.1),[§4\.1\.1](https://arxiv.org/html/2609.27248#S4.SS1.SSS1.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.5.4.1.1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.5.8.1.1)\.
- \[28\]V\. Lad, J\. H\. Lee, W\. Gurnee, and M\. Tegmark\(2025\)The remarkable robustness of llms: stages of inference?\.Advances in Neural Information Processing Systems\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p5.1)\.
- \[29\]A\. Jaegle, F\. Gimeno, A\. Brock, O\. Vinyals, A\. Zisserman, and J\. Carreira\(2021\)Perceiver: general perception with iterative attention\.InInternational conference on machine learning,pp\. 4651–4664\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p8.1)\.
- \[30\]J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.Advances in neural information processing systems35,pp\. 23716–23736\.Cited by:[§2](https://arxiv.org/html/2609.27248#S2.p8.1)\.
- \[31\]S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational conference on machine learning,pp\. 2397–2430\.Cited by:[§3\.2](https://arxiv.org/html/2609.27248#S3.SS2.p1.1)\.
- \[32\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§3\.2](https://arxiv.org/html/2609.27248#S3.SS2.p1.1),[§4\.1\.2](https://arxiv.org/html/2609.27248#S4.SS1.SSS2.Px1.p1.1)\.
- \[33\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§4\.1\.1](https://arxiv.org/html/2609.27248#S4.SS1.SSS1.Px1.p1.1)\.
- \[34\]A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. Tellex\(2019\)OpenWebText corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§4\.1\.2](https://arxiv.org/html/2609.27248#S4.SS1.SSS2.Px2.p1.1)\.
- \[35\]X\. Fang, Z\. Chen, K\. Lan, L\. Ma, S\. Ding, Y\. Liang, X\. Zhao, F\. Wen, Z\. Zhang, G\. Zhang,et al\.\(2025\)Creation\-mmbench: assessing context\-aware creative intelligence in mllms\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 447–456\.Cited by:[§4\.1\.2](https://arxiv.org/html/2609.27248#S4.SS1.SSS2.Px3.p1.1)\.
- \[36\]K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu\(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[Table 1](https://arxiv.org/html/2609.27248#S4.T1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.4)\.
- \[37\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[Table 1](https://arxiv.org/html/2609.27248#S4.T1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.4)\.
- \[38\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2019\)Bertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[Table 1](https://arxiv.org/html/2609.27248#S4.T1),[Table 1](https://arxiv.org/html/2609.27248#S4.T1.4)\.
- \[39\]T\. Zhang, Y\. Li, Y\. Chou, J\. Chen, A\. Yuille, C\. Wei, and J\. Xiao\(2025\)Vision\-language\-vision auto\-encoder: scalable knowledge distillation from diffusion models\.arXiv preprint arXiv:2507\.07104\.Cited by:[Figure 6](https://arxiv.org/html/2609.27248#A1.F6),[Figure 6](https://arxiv.org/html/2609.27248#A1.F6.4),[2nd item](https://arxiv.org/html/2609.27248#S4.I1.i2.p1.1),[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p2.1),[Table 5](https://arxiv.org/html/2609.27248#S4.T5),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.4),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.6.1.1)\.
- \[40\]K\. Cheng, W\. Song, J\. Fan, Z\. Ma, Q\. Sun, F\. Xu, C\. Yan, N\. Chen, J\. Zhang, and J\. Chen\(2025\)Caparena: benchmarking and analyzing detailed image captioning in the llm era\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 14077–14094\.Cited by:[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p4.1),[Table 5](https://arxiv.org/html/2609.27248#S4.T5),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.4)\.
- \[41\]J\. Hessel, A\. Holtzman, M\. Forbes, R\. Le Bras, and Y\. Choi\(2021\)Clipscore: a reference\-free evaluation metric for image captioning\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 7514–7528\.Cited by:[3rd item](https://arxiv.org/html/2609.27248#S4.I1.i3.p1.1),[Table 5](https://arxiv.org/html/2609.27248#S4.T5),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.4)\.
- \[42\]G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang,et al\.\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.3.1.1)\.
- \[43\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2024\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 26296–26306\.Cited by:[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.4.1.1)\.
- \[44\]P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.5.1.1)\.
- \[45\]L\. Chen, J\. Li, X\. Dong, P\. Zhang, C\. He, J\. Wang, F\. Zhao, and D\. Lin\(2024\)Sharegpt4v: improving large multi\-modal models with better captions\.InEuropean Conference on Computer Vision,pp\. 370–387\.Cited by:[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p2.1),[Table 5](https://arxiv.org/html/2609.27248#S4.T5.5.7.1.1)\.
- \[46\]X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer\(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p1.2)\.
- \[47\]C\. Schuhmann, R\. Beaumont, R\. Vencu, C\. Gordon, R\. Wightman, M\. Cherti, T\. Coombes, A\. Katta, C\. Mullis, M\. Wortsman,et al\.\(2022\)Laion\-5b: an open large\-scale dataset for training next generation image\-text models\.Advances in neural information processing systems35,pp\. 25278–25294\.Cited by:[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p2.1)\.
- \[48\]Y\. Onoe, S\. Rane, Z\. Berger, Y\. Bitton, J\. Cho, R\. Garg, A\. Ku, Z\. Parekh, J\. Pont\-Tuset, G\. Tanzer,et al\.\(2024\)Docci: descriptions of connected and contrasting images\.InEuropean Conference on Computer Vision,pp\. 291–309\.Cited by:[§4\.3](https://arxiv.org/html/2609.27248#S4.SS3.p4.1)\.
- \[49\]R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean\(2023\)Efficiently scaling transformer inference\.Proceedings of machine learning and systems5,pp\. 606–624\.Cited by:[§5](https://arxiv.org/html/2609.27248#S5.p1.1)\.

## Appendix: Repurposing Pre\-trained LLMs as High Fidelity Continuous Text Autoencoders

### Outline

[A](https://arxiv.org/html/2609.27248#A1): Captioning Model Quality AnalysisA detailed qualitative analysis, identifying high aesthetic fidelity but critical failures in symbolic grounding and OCR\.

[B](https://arxiv.org/html/2609.27248#A2): Attention Masking StrategyDetailed breakdown of the attention visibility matrix for LLMAE\.

[C](https://arxiv.org/html/2609.27248#A3): LLMAE Latent SmoothnessLatent robustness to Gaussian noise perturbations andDKLD\_\{\\text\{KL\}\}calculation equation\.

[D](https://arxiv.org/html/2609.27248#A4): LLMAE Latent InterpolationLinear interpolation examples to observe phrase\-level recombination and syntactic preservation\.

[E](https://arxiv.org/html/2609.27248#A5): ICAE\-style Autoencoder with Gemma\-270MComparative baseline ablation when using the ICAE architecture instead of LLMAE\.

[F](https://arxiv.org/html/2609.27248#A6): Method DetailsDetailed specifications for LLMAE and the latent diffusion model for captioning\.

## Appendix ACaptioning Model Quality Analysis

1234560010010020020030030075 \(12\.5%\)255 \(42\.5%\)3 \(0\.5%\)38 \(6\.3%\)55 \(9\.2%\)174 \(29\.0%\)CountFigure 5:Distribution of VLM\-Judge scores across the evaluation dataset\.VLM Score Judge Prompt:Your role is to serve as an impartial and objective evaluator of an image caption generated by a Large Multimodal Model \(LMM\)\. Based on the single image input, assess the caption on three main criteria:leftmirgin=\*, noitemsep, topsep=4ptCoverage of image elements – how well the caption mentions the salient objects, their attributes, actions, and contextual details\.leftmiirgiin=\*, noiitemsep, topsep=4ptAbsence of hallucinations – the caption must not invent objects, attributes, counts, spatial relations, or other details not present or implied by the image\.leftmiiirgiiin=\*, noiiitemsep, topsep=4ptObject spatial layout consistency – whether spatial relationships \(left/right, above/below, front/behind, center, background/foreground\) are described accurately\.label=∙\\bullet, noitemsep, topsep=2ptAny incorrect or invented spatial relation is a hallucination\.label=∙\\bullet, noitemsep, topsep=2ptOmitting an obvious spatial relation reduces coverage\.label=∙\\bullet, noitemsep, topsep=2ptStating a relation that is ambiguous or uncertain in the image is also a hallucination\.Evaluation protocol:
Start with a brief explanation of your evaluation process\. Then assign one rating using the scale below\. Output only the rating number—no extra text, symbols, or commentary\.6Comprehensive coverage, correct spatial layout, no hallucinations5Very informative, correct spatial layout, no hallucinations, minor omissions4Moderate coverage, correct spatial layout, no hallucinations, several omissions3Limited coverage, minimal spatial detail, no hallucinations2Informative but contains at least one hallucination \(object or spatial\)1Limited coverage and at least one hallucination \(object or spatial\)0Not informative and/or multiple hallucinationsFigure 6:System prompt used for the VLM Score Judge evaluation, adapted exactly from\[[39](https://arxiv.org/html/2609.27248#bib.bib31)\]\.We provide the distribution of VLM Judge Scores of our captioning model on the CapArena test set in Fig\.[5](https://arxiv.org/html/2609.27248#A1.F5)\.

![Refer to caption](https://arxiv.org/html/2609.27248v1/images/test_03194.jpg)Caption:“The image is a close\-up shot of Convention Tennis Tennis against a blue corrugated backdrop\. The prominent lettering “CorrespondingON" features a stylized depiction of a horse head with a mane and tail, both outlined in bright orange with a subtle gradient highlight\. The horse is rendered in profile, facing right, with its head raised, giving a slight frown\. The backdrop is a vibrant blue, vertically ribbed metal, creating vertical shadows that suggest it is daytime\. Below the horse’s emblem, the words V̈ICKäre printed in a classic, bold orange font\. Below the horse, the words C̈orrespondence TNINSäre written in the same orange type, slightly smaller font\. The overall lighting is bright and even, emphasizing the contrast between the blue metal backdrop and the vivid colors\. The mood conveyed is sleek and bold, appropriate for school team visual’s nod to tennis players\.”VLM Judge Score:1/6VLM Judge Explanation:The caption contains several inaccuracies and hallucinations\. It incorrectly describes the text as “Convention Tennis Tennis" and “CorrespondingON," while the actual text is “MAVERICK TENNIS\." It also mentions “Correspondence TNINS" and “VICK," which are not present\. The description of the horse’s mane and tail is inaccurate, as the image only shows the horse’s head\. The spatial layout and lighting description are mostly correct, but the caption’s errors significantly impact its accuracy\.Figure 7:Qualitative demonstration of the captioning model’s lack of grounding\.### A\.1Categorical Performance Analysis

To analyze the failure modes and performance benchmarks of our captioning model, we utilized an LLM Judge to aggregate and categorize the qualitative feedback and numerical scores from the full set of per\-image VLM Judge explanations generated by GPT\-4o\. The following generated LLM generated summary findings highlight a significant performance gap between aesthetic scene interpretation and symbolic grounding:

- •Environmental and Aesthetic Fidelity \(Primary Strength\):The model demonstrates high fidelity in describing natural landscapes and organic textures \(Scores: 5–6\)\. It utilizes an advanced vocabulary to describe environmental nuances such as moss\-covered surfaces, leaf translucency, and specific floral species\. Furthermore, it excels in atmospheric reasoning, correctly identifying lighting conditions such as dappled sunlight, diffused overcast light, and artificial nighttime illumination\.
- •Compositional and Spatial Awareness:There is strong evidence of high\-level compositional understanding\. The model consistently identifies camera perspectives \(e\.g\., macro, low\-angle, aerial\) and correctly interprets the spatial relationship between foreground framing elements and background subjects\. This suggests an effective global awareness of the scene’s visual hierarchy\.
- •Symbolic and OCR Grounding \(Critical Failure\):Systematic failure occurs in tasks requiring the decoding of textual or symbolic data \(Scores: 1–2\)\. The model frequently exhibits "symbolic hallucination," where it confidently generates plausible but entirely incorrect text for signs, murals, and labels \(e\.g\., misreadingMAVERICK TENNISasConvention Tennis Tennisin Fig\.[7](https://arxiv.org/html/2609.27248#A1.F7)\)\. This indicates a lack of robust low\-level OCR integration\.
- •Quantitative Reliability and Individuation:The model struggles with discrete object counting and instance individuation\. It consistently misreports quantities \(e\.g\., reporting "ten birds" for significantly larger flocks\) and occasionally hallucinates secondary objects to "fill" the semantic space of a scene\.
- •Domain\-Specific Technical Accuracy:Performance degrades in industrial or specialized contexts due to "false specificity\." While the model correctly identifies general categories \(e\.g\., "vehicle"\), it often hallucinates incorrect brand names, model numbers, or mechanical components \(e\.g\., attributing multiple turrets to single\-turret tanks\), suggesting a limit to its fine\-grained classification capabilities\.
- •Linguistic Consistency:A recurring failure mode involves "lexical stuttering" or repetitive token generation within long\-form descriptions \(e\.g\.,"fleece and fleece"or"low\-angle, low\-angle"\)\. This suggests periodic breakdowns in the model’s self\-attention mechanism during the autoregressive generation of complex captions\.

## Appendix BAttention masking strategy

Table[6](https://arxiv.org/html/2609.27248#A2.T6)provides a detailed breakdown of the attention masking strategy for LLMAE\.

Table 6:Attention masking strategy managing visibility between the encoding Input, Embed tokens, and decoder Output targets\. Rows denote queries; columns denote keys\. For a query token,Visiblekeys are subject to causal masking to maintain autoregressive consistency, whileMaskedkeys prevent information flow\. Layer 0 serves as a unique interface where the Output tokens are permitted to attend to the Embed tokens, effectively providing a global initialization for the reconstruction segment\.
## Appendix CLLMAE latent smoothness

To measure prior alignment in Table[4](https://arxiv.org/html/2609.27248#S4.T4), we treated each coordinate\(t,d\)\(t,d\)as a marginal distributionqt​dq\_\{td\}and compute empirical statisticsμ^t​d=1N​∑Nzn,t,d\\hat\{\\mu\}\_\{td\}=\\frac\{1\}\{N\}\\sum\_\{N\}z\_\{n,t,d\}andσ^t​d2=1N​∑N\(zn,t,d−μ^t​d\)2\\hat\{\\sigma\}^\{2\}\_\{td\}=\\frac\{1\}\{N\}\\sum\_\{N\}\(z\_\{n,t,d\}\-\\hat\{\\mu\}\_\{td\}\)^\{2\}\. Same as the KL regularization loss calculation, we computeDK​LD\_\{KL\}analytically using Eq\.[1](https://arxiv.org/html/2609.27248#A3.E1)\.

DKL\(qt​d∥𝒩\(0,1\)\)=12\(σ^t​d2\+μ^t​d2−1−lnσ^t​d2\)D\_\{\\text\{KL\}\}\(q\_\{td\}\\parallel\\mathcal\{N\}\(0,1\)\)=\\frac\{1\}\{2\}\\left\(\\hat\{\\sigma\}^\{2\}\_\{td\}\+\\hat\{\\mu\}^\{2\}\_\{td\}\-1\-\\ln\\hat\{\\sigma\}^\{2\}\_\{td\}\\right\)\(1\)
We further investigate the robustness of LLMAE autoencoding through noise perturbations: we apply additive i\.i\.d\. Gaussian noise to every scalar dimension:zn,t,d′=zn,t,d\+σ​ϵn,t,dz^\{\\prime\}\_\{n,t,d\}=z\_\{n,t,d\}\+\\sigma\\epsilon\_\{n,t,d\}, whereϵn,t,d∼𝒩⁡\(0,1\)\\epsilon\_\{n,t,d\}\\sim\\mathcal\{N\}\(0,1\)\. As shown in Table[7](https://arxiv.org/html/2609.27248#A3.T7), the generative prior of Gemma 3 effectively denoises the latent signal, maintaining high performance up toσ=0\.5\\sigma=0\.5\.

Table 7:Quality w\.r\.t\. noise augmentation\.
## Appendix DLatent Interpolation

To better understand the geometry of the learned text autoencoder, we perform a simple latent interpolation experiment\. Given two input textsxAx\_\{A\}andxBx\_\{B\}, we encode them into latent representations

zA=E⁡\(xA\),zB=E⁡\(xB\),z\_\{A\}=E\(x\_\{A\}\),\\qquad z\_\{B\}=E\(x\_\{B\}\),and decode a midpoint linear interpolant:

zm​i​d​p​o​i​n​t=zA\+zB2z\_\{midpoint\}=\\frac\{z\_\{A\}\+z\_\{B\}\}\{2\}This experiment is not intended to show that Euclidean interpolation corresponds to semantic interpolation\. Rather, it probes whether intermediate latent states remain decodable and what kinds of structure are preserved between distant texts\. We observe a desirable property of the autoencoder that the interpolated latents often decode into recognizable, locally coherent text rather than complete garbage\. However, the decoded midpoint is usually not a meaningful semantic average\. Instead, it frequently preserves the global syntactic frame of one or both endpoints while swapping entities, modifiers, actions, or causal clauses\. This suggests that the latent space supports high\-resolution phrase\-level recombination, even though linear paths through it do not reliably preserve semantic consistency\. We provide examples in Table[8](https://arxiv.org/html/2609.27248#A4.T8)\.

Table 8:Linear interpolation in the text autoencoder latent space\. Midpoints generally remain fluent and often preserve sentence\-level structure, but they do not correspond to clean semantic averages\. Instead, they exhibit phrase\-level recombination: sometimes producing coherent hybrids and sometimes producing binding errors\.
## Appendix EICAE\-style autoencoder with Gemma\-270M

Table 9:Reconstruction quality for ICAE\-style training on Gemma\-270MTo evaluate the utility of the LLMAE architecture, we implemented a baseline following the In\-context Autoencoder \(ICAE\)\[[27](https://arxiv.org/html/2609.27248#bib.bib32)\]framework\. This baseline utilizes a soft prompting paradigm where the K latent tokens serve as a compressed "memory tokens" prefix to the input tokens for autoregressive decoding\. We instantiated this model using the same Gemma\-270M backbone and training data we use for LLMAE\.

Our experiments indicate that this soft prompting approach is insufficient for high\-fidelity text autoencoding for the lightweight Gemma backbone\. This performance bottleneck remained non\-trivial with or without employing curriculum learning \(sorting training samples by sequence length\)\. The inability of the ICAE\-style prefix to recover fine\-grained semantic and syntactic details suggests that simply treating latents as a soft prompt does not provide the necessary structural guidance for long\-form reconstruction, at least in the Gemma 270M backbone\. We provide the resconstruction evaluations in Table[9](https://arxiv.org/html/2609.27248#A5.T9)\.

## Appendix FMethod details

### F\.1LLMAE

LLMAE trains Gemma\-3\-270M\-PT \(not instruction\-tuned\) with LoRA on all attention projections for 2 epochs over 1\.4M stratified text samples \(1M Pile\-uncopyrighted \+ 400K C4\-RealNewsLike\)\. The test sets for reconstruction are 500 samples each\. LLMAE utilizes LoRA adapters \(737K parameters\), 256 latent token embeddings \(164K parameters\), and an additive Codec \(9\.8M\), totalling 10\.7M trainable parameters\. The input layout is\[input tokens\|\|256 latent tokens\|\|input copy\]with the LM loss computed only on the reconstruction segment\. Texts are sorted by length \(curriculum learning\) and latent activations at layer 11 are regularized with KL penalties toward𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)\. Output logits are regularized with KL toward a frozen reference model\. Gaussian noise \(σ=0\.1\\sigma\{=\}0\.1\) is added to latent activations during training to encourage a smooth latent space\. The model is then further fine\-tuned for 2 additional epochs with a lightweight additive Codec inserted at layer 11, whose compressed latents are regularized toward𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)for a smooth latent space\. We note that the second row in Table[4](https://arxiv.org/html/2609.27248#S4.T4)was also trained for 2 additional epochs for a fair ablation\. The full parameter configuration is listed in Table[10](https://arxiv.org/html/2609.27248#A6.T10)\.

Table 10:LLMAE Hyperparameters
### F\.2Latent Diffusion for Image Captioning

The score network and diffusion process are based on the architecture utilized by COSMOS\[[21](https://arxiv.org/html/2609.27248#bib.bib37)\], incorporating three primary modifications to align with the LLMAE latent space\. First, the architecture is resized to256×640256\\times 640to ensure compatibility with the LLMAE latent token dimensionality, which consists of 10 attention heads of size 64 and a SwiGLU FFN width of 2560\. Second, we introduce image conditioning by projecting SigLIP features through a shared linear layer \(1152→6401152\\to 640\)\. These features are integrated via a per\-block cross\-attention sublayer utilizing RMSNorm pre\-norms on both query and key/value sequences\. A learnable scalar residual gate, initialized to zero, is applied to these sublayers to allow the model to begin as a purely unconditional denoiser and gradually learn to incorporate image signals\. Third, classifier\-free guidance \(CFG\) is implemented by replacing the image conditioning with a learned null embedding with a0\.10\.1probability during training, enabling CFG extrapolation during inference\. All other structural and procedural components—including the Tanh VP\-SDE schedule \(d=5d=5\), U\-Net skip connections,x0x\_\{0\}\-prediction parameterization, self\-conditioning, per\-dimension latent normalization, and the Euler reverse solver—remain identical to the original COSMOS diffusion implementation\. In our primary configuration, the autoencoder is our pre\-trained LLMAE \(278M; Gemma\-3\-270M backbone with a 9\.8M Codec\) operating on a compact 256×640 latent space, and the score network is a 12\-layer transformer with hidden dimension 640 \(111M parameters\), giving a total inference footprint of 819M parameters with only the score network trained\. Replacing LLMAE with the COSMOS autoencoder \(356M; 208M encoder, 149M decoder\) expands the latent space to 512×768, which requires a wider score network \(hidden dimension 768, 159M parameters\) to match; the total system grows to 943M parameters, with the trained fraction increasing from 111M to 159M \(13\.6% to 16\.9% of total\)\. The full parameter configuration is listed in Table[11](https://arxiv.org/html/2609.27248#A6.T11)\.

Table 11:Hyperparameters for Latent Diffusion for Captioning