Proposes treating semantic compression as a diffusion noise function for handling massive context beyond model windows, using multi-pass reading at decreasing compression levels. Untrained-model experiments show components work in isolation but the full chain needs training to resolve binding bottleneck.
I'm proposing a way to handle massive context longer than a model's context window by treating semantic compression as the noise function of a diffusion-like process. Instead of denoising masked tokens into coherent text (like DiffusionGemma or Nemotron-Diffusion do for generation), the model reads the source document in multiple passes at decreasing compression levels, heavy summary first, verbatim last all the while it iteratively refines an "integration state" (the output) through structured edit operations like add/replace/remove. The context window only needs to fit the current compressed view, not the full source. Three pieces: Context Diffusion: Multi-pass reader that refines an integration state across passes, each conditioned on a different compression level of the source. The source stays on disk and is never reconstructed into the window. The architectural shape converges heavily with Zhang, Kraska & Khattab's Recursive Language Models (2025) (I found their paper after writing most of mine and don't claim priority on the multi-pass structure). Diffusion-based Semantic Compression (DiSCo): The framing I think is novel. Using semantic compression as the noise function, so the "noised" view is much shorter than the source and context length is managed by the compressor, not the model window. This is a different noise domain from masked or vocabulary-level diffusion. Pass-Conditioned Reading: Training conditioned on position in the multi-pass schedule so the model learns different behaviour for early passes (broad understanding from coarse views) vs late passes (precise retrieval from verbatim views). What I actually tested and the honest result: Experiments using off-the-shelf models No fine-tuning - this was an untrained floor test. The outcome so far: "not unfeasible," not "it works": -Components worked better in isolation than the full record>retain>compose chain did. -The bottleneck is retention/recombination across passes. -Pre-registered kill-conditions failed and were reported. -The small signals were enough to justify training a small model, but not enough to call the architecture validated yet. Methodology was deliberately disciplined - kill conditions, published negatives, documented self-corrections. Hopefully that's what makes a small-scale claim credible when you don't have a lab behind you. Two questions for anyone still reading: -Prior Art: Has anyone seen compression-as-noise (using length-reducing semantic compression as the noise schedule in a diffusion-style iterative process) in the diffusion-LM literature? I've searched extensively and haven't found it, but I could easily be missing something. -The next step is model training with synthetic data to test whether training resolves the binding bottleneck. I'm an independent researcher on consumer hardware who is has not done model training before. I'd appreciate any guidance or if anyone has compute access or wants to collaborate, I'm interested. Proposal: https://github.com/dev-boz/diffusive-semantic-compression Experiments + full findings: https://github.com/dev-boz/pass-conditioned-reading Archived on Zenodo (DOI: 10.5281/zenodo.20695611), CC-BY-4.0. This is the first thing I've published. Expecting to have it torn apart.
An analysis of how six AI coding agents (Claude Code, Codex CLI, OpenCode, Cline, Cursor, Amp) converge on layered progressive compression for long contexts, differing in what they protect (user messages, stateful tool outputs) and whether they inform the model of compression, with tradeoffs between cost and accuracy.
This paper presents Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that efficiently handle long contexts through architectural search and large-scale pretraining, outperforming traditional KV cache methods in accuracy, speed, and memory usage.
This paper introduces Residual Context Diffusion (RCD), a module that recycles discarded token representations in diffusion language models to improve efficiency and accuracy, achieving 5–10% better accuracy and up to 4–5x fewer denoising steps on challenging reasoning tasks.
This paper proposes Telegraph English, a readable symbolic format for context compression that outperforms matched-budget baselines on multi-hop QA datasets, preserving entity content more densely.
The paper proposes Nemotron-TwoTower, a diffusion language model that decouples context representation and denoising using a frozen autoregressive tower and a trainable diffusion denoiser, achieving 98.7% of baseline quality with 2.42x throughput.