@Pavel_Izmailov: New paper: Latent Context Language Models (LCLMs)! Idea: encode 16 tokens as 1 latent token, and have the LLM work on t…
Summary
Introduces Latent Context Language Models (LCLMs), which encode 16 tokens as 1 latent token to improve performance, speed, and memory usage.
View Cached Full Text
Cached at: 06/10/26, 09:57 PM
New paper: Latent Context Language Models (LCLMs)!
Idea: encode 16 tokens as 1 latent token, and have the LLM work on top of the latent tokens. Result: general-purpose model with much better performance / speed / memory usage frontier. https://t.co/ldsBOVkmFF
Similar Articles
End-to-End Context Compression at Scale
This paper presents Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that efficiently handle long contexts through architectural search and large-scale pretraining, outperforming traditional KV cache methods in accuracy, speed, and memory usage.
Context Language Models
This paper introduces Context Language Models (CLMs), which natively manage their own context by treating it as a file, achieving superior performance over state-of-the-art methods and enabling in-context and parametric learning of context-management strategies.
Copy-Paste Language Models
The paper proposes Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head alongside the standard LM head to let models explicitly point to and copy tokens from context, mitigating the LM head gradient bottleneck. Pretrained models from 135M to 1.7B parameters show better training efficiency, downstream performance, and long-context operating points with negligible overhead.
@omarsar0: // Latent Semantic Memory for LLMs // This paper presents some really interesting findings about agent memory. They fin…
The paper introduces Memorilla, a memory system that compresses an entire document collection into 16 query-dependent memory tokens for frozen LLMs, outperforming full-context inference on FactKG (84.33% vs 80.43%) while using 108x fewer tokens and beating baselines like Mem0 and MemoRAG across five of eight benchmarks.
@dair_ai: Great paper for the weekend ahead. Meta presents Context Language Models, which can natively manage their own context.
Meta presents Context Language Models (CLMs), a paper exploring how effective it is to give language models native write access to manage their own context, with potential benefits for agentic workflows.