Copy-Paste Language Models
Summary
论文提出 Copy-Paste Language Models（CPLMs），在传统语言建模头之外加入轻量 Copy-Paste 头以增强模型的上下文复制能力，在 135M 到 1.7B 参数规模的预训练中验证了训练效率、下游性能及长上下文检索的提升，且开销极小。
View Cached Full Text
Cached at: 10/16/26, 07:06 PM
Paper page - Copy-Paste Language Models
Source: https://huggingface.co/papers/2610.19247
Abstract
Copy-Paste Language Models enhance in-context copying by adding a lightweight head that improves training efficiency, downstream performance, and long-context retrieval with minimal overhead.
Transformersare intuitively considered to be efficient at copying from their inputs. We actually show that the ability ofLLMsto copy in context can be improved through explicit modeling. We proposeCopy-Paste Language Models(CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points inlong-contextandretrievalsetups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM headgradient bottleneckthrough an alternative gradient path.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2610\.19247
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.19247 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.19247 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.19247 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Copy-Paste Language Models
The paper proposes Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head alongside the standard LM head to let models explicitly point to and copy tokens from context, mitigating the LM head gradient bottleneck. Pretrained models from 135M to 1.7B parameters show better training efficiency, downstream performance, and long-context operating points with negligible overhead.
From Copying to Composition: How Language Models Develop In-Context Learning
本文提出一种受控诊断任务，区分语言模型在上下文学习中的“复制示例”与“组合应用规则”两种行为，发现提示控制、答案偏好、无候选构造和迁移能力在训练中沿不同时间线发展，且合成定义文档训练能提升组合式响应。
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
该论文研究无分词器的字节级语言模型，发现配合 token-superposition 训练与哈希嵌入后，扁平 Transformer 随规模扩大能持续超越子词模型，并能自发学习出类似分词器的局部文本抽象，据此可将投机解码的接受率提升 3.4 倍。
@realchendahuang: Three years ago, when large language models had just swept the globe, the entire tech world was in an awkward state of being both thrilled and constrained everywhere. At the time, the industry broadly agreed on several almost insurmountable technical hurdles: model memory windows were too short — input just a little longer and it couldn't remember; the underlying compute architecture was extremely compute-hungry, with computation for long text exploding exponentially; the cost per call stayed sky-high, so enterprises didn't dare deploy them broadly in business use; and so-called autonomous agents would completely lose control under probabilistic drift if they ran just a few extra steps...
A deep retrospective on three years of LLM technical evolution: long-context handling shifted from RAG to prefix caching and million-token context windows; the Transformer still wins over linear-complexity architectures thanks to the hardware ecosystem of GPU tensor cores; inference costs were democratized through quantization, distillation, and task-based routing; agents moved toward pragmatic engineering paths like progressive disclosure, command-line sandboxes, and test feedback loops; and hallucination remains unsolved.
Learning to Learn a Language
This paper introduces the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained exclusively on samples from a synthetic structural causal model, enabling it to learn languages in context with frozen weights despite never seeing real text. Results show near-competitivenatural-language modeling, arithmetic, and compression capabilities below gzip/PPMd, suggesting meta-learning of language rather than memorization.