in-context-learning

ICCLHerd 提出一种基于 herding 的 LLM 训练数据选择框架，利用 von Mises–Fisher 核平衡 mode-seeking 与 mode-averaging，并以 in-context learning 作为验证代理进行超参数搜索，实验表明其优于现有数据选择算法。

该论文提出 ECI2D 方法，通过将语言推理链抽象为可执行程序并施加因果干预（do-operator）来筛选自生成的上下文示例，解决仅靠观察过滤示例时的观测等价问题，在数学、逻辑与常识推理基准上取得显著提升。

This paper investigates how different evaluation setup choices affect base language model benchmarking by examining 8 design aspects across 26 2B-parameter models and larger LLMs in four languages, proposing monotonicity, non-randomness, and ranking consistency as meta-measures to guide reliable evaluation setup selection during pretraining.

本文提出一种受控诊断任务，区分语言模型在上下文学习中的“复制示例”与“组合应用规则”两种行为，发现提示控制、答案偏好、无候选构造和迁移能力在训练中沿不同时间线发展，且合成定义文档训练能提升组合式响应。

The paper proposes CB-Agent, a concept-based in-context learning method that fits a reusable concept vector from expert trajectories to score candidate actions for LLM agents, achieving top cross-benchmark averages on TextWorld, ALFWorld, and ScienceWorld.

该报告大规模评测了 GPT-6-Astra 在 12 项机器人基准上的具身能力，指出其已跨越具身能力门槛但尚未实现实用具身智能，并探讨了 System 2/System 1 协同与真实部署中的成本、执行时间等关键挑战。

论文提出 Copy-Paste Language Models（CPLMs），在传统语言建模头之外加入轻量 Copy-Paste 头以增强模型的上下文复制能力，在 135M 到 1.7B 参数规模的预训练中验证了训练效率、下游性能及长上下文检索的提升，且开销极小。

论文提出 GIST（Generator-state-Informed Supervision for Tabular Foundation Models），利用生成器中间状态的条件标签分布替代硬标签监督，以降低表格基础模型预训练中的采样方差，在 TabArena 和 TALENT 基准上取得更好的平均排名。

该论文提出将 Transformer 网络解释为混合概率推理过程的可学习近似：自注意力对应在输入条件化、动态结构的马尔可夫随机场上执行 max-product 信念传播，而层间深度演进则对应隐式的期望最大化（EM）过程。基于此视角，作者提出 meta-model 范式，将自注意力、混合专家和门控单元等机制统一起来。

该论文提出了一种全晶体管物理学习transformer，通过在双电路中引入额外电路元件恢复双向信息流，并利用亚阈值MOSFET的指数电流定律构建注意力机制，仿真表明该电路可被训练执行简单的上下文学习任务。

论文提出 TabWM（表格世界模型）与 WorldRel 生成器，通过在上下文中观察执行样本来推断关系型系统的动态，预测动作导致的稀疏结构化编辑，从而将关系学习从状态建模扩展到世界建模与动作规划。

The paper proposes SFT-as-context, a two-pass inference method where the parent model uses the fine-tuned model's response as context, mitigating catastrophic forgetting of general capabilities after supervised fine-tuning while preserving task-specific improvements.

本文提出"空间感应头"（spatial induction heads），一种两层 gather-and-match 电路，解释 Transformer 如何在多维细胞自动机的序列化数据中进行上下文学习。研究表明，该电路可逼近贝叶斯后验并在确定性规则上实现精确预测，训练后的两层 Transformer 能泛化到未见过的规则。

This paper extends prior-fitted networks (TabPFN) to natural language by training a 300M-parameter byte-level transformer entirely on synthetic sequences drawn from randomly sampled recurrent causal models — a 'prior over languages'. With frozen weights, the model learns to predict real languages in context across six languages, and can also count, compare numbers, add approximately, and predict deterministic sequences like primes and Kolakoski, showing in-context language learning can emerge from a synthetic non-linguistic prior.

论文提出 GAMA，一个面向生物医学命名实体识别的 guideline 增强多智能体框架，通过从标注数据中归纳并验证注释规则构建 guideline 记忆，结合规划、编码与验证模块实现 schema 约束的实体抽取，在五个 BioNER 数据集上超越现有 LLM 基线。

This paper proposes activation alignment, a lightweight linear transformation trained on synthetic unlabeled data to map partial-context student model activations toward full-context teacher activations, enabling faster tabular in-context learning with little performance loss. Evaluated on 38 TabArena datasets with TabPFN-3 and TabFM, the method recovers nearly half the teacher's advantage in low-data regimes while running on commodity hardware without a GPU.

A Berkeley paper finds that LLMs often retain outdated preferences or deadlines in context and keep attending to stale values despite knowing the update; providing the current state explicitly (or steering attention to the newest value) dramatically improves performance without retraining.

The paper proposes ViGAR, a hierarchical world-action model for long-horizon compositional robotic manipulation that factorizes tasks into a visual subgoal planner and a subgoal executor, achieving 82.00% and 67.02% success on the RoboTwin Clean2Random benchmark and outperforming the strongest baseline by 12.86 percentage points.

LoopICL decouples parameter count from depth by using a single weight-tied transformer block with dual cell/row streams and a learned exit gate, matching TabICL v2 on TabArena/TALENT at the same FLOPs while using ~90% fewer parameters.

This SAP-led paper studies support-set target leakage in relational in-context learning, constructing 20 controlled target-derived features across 13 RelBench tasks and showing that leakage effects are strongly task- and model-dependent and can reverse conclusions between model variants, undermining evaluation reliability for relational foundation models.