in-context-learning

This paper extends prior-fitted networks (TabPFN) to natural language by training a 300M-parameter byte-level transformer entirely on synthetic sequences drawn from randomly sampled recurrent causal models — a 'prior over languages'. With frozen weights, the model learns to predict real languages in context across six languages, and can also count, compare numbers, add approximately, and predict deterministic sequences like primes and Kolakoski, showing in-context language learning can emerge from a synthetic non-linguistic prior.

论文提出 GAMA，一个面向生物医学命名实体识别的 guideline 增强多智能体框架，通过从标注数据中归纳并验证注释规则构建 guideline 记忆，结合规划、编码与验证模块实现 schema 约束的实体抽取，在五个 BioNER 数据集上超越现有 LLM 基线。

This paper proposes activation alignment, a lightweight linear transformation trained on synthetic unlabeled data to map partial-context student model activations toward full-context teacher activations, enabling faster tabular in-context learning with little performance loss. Evaluated on 38 TabArena datasets with TabPFN-3 and TabFM, the method recovers nearly half the teacher's advantage in low-data regimes while running on commodity hardware without a GPU.

A Berkeley paper finds that LLMs often retain outdated preferences or deadlines in context and keep attending to stale values despite knowing the update; providing the current state explicitly (or steering attention to the newest value) dramatically improves performance without retraining.

The paper proposes ViGAR, a hierarchical world-action model for long-horizon compositional robotic manipulation that factorizes tasks into a visual subgoal planner and a subgoal executor, achieving 82.00% and 67.02% success on the RoboTwin Clean2Random benchmark and outperforming the strongest baseline by 12.86 percentage points.

LoopICL decouples parameter count from depth by using a single weight-tied transformer block with dual cell/row streams and a learned exit gate, matching TabICL v2 on TabArena/TALENT at the same FLOPs while using ~90% fewer parameters.

This SAP-led paper studies support-set target leakage in relational in-context learning, constructing 20 controlled target-derived features across 13 RelBench tasks and showing that leakage effects are strongly task- and model-dependent and can reverse conclusions between model variants, undermining evaluation reliability for relational foundation models.

The paper formalizes support-set target leakage in relational foundation models during in-context learning, where labeled support examples contain target-derived features unavailable at query time. Using 14 synthetic leaker types on RelBench databases, the authors show target-table leakers cause the clearest degradation and that Integrated Gradients can rank and partially mitigate the problematic columns.

This paper measures the in-context binding capacity of language models, finding that recall scales with parameter count according to a power law and analyzing factors like pretraining recipes and interference.

The paper proposes Q-Target Pretrained Transformers (QTPT) for in-context reinforcement learning, replacing behavior cloning with a Bellman-style Q-target objective to improve robustness to weak or suboptimal data quality.

该论文提出 SimpleICL，一个极简且可复现的机器人上下文学习（ICL）框架，通过明确的问题定义解决视觉示范带来的提示歧义，并在仿真与真实环境中取得强性能，同时开源数据与训练管线。

研究发现预训练 transformer 在遵循上下文引用时极少使用其深度（仅可靠遵循 1.4-3.6 行），而一个冻结所有权重、只在早期层添加的 rank-8 LoRA 即可大幅扩展该能力——Qwen3-8B 在 24 行链上的精确准确率从 15.5% 提升至 99%。LoRA 起到'接力'作用，使程序行通过中间层传递链身份信息。

ALICE is a foundation model for zero-shot, in-context mutual information estimation, trained only on synthetic distributions and using rectified-flow velocity fields to estimate MI without per-distribution training. It achieves competitive accuracy across biology, genetics, and neuroscience benchmarks.

Acacia is a graph foundation model trained exclusively on the Common Crawl web graph, supporting arbitrary features, various tasks like node classification and link prediction, and in-context learning without relying on pretrained LLMs.

This paper reviews in-context learning methods for robots, categorizing them into four interface families and analyzing their application across manipulation and navigation tasks.

This paper introduces FakeContext-bench to evaluate how well large language models distinguish between contextual information and factual knowledge, and proposes Jurisdiction In-Context Learning (J-ICL) to enhance both in-context learning performance and resistance to misleading context.

This paper presents the first systematic evaluation of large language models for post-OCR correction in Hindi and Marathi, comparing in-context learning retrieval strategies and demonstrating the effectiveness of CharBM25.

CausalWM is a causal world model that uses Chain-of-Thought to predict future frames by capturing physical dependencies step-by-step, ranking top on benchmarks like TriWorldBench and PAI-Bench.

This paper presents RoboDawn, a method to transfer Vision-Language Model intelligence to robotic control, achieving state-of-the-art results on benchmarks with zero-shot and one-shot learning and successful real-world applications.

This paper proposes using state space models to efficiently select demonstrations for long-context language model prompts, reducing computational cost and improving performance.