in-context-learning

This paper presents a mechanistic analysis of induction in masked diffusion language models, identifying a bidirectional induction circuit and showing that these models use the global fraction of masked tokens as an implicit timestep.

This paper constructs a transformer with linear self-attention that performs in-context learning of the closed-form least squares solution for simple linear regression, using layer normalization to approximate the analytical solution rather than gradient descent.

This paper proposes a reformulation to apply tabular foundation models (TFMs) to discrete choice estimation, addressing the structural gap of row-independent assumptions. The best reformulation outperforms hierarchical Bayesian estimation by 8% in holdout log-likelihood and 3.6% in hit rate while running 16 times faster.

This paper proposes a framework to test whether LLM estimates obey statistical self-consistency (law of total probability) across subpopulations, finding widespread violations and the 'macro fallacy' where fine-grained estimates align better with human data.

A researcher discusses the challenge of continual learning in LLMs, comparing them to amnesiac interns, and explores approaches like extending context windows, building stateful memory, and compressing context into latent representations, citing their work on Still.

This paper proposes a self-evolving in-context learning framework for direct pilot-to-beamformer design in multi-user MISO systems, integrating a Transformer backbone with a pilot encoder-decoder network and curriculum learning to handle multiple channel models without retraining.

This paper evaluates state-of-the-art tabular models as encoders in image-tabular multimodal learning, addressing the challenge of using In-Context Learning models that require labels for embedding training and test instances.

AbICL proposes an in-context learning framework for antigen-specific antibody affinity ranking, combining a pretrained structural encoder with a context ranking head to leverage labeled demonstrations for test-time adaptation without gradient updates.

TOFFEE is a system that uses Monte Carlo Tree Search with adaptive model selection and cross-task prefix reuse to synthesize high-quality data agent trajectories at scale. These trajectories can be used for fine-tuning or in-context learning to improve data agent performance in heterogeneous enterprise environments.

Anjney Midha reflects on the value of in-context learning in AI and life, recalling his investment in Anthropic's 2021 seed round as an intuitive but often misunderstood concept.

This paper studies transformers trained on Markov chains and identifies that induction heads implement soft context-matching and Dirichlet-style smoothing, showing that transformers regularize in-context estimation rather than simply counting n-grams.

This paper studies how memory architecture affects language emergence in LLM agents playing a Lewis signaling game, finding that persistent private notebook memory outperforms stateless agents and prevents high-capacity collapse.

This paper theoretically analyzes linear transformers for in-context learning under domain generalization, establishing dimension-independent convergence rates and proposing novel activation and loss designs for linearizing pretrained softmax LLMs.

This paper investigates parametric memorization in tabular foundation models that use in-context learning, introducing a probing framework (IclMem) to separate context-based predictions from memorization. It finds moderate memorization signals under specific conditions but notes they largely vanish under realistic training scenarios.

该论文提出贝叶斯上下文实验者（Bayesian in-context experimenters），通过训练Transformer模仿贝叶斯后验Neyman教师策略，实现自适应平均处理效应（ATE）估计，并采用混合专家Transformer处理未知平滑性，理论证明可通过监督预训练学习该策略。

This paper investigates whether tabular in-context learning models, pretrained on synthetic causal tables, can generalize to predict biomolecular properties from limited labeled data. The authors find that these models are competitive for protein fitness regression but that representation choice is crucial for small-molecule classification.

This paper challenges the assumption that reranking always improves few-shot selection for LLMs, proposing a training-free gated reranking approach that uses model uncertainty to decide when to rerank, reducing computational costs by 15-80% while slightly improving performance.

Google Research introduces TabFM, a zero-shot foundation model for tabular data that uses in-context learning to perform classification and regression without requiring manual model training or hyperparameter tuning.

This paper proposes LC-ICL, a novel few-shot technique that uses both correct and incorrect examples with error-cause labels to improve large language models' performance on information extraction tasks like named entity recognition and relation extraction.

This paper tests the assumption that LLMs judge better than they generate in in-context QA, finding generation accuracy exceeds self-evaluation on most benchmarks, with evaluation attending less to context. The findings challenge core assumptions in self-evaluation pipelines.