in-context-learning

This paper reveals a counterintuitive phenomenon where correct demonstrations in in-context learning can degrade model accuracy, introducing task preserving perturbations to study the gap between exemplar correctness and utility.

This paper studies retrieval-augmented generation as an in-context optimization process, showing that linear self-attention can implement gradient descent on a unified RAG objective. It proposes a lightweight method for frozen RAG LLMs that predicts context-conditioned updates, improving performance across multiple QA benchmarks.

This paper presents MEMOR-E, a mobile quadruped robot with a tablet interface that uses fine-tuned and in-context learning with LLMs to provide personalized, stage-aware cognitive assistance for Alzheimer's patients, including medication reminders and memory interactions, with explainable AI for caregiver oversight.

This paper proposes a method to improve in-context learning by optimizing the continuous embeddings of a fixed few-shot prompt at test time, using a self-supervised confidence proxy derived from the model's log-probabilities without requiring fine-tuning or token generation.

This paper introduces Reflection-Augmented Scaling (RAS), a method that uses execution feedback from failed Cypher queries to iteratively refine query generation via in-context learning, reducing execution error rates by 41-50% across multiple datasets and models.

Discusses that the mathematics used by AI is mainly linear algebra, calculus, etc., from before the 19th century, but emerging phenomena such as Scaling Law, emergent abilities, double descent, in-context learning, and representation geometry lack mathematical explanation. Analogizes to the clouds in physics in 1900, suggesting it may drive the development of 21st-century mathematics.

This paper adapts classical class imbalance techniques to Prior-Data Fitted Networks (PFNs) for tabular classification, finding that thresholding and downsampling perform well due to PFNs' calibration and limited-data capabilities.

This paper proposes using distributional alignment between task vector-based and in-context learning inference as a criterion for designing task vectors, and introduces Linear Task Vector (LTV) that minimizes next-token probability discrepancy via closed-form linear mapping. LTV achieves 9.2% average accuracy improvement over baselines across eight benchmarks and five LLMs.

TabPFN-MT extends PFNs to multitask in-context learning for tabular data, achieving state-of-the-art on small-to-medium datasets while reducing inference cost from O(T) to O(1) forward passes.

This paper proposes that in-context learning in LLMs operates through low-dimensional concept subspaces, where task-relevant information concentrates in a small fraction of the representation space, supported by experiments on Llama-3-8B and Qwen2.5-7B.

This paper introduces ACIL, an automatic Chain-of-Thought framework to enhance In-Context Learning by generating and pruning reasoning chains, improving LLM performance on complex tasks.

SurvivalPFN is a prior-data fitted network that amortizes Bayesian inference for survival analysis via in-context learning, achieving strong predictive performance across 61 datasets without task-specific training or hyperparameter tuning.

FashionChameleon is a real-time, interactive framework for human-garment video customization that uses teacher-student distillation and in-context learning to enable multi-garment switching while maintaining motion coherence, achieving 23.8 FPS on a single GPU.

This paper investigates many-shot chain-of-thought in-context learning for reasoning tasks, revealing that standard scaling rules do not transfer and proposing Curvilinear Demonstration Selection (CDS) for improved ordering, achieving up to 5.42 percentage-point gain.

This paper investigates whether LLMs learn in-context through latent structure inference or local pattern matching, using mechanistic interpretability methods like PCA and activation patching on a graph random-walk task.

This paper investigates using Large Language Models, specifically Claude, interfaced with a Computer Algebra System (Maple) to perform algorithmic computations in theoretical physics, such as analyzing cosmological perturbations.

A fast-slow learning framework for LLMs combines fixed slow weights with optimized fast context weights, achieving up to 3x better sample efficiency and reduced catastrophic forgetting in continual learning scenarios.

This paper introduces In-context Sparse Attention (ISA), a framework that significantly reduces computational costs in video editing by pruning redundant context and using dynamic query grouping. The authors demonstrate the method's effectiveness with LIVEditor, achieving near-lossless acceleration and state-of-the-art results on multiple video editing benchmarks.

Georgetown researchers boost low-resource Coptic-to-English translation by augmenting in-context prompts with Universal Dependencies syntactic parses alongside bilingual glosses, setting a new state-of-the-art.

LiFT is a longitudinal instruction fine-tuning framework that unifies diverse temporal NLP tasks under a shared instruction schema with curriculum-based training. Evaluated across OLMo, LLaMA, and Qwen models, LiFT consistently outperforms base-model in-context learning, especially on out-of-distribution data and rare change events.