in-context-learning

This paper introduces 'pigeonholing,' a phenomenon where bad prompts cause LLMs to collapse and repeat errors, leading to a 38-40% performance drop. Experiments across 10 tasks and 10 models show worsening with more conversation turns, and propose RLVR with synthetic errors as a mitigation.

This paper extends contextual entrainment from token-level to sentence-level, showing that even counterfactual sentences in prompts increase their probability during inference. The effect decreases with model size and is driven by 2-4% of attention heads, which can be ablated without performance loss.

This paper empirically studies cross-lingual transfer in in-context learning across seven tasks, six models, and typologically diverse languages, showing that fine-tuning based expectations do not consistently apply and offering new heuristics for source language selection.

This paper introduces SwiftTrans, a framework that addresses the overlooked issue of runtime efficiency in LLM-based code translation, achieving consistent improvements in both correctness and efficiency across benchmarks.

This paper presents Brick-DICL, a dynamic in-context learning framework for automated classification of building management system points into Brick ontology classes. It uses retrieval-augmented generation and multi-model filtering to improve accuracy and reduce manual verification effort.

This paper introduces the Call Playbook dataset for classifying real-world B2B conversations and proposes methods to distill examples into compact, interpretable task instructions, achieving 99% token reduction and up to 7% AUC improvement over traditional in-context learning.

This paper introduces Grammatical Error Representation (GER), a novel method for retrieving in-context demonstrations based on error patterns rather than semantic similarity, significantly improving multilingual grammatical error correction performance in LLMs with in-context learning.

This paper proposes CRUMB, a three-stage inference wrapper that clusters test queries and selects a distributionally matched training subset via MMD minimization to enable efficient Prior-Fitted Network inference on large datasets, achieving state-of-the-art context selection on 51 TabArena datasets.

This paper proposes RoVE, a parameter-free modification to Rotary Position Embeddings that makes value pathways position-sensitive by rotating values simultaneously with keys, transforming RoPE attention into attentive convolution. Experiments on GPT-2 models show consistent gains in few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval.

This paper identifies and formalizes the 'structural attention tax' phenomenon, where the format of retrieved content (e.g., knowledge graph triples) independently distorts LLM attention distribution regardless of semantic relevance, leading to compressed demonstration attention. It provides a formal framework, empirical evidence across models and benchmarks, and proposes structure-aware mitigation strategies.

This paper introduces finite certificates for verifying determinacy and emergence in language model in-context behavior, providing theoretical criteria and experimental validation on contemporary models.

CL-Bench is a new expert-validated benchmark across six domains that evaluates whether LLM-based agents genuinely learn from sequential experience. It finds that naive in-context learning often outperforms dedicated memory systems, indicating current architectures add overhead rather than genuine learning.

This paper proposes a domain-aware coreset construction pipeline that enables a tabular foundation model to predict flood depth with only 0.7% of the training data, achieving 98.5% of the supervised reference accuracy and allowing transfer across watersheds without retraining.

This paper proposes a Perceiver-style architecture pretrained on synthetic bag-structured data to enable efficient, task-adaptive classification from few labeled examples in multiple instance learning, outperforming supervised baselines across twelve benchmarks.

Google researchers propose a 'Sleep' paradigm for continual learning that consolidates short-term in-context knowledge into long-term model parameters via distillation and replay. A 'Dreaming' stage uses reinforcement learning to generate synthetic curricula for self-improvement.

This paper introduces DOPA, a demonstration search framework that uses an out-of-distribution proxy to retrieve robust demonstrations for LLMs when the target domain is inaccessible, enhancing in-context learning performance under distribution shift.

Large language models can improve translation for low-resource languages through structured linguistic reasoning traces, with the most significant benefits occurring during inference rather than training.

This paper proposes using pairwise queries to improve selective classification for binary classification, particularly where confidence estimates are inconsistent, as in LLM in-context learning. Theoretical conditions and experiments on synthetic and real datasets show that pairwise query-based algorithms achieve better accuracy-cost tradeoffs than raw confidence estimates.

This paper investigates how large language models reorganize representational geometry during in-context learning, showing that ICL performance correlates with the geometric structure of tasks and that successful ICL involves increasing separability of representations.

This paper introduces a lightweight multimodal LLM-based framework for cost-effective defect grading of power transmission equipment, using in-context learning and chain-of-thought to generate training data and fine-tuning Qwen3-VL-8B for state-of-the-art performance.