in-context-learning

This paper proposes LC-ICL, a novel few-shot technique that uses both correct and incorrect examples with error-cause labels to improve large language models' performance on information extraction tasks like named entity recognition and relation extraction.

This paper tests the assumption that LLMs judge better than they generate in in-context QA, finding generation accuracy exceeds self-evaluation on most benchmarks, with evaluation attending less to context. The findings challenge core assumptions in self-evaluation pipelines.

The article argues that training AI on millions of verifiable tasks across diverse RL environments could lead to AGI, and that scaling may overcome current limitations like sample inefficiency. It also examines why progress on computer use has been slower due to lack of grindable environments.

This paper evaluates the robustness of tabular foundation models to biologically inspired distribution shifts in microbiome data, finding that protecting discriminative features is insufficient and zero-imputation is the most harmful perturbation.

This paper introduces 'pigeonholing,' a phenomenon where bad prompts cause LLMs to collapse and repeat errors, leading to a 38-40% performance drop. Experiments across 10 tasks and 10 models show worsening with more conversation turns, and propose RLVR with synthetic errors as a mitigation.

This paper extends contextual entrainment from token-level to sentence-level, showing that even counterfactual sentences in prompts increase their probability during inference. The effect decreases with model size and is driven by 2-4% of attention heads, which can be ablated without performance loss.

This paper empirically studies cross-lingual transfer in in-context learning across seven tasks, six models, and typologically diverse languages, showing that fine-tuning based expectations do not consistently apply and offering new heuristics for source language selection.

This paper introduces SwiftTrans, a framework that addresses the overlooked issue of runtime efficiency in LLM-based code translation, achieving consistent improvements in both correctness and efficiency across benchmarks.

This paper presents Brick-DICL, a dynamic in-context learning framework for automated classification of building management system points into Brick ontology classes. It uses retrieval-augmented generation and multi-model filtering to improve accuracy and reduce manual verification effort.

This paper introduces the Call Playbook dataset for classifying real-world B2B conversations and proposes methods to distill examples into compact, interpretable task instructions, achieving 99% token reduction and up to 7% AUC improvement over traditional in-context learning.

This paper introduces Grammatical Error Representation (GER), a novel method for retrieving in-context demonstrations based on error patterns rather than semantic similarity, significantly improving multilingual grammatical error correction performance in LLMs with in-context learning.

This paper proposes CRUMB, a three-stage inference wrapper that clusters test queries and selects a distributionally matched training subset via MMD minimization to enable efficient Prior-Fitted Network inference on large datasets, achieving state-of-the-art context selection on 51 TabArena datasets.

This paper proposes RoVE, a parameter-free modification to Rotary Position Embeddings that makes value pathways position-sensitive by rotating values simultaneously with keys, transforming RoPE attention into attentive convolution. Experiments on GPT-2 models show consistent gains in few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval.

This paper identifies and formalizes the 'structural attention tax' phenomenon, where the format of retrieved content (e.g., knowledge graph triples) independently distorts LLM attention distribution regardless of semantic relevance, leading to compressed demonstration attention. It provides a formal framework, empirical evidence across models and benchmarks, and proposes structure-aware mitigation strategies.

This paper introduces finite certificates for verifying determinacy and emergence in language model in-context behavior, providing theoretical criteria and experimental validation on contemporary models.

CL-Bench is a new expert-validated benchmark across six domains that evaluates whether LLM-based agents genuinely learn from sequential experience. It finds that naive in-context learning often outperforms dedicated memory systems, indicating current architectures add overhead rather than genuine learning.

This paper proposes a domain-aware coreset construction pipeline that enables a tabular foundation model to predict flood depth with only 0.7% of the training data, achieving 98.5% of the supervised reference accuracy and allowing transfer across watersheds without retraining.

This paper proposes a Perceiver-style architecture pretrained on synthetic bag-structured data to enable efficient, task-adaptive classification from few labeled examples in multiple instance learning, outperforming supervised baselines across twelve benchmarks.

Google researchers propose a 'Sleep' paradigm for continual learning that consolidates short-term in-context knowledge into long-term model parameters via distillation and replay. A 'Dreaming' stage uses reinforcement learning to generate synthetic curricula for self-improvement.

This paper introduces DOPA, a demonstration search framework that uses an out-of-distribution proxy to retrieve robust demonstrations for LLMs when the target domain is inaccessible, enhancing in-context learning performance under distribution shift.