icl

This thread presents research on whether new facts can be added to an LLM's weights without breaking the model, and finds that it breaks unexpectedly, making compressed KV caches and in-context learning more promising for continual learning.

This paper reveals a counterintuitive phenomenon where correct demonstrations in in-context learning can degrade model accuracy, introducing task preserving perturbations to study the gap between exemplar correctness and utility.