knowledge-preservation

AI companies are secretly buying and destroying physical books to obtain training data, locking knowledge away, and Anna's Archive is urging volunteers to scan rare books to prevent cultural heritage loss.

TRACE introduces a two-stage curriculum to preserve parametric tool knowledge in enterprise LLMs while enabling fast single-beam greedy decoding, achieving improved accuracy and recall over baselines.

This paper introduces Attention-Shifting (AS), a novel framework for selective machine unlearning in LLMs that balances effective removal of sensitive information while preventing hallucinations and preserving model utility. The method uses importance-aware attention suppression and retention enhancement to achieve up to 15% higher accuracy preservation compared to existing unlearning approaches on standard benchmarks.

This paper investigates how supervised fine-tuning (SFT) increases hallucinations in LLMs by causing knowledge degradation and proposes a self-distillation-based method to mitigate this issue while preserving pre-existing factual knowledge. The authors identify semantic interference among overlapping representations as the primary mechanism behind SFT-induced hallucinations and demonstrate solutions including parameter freezing and self-distillation.