knowledge-injection

KItCAT introduces a lightweight training strategy for auto-regressive LLMs that uses input corruption to generate diverse training inputs, improving knowledge injection from niche documents without costly paraphrasing.

This paper proposes Golden-GRPO Injection (GRIN), a mixed-policy reinforcement learning framework for continual knowledge injection in large language models, overcoming limitations of supervised fine-tuning. Experiments demonstrate superior performance on knowledge absorption benchmarks.

This paper introduces RoCo-ACE, a rollout-conditioned online distillation objective for knowledge injection into multimodal large language models. It improves injected knowledge accuracy while limiting drift in non-updated behaviors.

TokenMem injects knowledge into frozen LLMs via a dedicated cross-attention channel, training a thin gating adapter through two-phase curriculum to improve knowledge compliance under counterfactual knowledge, achieving 69-70% KC compared to 20-52% for vanilla RAG.

Presents a lightweight knowledge-injection framework for zero-shot ICU delirium prediction that augments structured EHR data summaries with external clinical knowledge at inference time, improving AUROC by up to 8.57 percentage points on LLaMA models without fine-tuning.

This paper investigates whether layer-wise differences exist in Mixture-of-Experts (MoE) models and proposes EAACD, an expert-aware adaptive contrast decoding method that leverages expert activation patterns in higher layers to reduce hallucinations in LLMs for QA tasks.

A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.

The paper investigates scaling laws for hypernetwork-based knowledge injection into LLMs, finding predictive power law scaling and reliable out-of-distribution generalization, establishing hypernetworks as a scalable alternative to LoRA and full fine-tuning.

Decoupled Mixture-of-Experts (DMoE) proposes a modular architecture for parametric knowledge injection, decoupling experts and router from the base model to enable efficient auto-regressive inference and mitigate catastrophic forgetting.

MixSD proposes a self-distillation method for knowledge injection in language models that aligns supervision with the model's native distribution, reducing catastrophic forgetting during fine-tuning. It achieves near-perfect memorization while retaining up to 100% of base capabilities, vastly outperforming standard SFT.