Session-Adaptive Orthogonal Distillation (SAOD)? Technology compresses 744B (1.5TB) to under 100GB?
Summary
Session-Adaptive Orthogonal Distillation (SAOD) is a technique that compresses a 744-billion parameter model (1.5TB) to under 100GB, greatly reducing storage and inference costs.
Similar Articles
D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
This paper introduces D-OPSD, a novel training paradigm for step-distilled diffusion models that enables on-policy self-distillation during supervised fine-tuning. It allows models to learn new concepts or styles without compromising their efficient few-step inference capabilities.
Lossless Anti-Distillation Sampling
This paper proposes Lossless Anti-Distillation Sampling (LADS), a novel sampling scheme that counters multi-account distillation by correlating responses across accounts while preserving exact statistical fidelity for individual benign users. Theoretical analysis and experiments show LADS degrades distilled student performance on image, math, and code generation.
OPRD: On-Policy Representation Distillation
OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.
Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation
DO-ALL is a plug-and-play framework that uses dataset distillation to generate synthetic anchors that summarize source data, enabling stable long-term continual test-time adaptation without retaining original source data.
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
ShortOPD proposes a short-to-long on-policy distillation schedule that recovers pruned LLMs for free-form generation by focusing training on effective prefixes, achieving up to 9x improvement over unrecovered models and matching long-horizon distillation with a quarter of the training time.