Tag
This paper introduces SpecDrop, a parameter-free category-conditioned routing scheme for modular networks, showing that on vision tasks it achieves competitive accuracy while on fuzzy language partitions it reduces to no-routing baselines, suggesting granularity alignment matters more than router design.
This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.
This paper introduces AdaNAGED, a method that combines zero-order optimization, parameter-free adaptation, and non-Euclidean update geometry for memory-efficient fine-tuning of large language models, with theoretical convergence guarantees and validation on the OPT-1.3B model.
This paper identifies gradient oscillation and residual explosion as causes of training instability in Looped Transformers, and proposes Fully Looped Transformer with two parameter-free modifications (Fully Looped Architecture and Attention Injection) to stabilize training up to 12 loop iterations, achieving up to 13.2% improvement in downstream performance.