icml2026

The paper introduces MoFO, a momentum-filtered optimizer that mitigates forgetting in LLM fine-tuning by updating only parameters with large momentum magnitudes, preserving pre-trained knowledge without extra storage.

New paper introduces PoPE, a positional encoding that decouples content and position, addressing a fundamental flaw in RoPE used in many LLMs like Qwen, Gemma, DeepSeek. Presented at ICML2026.

This paper formalizes analogical reasoning in Transformers using category theory, introduces synthetic tasks to study its emergence, and reveals that it arises from geometric alignment of relational structures and functor application, with signatures also found in pretrained LLMs. The work was accepted as a Spotlight at ICML 2026.