Tag
The paper introduces Coffee, a plug-and-play framework that guides discrete diffusion models at inference time using compiled finite-state objectives, avoiding exponential enumeration of token completions while supporting hard constraints and learned soft objectives across symbolic, language, and biological benchmarks.
FRAC是基于分数阶动力学的新型选择性状态空间模型架构，用幂律长记忆替代传统SSM的指数遗忘，通过log间距指数模式和近似实现高效循环模块，并在1.3B参数语言模型上显著提升长上下文性能。
Stanford University's CS336 course 'Building Language Models from Scratch' has released 18 complete lecture videos, open-source assignments, and course materials, comprehensively covering the entire workflow of large language model engineering, suitable for self-learners to systematically study LLM development.
This paper introduces Complex KDA, an enhancement to Kimi Delta Attention that improves the expressivity of linear RNNs by enabling 2D rotations and state-tracking capabilities, achieving superior length extrapolation and language modeling performance.
IntBMoE is a novel Mixture-of-Experts method that decouples participation, execution, and materialization costs through block-level conditioning and sparse execution, demonstrating improvements in image classification, language modeling, and recommendation systems, with real-world deployment in AMap's generative recommendation system.
This paper introduces Complex KDA, an enhanced version of Kimi Delta Attention that combines a delta-rule transformation with a reflection to achieve greater expressivity, outperforming Transformers in some tasks while maintaining efficiency, with open-source code and models available.
This paper introduces a method using rotating sparse wiring on a hypercube to replace attention in neural networks, achieving comparable or better performance with significantly reduced computational cost and parameters.
DantinoX is an open-source JAX/Flax library that provides a unified framework for training and comparing autoregressive, masked diffusion, and flow-matching language models, enabling controlled experiments and benchmarking.
This tweet discusses the challenge of using probabilistic methods in AI writing models to replicate individual language decisions, potentially advancing AI-assisted writing.
This article promotes the PyTorch Conference North America, highlighting a session by Witold Czubala from UBS on using PyTorch-based transformer models for time-to-event prediction in wealth management from longitudinal email data.
The paper proposes Representation-based Masked Diffusion Model (RMDM), which leverages text representations to improve parallel token updates in masked diffusion models, enhancing generation quality especially in few-step sampling.
This paper introduces Looped GPT-BERT, which uses depth-wise parameter sharing to train a small language model with fewer parameters, achieving comparable performance to baselines in the BabyLM 2026 Strict-small setting.
Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.
This paper examines hierarchical byte-level language models and reveals that their design restricts character-level understanding, with pure byte-level models outperforming them on character manipulation tasks, establishing a trade-off between computational efficiency and fine-grained performance.
The paper introduces Gated Recurrent Transformer, a model that reuses a shared core across depth with adaptive update gates, achieving comparable or better quality than deeper models with fewer parameters and lower memory.
This paper investigates the tokens learned when tokenization is optimized jointly with language modeling, comparing tokenizer-free methods across multiple languages and finding that they produce distinct, efficient vocabularies for NLP.
This paper proposes Q-Interference, a memory-efficient quantum-inspired attention mechanism for GPT models that uses phase-aware scoring and an exact trigonometric factorization to improve token interaction without increasing memory overhead.
BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.
The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.
This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.