Tag
This tweet discusses the challenge of using probabilistic methods in AI writing models to replicate individual language decisions, potentially advancing AI-assisted writing.
This article promotes the PyTorch Conference North America, highlighting a session by Witold Czubala from UBS on using PyTorch-based transformer models for time-to-event prediction in wealth management from longitudinal email data.
The paper proposes Representation-based Masked Diffusion Model (RMDM), which leverages text representations to improve parallel token updates in masked diffusion models, enhancing generation quality especially in few-step sampling.
This paper introduces Looped GPT-BERT, which uses depth-wise parameter sharing to train a small language model with fewer parameters, achieving comparable performance to baselines in the BabyLM 2026 Strict-small setting.
Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.
This paper examines hierarchical byte-level language models and reveals that their design restricts character-level understanding, with pure byte-level models outperforming them on character manipulation tasks, establishing a trade-off between computational efficiency and fine-grained performance.
The paper introduces Gated Recurrent Transformer, a model that reuses a shared core across depth with adaptive update gates, achieving comparable or better quality than deeper models with fewer parameters and lower memory.
This paper investigates the tokens learned when tokenization is optimized jointly with language modeling, comparing tokenizer-free methods across multiple languages and finding that they produce distinct, efficient vocabularies for NLP.
This paper proposes Q-Interference, a memory-efficient quantum-inspired attention mechanism for GPT models that uses phase-aware scoring and an exact trigonometric factorization to improve token interaction without increasing memory overhead.
BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.
The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.
This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.
A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.
This paper from the Kathleen series shows that an attention-free, byte-level model with ~0.5M parameters can beat a parameter-matched transformer on WikiText-103 language modeling and generation, introduces a non-parametric 'Form Distance' metric for evaluating text realism, and demonstrates that retrieval-augmented decoding from the model's own training corpus improves generation quality.
This paper investigates whether Arabic character form-function relationships are arbitrary from an NLP perspective, showing that random character remappings to reduced rasm sets can achieve competitive performance across various NLP tasks.
PreDiff-LM proposes a hybrid attention mechanism that preserves causal attention for prompt tokens and bidirectional attention for masked target tokens, enabling adaptation of pretrained autoregressive models for discrete masked diffusion language modeling, achieving improvements in perplexity and downstream tasks over prior diffusion baselines.
This paper analyzes why deterministic JEPA-style latent prediction works for images but not for text, attributing the failure to high conditional variance in language where masked contexts admit multiple valid completions whose representations lack a coherent center.
This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.
A personal project where a 0.5M parameter language model was trained on 1 billion tokens from the Fineweb-edu dataset.
Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.