Tag
The paper introduces Mela, a memory-augmented transformer architecture inspired by human memory consolidation, featuring a Hierarchical Memory Module that improves long-context language modeling performance.
A researcher named HongcanGuo teases a brand-new approach to text modeling, but the tweet provides no technical details.
This paper introduces Trajectory-Shaped Discrete Flow Matching (TS-DFM), which replaces blind stochastic jumps with guided navigation to significantly improve text generation efficiency and reduce computational costs. The method achieves superior perplexity and speed compared to traditional multi-step baselines while maintaining unchanged inference costs.
This paper introduces TextLDM, a method that adapts visual latent diffusion transformers for language modeling by mapping discrete tokens to continuous latents. It demonstrates that this approach, enhanced by representation alignment, matches GPT-2 performance and unifies visual and text generation architectures.
Introduces Token Time Continuous Diffusion (TTCD), a new diffusion language model that operates in continuous space with per-token times, outperforming discrete models at high speedups in conditional generation and Sudoku solving.
LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.
Andrej Karpathy's 'Neural Networks: Zero to Hero' is a free course covering neural networks from basics to modern architectures like transformers, with YouTube lectures and Jupyter notebooks. It includes hands-on implementations of micrograd and makemore.