language-modeling

Tag

Cards List
#language-modeling

@kplikethebird: One of the hardest parts of getting AI to write like you is getting it to make the same language decisions you would, w…

X AI KOLs Timeline ↗ · 2026-09-15 Cached

This tweet discusses the challenge of using probabilistic methods in AI writing models to replicate individual language decisions, potentially advancing AI-assisted writing.

0 favorites 0 likes
#language-modeling

@PyTorch: Learn how PyTorch can extend language modeling beyond classification to practical time-to-event prediction. Witold Czub…

X AI KOLs Following ↗ · 2026-09-14 Cached

This article promotes the PyTorch Conference North America, highlighting a session by Witold Czubala from UBS on using PyTorch-based transformer models for time-to-event prediction in wealth management from longitudinal email data.

0 favorites 0 likes
#language-modeling

Representation-based Masked Diffusion Model

arXiv cs.CL ↗ · 2026-09-14 Cached

The paper proposes Representation-based Masked Diffusion Model (RMDM), which leverages text representations to improve parallel token updates in masked diffusion models, enhancing generation quality especially in few-step sampling.

0 favorites 0 likes
#language-modeling

Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

arXiv cs.CL ↗ · 2026-09-10 Cached

This paper introduces Looped GPT-BERT, which uses depth-wise parameter sharing to train a small language model with fewer parameters, achieving comparable performance to baselines in the BabyLM 2026 Strict-small setting.

0 favorites 0 likes
#language-modeling

Kalman Delta Networks: Uncertainty-aware Associative Memory

Hugging Face Daily Papers ↗ · 2026-09-07 Cached

Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.

0 favorites 0 likes
#language-modeling

Toppling the Hierarchy in Byte-level Language Modeling

arXiv cs.CL ↗ · 2026-09-02 Cached

This paper examines hierarchical byte-level language models and reveals that their design restricts character-level understanding, with pure byte-level models outperforming them on character manipulation tasks, establishing a trade-off between computational efficiency and fine-grained performance.

0 favorites 0 likes
#language-modeling

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation in Transformers

Hugging Face Daily Papers ↗ · 2026-08-25 Cached

The paper introduces Gated Recurrent Transformer, a model that reuses a shared core across depth with adaptive update gates, achieving comparable or better quality than deeper models with fewer parameters and lower memory.

0 favorites 0 likes
#language-modeling

What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?

arXiv cs.CL ↗ · 2026-08-19 Cached

This paper investigates the tokens learned when tokenization is optimized jointly with language modeling, comparing tokenizer-free methods across multiple languages and finding that they produce distinct, efficient vocabularies for NLP.

0 favorites 0 likes
#language-modeling

Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention

arXiv cs.CL ↗ · 2026-08-19 Cached

This paper proposes Q-Interference, a memory-efficient quantum-inspired attention mechanism for GPT models that uses phase-aware scoring and an exact trigonometric factorization to improve token interaction without increasing memory overhead.

0 favorites 0 likes
#language-modeling

BCMT: Blockwise Causal Memory Transformer

arXiv cs.CL ↗ · 2026-08-17 Cached

BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.

0 favorites 0 likes
#language-modeling

Microsoft's Full-bandwidth Transformers (26 minute read)

TLDR AI ↗ · 2026-08-17 Cached

The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.

0 favorites 0 likes
#language-modeling

TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

arXiv cs.LG ↗ · 2026-08-11 Cached

This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.

0 favorites 0 likes
#language-modeling

Full-bandwidth transformer

Hugging Face Daily Papers ↗ · 2026-08-09 Cached

A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.

0 favorites 0 likes
#language-modeling

Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

arXiv cs.CL ↗ · 2026-08-06 Cached

This paper from the Kathleen series shows that an attention-free, byte-level model with ~0.5M parameters can beat a parameter-matched transformer on WikiText-103 language modeling and generation, introduces a non-parametric 'Form Distance' metric for evaluating text realism, and demonstrates that retrieval-augmented decoding from the model's own training corpus improves generation quality.

0 favorites 0 likes
#language-modeling

Character Iconicity vs. Arbitrariness: An Arabic NLP Perspective

arXiv cs.CL ↗ · 2026-08-05 Cached

This paper investigates whether Arabic character form-function relationships are arbitrary from an NLP perspective, showing that random character remappings to reduced rasm sets can achieve competitive performance across various NLP tasks.

0 favorites 0 likes
#language-modeling

PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

arXiv cs.AI ↗ · 2026-07-29 Cached

PreDiff-LM proposes a hybrid attention mechanism that preserves causal attention for prompt tokens and bidirectional attention for masked target tokens, enabling adaptation of pretrained autoregressive models for discrete masked diffusion language modeling, achieving improvements in perplexity and downstream tasks over prior diffusion baselines.

0 favorites 0 likes
#language-modeling

The JEPA Paradox in Language: The Geometry of Linguistic Alternatives

arXiv cs.CL ↗ · 2026-07-28 Cached

This paper analyzes why deterministic JEPA-style latent prediction works for images but not for text, attributing the failure to high conditional variance in language where masked contexts admit multiple valid completions whose representations lack a coherent center.

0 favorites 0 likes
#language-modeling

Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

arXiv cs.LG ↗ · 2026-07-27 Cached

This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.

0 favorites 0 likes
#language-modeling

I trained a 0.5M model on 1B tokens of Fineweb-edu dataset.

Reddit r/LocalLLaMA ↗ · 2026-07-23

A personal project where a 0.5M parameter language model was trained on 1 billion tokens from the Fineweb-edu dataset.

0 favorites 0 likes
#language-modeling

Gigatoken (GitHub Repo)

TLDR AI ↗ · 2026-07-22 Cached

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback