decoder-only

Linum introduces Pyramid-JiT (P-JiT), a decoder-only pixel-space diffusion architecture that eliminates the VAE and predicts the target image at multiple resolutions along the DiT trunk, reaching Linum v2's FD-DINOv2 score with 11.3× fewer training samples and 4.3× fewer GPU-hours at 4× the pixels. Code and weights are released under Apache 2.0 as a research artifact en route to Linum v3.

A 3D visualization explaining the architectural differences between Deepseek Flash v4.1 and a typical decoder-only transformer.

The paper explores K/V-cache interventions for persona control in decoder-only language models, finding dissociations between representation alignment and behavioral expression, with mid-layer interventions being effective for persona expression while preserving lexical diversity.

UEmbed is a decoder-only multimodal embedding model that produces both sparse and dense representations in a single forward pass, released at 2B, 4B, and 9B scales. It outperforms existing public-data-trained multimodal embedding models on MMEB-v2 and remains competitive on BEIR.

Modus is a decoder-only model that predicts any modality from any combination of others, achieving strong performance across diverse benchmarks without modality-specific heads or losses.

This paper demonstrates that scaling point-in-time language models—trained exclusively on text available up to each calendar date—can substantially narrow the performance gap with unrestricted models, enabling valid backtests and causal inference in finance and social sciences. The authors train decoder-only transformers up to 4B parameters on 1 trillion chronologically filtered tokens and release the full pipeline.

Inkling is a 975B-parameter sparse mixture-of-experts multimodal model accepting text, image and audio inputs and generating text outputs. Released with open weights for research, fine-tuning, and integration.

UniSE is a unified, prompt-free, autoregressive speech enhancement model based on a decoder-only language model, supporting multiple tasks like speech restoration, target speaker extraction, and speech separation in a single model.

PartRep proposes a selective prompt repetition method for decoder-only LLMs that appends only the most informative tokens (selected via NLL) instead of the full prompt, reducing KV cache and prefill FLOPs while retaining most of the accuracy gains across multiple benchmarks.

This paper studies the problem of recovering input token sequences from last-layer hidden states of decoder-only language models using continuous embedding-space optimization, revealing that high-frequency function words are the main failure points while content words recover almost perfectly, achieving up to 97.5% exact-match rate.

The paper introduces SPEED, a layer-asymmetric KV visibility policy that reduces long-context inference costs by processing prompt tokens only in lower layers during prefill while maintaining full-depth attention during decoding.

River-LLM proposes a training-free early-exit framework for decoder-only LLMs that uses KV-sharing to eliminate KV-cache gaps, achieving 1.71–2.16× speedup without quality loss.