decoder-only

UEmbed is a decoder-only multimodal embedding model that produces both sparse and dense representations in a single forward pass, released at 2B, 4B, and 9B scales. It outperforms existing public-data-trained multimodal embedding models on MMEB-v2 and remains competitive on BEIR.

Modus is a decoder-only model that predicts any modality from any combination of others, achieving strong performance across diverse benchmarks without modality-specific heads or losses.

This paper demonstrates that scaling point-in-time language models—trained exclusively on text available up to each calendar date—can substantially narrow the performance gap with unrestricted models, enabling valid backtests and causal inference in finance and social sciences. The authors train decoder-only transformers up to 4B parameters on 1 trillion chronologically filtered tokens and release the full pipeline.

Inkling is a 975B-parameter sparse mixture-of-experts multimodal model accepting text, image and audio inputs and generating text outputs. Released with open weights for research, fine-tuning, and integration.

UniSE is a unified, prompt-free, autoregressive speech enhancement model based on a decoder-only language model, supporting multiple tasks like speech restoration, target speaker extraction, and speech separation in a single model.

PartRep proposes a selective prompt repetition method for decoder-only LLMs that appends only the most informative tokens (selected via NLL) instead of the full prompt, reducing KV cache and prefill FLOPs while retaining most of the accuracy gains across multiple benchmarks.

This paper studies the problem of recovering input token sequences from last-layer hidden states of decoder-only language models using continuous embedding-space optimization, revealing that high-frequency function words are the main failure points while content words recover almost perfectly, achieving up to 97.5% exact-match rate.

The paper introduces SPEED, a layer-asymmetric KV visibility policy that reduces long-context inference costs by processing prompt tokens only in lower layers during prefill while maintaining full-depth attention during decoding.

River-LLM proposes a training-free early-exit framework for decoder-only LLMs that uses KV-sharing to eliminate KV-cache gaps, achieving 1.71–2.16× speedup without quality loss.