Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
Summary
The paper shows that long-context generation quality in open-weight LMs is primarily a sampling problem rather than a training problem: samplers like top-nsigma that keep a small, steady candidate-token pool preserve accuracy as temperature rises, while top-p/top-k collapse (e.g., 0 vs 46-62 of 100 SWE-bench issues resolved at temperature 2).
Similar Articles
@AdinaYakup: Hy4's weights are now available on the hub https://huggingface.co/tencent/Hy4 - 770B/49B - 1M context - Custom license …
Tencent's Hy4 MoE flagship model weights are released on Hugging Face, featuring 770B total/49B active parameters, 1M context, multimodal text-image-video understanding, and optimized inference via DFly2 speculative decoding.
Hy4 released: 770B MoE, 49B active parameters, and 1M context window
Hy4 is released as a 770B Mixture-of-Experts model with 49B active parameters and a 1M-token context window, marking a notable release in the large-scale LLM space.
Sensenova-Loop in 8GB VRAM
Release of mixed-precision GGUF quantizations of SenseNova-Loop-9B sized for 8 GB GPUs, offering up to ~131K context with lower KL divergence than standard Q4_K_M quants, requiring a patched llama.cpp runtime.
A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises
A paper reporting an industrial bakeoff evaluating ten agentic AI platforms across nine capability dimensions for regulated financial enterprises, revealing a halo effect in demonstration-based scoring and proposing a gate-then-score evaluation protocol.
Periodic and Harmonic Periodic RoPE for Length Extrapolation in LLMs
The paper proposes Periodic RoPE (P-RoPE) and Harmonic Periodic RoPE (HP-RoPE), which repeat rotation angles within sliding windows to eliminate position exhaustion, achieving near-flat perplexity from 768 to 16k and 99-100% passkey accuracy at 16k, far surpassing standard RoPE, NTK, and YaRN.