transformers

Hugging Face introduces transformers-mic, a Model Integration Contract (MIC) standard that packages tiny published fixtures, runnable usage paths, and human-reviewed baselines to catch model regressions on cheap CPU runners before they reach the Transformers main branch.

The team shares Latent Sparse Attention (LSA), an open research release enabling subquadratic attention that matches full-attention accuracy at ultra-long context, making sparse attention up to 22.7× faster and full attention up to 72.4× faster.

The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.

A Zyphra Research paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention method using a hierarchical tree-structured indexer over the KV cache, which can be adapted to pretrained models via short KL distillation and scales dramatically better beyond 1M context.

This paper shows that zeroing out 729 task-redundant attention heads in GPT-2-XL at inference time reduces verbatim memorisation by 91.5% while slightly improving WikiText-103 perplexity, offering a training-free way to mitigate privacy leakage from LLMs.

This paper proposes a realizability theory for length generalization in a controlled single-head attention setting, deriving via a simulation transfer operator why relative and absolute positional encodings differ in extrapolation behavior, with experiments on 2625 trained models confirming the structure.

该论文从随机最优控制的角度为 Transformer 在非马尔可夫数据上的近无损下一词预测提供了数学理论，构造了深度 M+1 的 Transformer 实现截断滤波器，并在平稳遍历性或滤波稳定性条件下证明误差可随上下文长度和数据量任意减小。

The paper introduces two kernel-based interpolatory KV-cache compression methods, GAIM and Q-DAIM, along with linear-time surrogate variants and streaming attention extensions, achieving performance comparable to state-of-the-art on LongBench-E.

Free open-source 'AI Engineering from Scratch' curriculum teaches building backprop, tokenizers, attention, transformers, hybrid RAG, and agents from raw math across Python, TypeScript, Rust, and Julia, with interactive in-terminal tutoring and runnable simulations.

A speculative discussion about the next architectural innovations likely to appear in AI models, including looped transformers, Gated Delta Net 2 variants, attention residuals, and a potential revival of diffusion transformers without quality loss.

A survey and position paper arguing that embedding research across modalities is converging via contrastive learning and Transformers, proposing a framework of coverage, comparability, and compositionality with transformation, alignment, and joint embedding as paths toward omnimodal unification.

Show HN: A project that compiles integer neural networks into ZDoom maps so DOOM's voodoo dolls and sector properties perform the arithmetic, verified bit-exact against NumPy. It's extremely slow but fully verifiable, and lets players draw digits for DOOM to classify in real time.

Mistral Large 4 (codenamed Le Chonk) is released on Hugging Face Hub — a multimodal, multilingual Mixture-of-Experts model with 1T total/49B active parameters and a 1M-token context window, featuring day-0 support in transformers, vLLM, and llama.cpp along with quantized GGUF versions.

Transformers v5.22.0 发布，新增对 Mistral Large 4 (Le Chonk) 模型的支持。

This tweet compiles 6 ML practice/interview questions related to Anthropic, covering data cleaning, tokenizer implementation, Transformer architecture, inference latency optimization, and distributed training pipeline design.

At EMNLP 2026, T-Tech researchers presented LIMe, a lightweight attention extension that reuses key-value buffers from earlier layers to mitigate representation collapse, reducing LLM training compute by ~15% at equal quality versus baseline LLaMA on ~1B-parameter models.

SpecSlot 是一种像素空间 rectified-flow Transformer，通过将空间锚点扩展为分配不相交 DCT 系数带的多个槽位，并仅让最低频槽参与全局注意力，在不增加计算成本的情况下提升图像细节生成质量。

AB-Attention提出一种免训练的块稀疏注意力方法，通过块内聚类选择、包含概率加权采样和细粒度核收集来近似稠密注意力，在Wan2.1和HunyuanVideo上跳过89-92%注意力计算，同时PSNR提升最高4.4dB，端到端加速最高2.23倍。

A Stanford CS 312 "Deep Learning Alchemy" course shifts the burden of theory construction onto students (and their Claude assistants), having them empirically probe transformer components via ablations on a 614M-token depth-8 transformer — prompting reflection that maybe memorized ML knowledge was never that critical.

该论文研究语言模型在处理花园路径句后是否会保留被放弃的句法分析残余，发现残余效应主要储存在名词短语的状态中，并可通过激活修补（activation patching）在32B和70B模型中近乎完全消除该残余。