Tag
该论文提出将层状态轨迹作为可观测对象来研究 Transformer 推理过程，发现真实层序产生的路径比端点保持的乱序更对齐，并通过局部转移执行器在 Qwen 等模型上选择性地控制内部轨迹与 top-1 候选边界。
The article shares a dedicated chapter on inference from the book 'How to scale your model', covering key optimization techniques for transformer models such as KV cache and latency analysis.
Extropic introduces Z1T, a sparse transformer-like model optimized for their probabilistic Z1 hardware to enhance energy efficiency in transformer inference through algorithmic-hardware co-design.
FlashAttention-V introduces a blocked FlashAttention optimization for scalable vector architectures, achieving up to 42× speedup in transformer inference for small language models on CPUs and identifying quantization bottlenecks.
A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.
DepthWeave-KV is a token-adaptive cross-layer residual factorization method for compressing KV cache in long-context transformer inference, achieving 8.3x memory reduction and 72.8 tokens/s at 64K context while preserving near-full-cache task quality across benchmarks.
Researchers introduce Programmable KV Cache, a method for editing and composing KV caches to avoid re-prefilling long contexts during LLM agent inference, achieving 53–398× reduction in p90 time-to-first-token while maintaining decision identity.
Etched made a high-profile debut, announcing an AI chip and full inference cluster built specifically for Transformer inference. It has secured over $1 billion in customer contracts and $800 million in funding, with the first cabinet set to ship this summer.
A custom digital chip designed gate-by-gate achieves over 56,000 tokens/sec running a Transformer with KV cache at just 80 MHz, prototyped on an FPGA.
This article provides a comprehensive overview of the complete technology stack for cloud deployment of Transformer inference, covering application scenarios, workload definition, models, inference engines, hardware, observability, and performance optimization, along with future trends.