inference

Tenstorrent raised $1.5 billion at a $6.2 billion valuation, more than doubling its late-2024 valuation, to scale AI chips tailored for serving open-weight models and video generation after deciding to remain independent amid acquisition interest from Intel and Qualcomm.

Hugging Face 主播了一场名为 Local AI 202 的直播，介绍如何通过推测解码（Speculative Decoding）大幅提升本地 AI 推理速度，现提供回放。

Maxime Rivest argues that any successful paid LLM inference run should be saved, since it constitutes a valuable example dataset, a view echoed by Gal Elmalah who frames data as a future asset.

Modular announced it is an official inference partner powering MiniMax's M3.1 Flash model, nicknamed Space Bunny. The partnership highlights Modular's MAX platform for serving new MiniMax models.

incoai发布了DFlash 2草稿模型（Qwen3.8-Flash-Next-DFlash2），这是一种用于推测解码的块扩散式草稿器，可在单次前向中预测整个token块，并通过SGLang实现无损解码加速，在接受长度上优于内置MTP。

Show HN: STROT，一个开源「决策编译器」，用 LLM 编译一次意图，之后在本地以 144M 小模型零成本执行分类、评分与路由决策，附带校准与置信度门控保证。

TorchNeuron 作为 PyTorch 的原生 Trainium 后端发布公测，Hugging Face 的 Transformers、Diffusers 等库可直接将其作为普通设备使用，显著提升了模型支持率并取代 Optimum Neuron 的复杂封装。

Charles Hoskinson claims Apple's upcoming Mac Studio M5 Ultra with 512GB of memory could be chained via Thunderbolt 5 to run a 2-trillion-parameter frontier model locally for roughly $100k, far cheaper than an equivalent NVIDIA setup.

A user praises DeepSeek v4.1 Flash 1017 as the most impressive open-source model they've worked with, noting it can achieve state-of-the-art results with the right settings and runs on 4x RTX PRO 6000 or DGX Spark hardware.

作者讲述了在封闭网络环境下用数十张 H200 显卡运行未量化版 Qwen 3.8 27b 的经历，推理速度极快、中文输出表现优秀，且模型能自行理解基础设施与代码架构，凸显了本地大算力给企业带来的竞争优势。

Reflection AI announced that its Beam inference service served 1 trillion tokens in 24 hours and reset usage limits for all users to celebrate the milestone.

llama.cpp の CUDA バックエンドにおいて、低ビット型（low-bit types）向けの MMVQ（マトリックス・ベクトル量子化積）演算を高速化する GitHub プルリクエスト (#30517) が公開された。量子化されたモデルの GPU 推論性能の改善を目指すもの。

作者将 adaptive KV streaming 从 CUDA 移植到 ROCm/HIP，并结合 RDNA 4 内核优化，使 Qwen3-27B (IQ4_XS) 能在单张 16GB RX 9070 XT 上以完整 262k 上下文运行，MTP 解码达到 80+ t/s。项目包含大量内核级优化与功耗调优结果，已发布到 GitHub 分支。

A user shares benchmarks showing that raising the --adapt-every parameter from 1 to 6 in Strata boosts token generation speed by 9-55% across multiple Qwen 3.8 Flash Next prompts, with the largest gains on long-context code descriptions.

StartLux Labs has open-sourced the full-size StartLux-Decision model family (0.8B~27B), supporting Instant single-forward fast answers and AutoThink adaptive reasoning, with its original RLCD² reinforcement learning method dramatically boosting reasoning efficiency — holding a dominant lead on the Decision Index 0.3 leaderboard.

Jaya Gupta and Ashu Garg argue the AI economy is shifting from model training and inference to agent execution — where CPU-based work like scheduled checks, storage, and transaction recovery dominates cost — and that application builders, not model or GPU providers, will control who gets paid.

Phancy Group and RiseUnion launched ModelSphere, an open-source LLM deployment platform on GitHub under Apache 2.0, targeting production serving on mixed GPU fleets with Kubernetes, Helm, SGLang, and vLLM integration. Reception is still early, and its listed features like autoscaling and shared KV cache remain unverified project claims.

A local LLM user reports that sacrificing context length (96K→75K) to run higher-quality quantizations (IQ4_XS→UD-Q4_K_XS with KV-Q8) on a Qwen 27B model actually improved overall task completion time, cutting an 8-hour coding task to under 6 hours due to fewer error-prone drafts.

Hugging Face hosts a live stream with the Reflection team about how they built Beam, which is being released with open weights under Apache 2.0 in collaboration with 40+ partners, including weights, tokenizer, and reference implementation in BF16, FP8, and NVFP4 formats.

David Bellamy shares a concise curriculum for learning core RL and inference systems ideas, citing the HF Ultra-Scale Playbook, Baseten's inference engineering guide, Lambert's RLHF book, and PMPP for kernels, supplemented by blog posts.