inference

AbliteratedOutTheBox 是一个 llama.cpp 分支，可在推理时对任意加载的模型实时执行 abliteration（去除对齐/拒绝机制），无需微调、无需额外文件，也不占用额外显存。

llama.cpp 合入 PR #30928，新增 FlintForCausalLM 模型架构支持，由 alexey-rsi 提交。

A user successfully ran the 748B-parameter DeepSeek-V4.1-Flash-1017 model at 7.2 tok/s on a €2k used PC with an RTX 3090, using a 1-bit Unsloth UD-IQ1_S GGUF quantization and a custom llama.cpp build with on-demand Engram table loading.

Flint scores 54.9% on Terminal-Bench 2.1 with only ~2B active parameters, outperforming other small MoE models and some dense ~30B models while running at 125+ tok/s on a MacBook Pro with ~24GB memory.

Part 1 of Amazon's tutorial series on profiling TorchNeuron workloads on AWS Trainium, covering how to configure torch.profiler with the Neuron backend, read key_averages() output, and trace operations through Neuron Explorer.

A user shares their positive experience running Qwen 3.8 Flash Next in IQ4 xs quantization via strata, comparing its quality favorably to models they used professionally last year.

NVIDIA在GTC柏林展示其本地AI硬件全线产品，包括RTX Spark、DGX Spark 64GB和DGX Station，并宣布多项新产品、软件更新及对Nemotron、DeepSeek、Qwen等新模型的首日支持。

Qwen3.8-Flash-Next is released in a 1.66bpw EXL3 quantization that fits entirely within a 36GB RTX 5090 with no offloaded experts, delivering stable 200-300 t/s inference without reasoning loops.

A developer shares custom general and fused LoRA kernels for llama.cpp that restore prompt-processing throughput on Strix Halo from ~300 t/s back up to ~1400 t/s when running LoRA adapters, and invites others to help upstream the changes.

Mouse-1 是一个面向 agent harness 的轻量决策模型，能处理 agent 执行过程中的大量小决策，宣称比 GPT-6 Sol 便宜 163 倍、快 65 倍，可与现有技术栈并行接入。

DeepSeek V4.1 Flash 的大规模推理部署文章，介绍 DeepInfer 结合分布式推理、缓存、负载均衡与投机解码提升效率，DeepSeek 单集群日处理 155 万亿 token，每 GPU 小时收入 38 美元，并开源了大部分推理框架。

PyTorch Conference North America's Demo Theater features 14 sponsor presentations covering training, inference, compilers, and deployment, including demos from IBM, Datadog, Modal, Arm, and Red Hat.

Engram is described as DeepSeek's most significant breakthrough of 2026, coming shortly after the company open sourced its entire inference serving stack for free.

Live from PyTorch Con 2026, speakers from IBM, Google Cloud, and Red Hat present "Open Source Inference at Production Scale" with a live LLM demo underway.

Simon Mo of Inferact presented the state of the vLLM project at PyTorchCon 2026, highlighting KV cache management and kernel optimizations across TPU, AMD, and NVIDIA hardware to address LLM performance and scalability challenges.

Live from #PyTorchCon 2026, Amazon's Robert Hundt introduces TorchNeuron, the native PyTorch backend for AWS Trainium, supporting eager execution and torch.compile for training and serving via vLLM-Neuron, plus Neuron Explorer for profiling down to hardware instructions.

swyx points out DeepSeek shared more substantive details on their Zhihu page, including claims that their in-house inference framework DeepInfer achieves ~155 trillion tokens/day per cluster and $38 per GPU-hour revenue, implying an ~80% serving margin versus typical $5-10/GPU-hour costs.

Hugging Face 发布 AnyLanguageModel 1.0 及配套 swift-huggingface、swift-xet 1.0，作为 Apple Foundation Models 框架的即插即用替代，支持本地与多种托管推理提供商的统一 API。

AWS 与 PyTorch 宣布推出 TorchNeuron——一个面向 AWS Trainium 芯片的原生 PyTorch 开源后端，支持 eager 执行、torch.compile 与 NKI 自定义内核，模型代码只需 `.to('neuron')` 即可运行，将于 10 月 20 日以 Apache 2.0 公开 Beta。

Taco 提出一种跨平台 LLM 压缩方法，通过纯整数推理、精确的 float32 到整数转换与一致性测试向量，确保编码端与解码端在不同 GPU/CPU 硬件上产生完全相同的输出，并在 7 个后端上验证了哈希一致。