inference

Particle-1, a coding-agent-focused model emphasizing token efficiency (~45% fewer output tokens than DeepSeek V4.1 Flash with comparable accuracy), is now available in OpenClaw v2026.10.10, with zero data retention for standard inference.

A pull request to llama.cpp adds Nemotron 3.5 Transcribe support in the mtmd (multimodal) module, extending the library's audio transcription capabilities.

用户分享使用 Strata 替换 llama.cpp 运行 Qwen 3.8 Flash Next 后，解码速度从 70-80 tk/s 提升至 150 tk/s，并顺带测试了 DeepSeek v4.1flash 2.25bit 量化在多 GPU 上的表现。

Olla v0.1.0 releases a lightweight open-source Go proxy that unifies multiple local LLM inference runtimes (Ollama, vLLM, SGLang, llama.cpp, MLX, etc.) behind a single endpoint with model-aware routing, load balancing, failover, sticky sessions, and OpenAI/Anthropic-compatible APIs. The update adds Hugging Face TEI support, a reworked dashboard, configurable retry behavior, and improved model capability detection.

Revolut moved ~2,000 engineers onto self-hosted GLM models for coding, retiring Claude Code and Codex to an exception list — running on 56 Nvidia B200s at a blended 30 cents per million tokens (about a tenth of prior Anthropic/OpenAI spend), with engineering lead time improving 70%.

NVIDIA released NemotronLabs RE-USE, a 9.6M-parameter multilingual universal speech enhancement model that restores speech quality across diverse degradations and sampling rates, outperforming larger open-source USE models while being ready for commercial use.

Strata's latest update significantly boosts inference performance for AMD R9700 users running Flash Next, roughly doubling token output from ~70 tok/s to 140-160 tok/s while still trailing Radiance. The user notes quantization limits (Q5 falling just short of VRAM for split-engine mode) and hopes future updates close the gap further.

The author reports that RTX 8000 48GB prices have fallen to around $2100 while RTX 3090 prices have roughly doubled, arguing the RTX 8000 may now be the best value 'do everything' option for running large local models in VRAM for home-lab setups.

A pull request to llama.cpp adds support for the Clover GPU/accelerator backend, extending the library's hardware compatibility for LLM inference.

Applied Compute announced it has crossed $100M in annualized run rate and raised $350M at a $3.25B valuation led by Elad Gil, with 40% of customers using it for both training and inference.

MiniMax thanked community builders who shrank M3.1 Flash open weights from 252.4 GB to as low as 93 GB in 10 days, with quantized variants ranging from GGUF to EXL3 and MXFP4 delivering strong accuracy and throughput on consumer and datacenter hardware.

AMD is pouring substantial engineering and compute resources into vLLM to bypass NVIDIA’s CUDA moat, drive the standardization of inference software, and break hardware lock-in—benefiting the entire ecosystem while yielding asymmetric returns for the two companies.

The author argues that accumulating efficiency gains will drive consumer hardware prices up while data center hardware prices fall, quoting a report of a capacity surge with providers offering training compute at heavily discounted rates.

The author benchmarks Speedy against Avalon and Strata for running a Qwen3.8 model locally, finding Speedy dramatically faster: 4x faster reading, generation tok/s around 105-110, and 26-31% shorter total wall-clock time than Avalon.

A user shares a detailed setup guide for running DeepSeek-V4.1-Flash-1017 (~35 tok/s) and Qwen3.8-Flash-Next (~130-140 tok/s) on older consumer hardware (RTX 3090 Ti, 128GB RAM, Threadripper 3990X), including an NVIDIA firmware-throttle unlock via an open-source kernel module and an Xolotl inference engine build.

tandem.cpp is a patch set for llama.cpp that offloads MoE models overflowing VRAM by keeping hot experts on GPU while the CPU handles the rest, achieving 97.3 tok/s on a single RTX 4090 (3.3× faster than llama.cpp) for Qwen3.8-Flash-Next, with gains for other MoE models too.

WiCi is a systems research lab shipping the first wireless eGPU, letting devices access a nearby GPU over Wi-Fi 7 for local AI inference and shared compute while keeping models and data under local control.

ik_llama.cpp now fully supports PrismML's Ternary Bonsai 2 with native CUDA offload, lossless weight repacking, and fast CPU prompt ingestion via -rtr 1, eliminating prior sign tensor bottlenecks. The post provides a setup guide covering PQ2_0 conversion and recommended command-line flags.

Bloomberg 报道 AI 推理基础设施公司 Baseten 将融资约 20 亿美元，估值最高达 300 亿美元，较六月估值翻倍以上；其年化收入据称在 2026 年第一季度从 2 亿美元跃升至 6 亿美元，反映出推理需求的快速增长。

A lawyer and PLI Labs founder publishes full benchmark data for running DeepSeek-V4.1-Flash-1017 (2.75 bpw EXL3) across two DGX Sparks with a 3090 box handling RDMA-served tables and a speculative drafter, hitting 2,600+ tok/s prefill to 256K context and 322 tok/s structured output. All code, launch scripts, and raw JSON are released publicly, with the author openly noting where his build loses to competing 2x Spark setups.