Tag
NVIDIA showcases the high-throughput performance of serving the Qwen3-8B 2.4T parameter model on GB300 NVL72 hardware, achieving over 4k tokens per second per GPU.
User seeks advice on the best operating system (Windows or Linux) and inference server to run the Qwen3.8.27b model on a dedicated AI rig with RTX 5090 and 96GB RAM for optimal performance.
A 397-billion-parameter AI model has been successfully run on an iPhone, demonstrating on-device AI capabilities with a mixture-of-experts design, though facing challenges in speed, storage, and heat.
A new paper on arXiv introduces an open-source library called LLMRouter with over 16 router implementations and a benchmark xRouteBench, demonstrating that learned routers can outperform fixed-model baselines by 14.6%.
The article clarifies that GPU performance in AI inference is limited by memory bandwidth rather than compute power, using the NVIDIA H100 as an example to explain GPU architecture and its effect on token generation rates.
The article compares confidential inference APIs from Privatemode, Tinfoil, NEAR AI, and Chutes, highlighting their security features like end-to-end encryption and trusted execution environments, along with tradeoffs in model selection and verification maturity.
NInfer adds day-0 support for the Qwen3.8-27B model, achieving around 200 tokens per second on a single RTX 5090 with speculative decoding, and includes engine improvements like concurrent requests and kernel optimizations.
French startup Kog is using software optimization to dramatically speed up LLM inference on standard datacenter GPUs, claiming up to 30x faster throughput. After a demo of 3,000 tokens per second with its open-sourced Laneformer 2B model, the company is now focusing on larger models and targeting software engineering workflows.
Cascadia has launched a distributed AI inference system designed for Intel hardware, enabling scalable and efficient inference workloads.
A hobbyist describes building a low-power llama.cpp server using an Intel N100 motherboard and a refurbished RTX 5060 Ti, sharing performance numbers, power consumption, and model choices.
An analysis of AI inference hardware, comparing Taalas and Groq's approaches to etching model weights into silicon, and noting recent investments by Nvidia and AMD.
The vLLM Conference is coming up in San Francisco, Aug 24–26, hosted by Inferact at Ray Summit, featuring speakers from major AI infrastructure companies.
SK hynix and SanDisk unveiled the High Bandwidth Flash (HBF) standard to bridge the performance gap between HBM memory and SSDs, targeting up to 3TB/s bandwidth and 512GB capacity to speed AI inference.
Astra, an unreleased AI system, produced machine-checkable Lean 4 proofs for 10 long-open math problems at roughly $2,000 inference cost, sparking debate about the true cost and significance of AI-discovered mathematics.
The author observes that Nvidia's desktop 70-class GPUs have stayed at 12GB VRAM across two generations, and suggests Nvidia may be intentionally limiting memory to preserve demand for higher-margin AI-focused hardware.
The author shares their storage server build optimized for local AI inference, anticipating a rumored 26T-a3b model called "Le Chaton FAT" and using high-capacity NVMe drives with ZFS for model storage.
Y Combinator hosted a Paper Club where researchers presented innovations in multi-GPU kernel optimization, including ParallelKittens, a CUDA framework that simplifies development of overlapped multi-GPU kernels and achieves significant speedups across workloads.
Discussion of viable low-cost hardware configurations to run the Kimi K3 AI model locally.
Testing reveals that Intel consumer platforms like Z890 with Arrow Lake CPUs have hardware/firmware limitations that prevent proper PCIe Peer-to-Peer (P2P) communication between multiple GPUs, making them unsuitable for multi-GPU AI workloads despite adequate lane counts.
AMD and Cerebras announced a joint AI inference solution combining AMD Helios rackscale solutions with Cerebras Wafer-Scale Engine, aiming for ultra-low latency and high throughput. The disaggregated inference workflow is expected to deliver up to 5x higher tokens per second per watt.