gpu

This paper introduces HYMELL, a hybrid analytical-machine-learning framework for estimating LLM inference latency and energy across prefill and decode phases, validated on NVIDIA H100 with under 5% error for LLaMA 3 8B.

Notes that RTX PRO 6000 GPUs could be obtained cheaply a year ago, but prices have since tripled.

A purported RTX 5090 with 96GB of VRAM has been spotted on Alibaba, hinting at a possible new GPU variant from Nvidia.

Enabling PCI-E peer-to-peer (P2P) for consumer Nvidia GPUs with patched drivers and vLLM environment variables yields roughly 25% prefill throughput improvement for free, as demonstrated by benchmarks.

A tweet highlights the large bandwidth difference between RTX PRO 6000 (1.8TB/s) and DGX Spark (273GB/s), arguing that higher bandwidth is crucial for local AI agents and agentic swarms.

Highlights a paper introducing ZeRO (Zero Redundancy Optimizer), a memory optimization system for efficiently training extremely large models on limited GPU memory.

Ahmad Osman shares performance numbers from running DeepSeek V4 Flash 0731 on an NVIDIA DGX Station.

A llama.cpp PR proposes switching quantized-KV decode from the VEC to TILE SYCL kernel on Intel Battlemage, reportedly delivering up to ~169% faster decode at 118K context. The PR is open with caveats and limited independent verification.

A developer is building Octane, a speculative decoding runtime for local LLM inference on consumer hardware, aiming for 2-3x speedup with exact output quality. Currently in active development with paged KV cache, continuous batching, and batched attention implemented.

A research paper presenting a learning-to-rank framework for selecting efficient tensor-network contraction plans for GPU-accelerated quantum circuit simulation, using gradient-boosted rankers trained from GPU measurements.

vLLM, open-sourced by UC Berkeley's Sky Computing Lab, is a high-performance LLM inference and serving library. Through PagedAttention and continuous batching, it dramatically improves throughput and reduces VRAM waste, while being compatible with the OpenAI API and various hardware.

vast.ai now supports Hugging Face Storage Buckets as a cloud connection, allowing rented GPU instances to pull datasets and checkpoints directly from HF buckets and push results back without manual transfers.

The author ports vLLM's serving stack to C++20, creating a 66 MiB binary with no Python at inference, token-for-token verifiable against vLLM and competitive in throughput on tested hardware.

A blog post announcing the addition of recursive functions to the Futhark programming language, explaining the historical challenges of recursion on GPU backends and the design trade-offs involved.

A PSA recommending that buyers of GPU compute include late fees on delivery as a standard contract clause in the current market, arguing it benefits everyone.

YC shares a podcast episode with Starcloud CEO Philip Johnston, whose company launched an Nvidia H100 GPU into orbit and trained the first LLM in space, raising $200M and planning 88,000 satellites.

A technical guide demonstrating how to serve five specialized small models (SLM, OCR, NER, reranker, object detector) on a single GPU using open-source tools, covering memory management, batching, and the Superlinked Inference Engine.

A discussion about the next evolution of frontier AI usage, questioning whether cloud-based agent workflows and massive distributed compute will replace local setups, referencing Tibo's tweet and OpenAI's recent math results.

A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.

At the Future of Memory and Storage conference, NVIDIA announced open sourcing its cuFile APIs for GPU-direct storage and featured its Vera CPU delivering up to 3.21x higher throughput than x86 in compression and encryption pipelines, addressing AI's growing storage demands.