gpu

The article discusses how 16GB of VRAM is the realistic high-end limit for most users due to financial constraints, but recent AI model improvements like Qwen 27B quants are enabling more capabilities on such hardware, with hopes for future architectural innovations.

Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.

A tweet suggests that future AI inference hardware may not come from current providers like NVIDIA, highlighting acquisitions of startups such as Groq because GPUs are not optimally designed for inference.

A tweet suggests that offering NVIDIA DGX Station or GB300 as a sign-on bonus would instantly attract hires in the tech industry.

A tweet explains the core components of inference engineering: loadable encodings, backend operations, and native hardware arithmetic, with a discussion on kernels and GPU efficiency.

AMD is planning a 10% price increase across its GPU and chipset products, with potential hikes for CPUs as well.

This paper investigates placement policies for KV cache across GPU, CPU, and SSD tiers to optimize LLM serving for long-lived sessions, evaluating policies like recency and reuse-frequency and finding workload-specific recommendations for migration and prefetch.

Version 1.0 of auto-gpu-kernel has been released, a meta-harness tool that autonomously generates high-performance GPU kernels.

The article advises against buying an RTX 5090 for $9K by suggesting to purchase it cheaper in Taiwan for about $4K, including travel expenses and a vacation.

A user seeks advice on choosing between different vendor versions of the Radeon AI PRO R9700 GPU, considering future expansion and cooling options.

The Nvidia RTX 5090 GPU has vanished from US online retail, with third-party sellers demanding up to $9,500, driven by AI demand and raising scam risks.

K2 Horizon 7B, a compact AI model, achieves impressive benchmark scores rivaling larger models and shows solid performance in initial testing for tasks like compiling llama.cpp for CUDA.

The paper presents tritonBLAS, an analytical model for optimizing GPU GEMM kernel parameters without runtime autotuning, achieving near-optimal performance and significantly reducing compilation time.

NVIDIA announces the RTX PRO 5500 Blackwell Workstation Edition, a professional GPU with 84GB GDDR7 memory built for enterprise AI and graphics workloads.

A user asks whether trading an RTX 5090 for a Mac Studio M5 Ultra is a sensible upgrade for coding, based on memory bandwidth and cost differences.

A China-modified Nvidia RTX 5090 with 96GB of VRAM is available on Alibaba for under $4,000, offering three times more memory at 65% of the original cost.

A user in a tech forum is seeking advice on GPUs for running AI models like DSV4 Flash locally with good performance, comparing AMD and NVIDIA options within a $15k budget.

This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.

Apple has announced the A20 Pro SoC for the iPhone 18 Pro, featuring a 7-core GPU with 40% faster graphics performance, a 32-core Neural Engine, and 50% more memory bandwidth, all manufactured on a 2 nm process.

The author is experimenting with an adaptive memory governor for PyTorch to prevent CUDA OOM errors on 8GB GPUs, sharing code and seeking community feedback.