Tag
AMD 发布了针对其 Ryzen AI NPU 优化的 Stable Diffusion 3.5 Medium，可在 Hugging Face 上获取，实现高效的本地文本到图像生成。
Simon Mo of Inferact presented the state of the vLLM project at PyTorchCon 2026, highlighting KV cache management and kernel optimizations across TPU, AMD, and NVIDIA hardware to address LLM performance and scalability challenges.
A discussion post asking whether open-weight local models will soon be capable of self-optimizing their own runtime stack — e.g. dissecting a large repo like llama.cpp and tuning it for specific hardware — or whether smaller distilled models like Qwen, DeepSeek, and GLM variants could already do it.
This paper explores approximating softmax in pretrained LLMs for kernel acceleration, demonstrating performance gains like up to 25.8% speedup on Blackwell B200 with minimal perplexity impact.
Overspill is an experimental disk tier for FreeToken that allows running large MoE models like DeepSeek-V4-Flash on limited consumer hardware by using memory-mapping and caching optimizations, achieving approximately 3 tokens per second on a setup with a 12 GB GPU and 64 GB RAM.
The author provides a detailed introduction to using Claude Opus-5.5 for kernel development and AI work, sharing practical experiences with local model workflows, quantization tasks, and hardware selection.
The article details a test of the Qwen3.8-Flash-Next AI model running locally on Intel V620 GPUs, where it generated a 3D game from a sloppy prompt in about 3 hours using the OMP harness.
The user has implemented nvfp4 KV cache support for the Qwen3.8 model on a heterogeneous GPU setup using custom CUDA kernels and quantization to optimize performance.
The article describes an optimization for the DeepSeek-V4.1-Flash AI model on Metal hardware, where the attention kernel was improved to only launch necessary tiles, reducing compute waste and enhancing inference speed.
Ahmad points out that the GPU he suggested at $2k has risen to $8k, and OsmanticAI is developing ODS to enable running local AI models on any machine, even for those with limited GPU resources.
The user describes a hardware configuration with dual RTX 3090 GPUs and an AMD EPYC CPU for running the Qwen3-Flash-Next model using llama.cpp, achieving 38 tokens per second in single-stream inference, and seeks advice on whether to add a third GPU or upgrade the CPU to improve performance, especially for running multiple parallel agents.
Ahmad Osman, founder of Osmantic, argues that open-source AI could compound like Linux, highlighting the benefits of developer collaboration and the risks of regulatory capture.
Optimizations for GLM 5.3 Flash on Apple M3 Ultra achieve up to 550 t/s prefill and 38 t/s inference speed through kernel fusion and efficient memory use, without quality loss.
The author explains how speculative decoding affects coding agent speed, with higher acceptance rates on boilerplate code leading to faster typing, and discusses other factors like cache misses that impact performance.
The PyTorch Conference North America will be held in San Jose, featuring sessions on agentic search, hardware-guided workflows, and AI performance optimization with speakers from major tech companies.
The tweet describes the goal of ODS to make local AI deployment seamless and plug-and-play across various hardware, models, and kernel optimizations, while celebrating almost 6,000 GitHub stars.
The article details an open-source tool 'Sunny Narrator' for translating fiction books using a two-model AI pipeline on Tesla P40 GPUs, with optimized llama-server configurations achieving 40-70 tokens per second via MTP speculative decoding.
This article is a practitioner's guide to running local AI models for agent work, highlighting hardware trade-offs and introducing Magnitude, an open-source inference server that simplifies configuration for optimized performance.
The author demonstrates how to run a 182B Qwen model on a 4080 GPU with 64GB DDR5 by offloading ngrams to SSD, achieving faster and more intelligent performance than a 27B model, making large model inference accessible on consumer hardware.
The article provides optimized settings for using the Qwen 3.8 model with llama.cpp, highlighting performance gains in token generation speed and context window management on an RTX 3090 GPU.