Tag
The article provides optimized settings for using the Qwen 3.8 model with llama.cpp, highlighting performance gains in token generation speed and context window management on an RTX 3090 GPU.
This article provides a detailed guide on running the Qwen3.8-Flash AI model on an RTX 3090 with 64GB RAM, discussing performance metrics, quantization settings, and deployment steps using tools like llamacpp.
Scott Hanselman debugs slow 10 Gigabit Ethernet performance by adjusting Intel NIC receive buffers and disabling Large Send Offload, identifying a Windows-specific issue.
Reverse-engineered the Axera AX8850 NPU's int8 weight format to enable direct GGUF inference in llama.cpp, achieving up to 24.5 t/s decode and 716 t/s prefill on a Raspberry Pi 5, outperforming the vendor's runtime by 1.5×.
The article details progress in achieving over 50 tokens per second inference speed for DeepSeek-V4-Flash on M3 Ultra using the DwarfStar inference engine, with improvements in performance and quality metrics.
A user shares how they achieved a 2x performance boost in AI inference by splitting a large MoE model between a Strix Halo APU and an R9700 GPU, detailing configurations and code modifications.
An article reports that enabling CPU affinity on a 24-core build machine unexpectedly slowed down builds, highlighting complexities in optimizing multi-core processing.
A user forked the Ninfer inference engine to run on CMP170HX GPUs, doubling performance for the Qwen3.6-35B model with llama.cpp and sharing the configuration details.
The user switched from LM Studio to vLLM for serving Qwen 3.8 models, achieving 143 tokens per second and reduced GPU temperatures, referencing a helpful GitHub repository.
A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.
NVIDIA shares debugging lessons from its Exemplar Cloud program, detailing how configuration issues in SMMU power management, NUMA placement, NCCL queue-pair concurrency, and hardware defects cause 8-12% training throughput gaps on AI clusters, and how to diagnose and fix them.
A guide on optimizing MTP (Multi-Token Prediction) performance by tuning n_max parameter, with benchmark results for various models like Gemma-31b and Qwen on P100 and V100 GPUs.
The author shares techniques that improved training pipeline performance from 36 to 47 steps per minute.
ClickHouse Managed Postgres scales PgBouncer to 4x throughput by running a fleet of processes with SO_REUSEPORT, enabling multi-core utilization and solving cancellation forwarding via peering.
Benchmarking shows that optimizing Java code (e.g., reducing SLF4J logging) has a far greater impact on latency than choosing a garbage collector, especially at high percentiles.
This article describes using the Codex AI agent to automatically migrate terminal shell configuration from Oh My Zsh to Zinit + Starship + Rust toolchain, demonstrating the AI's ability to perform engineering steps such as backup, key isolation, and performance analysis, ultimately achieving an order-of-magnitude improvement in startup speed.
This article provides a comprehensive overview of the complete technology stack for cloud deployment of Transformer inference, covering application scenarios, workload definition, models, inference engines, hardware, observability, and performance optimization, along with future trends.
This article describes how to use the SYCL backend with llama.cpp to achieve over 60 tokens per second on the Qwen 3.6-35B-A3B model using an Intel Arc Pro B70 GPU, with the entire model and KV cache in VRAM.