Tag
Krasis, a MoE-focused runtime, enables running the 397B-parameter Ornith model on a single RTX PRO 6000 Blackwell 96GB GPU with ~20-24 tok/s decode by dynamically managing expert residency in VRAM.
A detailed guide on running the 27B Bonsai model on hardware with only 8GB VRAM using a 1-bit quantized version and the PrismML fork of llama.cpp, including exact server commands and configuration.
This paper proposes an efficient GPU inference method for LLMs with moderate unstructured sparsity, introducing a three-layer matrix storage format and a SpMM kernel that jointly utilizes sparse tensor cores and CUDA cores, achieving up to 1.64× kernel-level speedup over SpInfer and up to 1.41× end-to-end speedup over FlashLLM.
Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.
Demonstration of running Qwen3.6-27b on GPU at 100 TPS and Hy3-299B on CPU at 25 TPS simultaneously, marking a milestone in local LLM inference.
A detailed home lab setup with 6x P40 GPUs running a quantized MiniMax M2.7 model, including hardware specs, benchmark results, and optimal configuration using llama.cpp.
A detailed guide on running the Qwen3.6-35B-A3B APEX model on an RTX 3090, comparing two llama.cpp forks and quantization methods for optimal speed and quality.
AutoMegaKernel is an open-source agent harness that compiles any HuggingFace model into a single persistent megakernel, fusing the entire forward pass into one GPU launch to reduce overhead. It achieves up to 1.33x speedup over CUDA-graphed cuBLAS on inference-class GPUs like L4 and L40S, while proving schedules deadlock- and race-free.
ExecuTorch now has an MLX delegate that enables GPU-accelerated inference for PyTorch models on Apple Silicon Macs, supporting LLMs, speech-to-text, and MoE models with quantization via TorchAO.
A user reports an issue where the Qwen 3.6 model stops mid-task when served via vLLM with specific Docker and speculative decoding configurations.
A benchmark analysis of Qwen 3.6 27B MTP on 4x RTX 3090 GPUs, demonstrating that using NVLink for tensor parallelism yields significant throughput improvements (up to +53%) over PCIe configurations.
Anyscale is hosting a hands-on virtual lab session teaching developers how to build and scale data pipelines with Ray, covering video data curation, distributed GPU inference, and CPU/GPU streaming pipelines.
Quantized 27B Qwen3.6 model achieves 200 tok/s peak (136 avg) with 256k context and 10 agents on a single 49W GB10 GPU using Dflash+DDTree optimizations.
User reports successfully running a 35B-parameter mixture-of-experts model at 768K context length using Q4_K_M quantization and YaRN on an RTX 3090 via a llama.cpp fork, offloading only 8 experts to CPU while maintaining acceptable performance.