Tag
The user switched from LM Studio to vLLM for serving Qwen 3.8 models, achieving 143 tokens per second and reduced GPU temperatures, referencing a helpful GitHub repository.
FreeToken is an open-source inference framework developed by researchers from UC Berkeley, MIT, and UT Austin, specifically designed for running ultra-large MoE models on personal computers, achieving efficient inference by dynamically allocating GPU, CPU, and memory resources.
A developer achieved up to 381 tok/s inference speed on a single RTX 3090 with the Qwen3.8-27B model using optimized techniques like DFlash2 and prefix caching, particularly effective for document-based tasks like RAG and coding assistants.
The article presents a methodology for optimizing the serving of DeepSeek-V4-Pro, a 1.6-trillion-parameter MoE model, on H20 GPUs, achieving significant performance improvements through scenario-specific configurations and optimizations.
PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.
KernelArc is a multi-agent framework that uses strategy-specialized agents to autonomously optimize GPU kernels across heterogeneous workloads, achieving top rankings on NVIDIA GPU benchmarks.
A developer achieved real-time monocular depth estimation by optimizing a 448x448 model to run at ~8 ms on an M4 Pro using TypeGPU, allowing seamless GPU integration for depth-aware lighting.
This paper proposes Agentic ESOpt, a method using evolution strategies to enable scalable full-parameter fine-tuning of long-horizon LLM agents with minimal GPU memory requirements.
The article introduces Gonka, a decentralized inference network that enables access to the V4-Flash AI model with full 1M context without requiring local GPU ownership, using an OpenAI-compatible interface.
A user shares positive experiences using the Qwen 3.8 27b model with DeepSeek Harness, praising its stability and long-context handling, but mentions speed limitations and hopes for future model releases.
A blog post detailing how the author used Codex to optimize a kernel in a GPU Mode contest, achieving a 232x speedup in QR decomposition and sharing learnings on auto-research.
French startup Kog is using software optimization to dramatically speed up LLM inference on standard datacenter GPUs, claiming up to 30x faster throughput. After a demo of 3,000 tokens per second with its open-sourced Laneformer 2B model, the company is now focusing on larger models and targeting software engineering workflows.
TTT-Discover is a framework that trains LLMs at test time using reinforcement learning on individual problems, setting new records in GPU kernel engineering and improving mathematical bounds.
This paper studies GPU control gates for LLM-agent services, analyzing concurrent cohort scheduling and on-device routing versus host redispatch to reduce host round trips and improve GPU utilization.
Introduces v100-skinny, a custom kernel library enabling fast NVFP4 inference on V100 (sm70) GPUs, achieving 366 t/s for Qwen3.6 27B in best-case extraction, with lower speeds for structured generation and code.
NVIDIA and PyTorch explain how kernel fusion in CUDA improves GPU memory bandwidth by combining multiple operations into a single kernel, reducing round-trips through global memory and kernel launch overhead.
Meta's FBTriton is a downstream fork of OpenAI's Triton compiler that enables rapid development of GPU optimizations like TLX and autoWS while staying synced upstream. This blog details its continuous upstream ingestion strategy, hierarchical L1/L2/L3 validation framework, and the practical challenges of balancing innovation with production stability.
Nifer is a tool that achieves 700 tokens per second inference on Qwen 3.6 35B without thinking, specifically optimized for the RTX 5090, with support for full 250k context.
A user shares their experience pushing Qwen 3.6 27B to 262K context with coherent results, and discusses using Rope/Yarn scaling to go higher, along with kv-cache swapping strategies for RTX 3090 Ti.
LongStraw is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, demonstrating execution capacity up to 2.1M positions on Qwen and GLM models.