Tag
A developer got GPT-OSS 120B running locally on a 4070 Ti with 32GB RAM by exploiting its MoE architecture, streaming cold experts from NVMe and caching hot experts on GPU, reaching 21 tok/s with a top-1 approximation.
A local experiment comparing Qwen 35B-A3B MoE and Qwen 27B dense on coding-maintenance tasks, finding the MoE model ~3.9Γ faster with a smaller quality gap than expected.
Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.
Actual, a local inference stack with low CPU impact, now natively supports Hermes Agent by NousResearch, letting users leverage personal inference capacity from anywhere.
Ling-3.0-flash MXFP4, a quantized model, has been released and runs locally on a single DGX Spark, achieving ~80 tok/s decoding and 2,500-3,500 tok/s long-input prefilling, enabling private on-device inference for coding, agents, and offline batch jobs.
Qwen3-TTS voice cloning has been merged into mainline llama.cpp, enabling local text-to-speech with voice cloning from short reference audio via the llama-tts binary, supporting multiple languages. Limitations remain, including only the Base model and no server endpoint yet.
VibeVoice 1.5B runs locally on an iPhone with ~2.2 GB memory and up to 1.28Γ real-time speed; the author plans to release the xcframework and code for audio.cpp.
Liquid AI releases LFM2.5-2.6B, a compact agentic model designed to run entirely on-device, enabling free inference, low latency, and privacy. The post details its training pipeline including SFT, teacher specialization, distillation, and agentic RL.
Detailed explanation of how to run the MiniMax H3 video generation model locally on an RTX 4080 16GB using ComfyUI's native workflow, including model download, directory configuration, parameter tuning, and common pitfalls.
Hugging Face page for DeepSeek-V4-Flash-0731-GGUF, a GGUF quantized version of DeepSeek V4 Flash with an updated template supporting reasoning levels, ready for use with llama.cpp, vLLM, Ollama, and other local inference tools.
A user shares experiments running a 4-bit quantized DeepSeek v4 Flash on a 32GB M5 MacBook Air, achieving roughly 50 tokens/s prefill and 1 token/s decode using streamed experts and other tricks.
atomic.chat released 14 quantized GGUF builds of DeepSeek V4 Flash 0731, from lossless BF16 to 1-bit. They recommend AD-IQ2_M for 128GB hardware, which matches the original's token choice 83.6% of the time.
Announcement that GGUF quantizations of MiniMax H3 are available, with the Q2 version being only 8.49 GB for lower-end GPUs.
A user expresses excitement about MinMax-H3, highlighting that this video quality is now achievable 100% locally.
DeepSeek-V4-Flash-0731 is released in GGUF format at 2.88 bits per weight (95 GiB), making the 284B-parameter MoE runnable on 128 GB machines. Includes a detailed operating guide and a required TurboQuant llama.cpp fork.
This Hugging Face repository provides community-compiled quantized and pruned weights for MiniMax H3 (Hailuo 3.0), enabling local text/image/audio-to-video generation on consumer GPUs with 16-24GB VRAM. It includes INT4, INT8, and NVFP4 variants with hardware-specific guides.
Daniel Han of Unsloth validates that Qwen3.8-27B will run in only 17GB VRAM, making it accessible for local inference.
Qwen announced Qwen3.8, including a new 27B model, generating excitement for local deployment.
A GitHub patch allows running DeepSeek V4 Flash in LM Studio on 128GB Macs by sideloading antirez's llama.cpp fork, working around struct-layout drift, decoding splits, and code-signing issues.
The author shares their storage server build optimized for local AI inference, anticipating a rumored 26T-a3b model called "Le Chaton FAT" and using high-capacity NVMe drives with ZFS for model storage.