Tag
An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.
MawForge presents a memory-bounded approach for serving large Mixture-of-Experts (MoE) language models on constrained unified-memory machines by storing the full model on disk and materializing expert tensors into a bounded cache on demand. Experiments on a MacBook Pro M5 Pro demonstrate effective serving of 34GB and 25GB models within a 24GB memory envelope, with analysis of cache size trade-offs and speculative decoding results.
A new model enables generating 3D models from a single image locally on Apple Silicon devices and iPhones, using less than 2GB RAM and completing in under 20 seconds.
Successfully ran the 75B Nemotron Puzzle model locally on a 64GB M2 Max Mac, demonstrating large model inference on consumer hardware.
An AI engineer released an open-source project teaching how to build a local RAG system from scratch and a production-grade agentic architecture with LangGraph, hybrid retrieval, caching, and observability.
Demonstrates running two 80B Qwen models simultaneously on a MacBook Pro using BBQ-FP4 quantization, claiming functionally lossless performance and speed.
This post details running GLM 5.2 on a 4xGB10 setup with a 100G switch, achieving ~25 tok/s decode and ~650 tok/s prefill at 330k context. It includes hardware costs, performance benchmarks with Depth Prefill, and notes on model pruning for longer context.
Tencent Hy3 is a new open-source frontier-level AI model with 295 billion parameters (21 billion active) that tops benchmarks and can run locally on consumer hardware. It is available for free in Cline.
The author built Kivarro, an all-in-one local inference workbench, and is seeking feedback from users who run models locally.
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
Poolside releases Laguna XS 2.1, a 33B parameter Mixture-of-Experts model with 3B activated parameters per token, designed for agentic coding, with improvements on SWE-bench Multilingual and other benchmarks, now available under the permissive OpenMDW-1.1 license.
Describes a patch for llama.cpp that adds CUDA support for DeepSeek V4 Flash's context indexing, enabling full 1M token context on an RTX 5090 with significantly reduced VRAM usage and high throughput.
Another open-source tool on GitHub, Shimmy, is a single 5MB file written in Rust that provides fast and stable local inference with a full OpenAI-compatible API, targeting Ollama's pain points. It starts in under 100ms and uses about 50MB of memory.
Discussing the risks of relying on a single US frontier AI model and advocating for a diversified stack using Chinese alternatives like DeepSeek and Qwen, which can be run locally and are cheaper, for better resilience and data privacy.
VibeVoice 1.5B, a long-form multi-speaker TTS model, is now supported in audio.cpp, a native C++/ggml runtime, achieving 4.08x real-time speed on RTX 5090, 2.86x faster than Python baseline without quantization.
Cohere Labs releases North Mini Code, a 30B parameter (3B active) open-source coding model under Apache 2.0, optimized for code generation and agentic tasks, capable of running locally with 20GB RAM via 4-bit quantization.
Announces Ornith-1.0-35B, a coding fine-tune of Qwen3.6-35B-A3B that slightly outperforms the base model on aider benchmarks. Also promotes the club-3090 repository for running LLMs on RTX 3090s.
Release of Inflect-Nano-v2, a fixed-voice English TTS model with under 4M parameters for local text-to-waveform synthesis, supporting CPU or CUDA inference and long-text handling.
This article evaluates the performance of running GLM 5.2 (IQ2M quantized version) on Dual Strix Halo (256GB VRAM). The generation speed is only about 7 tokens/s, and coding tasks take twice as long as DeepSeek V4 Flash. Its cost-performance ratio is far inferior to other models, so it is not recommended for use with this hardware configuration.
Stable Diffusion XL (SDXL) can now run locally in the browser using WebGPU, enabling high-quality AI image generation directly on-device with open-source code.