Tag
A trimmed English-only GGUF version of Kimi K3 (IQ2-XXS) reduces model size from 711GB to 478GB by removing multi-language components, with early tests suggesting it may match or outperform the standard 2-bit version on coding tasks.
The author shares excitement for the upcoming Qwen 3.8 model, highlighting their experience with Qwen 3.6 27B for local LLM use, and discusses the potential of self-hosted AI to replace subscription-based frontier models.
A developer is building Octane, a speculative decoding runtime for local LLM inference on consumer hardware, aiming for 2-3x speedup with exact output quality. Currently in active development with paged KV cache, continuous batching, and batched attention implemented.
LabyrinthBench is a local, judge-free LLM benchmark that deterministically measures context recall under interference for multi-step agentic tasks. Initial experiments show that wiping context and re-injecting gate answers helped 7 of 9 models but backfired on 2, highlighting the complexity of context management strategies.
Unsloth AI announces DSpark, enabling DeepSeek-V4-Flash GGUF models to run ~1.4–2× faster locally, reaching 120 tokens/s with no accuracy change.
Olares is an open-source personal cloud OS from the beclab team, built on Kubernetes, specifically designed for 7x24 always-on AI Agents. It can convert old computers or NAS into a self-hosted AI platform, supporting local data storage and natural language operation.
User asks for performance numbers on Deepseek V4 Flash running via Colibri, focusing on high VRAM setups, long context prefill, and token generation speed for agentic workloads.
Introduces GraphARC, an MIT-licensed open-source tool that lets a model author agent graph topologies at runtime, with a deterministic admission gate for auditable execution, built on LangGraph and running locally via ollama or against cloud APIs.
Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.
A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.
User benchmarks DeepSeek v4 Flash against Qwen3.6-27B, Qwen3.5-122B, and Gemma 4 31B on a local coding benchmark, finding Flash wins overall but Qwen 122B performs surprisingly well with better first-try success and lower token usage.
The author had Claude Fable 5 play chess against a locally running DeepSeek V4 Flash. As a result, DeepSeek consumed tens of thousands of tokens per move and even timed out under extended thinking, yet it ended up with a better position. The game lasted 5 hours and was still unfinished.
A user reports that DeepSeek V4 Flash, running as a 2-bit quantized GGUF on dual RTX 3080s, is the first local model to score 100% on a real-world SQL benchmark, matching frontier models like Opus 4.7 and GPT-5.5.
Technical write-up on running DeepSeek-V4-Flash-0731 with full 1M context on a single RTX 5090 + 256GB DDR5 desktop using vLLM with CPU/RAM offloading, achieving ~800 tps prefill and ~15 tps decode.
Homebench is a zero-config terminal tool that benchmarks locally-run LLMs for speed, memory, and quality, presenting a live leaderboard. It supports Ollama, LM Studio, llama.cpp, vLLM, and OpenAI-compatible servers.
LM Studio users are concerned that the company is de-emphasizing its original local LLM app by redirecting attention and downloads to its new Bionic agent, while the main app receives few updates and is harder to find on the website.
A test by atomic.chat shows Qwen 3.8 Max outperforming Fable 5 at generating self-contained 3D physics scenes while costing about 7x less per run.
A reply celebrates Unsloth AI's upcoming Qwen3.8-27B model, which will run on 17GB RAM/VRAM setups, and notes TurboQuant+ is already integrated into many inference engines — great news for RTX 3090 users.
A developer enthusiastically recommends KAT Coder 2.5 dev, claiming it is faster, more accurate, and uses fewer tokens than Qwen 3.6 35b a3b, and outperforms Gemma 4 models on their setup, with a GitHub repo containing detailed benchmarks.
Alibaba announces Qwen3.8-27B open-weights release, capable of running locally on 17GB RAM/VRAM, alongside the larger Qwen3.8-Max.