We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.

Reddit r/LocalLLaMA Models
long-context kv-cache inference-engine open-source local-llm gpu-optimization

Summary

Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎（MIT 开源），通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache，使 262K 长上下文能在 16 GB GPU 预算内运行，并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。

Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, and it grows with every token, so sooner or later you either cut the context or move to the cloud. We wanted long sessions to stay on your own GPU. So we built Volundr so that 54 of its 72 layers keep a small, fixed-size state instead of a growing cache. Why we wrote our own engine Existing servers aren't really built for that kind of model, so we wrote Agens Engine (MIT) around it. Two things it does that we're proud of: It remembers where a conversation left off, so a follow-up doesn't re-read the whole history. It stays inside a GPU memory budget you choose. Long context spills to system RAM instead of growing VRAM. The video One uncut take, real time. A 110K-token novel takes ~69 s to read cold. The follow-up question is answered in 0.5 s, because the engine just picks up where it left off. GPU memory peaks around 15.6 GiB. To be upfront: we recorded it on a 48 GB card with the engine held to a 16 GiB plan. We don't own a real 16 GB card yet. The best real-world numbers so far came from you. mmssix on HF tried it on a 24 GB card we'd never tested and got 86.4–106.7 tok/s. That's in the README now. Thank you! What's next We read every comment last time, and you were right about a lot: MoE: more of you could run it with an MoE variant. We're exploring one for a future version. Engram: the memory tables should be bigger. That's where we're heading. Image generation on the same GPU: because the LLM's memory stays flat, a 24 GB card has room left over. We'd like to run an image-generation model right next to it, with no model switching. It's a goal; we haven't built or tested it yet. Windows and Mac: Windows (starting with WSL2) is on our roadmap, and we're looking at Apple Silicon. A small box: we're also working on a compact desktop that runs all of this out of the box. We're a small team and there's still a lot to improve. If you give it a try, especially on a real 16 GB card, please tell us honestly how it goes. Every bit of feedback goes into training our next version. What we're hoping for is simple: a world where tokens are free, running on hardware you own. Engine: https://github.com/BlockWayz/agens-engine Weights: https://huggingface.co/collections/Blockway/agens-volundr-32b-v1
Original Article

Similar Articles

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

Reddit r/LocalLLaMA

Blockway released Agens Volundr 32B Preview, an Apache-2.0 32B dense model on a hybrid architecture where only 18 of 72 layers keep a KV cache, aiming for efficient long-context inference on consumer/local GPUs. The team reports competitive benchmarks versus Qwen3.8-27B on coding and math but weaker agentic performance, with a custom sglang build required for now.

I built Ninfer 4080 for 16GB class GPUs

Reddit r/LocalLLaMA

A community developer released Ninfer 4080, an optimized local inference runtime that runs the Qwen 3.8 27B GSQ model at 100k context on a 16GB RTX 4080, achieving up to ~2720 tok/s prefill and 262 tok/s decode via DFlash2 speculative decoding — significantly faster than general-purpose engines like llama.cpp or vLLM.

@servasyy_ai: https://x.com/servasyy_ai/status/2106563331976991217

X AI KOLs Timeline

This long post is a real-world test of running Qwen3.8-27B on two second-hand V100 32G GPUs. By rewriting the kernel of the inference engine NInfer, adding dual-GPU support and speculative decoding, generation speed on a 200K-token code task jumped from 36 tok/s to roughly 100 tok/s, beating the 80.9 tok/s of a single 4090 48G. The post also analyzes how VRAM bandwidth, quantization, and NVLink affect inference performance.