llama.cpp b9966 for sm-tensor

Reddit r/LocalLLaMA Tools
llama-cpp performance caching regex decode optimization open-source

Summary

llama.cpp b9966 introduces a fix for the -sm tensor mode that caches regex patterns, eliminating 29 recompilations per tensor per token on the decode thread, resulting in significantly reduced CPU overhead.

B9966 If you run -sm tensor in production you might want to grab this fix which removes 29 regex recompilations per tensor per token on the decode thread. Claude tell me in one sentence what this means: The code was rebuilding 29 regex patterns from scratch on every tensor of every token instead of building them once and reusing them, so the fix just caches them — same behavior, way less wasted CPU on the decode thread
Original Article

Similar Articles

Llama.cpp : Split Mode Tensor Fix Incoming?

Reddit r/LocalLLaMA

Llama.cpp is expected to receive a fix for split mode tensor crashes on multi-GPU setups, which currently cause VRAM exhaustion every 90-120 minutes. The fix also reportedly brings a ~35% throughput improvement over layer mode.

ggml-org/llama.cpp

GitHub Trending (daily)

llama.cpp is an open-source C/C++ library for efficient LLM inference on local hardware, supporting various quantization methods and multiple backends (CPU, GPU, etc.).