A prototype extension for llama-server enabling custom sampling logic, with an example loop detector, without maintaining a separate fork.
Just an idea and a prototype (made by Qwen3.6-27B-UD-Q6\_K\_XL via OpenCode) for allowing users to add custom sampling logic to llama-server without having to maintain their own entire fork and without having to make a wrapper that reimplements everything llama-server can do. Included is an example extension that detects and breaks one kind of loop that I've commonly seen with heavily quantized models, where they get stuck repeating the same 1-3 tokens. Other ideas for sampling that aren't in llama.cpp include different sampling parameters during thinking, tool calling, and normal generation; toggling grammars based on context; non-GBNF grammars; guaranteeing that only real tables are referenced in a generated SQL query; redacting PII in the sampler itself; and other experimental general sampling approaches. This was based on the latest master branch after MTP was merged; also works with speculative decoding. Posted for votes here: [https://github.com/ggml-org/llama.cpp/discussions/23028](https://github.com/ggml-org/llama.cpp/discussions/23028) Branch: [https://github.com/dpmm99/llama.cpp/tree/master-with-sampling-extensions](https://github.com/dpmm99/llama.cpp/tree/master-with-sampling-extensions) The example sampler extension is one fairly short file: [https://github.com/dpmm99/llama.cpp/blob/master-with-sampling-extensions/examples/sampling-ext/loop-detector.cpp](https://github.com/dpmm99/llama.cpp/blob/master-with-sampling-extensions/examples/sampling-ext/loop-detector.cpp) Vulkan Windows x64 release copy for convenience if you want to try it: [https://github.com/dpmm99/llama.cpp/releases/tag/dpmm99-0.1](https://github.com/dpmm99/llama.cpp/releases/tag/dpmm99-0.1) but here's your daily reminder not to trust random executables from the internet. ;) Example command: llama-server -np 1 -c 32768 --temp 0.1 -m Qwen3.6-27B-UD-Q6_K_XL-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 3 --sampling-ext-path sampling-ext-loop-detector.dll [the extension working in llama-server with Qwen3.6-27B using MTP](https://preview.redd.it/1pwpo5p9mi1h1.png?width=773&format=png&auto=webp&s=e9d8bda72bbc127f0b9cc5dcbaa4a73e62096b36)
This article introduces an experimental sampler for llama.cpp called 'scatter' that applies a local diffusion over rank-sorted tokens to smooth the probability distribution among top candidates, enabling more creative text generation without increasing tail probabilities.
SonicSampler presents a unified suite of tile-aware Triton kernels that vertically fuse the entire LLM sampling pipeline, supporting dynamic per-request behaviors and speculative verification, achieving up to 16x speedup over state-of-the-art baselines.
The article presents the official home for llama.cpp, an open-source local LLM inference engine, highlighting integration with the Pi coding agent via the pi-llama plugin and broad hardware optimization.
CachyLLama is a fork of llama.cpp that adds a persistent SSD-backed KV cache and multi-tier caching, dramatically reducing prompt reprocessing time for long local-agent sessions on slower hardware.
The author built a custom llama.cpp server and Mikupad UI to enable local inference and activation steering with Anthropic's open-weight Natural Language Autoencoders. A LoRA version is in development to reduce memory requirements.