ggml

A llama.cpp/GGML PR (26291) speeds up RPC-based 300GB model loads by ~300%, cutting load time from ~5 minutes to ~1.5 minutes on consumer hardware with the GGML_RPC_LOAD_THREADS setting.

LocalAI explains why it writes its own C/C++ inference backends, showing that its vllm.cpp port achieves comparable or better throughput and far smaller footprint than vLLM, with benchmarks across multiple models and hardware.

Release 0.4 of audio.cpp adds C++/GGML inference for Higgs Audio v3 TTS 4B (10x real-time) and Fish Audio S2 Pro, with full GGUF loading and Q8 speed/VRAM gains.

Release of C++/GGML based implementations of Supertonic 3, MOSS-TTS, IndexTTS2, and Irodori-TTS in audio.cpp, capable of generating 10 hours of audio in 3 minutes on an RTX 5090.

Celebrating the 10000th release of llama.cpp, a tool for running LLMs locally.

audio.cpp update introduces streaming support and four ASR/STT models (Nemotron 3.5, Higgs Audio STT, VibeVoice ASR, Hviske ASR) in native C++/GGML, achieving up to 2.41x faster than Python with competitive WER and VRAM usage.

audio.cpp releases a major update adding music/SFX generation and source separation with ACE-Step, HeartMuLa, Stable Audio 3, and HTDemucs, achieving up to 10x real-time speed for long music generation in native C++/GGML.

VibeVoice 1.5B, a long-form multi-speaker TTS model, is now supported in audio.cpp, a native C++/ggml runtime, achieving 4.08x real-time speed on RTX 5090, 2.86x faster than Python baseline without quantization.

The developer improved qwen3-tts.cpp to run 5x realtime on RTX 5080 and created a cross-platform desktop GUI with Kotlin Compose Multiplatform, featuring voice cloning, streaming, and speaker embedding management.

Introduces performance improvements by reducing synchronizations during split compute in the ggml backend, adds async CUDA copy capability, and makes sync relaxation more general across backends.

audio.cpp is a C++/ggml runtime that integrates 12 audio models including Qwen3-TTS, PocketTTS, and VeVo2, achieving TTS up to 5x faster than Python on CUDA.

Testing shows that default pipeline parallelism in llama.cpp wastes VRAM with no speed benefit; compiling with GGML_SCHED_MAX_COPIES=1 saves significant VRAM while maintaining identical inference speed.

parakeet.cpp is a fast, dependency-light C++17 inference pipeline for NVIDIA's NeMo Parakeet speech recognition models, built on ggml. It achieves byte-identical transcripts to NeMo with significant speedups on CPU and GPU.

NVIDIA's Parakeet speech-to-text models have been ported to pure C++/ggml, achieving byte-identical output to NeMo, up to 5x faster inference on GPU, and quantized GGUF variants for efficient deployment anywhere without Python or PyTorch.

The article highlights the new WebGPU backend in llama.cpp/ggml, enabling GPU-accelerated local AI model inference in browsers, developed by Reese Levine and team at USCS over the past year and a half.

llama.cpp adds Multi-Token Prediction (MTP) support for the Qwen3.6 family, delivering massive performance improvements for local AI inference on commodity hardware.

OpenMOSS is a standalone C++/GGML port of MOSS-TTS that provides a self-contained binary for text-to-speech and voice cloning, leveraging a Qwen3-8B backbone and 32 RVQ audio codebooks. It includes a CLI for one-shot synthesis and an HTTP server for repeated generations.

PFlash now supports running @poolsideai's Laguna-XS.2 (33B-A3B MoE) on a single RTX 3090, achieving 111 tok/s decode and 5.4x faster prefill than llama.cpp, with NIAH passes up to 131K context.

Pull request adds optimized x86 and generic CPU q1_0 dot-product kernels to ggml-cpu, improving quantized LLM inference speed.

GGML and llama.cpp have joined Hugging Face to ensure long-term sustainability of local AI development. Georgi Gerganov's team will maintain full autonomy over the projects while receiving resources to scale community support and improve integration between llama.cpp inference and transformers model definitions.