hubert.cpp, a C++ implementation of distilHuBERT [P]
Summary
A C++ implementation of distilHuBERT with no runtime dependencies, compiled-in weights, dynamic sizing, and on-par performance with ONNX Runtime, designed for easy integration into CMake projects.
Similar Articles
Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
Tiny-vLLM is a high-performance LLM inference engine implemented in C++ and CUDA, offering features like continuous batching and PagedAttention, and serves as an educational resource.
Designing the hf CLI as an agent-optimized way to work with the Hub
Hugging Face redesigned its `hf` CLI to be optimized for both human users and AI coding agents like Claude Code and Codex, with agent-aware output rendering and benchmarking showing up to 6× token savings versus no-CLI baselines on complex tasks.
Benchmark: ONNX Runtime vs HF Transformers vs GGUF for Parakeet TDT 0.6B on CPU-only hardware [D]
A benchmark comparing ONNX Runtime, HF Transformers, and GGUF for the Parakeet TDT 0.6B ASR model on CPU-only hardware shows ONNX Runtime achieves 37% faster inference than HF Transformers bfloat16, while GGUF prioritizes memory efficiency.
We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)
Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.
Announcing Humming: Fast, Flexible Kernels for Quantized Inference (19 minute read)
Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference that supports broad GPU coverage, diverse quantization formats, and outperforms existing implementations like Marlin, FlashInfer, and DeepGEMM.