@ZyphraAI: Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs ent…
Summary
Zyphra Research releases PUFFER, an incremental fuzzy deduplication system for LLM-scale datasets that runs entirely on CPU with 11-35x speedup over existing methods, open-sourced under Apache 2.0.
View Cached Full Text
Cached at: 09/02/26, 01:57 PM
Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets.
PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods.
We use PUFFER internally for our own training dataset and release it under Apache 2.0 license. https://t.co/I4XygVZkFi
Similar Articles
@Zai_org: https://x.com/Zai_org/status/2057216685040443743
This paper presents ZCube, a novel network architecture developed by Z.ai, Harnets.AI, and Tsinghua University to address topology-induced congestion in Prefill-Decode disaggregated LLM inference clusters. Production deployments on GLM-5.1 coding workloads achieved a 33% reduction in network CapEx, 15% throughput improvement, and 40.6% reduction in TTFT P99 latency.
Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
Proposes a scalable subdocument deduplication framework for LLM pretraining that separates duplicate detection from copy retention, using frequency- and length-aware policies. Experiments on FineWeb-Edu and a code web corpus show improved model performance.
HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks
HyperZip proposes an efficient LLM-based lossless data compression framework that uses diffusion-based LLMs with Multi-Token Prediction and a hypernetwork to adapt models to target data without fine-tuning, achieving a superior compression-rate/throughput trade-off over autoregressive baselines.
LFM3-1.7b seems impressive
The author tested Liquid AI's LFM3-1.7B model (Q8_0 GGUF via llama.cpp/ROCm) for compression and summarization, reporting strong prefill/decode throughput on Strix Halo and tensor-split 7900 XTX+XT setups, plus unexpectedly coherent performance at 32k-64k context lengths for a 1.7B model.
OpenZL
OpenZL is a compression library that generates specialized compressors for specific data formats, achieving high compression ratios at high speeds suitable for datacenter workloads like AI processing.