Tag
GVD introduces a unified framework for governed versioning and deduplication in document repositories, using bidirectional rule alignment and conflict resolution policies with local encoder models to achieve high performance on enterprise data.
The article describes testing to reduce reading cost in an AI verification pipeline while preserving high recall and minority evidence, noting challenges with current approaches and inviting community input.
Zyphra Research releases PUFFER, an incremental fuzzy deduplication system for LLM-scale datasets that runs entirely on CPU with 11-35x speedup over existing methods, open-sourced under Apache 2.0.
A truck driver with no coding background built an AI news aggregator using Next.js and Supabase that summarizes and deduplicates news from multiple sources.
The author describes building a memory layer for news-reading AI agents to handle deduplication and improve continuity across sessions, using SQLite and vector storage with a Python API, CLI, and MCP server.
Proposes a scalable subdocument deduplication framework for LLM pretraining that separates duplicate detection from copy retention, using frequency- and length-aware policies. Experiments on FineWeb-Edu and a code web corpus show improved model performance.
Engineering notes on optimizing frame selection for feeding video to LLMs, covering scene detection, deduplication strategies, and token budget management.
This paper presents a production extraction layer that converts heterogeneous documents into an ontology-aligned knowledge graph using a locally hosted tuned Qwen LLM, with ontology-guided prompts, multi-stage deduplication, and embedding-based resolution. Evaluation on intelligence corpora improved search recall from about 70 to 95 percent with no false merges.
A deliberately empty 16.5-trillion-parameter model uploaded to Hugging Face exposes that parameter counts are computed from safetensors headers alone, and that Xet's content-defined deduplication reduces upload bandwidth enormously while storage quota still bills the full logical size.
Hugging Face introduces Storage Buckets, a scalable object storage service for AI teams with per-TB pricing, Xet deduplication, built-in CDN, and no git overhead, designed for datasets, model checkpoints, and ML artifacts.
A new study quantifies the damage caused by repeated data in language model pretraining, showing that even aggressive deduplication leaves harmful repetition that can waste up to a third of compute FLOPs.
This paper systematically studies the damage caused by exact document repetition during language model pretraining, showing that repeating a moderately sized subset a moderate number of times maximally harms performance, and that repetition can waste up to 33% of compute (as measured by compute-equivalent loss).
A detailed guide on optimizing knowledge graph ingestion for AI agents, presenting a five-step pipeline (extraction, resolution, embedding, deduplication, routing) to prevent graph corruption and improve retrieval quality.
This thread discusses best practices for building unified memory layers with knowledge graphs, emphasizing the separation of entity resolution (naming) from deduplication (identity) to avoid graph corruption. It also highlights using orchestration tools like PrefectIO to manage expensive LLM extraction pipelines with checkpointing and caching.
Hugging Face announces Storage Buckets, a storage solution for large, evolving training datasets with built-in CDN and deduplication, recommended by CommonCrawl.
Velonus is an open-source AppSec scanner for Python that runs five security tools in one command, normalizes findings, and deduplicates noise, with support for SARIF output and CI integration.