Tag
Agnos is a new tool that cleans documents for AI use and provides transparency by showing exactly what was removed during preprocessing.
Zyphra Research releases PUFFER, an incremental fuzzy deduplication system for LLM-scale datasets that runs entirely on CPU with 11-35x speedup over existing methods, open-sourced under Apache 2.0.
Microsoft has launched the MarkItDown tool, which converts various document formats (such as PDF, Word, Excel) into Markdown, making it easier for large models and RAG systems to process enterprise data.
This paper presents SSAS (Syntactic & Semantic Context Assessment Summarization), a framework designed to improve consistency in LLM-based sentiment prediction by reducing noise and variance through hierarchical classification and iterative summarization. Empirical evaluation on three industry-standard datasets shows up to 30% improvement in data quality and reliability for enterprise decision-making.