chunking

This paper compares chunking and embedding strategies for RAG systems in Turkish, evaluating their impact on document question answering through a systematic study with multiple configurations.

An educational article from MachineLearningMastery covering seven chunking strategies for RAG pipelines, from fixed-size token chunking to advanced semantic and hierarchical approaches, with guidance on when to use each.

A prediction that Git can't be replaced, but content-defined chunking and Cloudflare Artifacts will improve large file support and commoditize repo hosting.

This article discusses five chunking strategies for RAG systems, highlighting the tradeoffs between retrieval precision and reasoning context, and emphasizing that proper chunking is crucial for effective retrieval.

Jerry Liu highlights the engineering challenges of productionizing agentic retrieval systems, emphasizing that success depends on careful tuning of chunking, synchronization, reranking, and tool API design rather than novel techniques.

A developer shares three common reasons AI agents fail in production: poor RAG chunking, demo-only prompts, and lack of fallback logic, emphasizing that model quality is rarely the main issue.

A developer recounts debugging RAG systems, discovering that fixed-size chunking breaks sentence boundaries, vector search fails for exact identifiers (solved with BM25), and stale indexes cause confident wrong answers.

Tested chunking and embeddings across three production websites for RAG retrieval, finding that a yield score (ratio of high/medium chunks) predicts corpus quality and reranking effectiveness.

A developer shares the failure modes encountered while debugging a RAG system, including issues with chunking, stale indices, and hybrid search, along with practical fixes like sliding window chunking and contextual retrieval.

The author argues that most agent RAG failures are due to retrieval problems—specifically chunking errors, lack of freshness signals, and reliance on pure vector search—rather than the LLM, and recommends structural chunking, decay-based ranking, and hybrid BM25+vector search.

This paper evaluates various chunking strategies for retrieval-augmented generation on German legal code, finding that structure-aligned methods like section-based retrieval outperform more complex approaches.

CHOP is a framework for improving RAG systems on multi-document retrieval by using context-aware metadata and LLM-based chunk relevance evaluation to reduce semantic conflicts and hallucinations. The approach achieves 90.77% Top-1 Hit Rate through intelligent chunking and contextual preservation strategies.

Introduces Adaptive Chunking, a framework using five intrinsic document metrics to select optimal chunking strategies for RAG, improving answer correctness from 62-64% to 72% and question resolution rate by over 30%.