Great RAG paper from IBM.

There are some really good ideas on how to solve common RAG issues.

It’s well known that retrievers chunk long documents by length, which discards the hierarchy the document already has.

So they propose using a table of contents.

A table of contents helps to encodes exactly the global structure that chunking throws away.

STAIR uses that table of contents as the addressing scheme for a generative retriever, so the model stores and retrieves information from its own parameters against a structure the corpus supplies.

On SearchTome, it reaches Recall@1 of 82.6 percent against 76.9 percent for a fine-tuned Differentiable Search Index, a statistically significant gap, with BM25 at 59.5 percent and DPR at 68.7 percent.

Hallucination stays below 0.05 percent, which is the standing objection to generative retrieval and the reason grounding the address space in a real hierarchy is worth the extra structure. The ablations also show it generalizes where very few training samples exist.

Paper: https://academy.dair.ai/papers/stair-structure-aware-information-retriever-a-novel-dataset-and-llm-based-retrie-2609.03874…

STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation

Source: https://academy.dair.ai/papers/stair-structure-aware-information-retriever-a-novel-dataset-and-llm-based-retrie-2609.03874 Retrieval · DataVineet Kumar, Meghanadh Pulivarthi, vishwajeet kumar, Jaydeep Sen, Riyaz Ahmad Bhat, Sachindra Joshi

Chat with Paper

First page

The curator’s take

Vineet Kumar and colleagues at IBM introduce STAIR, a generative retriever that stores and addresses a corpus through its table of contents rather than through length-based chunks.

Ask this paper

Question about this paper Key points01

Chunking discards global structure. Standard retrievers split long documents by length, which throws away the hierarchy a table of contents already encodes. STAIR keeps that hierarchy as the addressing scheme.

02

Recall@1 of 82.6% on SearchTome against 76.9% for a fine-tuned Differentiable Search Index, a statistically significant difference, and well above BM25 at 59.5% and DPR at 68.7%.

03

Hallucination below 0.05% in the generative IR setting, which is the main risk of generative retrieval and the reason ToC grounding is worth the added structure.

04

Generalizes with very few training samples, according to the ablations, which matters because generative retrievers usually need dense per-document supervision.

05

SearchTome is released: 18 books across 6 domains, built to evaluate structure-aware retrieval specifically.

AbstractRetrieval Augmented Generation (RAG) is a key component for generating accurate and hallucination free answers using Large Language Models (LLMs). LLMs are improving at handling long context, but still suffer from “lost in the middle” problem. Thus, precise and accurate retrieval is important. Current retrievers chunk long context into length-based manageable chunks - in the process throwing away rich and informative semantic global structure in the corpus. We introduce a novel retrieval system STAIR that empowers an LLM to exploit global structure in a corpus such as a Table of Contents (ToC) to efficiently store and retrieve information from its model parameters. Our thorough and careful ablation studies with a finetuned Differentiable Search Index (DSI) system show that ToC helps build a low hallucination (less than 0.05%) generative Information Retrieval (IR) system and can generalize to examples where very few training samples are available. To further research in this novel direction of ToC based retrieval we release SearchTome - a diverse benchmark created from 18 books across 6 diverse domains to further research in this novel direction. STAIR achieves a high Recall@1 score of 82.6% on SearchTome as compared to DSI (76.9%), where the difference is found to be statistically significant. STAIR easily beats other strong baselines such as BM25 (59.5%), DPR (68.7%) and out-of-the-box Mistral (13.8%).