Let RAG Grow a Brain: No More Forced Answers for Composite Questions

By Article 5, your RAG is ready for production — accurate retrieval, evaluation, updates, and no data leaks. But it still has a fatal flaw: it only retrieves once.

When a user asks, “What was our Q3 gross margin?” one retrieval is enough. But if they ask, “Compare our Q3 and Q2 gross margins: which changed more, and what were the main reasons?” — one retrieval won’t cut it. This question contains three sub-questions: Q3 gross margin, Q2 gross margin, and reasons for the change. You need to query three times, calculate the difference once, and then synthesize everything.

Ando@ando_w·Jul 9 Article Private Knowledge Base Part 5: Permission Isolation in Enterprise RAG — Control It from the Retrieval Stage

I have a friend in Hangzhou who built an enterprise knowledge base. In the second week it went live, something went wrong. The boss dropped a compensation adjustment plan into the knowledge base, thinking executives could easily look it up. Then a regular employee asked, “What’s the salary increase rate this year?” RAG… 24256.1K

When a single‑turn RAG encounters this kind of problem, it either fabricates an answer (hallucination) or stuffs all three sub‑questions into one query (retrieves nothing useful).

This article explains how to give RAG a brain — upgrading from “single‑turn Q&A” to “multi‑step reasoning”, also known as Agentic RAG.

1. Where Single‑Turn Retrieval Gets Stuck

Let’s be clear. The flow of single‑turn RAG is: user asks → retrieve once → feed results to LLM → LLM generates answer. One shot.

This flow works fine for factual queries because they have only one sub‑question. But in real business, users often ask composite questions:

• “Compare the pros and cons of A and B” — need to retrieve A, retrieve B, and compare them.

• “Why did sales drop in the last three months?” — need data for three months, market conditions in the same period, and internal actions.

• “Are there any clauses unfavourable to us in this contract?” — need to retrieve the whole contract in segments and evaluate each segment.

These questions share a common feature: one retrieval cannot get all the information, and even if it does, the LLM can’t digest it all. If you feed 50 chunks to the LLM, it will get lost in the noise and produce a vague, safe answer.

There’s another scenario where single‑turn RAG completely fails: external tools are needed. The user asks, “Has the order placed last week by this customer arrived?” Retrieval can only find contract information, not logistics status — the logistics data lives in another system, requiring an API call.

So Agentic RAG needs to solve two things: (1) let the LLM decide how many times to retrieve and what to retrieve; (2) let the LLM call tools, not just retrieval.

2. The Core Idea of Agentic RAG

In one sentence: promote the LLM from “answer machine” to “commander”.

In single‑turn RAG, the LLM is passive — it answers whatever is fed to it. In Agentic RAG, the LLM actively orchestrates: after receiving the user’s question, it thinks “how many retrievals do I need, what should I retrieve first, should I call other tools, do I have enough information?” and then executes step by step until it can answer.

This process is called the “reasoning‑acting loop”, academically known as ReAct (Reasoning + Acting). The idea is simple:

• Think: LLM analyzes what information it already has and what’s missing.

• Act: If something is missing, go retrieve it, or call a tool.

• Observe: See what the action returned.

• Re‑think: If enough information, answer; otherwise, go back to step 1.

Loop until the LLM decides “I can answer” or reaches a step limit.

Sounds mysterious, but it’s just a while loop. The difficulty isn’t in the algorithm — it’s in engineering: making each step controllable, observable, and interruptible.

3. Step One: Encapsulate Retrieval as a Tool

To let the LLM call retrieval, you must first turn retrieval into a “tool”. A tool is essentially a manual: it tells the LLM what the tool does, how to pass parameters, and what it returns.

Using Qwen3.7-Max’s function calling, define two tools: one to retrieve the knowledge base, and one to do calculations (difference, percentage, etc.).

import json
from openai import OpenAI

client = OpenAI(
    api_key="sk-你的key",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

MODEL = "qwen-max-latest"

# Tool definition: tell the LLM which tools are available
tools = [
    {
        "type": "function",
        "function": {
            "name": "search_knowledge_base",
            "description": "Retrieve information from the company’s private knowledge base. Use this when you need to query internal documents, contracts, reports, manuals, etc. Pass a specific query question; returns relevant document snippets.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The specific question to retrieve. Be specific, e.g., 'Q3 gross margin' rather than 'financial data'"
                    }
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "calculate",
            "description": "Perform mathematical calculations. Use this when you need to compute differences, percentages, sums, etc. Pass a math expression.",
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "Mathematical expression, e.g., '45.2 - 42.1' or '(45.2-42.1)/42.1*100'"
                    }
                },
                "required": ["expression"]
            }
        }
    }
]

One detail many overlook: the tool description must be written so the LLM can understand when to use it. Don’t write “database retrieval” — write “use when querying internal company documents”, “pass a specific question, not a broad term”. The LLM decides whether to call a tool based on the description. If the description is vague, the calls will be chaotic.

4. Step Two: Implement the Tool Execution Functions

The tool definitions are just the manual; you still need to write the actual execution. Plug in the retrieval pipeline from the previous five articles — vector retrieval + BM25 + RRF fusion + re‑ranking + permission filtering — all intact.

def search_knowledge_base(query: str, user_id: str = "u1") -> str:
    """Retrieve knowledge base, reusing the complete pipeline from the previous 5 articles"""
    # 1. Vector retrieval (with permission JOIN)
    vec_sql = """
        SELECT c.id, c.content, c.doc_id,
               1 - (c.embedding <=> %s::vector) AS vec_score
        FROM chunks c
        JOIN user_doc_acl a ON a.doc_id = c.doc_id AND a.user_id = %s
        WHERE 1 - (c.embedding <=> %s::vector) > 0.3
        ORDER BY c.embedding <=> %s::vector
        LIMIT 20
    """
    emb = get_embedding(query)  # text-embedding-v3, 1024‑dim
    cur.execute(vec_sql, (emb, user_id, emb, emb))
    vec_results = {r[0]: {"content": r[1], "vec_score": r[3]} for r in cur.fetchall()}

    # 2. BM25 retrieval (with permission JOIN)
    bm25_sql = """
        SELECT c.id, c.content,
               ts_rank_cd(c.tsv, plainto_tsquery('chinese', %s)) AS bm25_score
        FROM chunks c
        JOIN user_doc_acl a ON a.doc_id = c.doc_id AND a.user_id = %s
        WHERE c.tsv @@ plainto_tsquery('chinese', %s)
        ORDER BY bm25_score DESC
        LIMIT 20
    """
    cur.execute(bm25_sql, (query, user_id, query))
    bm25_results = {r[0]: {"content": r[1], "bm25_score": r[2]} for r in cur.fetchall()}

    # 3. RRF fusion
    all_ids = set(vec_results.keys()) | set(bm25_results.keys())
    rrf = {}
    for rank, cid in enumerate(sorted(vec_results.keys(), key=lambda x: -vec_results[x]["vec_score"])):
        rrf[cid] = rrf.get(cid, 0) + 1 / (60 + rank)
    for rank, cid in enumerate(sorted(bm25_results.keys(), key=lambda x: -bm25_results[x]["bm25_score"])):
        rrf[cid] = rrf.get(cid, 0) + 1 / (60 + rank)

    # 4. Take Top5 for re‑ranking
    top_ids = sorted(rrf.keys(), key=lambda x: -rrf[x])[:5]
    contents = [vec_results.get(cid, bm25_results.get(cid))["content"] for cid in top_ids]
    reranked = rerank(query, contents)  # re‑rank model from Article 3

    # 5. Return Top3
    result = []
    for idx in sorted(range(len(reranked)), key=lambda i: -reranked[i])[:3]:
        result.append(contents[idx])
    return "\n---\n".join(result) if result else "No relevant content found"

def calculate(expression: str) -> str:
    """Mathematical calculation"""
    try:
        # Only allow digits and operators to prevent injection
        safe = "".join(c for c in expression if c in "0123456789.+-*/() ")
        result = eval(safe, {"__builtins__": {}}, {})
        return f"{expression} = {result}"
    except Exception as e:
        return f"Calculation failed: {e}"

Note that the permission JOIN is still in search_knowledge_base — the permission model from Article 5 must not be dropped. Agentic RAG merely wraps single‑turn retrieval; the underlying permission isolation, hybrid retrieval, and re‑ranking remain unchanged. Many people forget permissions when implementing Agentic RAG: the LLM calls the tool with the user’s identity, but the tool doesn’t filter internally, rendering permissions useless.

5. Step Three: Multi‑Step Reasoning Loop

This is the heart of Agentic RAG. A while loop: LLM thinks → calls tools → observes results → thinks again, until it can answer or the steps are exhausted.

def agentic_rag(question: str, user_id: str = "u1", max_steps: int = 5) -> str:
    """Agentic RAG: multi‑step reasoning loop"""
    messages = [
        {
            "role": "system",
            "content": (
                "You are an enterprise knowledge base assistant. When answering user questions, "
                "if information is insufficient, you may call search_knowledge_base to retrieve the knowledge base, "
                "or call calculate to perform calculations. After each retrieval, decide if you have enough information; "
                "if so, answer; otherwise, continue retrieving. Maximum of 5 retrievals to avoid infinite loops."
            )
        },
        {"role": "user", "content": question}
    ]

    for step in range(max_steps):
        # LLM thinks: decide to call a tool or answer directly
        resp = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=tools,
            tool_choice="auto"  # auto lets the LLM decide
        )
        msg = resp.choices[0].message
        messages.append(msg)

        # If the LLM decides to call a tool
        if msg.tool_calls:
            for call in msg.tool_calls:
                name = call.function.name
                args = json.loads(call.function.arguments)

                # Execute the tool
                if name == "search_knowledge_base":
                    result = search_knowledge_base(args["query"], user_id)
                elif name == "calculate":
                    result = calculate(args["expression"])
                else:
                    result = f"Unknown tool: {name}"

                print(f"  [Step {step+1}] Call {name}({args}) → {result[:80]}...")

                # Put tool result back into the conversation
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": result
                })
        else:
            # LLM didn't call a tool — it has enough information, answer directly
            print(f"  [Step {step+1}] Direct answer")
            return msg.content

    # Steps exhausted, force the LLM to answer based on existing information
    messages.append({
        "role": "user",
        "content": "Maximum retrieval attempts reached. Please provide the best possible answer based on the information you have."
    })
    resp = client.chat.completions.create(model=MODEL, messages=messages)
    return resp.choices[0].message.content

Let’s run the gross margin comparison question and see how the LLM orchestrates itself:

answer = agentic_rag("Compare our company's Q3 and Q2 gross margins: which changed more, and what were the main reasons?")

The console will print an execution trace like this:

[Step 1] Call search_knowledge_base({'query': 'Q3 gross margin'}) → 2024 Q3 gross margin 45.2%, main...
[Step 2] Call search_knowledge_base({'query': 'Q2 gross margin'}) → 2024 Q2 gross margin 42.1%, main...
[Step 3] Call calculate({'expression': '45.2 - 42.1'}) → 45.2 - 42.1 = 3.1
[Step 4] Call search_knowledge_base({'query': 'reasons for Q3 gross margin increase'}) → Q3 gross margin increase mainly due to raw material...
[Step 5] Direct answer

The LLM decomposed the problem into four sub‑questions, retrieved three times, calculated once, and then synthesized the answer. You didn’t need to tell it beforehand “this is a comparison question” — it figured it out. This is the fundamental difference between Agentic RAG and single‑turn RAG: control is handed over to the LLM.

6. Preventing Runaway: Don’t Let the Agent Go Wild

Passing control to the LLM also introduces risks: it could get stuck in an infinite loop, retrieving again and again, never satisfied with its answer. Or worse, each retrieval returns nothing, but it still stubbornly tries ten more times, burning tokens, only to tell you “not found.”

Three valves must be installed:

First: maximum step limit. That’s max_steps=5 in the code above. Five steps are enough for most questions; anything more complex should be broken into multiple conversations. Don’t set this number too high — setting it to 20 could burn through half a month’s token budget in one query.

Second: deduplication. LLMs have a habit of repeatedly retrieving the same query. You can add a cache in the tool: if the same query has been retrieved before, return it directly — no need to hit the database again. This saves cost and prevents loops.

_search_cache = {}

def search_knowledge_base(query: str, user_id: str = "u1") -> str:
    cache_key = (query, user_id)
    if cache_key in _search_cache:
        return _search_cache[cache_key]  # cache hit, return directly
    # ... normal retrieval ...
    _search_cache[cache_key] = result
    return result

Remember to include user_id in the cache key (the pitfall from Article 5 — permissions must not leak through caching).

Third: cost monitoring. Record the tokens consumed in each loop and force termination when a threshold is exceeded. In an enterprise environment, without this, the month‑end bill will make you question your life choices.

total_tokens = 0
COST_LIMIT = 8000  # maximum 8000 tokens per Q&A

for step in range(max_steps):
    resp = client.chat.completions.create(...)
    total_tokens += resp.usage.total_tokens
    if total_tokens > COST_LIMIT:
        break  // over budget, force termination

7. When to Use Agentic RAG and When Not To

This is the most overlooked question. Agentic RAG is not a silver bullet; it comes at a cost: slower (answers may take tens of seconds), more expensive (multiple LLM calls), and less controllable (the LLM may orchestrate poorly).

Scenarios where you SHOULD use it: highly composite questions, requiring multi‑hop reasoning, or needing external tools. For example, “comparison”, “causation”, “verification” questions — the ones single‑turn retrieval can’t handle.

Scenarios where you should NOT use it: simple fact‑finding. The user asks, “What is the company’s address?” One retrieval answers it. If you make the LLM think three times and call two tools, the response time goes from 2 seconds to 15 seconds — the user will be gone.

The engineering approach is routing: use a lightweight classifier to separate questions into “simple” and “complex”. Simple ones go to single‑turn RAG, complex ones go to Agentic RAG. The classifier can be an LLM or a rule‑based one (e.g., question length, presence of keywords like “compare”, “analyze”, “reason”).

def route_question(question: str) -> str:
    """Routing: decide single‑turn or Agentic"""
    complexity_keywords = ["compare", "comparison", "analyze", "reason", "why", "difference", "change"]
    if any(kw in question for kw in complexity_keywords) or len(question) > 30:
        return "agentic"
    return "simple"

def rag(question: str, user_id: str = "u1") -> str:
    route = route_question(question)
    if route == "simple":
        return simple_rag(question, user_id)  # single‑turn, fast
    return agentic_rag(question, user_id)     # multi‑step, accurate

Don’t throw every question to Agentic RAG just to show off a fancy tech stack — that’s irresponsible to both the user’s patience and your wallet.

8. What You Gain from This Article

Now the RAG series is complete. Let’s review the full pipeline across six articles:

• Article 1: Minimum viable engineering — get RAG running.
• Article 2: Qwen model chain — embedding + LLM, fully domestic.
• Article 3: Query rewriting + hybrid retrieval + re‑ranking — maximize recall.
• Article 4: Evaluation set + incremental updates — measurable and iterable.
• Article 5: Permission model — enterprise‑grade deployment without leaks.
• Article 6: Agentic RAG — grow a brain, multi‑step reasoning.

All together, your RAG evolves from “can run” to “can fight”. Each article is a natural extension of the previous one — no jumps, code you can directly chain and run.

Agentic RAG is the end of this series, but not the end of RAG. Beyond this are GraphRAG (knowledge graph enhanced), Self‑RAG (the model decides whether to retrieve), and multi‑modal RAG (images, documents, tables together). Those are different stories. First, solidify these foundations.

If you’re serious about building enterprise RAG, these six articles are enough to go live. The rest will be taught by real users and real data.

References:

• ReAct original paper (Reasoning + Acting): https://arxiv.org/abs/2210.03629

• Qwen function calling documentation: https://help.aliyun.com/zh/model-studio/function-calling

• OpenAI tools API specification (Qwen compatible): https://platform.openai.com/docs/guides/function-calling

• Agentic RAG survey: https://arxiv.org/abs/2501.09136

Previous article: Private Knowledge Base Part 5: Permission Model — Post‑Retrieval Filtering Is Wrong