Agentic Retrieval Loops

By Michael Samuel Naeem · November 17, 2024 · 5 min read

A user asked your docs bot "Can I use SSO with the legacy API if my org migrated to the new auth service last quarter?" Single-shot RAG retrieved a chunk about SSO configuration from 2022 and a chunk about the new auth service launch — then synthesized a confident wrong answer because nobody taught the pipeline that "last quarter" requires checking migration timelines across two doc sets. Agentic retrieval loops exist for exactly this: questions where one embedding search is not enough.

Single-shot RAG breaks on multi-hop questions

Standard RAG embeds the user query, retrieves top-k chunks, stuffs them into a prompt, and generates an answer. That pipeline assumes the query embedding aligns with the chunks that contain the answer. Multi-hop questions violate that assumption because the answer requires intermediate facts the original query never mentions.

"Which compliance framework applies to our EU customers using the Frankfurt region?" needs at least two hops: identify which product those customers use, then find the compliance doc for that product in that region. One retrieval call rarely lands both chunks in top-k.

The agentic loop pattern

An agentic retrieval loop wraps the LLM and retriever in a cycle:

1. Plan — decompose the question into sub-queries or decide what to search next.
2. Retrieve — run one or more searches with the current sub-query.
3. Evaluate — assess whether retrieved context is sufficient, contradictory, or incomplete.
4. Refine — rewrite the query, search again, or proceed to answer.
5. Stop — exit when criteria are met or max iterations hit.

MAX_ITERATIONS = 4

def agentic_rag(question: str) -> str:
    context_chunks = []
    sub_query = question

    for i in range(MAX_ITERATIONS):
        new_chunks = retrieve(sub_query, top_k=5)
        context_chunks.extend(new_chunks)

        assessment = llm_evaluate(question, context_chunks)
        # assessment: { "sufficient": bool, "next_query": str | None }

        if assessment["sufficient"]:
            break
        if assessment["next_query"] is None:
            break
        sub_query = assessment["next_query"]

    return llm_generate(question, dedupe(context_chunks))

The evaluate step is what separates agentic RAG from blind re-querying. Without it, you are just running the same search multiple times with minor paraphrases.

Query decomposition strategies

LLM-generated sub-queries: Ask the model to break the question into independent search tasks before any retrieval. Works well for analytical questions with clear facets.

Question: Compare uptime SLAs for Enterprise vs Starter plans in APAC.

Sub-queries:
1. Enterprise plan uptime SLA APAC
2. Starter plan uptime SLA APAC
3. APAC region infrastructure redundancy

Hypothesis-driven retrieval: Generate a draft answer from initial chunks, identify claims lacking evidence, and retrieve specifically for those gaps. Self-RAG and corrective RAG variants use this pattern.

Tool-calling agents: Frame retrieval as a tool the agent invokes with explicit query strings. The agent's reasoning trace shows why each search happened — valuable for debugging and audit.

Handling contradictions and stale chunks

Loops should detect when new chunks contradict earlier ones. Prompt the evaluator explicitly:

If retrieved documents conflict, note the conflict and retrieve
with a query targeting the more recent or authoritative source.
Prefer documents tagged "current" over "deprecated".

Metadata filters — document version, effective date, status — reduce contradiction frequency but do not eliminate it. The loop's evaluate step is your last line of defense.

Stopping criteria that work in production

Hard limits prevent runaway cost: MAX_ITERATIONS = 4 is a common starting point. Soft stops improve quality:

• Coverage check: Does context address every entity in the question?
• Novelty check: Did the last retrieval return chunks already in context?
• Confidence threshold: Model rates its evidence sufficiency above 0.8.

Log iteration count, sub-queries, and chunk IDs per request. Questions that consistently hit max iterations indicate corpus gaps or bad chunking — not just a tuning problem.

Routing simple vs complex queries

Not every question needs a loop. A lightweight classifier — keyword heuristics, embedding distance to known simple patterns, or a small model — routes "What are your business hours?" to single-shot and "How does feature X interact with policy Y for migrated accounts?" to the agentic path.

This hybrid architecture keeps p95 latency acceptable while improving quality on the 15–20% of queries that actually need multi-hop retrieval.

Common production mistakes

Teams get agentic retrieval loops wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

RAG pipelines for agentic retrieval loops degrade when chunk boundaries split tables, embeddings go stale after doc updates, and retrieval metrics are measured offline only. Re-index incrementally and monitor answer faithfulness on live traffic samples.

Debugging and triage workflow

When agentic retrieval loops misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Self-RAG paper (Asai et al.)
• Corrective RAG (CRAG) research
• LangGraph — agentic workflows documentation
• LlamaIndex query engines and agents
• Anthropic — building effective agents