Is Classic RAG Dead? When to Build a RAG Agent Instead
Classic RAG is not dead. A single retrieve-then-generate pass is still the right build when questions are answerable from one lookup and latency matters. A RAG agent, which plans, calls retrieval as a tool and searches again when results fall short, earns its extra model calls on multi-step, ambiguous or multi-source questions.
What the "don't build RAG" headline is really about
The seed for this page is a 15-minute tutorial from the AI with Thiru channel, published 17 November 2026 under the title "Don't Build RAG in 2027 | RAG Agent Explained". Its description shows the actual content: how RAG works, combining an AI agent with RAG and MCP, a RAG agent implementation, and live tests in a UI and through an API, with the code on GitHub. So the video does not abandon retrieval. It wraps retrieval in an agent.
That is the real argument behind most "RAG is dead" headlines. They reject the naive pipeline, not retrieval. Both architectures below still retrieve; they differ in who decides when, where and how many times.
Two architectures, one retrieval step
Classic RAG is a fixed pipeline. Embed the question, fetch the top few chunks from a vector index, place them in the prompt, generate once. One retrieval, one model call, a predictable path.
A RAG agent, usually called agentic RAG, gives the model retrieval as a tool inside a loop. The model decides whether to search at all, rewrites the query, chooses between sources such as a vector index, a SQL database, web search or MCP tools, reads what comes back, and searches again if the results fall short. A January 2025 survey on agentic RAG by Singh and colleagues (arXiv 2501.09136) groups the patterns into four: reflection, planning, tool use and multi-agent collaboration.
The ideas predate the label. Self-RAG (Asai and colleagues, 2023) trained a model to decide when to retrieve and to critique its own use of passages. Corrective RAG (Yan and colleagues, 2024) added an evaluator that scores retrieved documents and falls back to other sources when they look weak. An agent loop generalizes both.
When single-shot RAG is still the right build
Anthropic's December 2024 guidance on building agents recommends finding the simplest solution possible and adding complexity only when needed, and notes that agentic systems often trade latency and cost for better task performance. For a large share of retrieval workloads, that trade is not worth making.
The questions are lookups. Product docs, internal policies, support macros, API references. If one good search usually contains the answer, a planner adds calls without adding accuracy.
Latency is a requirement. Support widgets, voice agents and inline assistants need a reply in about the time one model call takes. Every agent iteration is another serial model call.
Cost must be predictable. A pipeline costs roughly the same per question. An agent's cost depends on how many loops a question triggers.
You need to evaluate it. A fixed pipeline can be measured with standard retrieval metrics and fixed by tuning chunking, embeddings or reranking. An agent's failures are spread across planning, tool choice and stopping decisions, which makes them harder to isolate.
When a RAG agent earns its extra cost
Multi-hop questions. "Which customers on the old plan filed tickets about the outage?" needs one lookup to find the plan, another for the outage window, a third for tickets. A single query cannot express that.
Ambiguous or badly phrased queries. An agent can rewrite, split or broaden a query after seeing that the first results missed.
Several sources with different shapes. Numbers live in a database, explanations in documents, recent events on the web. An agent can pick the right tool per sub-question instead of forcing everything through one vector index.
Retrieval that often comes back empty or wrong. If your logs show many answers built on irrelevant chunks, a check-and-retry step catches some of them before generation.
Answers that feed actions. When the output files a ticket or changes a record, the extra verification pass is cheap insurance.
What the loop actually costs
Each iteration is another model call that reads the retrieved text, so tokens and latency stack with the number of loops. The failure modes are new as well: agents that search forever, agents that over-retrieve and bury the answer in a crowded prompt, and errors that compound across steps, a risk Anthropic's guidance names directly.
Three controls keep it in check. Cap iterations and token budget per question. Route first: a cheap classifier sends simple lookups down the single-shot path and only hard questions into the loop. Move narrow judgments, such as whether a passage is relevant, to something cheaper than the main model, an approach covered in the comparison of decision models and agentic RAG.
A practical path, and the layer underneath
Start with single-shot RAG and log every question, the chunks retrieved and whether the answer held up. Group the failures. If most are bad chunks, fix retrieval: chunking, embeddings, metadata filters, reranking. If most are questions that need several lookups or several sources, add an agent loop for that class of question only.
Either way, retrieval quality is the ceiling. An agent that searches a poor index three times gets three poor results. RDK provides the index as a layer both designs can call over MCP, the CLI or the Partner API. Its stacked retrieval answers 40 to 65 percent of queries from a private vault, 15 to 20 percent from the public network, and leaves 5 to 10 percent for the LLM as fallback, so the loop runs less often in the first place.
Frequently asked questions
- Is RAG dead?
- No. What critics call dead is the naive pipeline that embeds a question, grabs the top few chunks and generates once, applied to every problem. Retrieval itself is in every serious agent design. Single-shot RAG remains the right choice for lookup-style questions with tight latency budgets, and agentic retrieval is an extension for harder questions, not a replacement.
- What is agentic RAG?
- Agentic RAG puts retrieval inside an agent loop. Instead of one fixed search, the model decides whether to search, rewrites queries, chooses among sources such as vector indexes, databases and web search, checks the results and searches again when they fall short. A 2025 survey groups its patterns into reflection, planning, tool use and multi-agent collaboration.
- Is agentic RAG slower and more expensive than classic RAG?
- Usually, yes. Each planning step, retrieval check and retry is another model call that reads retrieved text, so latency and token cost grow with the number of iterations, and cost per question becomes less predictable. The trade pays off on multi-step or multi-source questions where single-shot retrieval fails, and rarely on simple lookups.
- Can an existing RAG pipeline be turned into a RAG agent?
- Yes, and incrementally. Expose the existing retriever as a tool the model can call, add a step that judges whether the results answer the question, and allow a capped number of query rewrites. Keep the original pipeline as the fast path and route only questions that fail or look multi-step into the agent loop.
- Does agentic RAG fix bad retrieval?
- Only partly. Query rewriting and retries can recover from a poorly phrased question, but if the right passage was never indexed, was split badly, or is drowned out by near-duplicates, more searches return the same weak results. Fix chunking, embeddings, metadata and reranking first, then measure whether an agent loop adds accuracy on what remains.