Kuba Rogut: Is RAG Dead or Evolving?

The AI community is buzzing with the notion that Retrieval Augmented Generation (RAG) might be dead, or at least significantly evolving. Kuba Rogut, a deployed engineer at Turbopuffer, dives into this topic in his talk, "RAG is dead, right?? How hybrid, tool-rich retrieval is becoming the default for serious agentic search." Rogut suggests that while RAG laid crucial groundwork, the field is moving towards more sophisticated "agentic retrieval" methods.

The "RAG is Dead" Phenomenon

Rogut opens by showcasing a wave of social media posts, particularly from Twitter (now X), all proclaiming "RAG is dead." This sentiment is fueled by the rapid advancements in large language models (LLMs) and the increasing demand for more capable AI agents. He presents Google Trends data indicating a sharp rise in interest for "RAG is dead" throughout late 2023 and early 2024, underscoring the timeliness of this discussion.

Clarifying RAG vs. Agentic Search

Rogut clarifies the common understanding of RAG, often simplified to vector search and passing retrieved context to an LLM. He contrasts this with what he terms "agentic search," which involves a more complex process. Agentic search gives agents a set of tools to progressively find and reason over external context. This iterative approach, as opposed to a single retrieval event, is key to its effectiveness.

The "old" way of RAG involved a single retrieval, stuffing context into the prompt, and crossing fingers for a good result. The "new" way, agentic retrieval, involves reasoning in steps, searching as needed, and fetching only what is useful. This iterative process is central to building more robust and reliable AI agents.

Case Study: Cursor's Approach to Codebases

To illustrate the practical application of these concepts, Rogut highlights the work of Cursor, a company that builds AI-powered code editors. He references a blog post from Cursor titled "Securely indexing large codebases." The post details how Cursor uses semantic search to provide a searchable index of a codebase, which is crucial for developers working on large projects.

Rogut points out that Cursor's evaluation showed significant improvements in response accuracy (12.5%) and reduced user dissatisfaction when using their retrieval methods. He notes that for teams of thousands of files, processing and indexing can take hours, and semantic search is not always available. Cursor's approach, which involves chunking, embedding, and indexing code, allows for faster retrieval of relevant information.

The data presented by Cursor indicates that their semantic search significantly improves agent performance. For instance, their models achieved a 12.5% higher accuracy in answering questions compared to previous methods. Furthermore, they observed a 2.6% increase in code retention and a 2.2% decrease in dissatisfied user requests.

The "Cache Compute" Advantage

Rogut emphasizes the concept of "embeddings are cached compute." This means that the initial processing and embedding of data are done upfront, and this "work" is then amortized across multiple retrieval operations. In contrast, the "per-session discovery" approach, more akin to traditional RAG, involves repeated processing for each agent and each task. This can lead to a significant token cost, as illustrated by the example of 6,314 tokens being used across repeated sessions for an agent.

Agentic retrieval, by contrast, indexes once and retrieves at runtime. The model has already "read" every file, making subsequent queries much more efficient. The example shows a query that, with agentic retrieval, results in only 424 tokens being used, a substantial saving.

From RAG to Agentic Retrieval: A Paradigm Shift

The core message is that retrieval is now iterative and tool-driven. Rogut references Jeff Dean from Google, who stated that "bigger context windows alone are not enough." What matters are staged retrieval, lightweight mechanisms that narrow down trillions of tokens to the millions you actually need. This sentiment aligns with the shift towards agentic retrieval.

The "old" method of retrieval was a one-time event, stuffing everything into the prompt and crossing fingers. The "new" method involves reasoning in steps, searching as needed, and fetching only what is useful. This iterative process is more efficient, more reliable, and ultimately leads to better AI agent performance.

Rogut concludes by suggesting that while the term "RAG is dead" might be provocative, it signifies a genuine evolution in how we approach AI agent development, moving towards more sophisticated and efficient retrieval mechanisms.