Alex Bowcut on RAG: Accuracy Over Obsolescence

The rapid advancement of large language models (LLMs) with increasingly vast context windows has sparked debate about the relevance of existing AI techniques. One such technique, Retrieval Augmented Generation (RAG), has been a cornerstone for enhancing LLM accuracy by providing relevant context. However, as LLMs become more capable of processing massive amounts of information directly, the question arises: Is RAG becoming obsolete?

In a recent discussion, Alex Bowcut, Head of Engineering at Sphere, argued that RAG is far from dead, particularly in applications where accuracy and traceability are non-negotiable. Bowcut, whose company builds AI systems for sales tax automation and compliance, highlighted that while larger context windows might streamline simpler queries, they don't replace the need for RAG in complex, high-stakes scenarios.

The full discussion can be found on TWIML's YouTube channel.

The Enduring Value of RAG

Bowcut explained that for specialized domains, the ability to pinpoint the source of information is as important as the answer itself. In sales tax automation, for example, simply getting an answer isn't enough; users need to know precisely which regulation or piece of legislation supports that answer. This is where RAG shines, by retrieving specific documents and citing their exact origin.

He elaborated, "I think for some use cases it's certainly true. I think for us, or at least for this particular problem, because we are so sensitive to accuracy and we're so sensitive to the exact right citation, as of today, I don't think agents are just searching over the file system, grepping over it, it's at a point where we could switch over and not lose accuracy."

This need for precision is particularly evident when dealing with rapidly changing legal and regulatory frameworks. Bowcut noted that while LLMs can ingest vast amounts of data, keeping them updated with the latest legal changes and ensuring they can cite specific, current regulations is a significant challenge. RAG systems, by design, can be more readily updated and queried for the most relevant and up-to-date information.

Automating Complexity in Compliance

Bowcut shared insights into Sphere's work, where they leverage AI to navigate the complexities of sales tax compliance across various jurisdictions. He explained that the process is far from straightforward due to the sheer volume and variability of regulations across different states and countries.

"We help companies with all their revenue-based compliance needs," Bowcut stated. "The main one of those is sales tax, both in the US and internationally. Because every US state, and potentially province has their own rules, and then internationally, every country has their own rules. And so, you have to understand how your product is taxed in each jurisdiction."

Traditionally, companies relied on large teams of human experts to manage this. Bowcut highlighted the inefficiency and cost associated with this approach, noting that it often involves manually sifting through vast amounts of legal text and documentation.

"Companies need to understand how their products are taxed in each jurisdiction," he continued. "And so, we've built a system where we essentially supercharge our tax experts, where we allow our internal tax experts to move almost over order of magnitude faster through this process. And the reason why is because of the way we've built our system, we can ingest these documents and then query over them, and then use the LLM to, you know, answer questions about them, but also to retrieve the specific pieces of data that we need to make a determination on whether a product is taxable or not."

The Role of AI in Data Analysis

Bowcut emphasized that their AI system, TRAM, is designed to augment the capabilities of tax experts. TRAM analyzes vast datasets, including legal documents and regulatory changes, to provide precise answers and support for tax decisions.

"TRAM," Bowcut explained, "allows our internal tax experts to move almost over order of magnitude faster through this process. And the reason why is because of the way we've built our system, we can ingest these documents and then query over them, and then use the LLM to, you know, answer questions about them, but also to retrieve the specific pieces of data that we need to make a determination on whether a product is taxable or not."

He further elaborated on the challenges of unstructured data, noting that while many companies still rely on manual processes or basic keyword searches, AI-driven approaches can unlock significant efficiencies. "We've all seen the sheer volume of data that you have to process to, you know, to figure out how to tax things correctly," Bowcut said. "And so, you know, the fact that we're able to automate that, especially with the advent of these large language models, is pretty cool."

Future of RAG and AI in Specialized Domains

Bowcut's perspective underscores a nuanced view of AI's role in specialized fields. While LLMs are becoming increasingly powerful, RAG and similar techniques remain crucial for ensuring accuracy, explainability, and compliance, especially in areas governed by complex and evolving regulations.

The discussion also touched upon the challenges of integrating AI into existing workflows, particularly in industries with strict compliance requirements. Bowcut's experience at Sphere highlights the importance of not just adopting AI, but tailoring it to specific needs and ensuring it complements, rather than replaces, human expertise.