Stop Babysitting AI Agents: Build a Context Engine

In the rapidly evolving world of AI, the challenge of managing autonomous agents has become a significant hurdle for many teams. Brandon Walsenuk, from Unblocked, recently addressed this issue at AI Engineer Europe, highlighting the critical need to move beyond simply providing agents with access to data and towards building robust context engines. His presentation, titled "Stop babysitting your agents: building a context engine for mergeable code," outlined the common pitfalls and offered a path forward for creating more effective and independent AI systems.

The Problem: Context is King, Not Just Access

Walsenuk began by drawing a parallel between human onboarding in a company and the current state of AI agents. Just as new hires initially lack context and require guidance, newly spawned AI agents often begin with a "zero context" state. This necessitates continuous human oversight and intervention, a process Walsenuk refers to as 'babysitting.' He argued that the core issue is not a lack of intelligence in these agents, but a deficit in understanding the broader context in which they operate.

He debunked several myths surrounding the creation of effective AI agent context. First, he stated that a 'naive RAG over my docs is a context engine' is a misconception. While Retrieval Augmented Generation (RAG) can provide access to data, it often fails to deliver true understanding or reasoning capabilities, leading to agents that simply stop looking once they find a superficial match, rather than exhaustively searching for the correct solution. Secondly, the idea that 'if I just connect enough MCPs, I'm done' is also flawed. Simply connecting multiple tools or data sources does not guarantee the agent will understand or correctly reason across them. Finally, he addressed the myth that 'a bigger context window will solve this.' While larger context windows can be beneficial, they don't inherently provide the necessary reasoning capabilities or solve the fundamental problem of understanding nuanced relationships within the data.

The Solution: Building a True Context Engine

Walsenuk emphasized that a proper context engine is essential for unlocking the full potential of AI agents. Such an engine should possess several key attributes:

• Unified System Context: It must merge signals from all relevant sources before delivery to the agent.
• Targeted Retrieval: It needs to retrieve only the information agents need based on the data graph.
• Conflict Resolution: Recency and authority signals should resolve contradictions in the data.
• Token Optimization: It should render compressed context to create relevant responses within prompt windows.
• Data Governance: Permissions and policies must be enforced automatically across systems.
• Personalized Relevance: Context should be scoped to the user, team, and work history.

He illustrated these points with a demonstration of Unblocked's 'Social Graph Builder,' an open-source tool that analyzes historical pull request data to map team collaboration and identify subject matter experts. This tool, Walsenuk explained, provides a foundational component for building a sophisticated context engine. The demo showcased how such an engine can ingest data from various sources, understand individual roles and relationships within an organization, and provide agents with the precise, contextual information they need to perform tasks autonomously and effectively.

Hard Lessons and Future Applications

Walsenuk shared three hard lessons learned from their experience building context engines:

1. Optimized for Access, Not Understanding: Simply wiring more tools didn't help agents understand tasks better.
2. Hid Conflicts Instead of Surfacing Them: When sources disagreed, the system often picked one truth and ignored the conflict, leading to errors.
3. Cached Answers Instead of Computing Them: Reusing stale answers, even if they once seemed correct, broke the system as code and requirements evolved.

He concluded by stressing that AI-generated code should feel as if it were written by a seasoned team member. This requires a context engine that can reliably provide the right information at the right time, enabling agents to operate with autonomy and intelligence, ultimately freeing up human engineers to focus on more complex, strategic tasks.