Supabase's Pedro Rodrigues on AI Agents and Context

Pedro Rodrigues, an AI tooling engineer at Supabase, recently presented at AI Engineer Europe on the topic of "Combine Skills and MCP to Close the Context Gap." His talk focused on the crucial role of providing AI agents with the right guidance and context to perform tasks effectively.

Understanding the "Skill Issue"

Rodrigues began by highlighting a common challenge in AI agent development: agents are smart, but they need the right 'guidance.' This guidance comes in the form of 'Skills,' which are essentially folders containing instructions, scripts, and resources that agents can discover and dynamically load to complete tasks. These skills are designed to address specific needs, such as security pitfalls, stale knowledge, or missing workflows.

He explained that Supabase has been actively working on building these skills for their product. The process involved writing a Supabase skill, which meant teaching AI agents about the Supabase product. Rodrigues noted that this endeavor was more complex than his previous experience writing his master's thesis, emphasizing the challenges in creating comprehensive and effective documentation for AI agents.

Principles for Building Effective AI Agent Skills

Rodrigues outlined three key principles for building successful product skills for AI agents:

• Describe intent, not means: Instead of telling agents how to do something, it's more effective to tell them what needs to be achieved and where to find the information. This involves providing clear instructions and pointing them to the right resources. For example, instead of detailing a specific search query, guide the agent to search Supabase documentation.
• If it can be skipped, it will be: This principle emphasizes the importance of including critical information directly within the skill file (inline) if it's essential and unlikely to change. Reference files are optional, but agents should be able to skip them if they are not needed. Security rules, for instance, are not optional and should be included inline.
• Be opinionated: Rodrigues stressed that product teams know their products best and should encode workflows that make sense for their specific use cases. This means being decisive about how agents should interact with data, such as using direct SQL queries for certain tasks or running advice before making changes.

Supabase Docs over SSH: An Experimental Approach

The presentation also touched upon an experimental feature: Supabase Docs over SSH. This offers a more intuitive way for agents to query documentation using a familiar bash interface. The idea is to allow agents to interact with documentation as if it were a local file system, enhancing their ability to retrieve and process information relevant to Supabase. Rodrigues mentioned that this feature is still experimental and they are actively seeking feedback.

Evaluating Skill Performance

To measure the effectiveness of their skills, Supabase conducted evaluations across various scenarios. They tested six Supabase-specific scenarios, four agent-only scenarios, and three test conditions. The evaluations compared a baseline performance, an 'MCP Only' condition (likely referring to a specific AI model or framework), and an 'MCP + Skill' condition. The results indicated that skills significantly improved agent performance across different models and tasks, particularly in areas like code generation and understanding complex data schemas.

Rodrigues highlighted that the data clearly shows that skills, when properly implemented, are crucial for bridging the context gap and enabling AI agents to perform tasks more accurately and efficiently. He encouraged the audience to consider how they could apply these principles to build their own product skills.

Getting Started with Supabase Skills

For those interested in learning more or getting started with Supabase agent skills, Rodrigues pointed to the Supabase blog and their GitHub repository. He also mentioned an ongoing giveaway where attendees could win a Mac Mini by scanning a QR code and signing up for Supabase. The presentation concluded with a Q&A session, where Rodrigues addressed questions about the practical implementation and distribution of these skills within organizations.