From Discord's AI Growing Pains to Promptfoo's Red Teaming Triumph

Ian Webster’s journey, culminating in Promptfoo’s $18M Series A, offers a compelling narrative of entrepreneurial pivot born from direct experience. Having spearheaded Discord’s AI chatbot, Clyde, Webster intimately understood the critical chasm between developing AI and deploying it safely to a massive user base. This foundational insight propelled him from a general evaluation tool to a specialized AI security red teaming platform, addressing a burgeoning need in the rapidly evolving AI landscape.

In a recent interview with Alessio Fanelli and Swyx on the Latent Space Podcast, Webster detailed the evolution of Promptfoo. His initial foray into open-source AI evaluation while at Discord quickly revealed that while "evals are table stakes," the true challenge lay in "systematically finding and fixing application-specific risks before they reach production." This realization underpinned his strategic shift towards proactive security.

A core insight from Webster is the crucial distinction between risks inherent to foundational models and those arising from application-level implementation. While major model providers like OpenAI and Anthropic dedicate teams of geniuses to mitigate issues like toxicity and bias in their base models, they cannot account for every application-specific vulnerability. Webster succinctly put it: "you can't fix stupid." This means that even the most robust foundation models are susceptible to mistakes in how they are integrated and utilized within an application, whether through improper RAG access controls or overly permissive agent API capabilities. This fundamental gap demands a specialized approach that goes beyond generic safeguards.

Promptfoo tackles this challenge head-on by employing AI-versus-AI techniques. Unlike static vulnerability databases or canned prompt injections, Promptfoo generates tailored attacks by understanding the application's specific purpose, features, access permissions, and business context. This plugin-based system, with over a hundred risk areas, allows for dynamic attack objectives and strategies. It simulates malicious user interactions, feeding results back to an "Attacker LLM" and a "Judge LLM" in a continuous feedback loop to refine attacks and identify vulnerabilities.

Related Reading

• CrowdStrike CEO: AI Needs Cybersecurity, Not Replacement
• Engineering AI Prompts: Google's Framework for Benchmarking and Automation
• AI's Volatility Paradox: Calm Indices Mask Deepening Tech Anxiety

The future of AI security, according to Webster, lies not in reactive measures but in proactive integration. He asserts that "security tooling needs to be closer to developers as opposed to kind of after the fact." This implies a shift left in the development lifecycle, embedding security testing within CI/CD pipelines. With agents becoming increasingly complex and runtimes extending to 30 minutes or more, current post-deployment monitoring methods will prove inadequate. The goal is to identify and mitigate risks before they ever reach production, making security an inherent part of the development process.

The AI security market is a greenfield opportunity, albeit one fraught with challenges. Webster noted the skepticism inherent in security sales, compounded by the novelty and rapid pace of AI development. In such an environment, open-source initiatives like Promptfoo play a vital role in building trust, offering transparency where "snake oil and smoke and mirrors" often prevail. Notably, over 10% of Fortune 500 companies are already Promptfoo customers, indicating that large enterprises, with their significant reputational and financial stakes, are recognizing the imperative of robust pre-deployment AI security. These companies are not merely checking a compliance box; they are seeking comprehensive solutions to manage the complex, application-specific risks that can lead to high-profile incidents.