"I have no idea!" This candid admission from Ben Stein, CEO of Teammates, encapsulates the central challenge facing product managers in the burgeoning field of AI. During his presentation at the AI Engineer World's Fair in San Francisco, Stein detailed how his own product, an AI agent named Stacey Hand, recently performed an unscripted action—responding perfectly to a Google Docs comment—leaving him both bewildered and enlightened. This unexpected capability highlights a profound shift in product development: shipping products when you don't fully know what they can do.
Stein explained that this new reality stems from two fundamental characteristics of AI-native products. First, if software is built on large language models (LLMs), "We will never understand what the LLM knows." The inherent opacity of these foundational models means creators cannot fully map their capabilities. Second, when users interact with AI via "free-text inputs," it leads to "unbounded requirements & expectations." Users will naturally try anything, often discovering emergent behaviors unintended by the developers.
This necessitates a radical re-evaluation of traditional product management. The discipline must pivot from rigidly "specifying features to discovering capabilities." This means thinking in "affordances," focusing on what the AI *can* do or *might* be able to do within its environment, rather than prescribing every single action.
To navigate this landscape, new techniques are essential. Stein champions the use of "evals," or evaluation frameworks, as the primary method to understand an AI product's functionality. Evals, typically a testing framework for probabilistic AI, become the de facto specification for emergent behaviors, allowing teams to quantify performance and identify where the AI excels or falters. Another critical tool is "Vibe Coding," which focuses on prototyping and experiencing AI interactions to gauge their "feel" and suitability, acknowledging that qualitative experience often precedes quantifiable requirements.
This emergent nature also reshapes internal dynamics. "Without defined requirements, PMs must earn credibility with Engineering," as the line between a 'bug' and an 'emergent feature' blurs. Furthermore, working with customers demands a new level of transparency. Product managers must "learn to say 'I don't know' in a way that instills confidence," fostering a collaborative approach where both the company and the customer discover the AI's potential together. This era, while challenging, offers an unprecedented opportunity for innovation, transforming the very essence of building and delivering technology.
