Notion's AI Lead on Token Costs & Strategy

Sarah Sachs, Lead of AI Engineering Teams at Notion, delivered a candid talk at AI Engineer World's Fair on the realities of building and scaling AI products, particularly focusing on the economics of AI tokens and strategic approaches to development. Titled "Token Town: How do you go from AI-pilled to AI-poor?", Sachs highlighted the significant cost barriers and market dynamics that companies face when integrating AI, emphasizing that the path to sustainable AI products lies beyond simply adopting the latest frontier models.

Navigating the AI Transformation Journey

Sachs outlined a four-stage model for AI transformation: AI as a thought partner, AI as an assistant, AI as teammates, and finally, AI as the system. She noted that while many companies are still grappling with the initial stages of AI adoption, the ultimate goal is to reach a point where AI becomes a critical workflow, with entire systems running autonomously. However, she pointed out that only a small percentage of companies have achieved this, citing a thesis at Notion that the prevalence of siloed data and the lack of a durable system of record hinder progress.

The High cost of AI tokens

A significant portion of Sachs' talk focused on the economic challenges of AI, particularly the cost of tokens. She illustrated this with examples of how reasoning models can increase output token usage unexpectedly, and how new model releases often come with significantly higher price tags. "Cost is a structural barrier to entry," Sachs stated, explaining that these costs make it difficult to serve products and build AI factories at scale. She argued that companies are often forced into unfair deals with token providers, leading to a loss of optionality and a precarious business model.

Winning on Product, Not Just Tokens

Sachs advocated for a strategic shift from focusing on token economics to prioritizing product differentiation. She stressed the importance of building "data flywheels" and understanding customer needs to determine when to prioritize capability, price, or latency. "I promise you, you don't always need what is usually the slowest but the most capable model out there," she advised.

She also introduced the concept of "product moats," which are built through compelling UI, orchestration, and integrations that justify the cost of reselling tokens. Her key message was to "bet on the frontier, not on the lab," encouraging companies to leverage the latest advancements without becoming overly dependent on a single provider.

The Model Agnostic Playbook

To navigate the complex AI token market, Sachs proposed a "model agnostic playbook" with several key tenets:

• Build for multi-model: Run infrastructure across all major providers to maintain flexibility.
• Evaluate on value, not tokens: Base decisions on cost-per-capability-per-second rather than just token count.
• Switch fast, switch often: Adapt quickly as new models emerge and tools evolve.
• Give frontier labs something back: Provide detailed evaluations and feedback to model providers.
• Forgo discounts for optionality: Prioritize flexibility over short-term cost savings to build long-term trust and growth.

Notion's own "Auto Model" was presented as an example of this approach, capable of switching between state-of-the-art models and handling 75% of their AI traffic without vendor lock-in.

Challenges and the Future

Looking ahead, Sachs identified security, sandboxes and computers, and multi-agent orchestration as key challenges for the next six months. She highlighted the "lethal trifecta" of security risks: access to private data, exposure to untrusted content, and the ability to externally communicate. She concluded by emphasizing that the AI market is still in its early stages and that companies have a responsibility to get it right for their customers.