Cognition's Jia Wu on Measuring AI Engineering Value

Jia Wu, Head of Forward Deployed Engineering at Cognition, detailed the company's unique approach to measuring the impact of AI coding agents. Unlike many in the market that focus on optimizing token usage, Cognition emphasizes tangible results for customers. Wu shared that their deployed engineering team has observed an impressive 82% reduction in the work targeted by their AI agents. This metric is achieved by measuring productivity before the agent is fully integrated and again once it's operational, ensuring the value is a real delta.

The Parabolic Rise of AI Productivity

Wu explained that the value derived from AI agents isn't linear. A single team adopting an agent can represent a step-change in productivity. However, as an entire enterprise integrates these tools, the impact can become parabolic. This occurs when the products Cognition builds and the customer's existing backlog begin to align, creating a compounding effect. This integration fundamentally changes the nature of deployed engineering roles.

Redefining the Deployed Engineer's Role

As the purely technical, coding-intensive aspects of engineering trend towards zero with AI assistance, the deployed engineer's job increasingly leans on business and people skills. Wu highlighted the importance of understanding the customer's true problem space. The role involves identifying the highest-leverage areas to direct the AI agent. Crucially, it requires effective communication to integrate these insights back into the product roadmap, as customer needs are paramount. Wu stated, "customers are the lifeblood."

Iterative Improvement Through Deployment

Each deployment is designed to derisk and improve subsequent ones. This is because the challenges encountered tend to recur in similar patterns across different companies. The underlying value proposition for Cognition's approach is clear and direct: correctness and customer success above all else. The goal is to ship difficult work and leave no potential value on the table.

Measuring Real-World Impact

Wu emphasized the importance of measuring outcomes that customers can actually see, rather than focusing on metrics like token usage. This involves a rigorous process of establishing a baseline before AI integration and then reassessing once the agent is fully activated within the customer's environment. The 82% reduction figure cited is a testament to this outcome-oriented measurement strategy. Wu also touched upon the concept of developers working alongside Devin, the AI agent, to achieve autonomous development. The core values driving Cognition's mission are correctness and unwavering customer success.