Mike Spitz on Post-Engineer Engineering Org

Mike Spitz, speaking at AI Engineer Europe, shared insights into building a "post-engineer engineering organization" by integrating AI agents into the development lifecycle. Spitz, who works at a sports data company that aids NFL and NBA teams, highlighted how his team's shift towards AI-augmented workflows has led to dramatic improvements in efficiency and output.

The PFF Case Study: Boosting Deployment Frequency

Spitz detailed a case study at PFF where the engineering team, initially 20 engineers working across multiple locations, faced challenges in keeping pace with competitors. By implementing AI agents, they observed a remarkable 25x increase in deployment frequency. While a team of two engineers leveraging AI was deploying five times per day, a team of ten without AI was deploying only once per week. Spitz noted that while smaller teams naturally tend to be faster, the AI integration amplified this effect significantly.

Measuring Success Beyond Speed

Beyond sheer speed, Spitz emphasized the importance of quality and customer satisfaction. The AI-augmented team achieved an average quality score of 8.6 out of 10, a marked improvement from the previous average of 7-7.5. This indicates that increased speed did not come at the expense of product quality; in fact, it led to better outcomes aligned with customer needs.

Rethinking Engineering Workflows

Spitz argued that traditional agile methodologies, such as daily stand-ups and sprint planning, are becoming less relevant in an AI-driven context. Agents can automatically update tickets, generate reports, and flag issues, minimizing the need for manual, human-centric rituals. The new workflow involves agents assisting in every stage of development, from specification and design documents to code creation, testing, and quality assurance. This shift allows engineers to focus on higher-level strategic tasks.

The Role of Humans in the AI Loop

While agents are taking on more responsibilities, Spitz stressed that human oversight remains crucial, particularly in areas like security, product feel, and scaling complex tasks. He outlined a tiered approach for human involvement:

• Verifiable, Deterministic Tasks: These are handled by agents using unit tests, end-to-end tests, and PR prerequisites.
• Agentic Review: Human steering and agentic review are employed for tasks requiring a degree of subjective judgment or brand alignment.
• Tiered Human in the Loop: Heavy human review is applied to system design and product feel, while lighter review occurs at the code level (except for security, which receives heavy review).

Spitz also highlighted the importance of building trust in agents to handle tasks autonomously, allowing humans to focus on the broader strategic picture.

The Playbook: Do's and Don'ts

For organizations looking to adopt AI in their engineering processes, Spitz offered a practical playbook:

Do:

• Start with boring, repetitive tasks that engineers dislike.
• Remove as much redundant process as possible.
• Encode your team's patterns as skills for agents.
• Build guardrails before granting full autonomy to agents.
• Pick your best engineers to build this out.

Don't:

• Try to onboard everyone at once.
• Create a one-size-fits-all approach; tailor it to your organization.
• Be conservative; the asymmetric risk of not adopting AI is too large.
• Try to do too much at once; a phased approach is necessary.

Spitz concluded by emphasizing that building an AI-augmented engineering organization requires a deliberate, phased approach, focusing on clear objectives and leveraging the unique strengths of both humans and AI.