PostHog's Joshua Snyder on Self-Driving Products

Joshua Snyder from PostHog presented on the concept of "Self-Driving Products," detailing how to engineer a pipeline that translates product signals into actionable pull requests. The core idea is to automate the process of identifying and fixing issues within a product, moving from a reactive, human-intensive workflow to a proactive, AI-driven one.

The "Self-Driving" Pipeline

Snyder outlined a five-stage pipeline for this self-driving product approach: Ingest, Group, Research, Assess, and Execute. The ingest phase involves filtering and normalizing signals from various sources like error tracking, session replays, experiments, analytics, and external feedback channels. These normalized signals are then grouped together to identify potential issues.

The research phase involves an AI agent that analyzes these grouped signals to understand what the problem might be. Following this, the assess phase determines the actionability of the identified issue. Finally, the execute phase involves writing code to fix the problem and, ideally, creating a pull request for review and merging.

From Observability to Automation

Snyder contrasted the current state of observability with the future vision. Today, signal detection leads to human investigation, analysis, and then the creation of a pull request. In the future, a background agent will detect a change, research the problem, create a pull request, and ship the feature behind a feature flag. This shift aims to drastically reduce the time and effort required to respond to product issues.

The current process is described as slow and potentially uninteresting for engineers. The proposed automated pipeline aims to streamline this by having agents handle the initial detection and research, freeing up human engineers for more complex tasks or creative development.

Key Learnings and Challenges

During the development of this pipeline, several key learnings emerged:

• Evals Matter: It's difficult to predict what might go wrong without testing on representative data.
• Embed the Right Thing: Embedding models should prioritize structural similarity over purely semantic similarity to group signals effectively.
• Agents Will Always Fix a Problem: If an agent is presented with a problem, it will attempt to solve it. Therefore, it's crucial to feed agents well-defined problems.
• Tokens Are Free (But Costs Matter): While tokens might seem free, the cost implications of ingesting vast amounts of data can lead to suboptimal architectural decisions.

The Pipeline in Action

Snyder elaborated on the specific steps within the pipeline. The ingest phase involves filtering and normalization, with a safety filter using an LLM classifier to identify and drop malicious injection attempts. The output of this phase is a normalized signal with attributes like source, type, content, and vector embedding.

The grouping stage focuses on cross-type clustering. For example, an error tracking signal about a "checkout null pointer bug" and a Slack message stating "Hey checkout is broken for me" would be grouped together. Similarly, analytics data showing a "funnel drop on pricing page" and an experiment showing a "price variance drop" would be clustered as a "pricing experiment regression." This grouping is weighted based on the significance and source of the signals.

Rewriting Queries with LLMs

A significant challenge discussed was how to effectively search through the vast amount of signal data. A naive approach using raw embeddings proved inefficient. The better approach involves rewriting queries using LLM-generated natural language queries. This allows the agent to retrieve more relevant signals by understanding the intent behind the problem description.

Research and Actionability

The research and actionability phase involves a research agent that uses various tools, including PostHog's own MCP tools, GitHub API, and external MCPs. The agent analyzes the reported problem, its context, and relevant data to determine actionability. The output is categorized into three buckets: immediately actionable (where the agent can create a task and write a fix), needs human input (requiring more context or decision-making from a human), and not actionable (for issues that are too trivial or lack sufficient data).

Closing the Loop

The presentation concluded by summarizing what has been built and outlining future directions. PostHog has built a system that ingests signals, groups them into reports assessed for actionability, and prepares pull requests ready for merging. The future vision includes automatically shipping experiments, deploying changes behind feature flags based on metric performance, and continuously learning from every outcome to refine the system.