Pinterest's Medic AI Tames Spark Failures

Troubleshooting failures in distributed systems like Apache Spark can be a complex and time-consuming task, often involving sifting through disparate data sources and dealing with ambiguous priorities. To address this challenge, Pinterest has developed Medic, an agentic diagnostics tool designed to streamline the process of identifying and resolving Spark job failures.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

Flink

$1.1B

A platform for managing and optimizing employee benefits and compensation.

LangGraph

$200M

LangGraph is an open-source framework for building, managing, and deploying long-running, stateful agents as graphs.

Pinterest

$1.1B

A visual discovery engine for finding ideas like recipes, home and style inspiration, and more.

Langfuse

$3M

Open-source LLM engineering platform for debugging, monitoring, and improving production-grade LLM applications.

The Motivation Behind Medic

Drasko Profirovic, a Staff Engineer at Pinterest, explained the motivation for building Medic. The data platform organization at Pinterest, like many others, faces a high bar for supporting partner teams. This often translates into a constant stream of questions and problems, particularly when dealing with distributed systems like Spark, which can be difficult to troubleshoot, especially for newcomers.

The challenge is compounded by the need to balance priorities: helping a team with a failing job versus unblocking another team with a looming deadline. Profirovic highlighted that while humans struggle with such trade-offs, Large Language Models (LLMs) offer the potential to scale capabilities on demand. The vision for Medic was to enable users to ask a simple question like, "Why did a job fail?" and receive a detailed report with evidence and context-specific suggested fixes, available across user interfaces like Slack and Airflow.

From Prototype to Multi-Agent Architecture

The development journey began by exposing Pinterest's data resources via the Model Context Protocol (MCP) to connect them with LLMs. This initial approach required careful prompting from human operators to reason about Spark jobs. To improve this, Pinterest created a single reasoning and acting agent, dubbed ReAct, which was given a comprehensive prompt covering problem-solving, response structuring, and common failure patterns.

However, early trials revealed shortcomings: prompt tuning became unsustainable as a single prompt struggled to handle all scenarios, leading to inconsistent response quality and issues with context window limitations. Large log outputs, for instance, could quickly consume tokens, halting the agent's reasoning. Moreover, the reliance on manual end-to-end testing made it difficult to ensure changes didn't introduce regressions.

To overcome these hurdles, Pinterest focused on improving observability and testability. They integrated OpenTelemetry to publish traces to LangFuse, allowing for better visualization of the agent's execution steps. A purpose-built end-to-end test harness was developed to snapshot production data, codify expectations as offline evaluations, and enable prompt tuning based on quantifiable quality metrics.

Enhancements in Log and Metric Handling

Addressing the noise in logs, especially benign exceptions, was a key area of improvement. Pinterest built an exception classifier pipeline that learns common exceptions in successful jobs, filters out red herrings, and ranks remaining exceptions by content relevance and recency. This led to an improved signal-to-noise ratio and reduced the LLM's tendency to focus on misleading exceptions.

Similarly, raw time-series metrics were found to be token-inefficient and problematic for long-running jobs. The solution involved a quarantined sub-agent that analyzed metrics, converted them into collaged graphs (similar to Grafana dashboards), and attached these images to the LLM conversation. This approach ensured a consistent input token count for any job duration, allowing the LLM to reason about patterns like executor drops or resource bottlenecks.

The Multi-Agent Advantage

The system was further refined by moving from a single ReAct agent to a multi-agent architecture, built on LangGraph's deep agent library. Each agent was assigned a dedicated prompt and a subset of MCP tools. This decomposition allowed for specialized roles, such as a supervisor agent orchestrating the workflow, a triage agent identifying failure hypotheses, research agents validating hypotheses, and a healer agent providing remediations. This modularity not only improved control over the system's behavior but also simplified the expansion of its scope, enabling the addition of new functionalities like Spark SQL optimization.

Learnings and Future Directions

Profirovic shared key learnings, emphasizing the success of the multi-agent architecture and the importance of improving signal-to-noise ratios for logs. A critical lesson was that using LangGraph workflows to represent system state could be brittle, favoring the reasoning and acting agent paradigm. Looking ahead, Pinterest plans to move from prompt tuning for domain-specific knowledge to agentic RAG (Retrieval Augmented Generation) and extend the Medic framework to other distributed systems like Flink and Trino.