JP Morgan's Ritvik Pandya on Learned Execution Graphs for APIs

Ritik Pandya, who leads the payments team at JP Morgan Chase, recently presented a talk on "Learned Execution Graphs for real-time anomaly detection & Drift Classification in APIs." The core idea revolves around leveraging execution graphs, which are short-lived graphs representing the flow of a request through various services, to identify deviations from normal behavior.

Pandya distinguished these execution graphs from persistent or property graphs commonly used in knowledge representation. While traditional graphs are queried for relationships and data, execution graphs are traces built from spans, describing how a specific request was processed. These graphs are inherently acyclic and capture the causal flow of operations, making them ideal for understanding and diagnosing performance issues.

Understanding Execution Graphs

The presentation outlined a typical request processing pipeline, starting from an edge layer where requests enter, moving through gateways, authentication, orchestration, and finally to various backend systems that might be called in parallel. This entire process can be visualized as a Directed Acyclic Graph (DAG). The DAG structure simplifies understanding the order of service execution and the context passed between different nodes.

The concept extends to handling complex scenarios like retries and loops, where each iteration can be represented as a distinct entity within the graph for easier tracking.

Anomaly Detection and Drift Classification

Pandya explained a tiered approach to anomaly detection. The first tier involves a quick check, akin to a boarding pass scan at an airport. If the request execution appears normal end-to-end, no further checks are needed. However, if a delay is detected, the system progresses to deeper analysis. This analysis involves identifying structural changes (like added or removed nodes) or scale deviations.

The presentation differentiated between anomalies and drifts. An anomaly is a single, isolated event that deviates from the norm, while a drift signifies a gradual change in patterns over time. The system aims to detect both. Drifts can be categorized into structural (changes in topology), performance (changes in execution time), or covariate (changes in input distributions), each requiring different handling.

The "Learned" Approach

A key aspect of the proposed solution is the "learned" component, which emphasizes modeling the distribution of normal graphs rather than relying on predefined rules. This approach is crucial because hand-written thresholds often fail on every deployment. Instead, the system learns baselines from normal traffic and the graph's normal structure. This allows for more accurate and adaptive anomaly detection.

The talk also touched upon the challenges of trace assembly, including late spans, missing spans, sampling bias, and cold starts for new endpoints. To address these, the system leverages tools like OpenTelemetry for data collection, Kafka for streaming, and stream processing frameworks for real-time analysis.

Real-World Application and Benefits

Pandya shared an example of a benchmark experiment using millions of traces, where injecting anomalies helped train the system. The results indicated that this approach could significantly reduce the mean time to discovery for issues. By localizing problems within the execution graph, the system enables faster resolution, which is particularly critical in payment processing where speed and reliability are paramount.

The presentation concluded by highlighting the importance of clear labeling, defining appropriate time windows for analysis, and ensuring explainability of the data to make informed decisions. The overall goal is to create a more automated and efficient system for monitoring and maintaining the health of complex API ecosystems.