Storage-derived lineage cannot run production debugging. LinkedIn Engineering says it rebuilt lineage around compute after storage signals blurred task mappings into a 24-way fan-in and fan-out.
I think the move settles a debate that catalog marketing kept vague for years, because lineage captured at read and write time cannot see projections, filters or column pruning. Compute can.
LinkedIn says tens of thousands of workflows over millions of datasets now emit tens of millions of events per day, with Spark, Trino, Flink SQL, Beam, Airflow and Flyte in the first instrumented set across more than 15 systems. Native engines emit through a lightweight SDK to one Kafka topic, then Flink builds an append-only log and a MySQL edge table for breadth-first traversal. For hot nodes the engine can return a partial result such as 5 expanded with thousands hidden, then expand on demand.
Each edge carries type, confidence and granularity, so the same graph serves dataset-to-dataset tracing, dataset-to-job debugging and workflow-to-workflow analysis, the company said. That choice also keeps VIEW and REFERENCE edges first class, which account for roughly 20% of production edges, per the company. It preserves policy context for governed views and resolves Hive tables to HDFS paths without forking graphs.
OvalEdge notes that DataHub, first built at LinkedIn, now bundles discovery, governance and lineage, while plain OpenLineage often gets outgrown and folded into a broader catalog. The fair concession is that compute instrumentation is costly and incomplete, and LinkedIn admits RDD jobs with arbitrary code get dataset-level lineage only plus lower-confidence declared or inferred fallbacks. It still beats storage inference because the confidence is explicit instead of hidden.
Authors Yang Yang, Alexander Ivaniuk and Siddharth Sodhani say the joint instrumentation, ingestion and retrieval rebuild rolled out across supported platforms in less than six months. If you run pipelines, don't trust lineage unless you know where it was captured and how confident it is.





