Technology

LinkedIn says storage lineage misleads

LinkedIn rebuilt data lineage around compute query plans to fix false mappings and support compliance and incident response.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Data pipeline lineage graph showing compute-based dependencies
Credit: LinkedIn Engineering

Storage-derived lineage cannot run production debugging. LinkedIn Engineering says it rebuilt lineage around compute after storage signals blurred task mappings into a 24-way fan-in and fan-out.

I think the move settles a debate that catalog marketing kept vague for years, because lineage captured at read and write time cannot see projections, filters or column pruning. Compute can.

LinkedIn says tens of thousands of workflows over millions of datasets now emit tens of millions of events per day, with Spark, Trino, Flink SQL, Beam, Airflow and Flyte in the first instrumented set across more than 15 systems. Native engines emit through a lightweight SDK to one Kafka topic, then Flink builds an append-only log and a MySQL edge table for breadth-first traversal. For hot nodes the engine can return a partial result such as 5 expanded with thousands hidden, then expand on demand.

Each edge carries type, confidence and granularity, so the same graph serves dataset-to-dataset tracing, dataset-to-job debugging and workflow-to-workflow analysis, the company said. That choice also keeps VIEW and REFERENCE edges first class, which account for roughly 20% of production edges, per the company. It preserves policy context for governed views and resolves Hive tables to HDFS paths without forking graphs.

OvalEdge notes that DataHub, first built at LinkedIn, now bundles discovery, governance and lineage, while plain OpenLineage often gets outgrown and folded into a broader catalog. The fair concession is that compute instrumentation is costly and incomplete, and LinkedIn admits RDD jobs with arbitrary code get dataset-level lineage only plus lower-confidence declared or inferred fallbacks. It still beats storage inference because the confidence is explicit instead of hidden.

Authors Yang Yang, Alexander Ivaniuk and Siddharth Sodhani say the joint instrumentation, ingestion and retrieval rebuild rolled out across supported platforms in less than six months. If you run pipelines, don't trust lineage unless you know where it was captured and how confident it is.

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsLinkedInSparkTrinoFlinkAirflowDataHubOpenLineageMySQL

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.