# LLM Provenance: Tracking Data Origins with Graffiti _Daniel Chalef of Zep AI discusses the critical challenge of provenance in LLM-generated data and how the Graffiti framework addresses it through temporal graph modeling._ **Published:** 2026-07-23 **Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/llm-provenance-tracking-data-origins-with-graffiti --- In the rapidly evolving world of AI, Large Language Models (LLMs) are celebrated for their ability to synthesize information from diverse sources. However, as Daniel Chalef, Founder of Zep AI and Graffiti, explained at the AI Engineer World's Fair, this synthesis process often leads to a loss of provenance, the crucial trail of how data was generated and where it originated. This lack of traceability poses significant challenges for debugging, ensuring legal compliance, and building trust in AI-generated outputs. LLM Data SynthesisDriver LLMs combine diverse sources, creating new information not verbatim from inputsFrom the article 4 mentionsHowever, as Daniel Chalef, Founder of Zep AI and Graffiti, explained at the AI Engineer World's Fair, this synthesis process often leads to a loss of provenance, the crucial trail of how data was generated and where it originated.leads toProvenance LossDrivercritical trail of how data was generated and where it originated gets obscuredFrom the articleHowever, as Daniel Chalef, Founder of Zep AI and Graffiti, explained at the AI Engineer World's Fair, this synthesis process often leads to a loss of provenance, the crucial trail of how data was generated and where it originated.Challenges AriseOutcomeFrom the article 3 mentionsThis lack of traceability poses significant challenges for debugging, ensuring legal compliance, and building trust in AI-generated outputs.Graffiti FrameworkCoretemporal graph modeling addresses provenance by tracking data origins over timeFrom the article 8 mentionsTo tackle these challenges, Chalef's team developed Graffiti, an open-source temporal graph framework.Graph for ProvenanceContextframework uses a graph structure to map relationships and transformations of dataFrom the article 9 mentionsGraffiti models the relationships between source data and derived artifacts, such as facts, as a knowledge graph.Metadata ChallengesDriverprojection and deletion of metadata pose specific hurdles within the frameworkFrom the article 4 mentionsGraffiti also incorporates metadata projection, enabling the tagging of episodes at ingestion.Improved TraceabilityEffectGraffiti enables clear understanding of data lineage for AI-generated contentFrom the articleThis lack of traceability poses significant challenges for debugging, ensuring legal compliance, and building trust in AI-generated outputs.informsFuture DirectionsOutcomeongoing work explores advanced features and broader applications of the framework ## The Provenance Problem in LLM Outputs Chalef highlighted a common failure mode: LLMs generate summaries, extracted facts, and structured records that may not appear verbatim in their source inputs. This synthesis can obscure the origin of information, making it difficult to ascertain its veracity or troubleshoot errors. For instance, an LLM might confidently state a patient has a penicillin allergy, but the fact could be derived from a mix of electronic health records, lab reports, and user-inputted chat data. If this origin isn't clear, a doctor might be misled in a critical treatment scenario. The intuitive solution of simply storing a source ID with each fact breaks down in the context of LLM pipelines. Unlike deterministic data pipelines, LLM outputs are often synthesized from multiple, evolving sources. Merging entities (like 'J. Smith' and 'John Smith' into one identity) and mutating data over time means that simple pointers to a single source become unreliable. Chalef emphasized that lineage, in this context, needs to be an evolving set that survives mutation. ## Graffiti: A Graph Framework for Provenance To tackle these challenges, Chalef's team developed Graffiti, an open-source temporal graph framework. Graffiti models the relationships between source data and derived artifacts, such as facts, as a knowledge graph. This graph structure allows for tracing a fact back to its origin through a simple graph walk. The framework is designed to handle the complexities of LLM-generated data, including: - **Multi-source facts:** A fact can be linked to every episode that produced it. - **Entity resolution:** When entities merge, their source lineage is unified. - **Temporal invalidation:** The graph captures when new data supersedes old facts, marking them as invalid and noting the source episodes responsible for the mutation. ## Metadata Projection and Deletion Challenges Graffiti also incorporates metadata projection, enabling the tagging of episodes at ingestion. These tags, such as 'EHR' or 'verified clinical source,' are inherited by all derived entities and facts, allowing agents to filter information based on its source's reliability. However, when dealing with multiple sources contributing to a single fact, determining the overall veracity becomes complex. Chalef illustrated this with a scenario where a fact has three parent episodes, two verified and one not. The system must decide whether 'any' parent or 'all' parents need to be verified, depending on the criticality of the fact. For instance, an allergy fact might require 'all' parents to be verified, while a consent fact might be acceptable if 'any' parent is verified. Data deletion also presents a significant challenge, particularly with privacy compliance and 'right to be forgotten' requests. Graffiti's graph model helps by identifying which facts are derived from specific source data intended for deletion. If a fact is supported by multiple sources, it can survive the deletion of one source. However, if a fact is derived solely from deleted data, it is removed from the graph. This ensures that shared facts persist as long as at least one source remains, while single-source facts are correctly purged. ## Key Insights and Future Directions Chalef summarized the key takeaways for building reliable LLM-driven systems: - LLM graph construction is inherently lossy and generative; lineage must be engineered into the data structure, not merely logged afterward. - Sources should be stored verbatim, with every derived node and edge linked back to them. - This approach provides compliance, verification, debugging, and filtering capabilities. He also acknowledged that graph construction, especially with Graffiti's method, is computationally expensive but noted that significant effort has been invested in optimizing cost and latency. Chalef encouraged the audience to explore the Graffiti framework on GitHub. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.