LLM Provenance: Tracking Data Origins with Graffiti

In the rapidly evolving world of AI, Large Language Models (LLMs) are celebrated for their ability to synthesize information from diverse sources. However, as Daniel Chalef, Founder of Zep AI and Graffiti, explained at the AI Engineer World's Fair, this synthesis process often leads to a loss of provenance, the crucial trail of how data was generated and where it originated. This lack of traceability poses significant challenges for debugging, ensuring legal compliance, and building trust in AI-generated outputs.

The Provenance Problem in LLM Outputs

Chalef highlighted a common failure mode: LLMs generate summaries, extracted facts, and structured records that may not appear verbatim in their source inputs. This synthesis can obscure the origin of information, making it difficult to ascertain its veracity or troubleshoot errors. For instance, an LLM might confidently state a patient has a penicillin allergy, but the fact could be derived from a mix of electronic health records, lab reports, and user-inputted chat data. If this origin isn't clear, a doctor might be misled in a critical treatment scenario.

The intuitive solution of simply storing a source ID with each fact breaks down in the context of LLM pipelines. Unlike deterministic data pipelines, LLM outputs are often synthesized from multiple, evolving sources. Merging entities (like 'J. Smith' and 'John Smith' into one identity) and mutating data over time means that simple pointers to a single source become unreliable. Chalef emphasized that lineage, in this context, needs to be an evolving set that survives mutation.

Graffiti: A Graph Framework for Provenance

To tackle these challenges, Chalef's team developed Graffiti, an open-source temporal graph framework. Graffiti models the relationships between source data and derived artifacts, such as facts, as a knowledge graph. This graph structure allows for tracing a fact back to its origin through a simple graph walk. The framework is designed to handle the complexities of LLM-generated data, including:

• Multi-source facts: A fact can be linked to every episode that produced it.
• Entity resolution: When entities merge, their source lineage is unified.
• Temporal invalidation: The graph captures when new data supersedes old facts, marking them as invalid and noting the source episodes responsible for the mutation.

Metadata Projection and Deletion Challenges

Graffiti also incorporates metadata projection, enabling the tagging of episodes at ingestion. These tags, such as 'EHR' or 'verified clinical source,' are inherited by all derived entities and facts, allowing agents to filter information based on its source's reliability. However, when dealing with multiple sources contributing to a single fact, determining the overall veracity becomes complex. Chalef illustrated this with a scenario where a fact has three parent episodes, two verified and one not. The system must decide whether 'any' parent or 'all' parents need to be verified, depending on the criticality of the fact. For instance, an allergy fact might require 'all' parents to be verified, while a consent fact might be acceptable if 'any' parent is verified.

Data deletion also presents a significant challenge, particularly with privacy compliance and 'right to be forgotten' requests. Graffiti's graph model helps by identifying which facts are derived from specific source data intended for deletion. If a fact is supported by multiple sources, it can survive the deletion of one source. However, if a fact is derived solely from deleted data, it is removed from the graph. This ensures that shared facts persist as long as at least one source remains, while single-source facts are correctly purged.

Key Insights and Future Directions

Chalef summarized the key takeaways for building reliable LLM-driven systems:

• LLM graph construction is inherently lossy and generative; lineage must be engineered into the data structure, not merely logged afterward.
• Sources should be stored verbatim, with every derived node and edge linked back to them.
• This approach provides compliance, verification, debugging, and filtering capabilities.

He also acknowledged that graph construction, especially with Graffiti's method, is computationally expensive but noted that significant effort has been invested in optimizing cost and latency. Chalef encouraged the audience to explore the Graffiti framework on GitHub.