TwelveLabs Builds Video Memory Layer

James Le, speaking at the AI Engineering World's Fair, introduced the concept of a 'memory layer' for video intelligence, arguing that current AI systems fail to truly understand video due to a lack of memory and a misapplication of language model principles.

The Limitations of Current Video AI

Le began by stating that most video AI systems today lack memory in a systemic sense. He highlighted that video is not merely a 'bag of frames' but a 'spatiotemporal volume' containing visual information, speech, sound, motion, OCR, camera changes, and more. The continuity and relationships within this volume are crucial for understanding, yet current approaches often treat video as a stack of images or a transcript, discarding this vital context.

He identified three core problems with existing AI stacks for video:

• Wrong Context: Forcing video into a text-based sequence by sampling frames or extracting transcripts loses the spatiotemporal relationships that define events.
• Wrong Memory: While text systems use retrieval, generation, and vector search for memory, video requires durable continuity, linking events across different files, camera angles, and time.
• Wrong Reasoning: Text-first models struggle to reason natively over motion, causality, and progression, elements that define video understanding.

Le emphasized that video's inherent properties, temporal dependencies, multimodality, density, ambiguity, and expense, make building memory for it particularly challenging.

TwelveLabs' Solution: A Video Memory Layer

Le introduced TwelveLabs' approach, which involves a stack starting with 'semantic chunks' representing meaningful temporal units. These are processed by a spatiotemporal embedding encoder called 'Morango,' generating vector embeddings. A 'spatiotemporal context store' preserves reusable structures like moments, entities, and metadata. Finally, their video context-aware language model, 'Pegasus,' acts as the reasoning layer.

He differentiated between search and memory, stating that search finds relevant moments, while memory preserves meaning and enables systems to answer more complex questions requiring entity timelines and cross-corpus evidence.

The Context Graph and Principles for Video Memory

Le proposed representing video collections as a 'context graph,' a durable, queryable representation connecting moments, entities, appearances, relationships, and corpus-level context. This graph allows applications to traverse video content from different perspectives and across time.

He outlined five key principles for building a video memory layer:

1. Ingest Once, Reason Many Times: Move expensive understanding into the preparation stage, similar to databases.
2. Store Primitives, Not Just Answers: Preserve moments, entities, appearances, and relationships for downstream workflows.
3. Ground Every Claim: Ensure evidence is traceable to specific timestamps within the video.
4. Let Intent Shape Memory: The memory layer should be configurable to suit different workflows and applications.
5. Keep the Layer Composable: Maintain an API-first approach, providing reusable layers for applications.

video workers and Harness Engineering

Le explained that while models provide answers, 'video workers' operate within a system, utilizing memory and evidence to plan tasks, retrieve relevant moments, and synthesize outputs. This worker paradigm requires specific capabilities such as memory access, task planning, retrieval, tool utilization, operating envelopes, output contracts, and evaluation.

Demos Showcasing Jockey's Capabilities

Le then demonstrated the capabilities of TwelveLabs' product, 'Jockey,' through three use cases:

• Sports Understanding: Identifying near-miss shots in World Cup footage, detailing the build-up and finish of goals, and tracking players like Lionel Messi across multiple videos.
• Security and Surveillance: Counting and classifying vehicles and pedestrians in traffic intersections, detecting safety events like red-light entries, and performing analysis in various conditions including rain.
• Advertising: Identifying moments in an Adidas commercial suitable for ad placement, such as slow-motion sequences, hero shots, and hard cuts.

These demos illustrated how Jockey can unlock applications in media and entertainment, public safety, commercial security, advertising, and the creator economy.

The Future of Video Intelligence

Le concluded by emphasizing that Jockey is TwelveLabs' effort to productize this video memory layer, offering knowledge stores, configurable ingestion, and an API. He stressed that it is not an application or editing platform but the underlying cognition infrastructure that enables these applications. Jockey is currently in private beta, and developers working on content assembly or organization workflows are encouraged to explore the platform.