Engram CEO on AI Memory & Context

In a recent episode of the Latent Space Cooking Show, Dan Biderman, co-founder and CEO of Engram, joined host Allen Park to discuss the challenges of AI memory and the limitations of long context windows. Engram is developing a novel memory layer for AI, aiming to help these systems truly understand and recall information within an organization, tackling the issue of "context rot" and enabling more efficient interaction with vast amounts of data.

From Special Forces to AI Research

Biderman shared his unique background, which includes a stint as an officer in naval special operations in the Israeli military, followed by studies in cognitive neuroscience and computational neuroscience. He explained how his military experience, which involved identifying and scaling unconventional solutions, fostered an entrepreneurial mindset crucial for founding an AI research company. His academic work focused on data efficiency and learning from limited examples, principles that underpin Engram's approach.

The "AI Memory Problem" and Engram's Solution

The core problem Engram addresses is the difficulty AI models have in retaining and reasoning over long, complex contexts. Biderman likened current LLMs to first-time kitchen visitors reading a cookbook for every dish, they can follow steps robotically but lack the intuition of an experienced chef. Engram's solution involves training models to "study" a corpus of knowledge, creating compact representations called "cartridges." These cartridges are designed to be a thousand times more compressed than textual representations, allowing models to operate with fewer tokens, be less confused, and achieve higher accuracy.

Biderman elaborated on the concept of "context rot," where models become less accurate as the amount of context increases. He explained that Engram's approach, which involves training models to internalize knowledge within their weights, aims to overcome this limitation. This method allows for more efficient and intuitive interaction with data, enabling AI to move beyond simply recalling information to developing a deeper understanding and even innovating, much like a human expert.

Scaling Knowledge and Future Ambitions

Looking ahead, Biderman anticipates that companies will soon grapple with knowledge workspaces containing trillions of tokens of proprietary data. He believes that at this scale, current methods like Retrieval-Augmented Generation (RAG) and even models with massive context windows will struggle. Engram's focus on creating these "cartridges" of knowledge, which are task-specific and corpus-specific, aims to provide a more scalable and efficient solution.

The long-term ambition for Engram is to enable every person to have a personalized AI model that learns from their specific knowledge and expertise. This ultimate form of continual learning, Biderman suggests, might eventually run on personal devices, fundamentally changing how individuals interact with and leverage AI.

Balancing Intelligence and Efficiency

Biderman emphasized that intelligence and efficiency are not mutually exclusive. He argued that doing "more with less" allows for tackling more ambitious tasks and longer-term problems. While the current AI paradigm has relied on "more with more" scaling, he believes the next phase will involve a greater emphasis on efficiency, a principle that Engram is built upon.

The conversation touched upon the potential for models to learn and discern valuable feedback, the importance of getting out of the model's way, and the ongoing research into token efficiency and model routing. Biderman also highlighted the need for robust infrastructure to support the deployment of these advanced AI systems, particularly for managing millions of endpoints and balancing AI workloads.