Trajectory's Ronak Malde on Scaling Continual Learning

Ronak Malde, founder of Trajectory, a platform focused on continual learning for AI, presented a compelling case for a new approach to AI development. Speaking at the AI Engineer World's Fair, Malde outlined the limitations of current AI scaling methods, which rely heavily on benchmarks that quickly saturate and become costly to maintain. He argued that the future of AI lies in its ability to learn continuously from real-world interactions, much like humans do.

The Benchmark Bottleneck

Malde began by illustrating the trend of rapidly scaling AI benchmarks over the past few years. While this has led to significant progress, he noted that these benchmarks are becoming increasingly time-consuming and expensive to train on. "We're seeing domains where it takes 4 hours, 6 hours, 24 hours or even several days in order to scale up benchmarks," Malde stated. More concerningly, he pointed out that these benchmarks are often "not tied to real world use cases where people are using AI."

The Trillion-Token Opportunity

In contrast, Malde highlighted the vast opportunity presented by the trillions of tokens generated daily through AI inference. This real-world data, capturing how models perform both well and poorly, should be the signal used for continuous learning. "This is actually how humans learn, right? We're continuously updating in the real world and getting smarter every single day," he explained. The field is increasingly recognizing this, with prominent AI experts and industry leaders like Ilya Sutskever, Andrej Karpathy, Satya Nadella, and Demis Hassabis all emphasizing the importance of continual learning.

Limitations of Current Algorithms

Malde then delved into the core problems hindering the widespread adoption of effective continual learning. He identified four key issues with current algorithms:

• Task distribution mismatch: Benchmarks often don't reflect real-world scenarios.
• Off-policy sampling: Methods learn from behavior that the model didn't actually generate.
• Rollout parallelism: Training requires massive infrastructure to replicate real-world environments, introducing bias.
• Sequence-level reward: A single score for an entire trajectory misses nuanced, per-token feedback.

He charted the evolution of these methods, from SFT (Instruction Fine-tuning) to DPO (Direct Preference Optimization) and GRPO (Generalized Reinforcement Learning Policy Optimization), noting that while each iteration has improved, none have fully solved these challenges.

Introducing On-Policy Self-Distillation (OPSD)

Malde introduced On-Policy Self-Distillation (OPSD) as a novel approach designed to overcome these limitations. The core idea is to use a model's own rollouts as the training data, guided by a 'teacher' model. In the case of self-distillation, when a superior teacher model isn't available, the model can leverage 'privileged information' or hints within its prompts to become its own teacher.

"You take what's called this hint, put it into the beginning of the prompt, and now you match the log props of the student without that hint to the teacher with that hint," Malde explained. This method, he asserted, "is an extremely powerful algorithm" that addresses the four key problems: it uses online task distributions, ensures on-policy sampling, requires only single parallel rollouts, and provides dense, per-token reward signals.

Scaling Challenges and Solutions

While OPSD shows great promise, Malde acknowledged the challenges encountered when scaling it up to larger models and longer tasks. He highlighted the "but wait" problem, where a model might get stuck in suboptimal loops, and the issue of "hint leakage," where privileged information might prematurely reveal the answer. To address these, Trajectory is developing techniques like step-level divergence weighting and residual guidance.

These advancements have allowed them to achieve significant improvements, as demonstrated by their work on a 12B parameter model for Mercor Apex Agents, which surpassed traditional RL methods. Malde concluded by emphasizing Trajectory's vision: "Agents that get better the more they're used." The company is building a platform to facilitate this continuous learning loop, turning every interaction into an opportunity for improvement.