Harvey Labs: Building AI Research on a Budget

In a world where frontier AI labs command billions in funding, application-layer companies face a daunting challenge in building their own advanced AI capabilities. Harvey, a company specializing in AI for legal and professional services, has developed a high-level playbook for establishing a research lab on a budget, as outlined by co-founder and president Gabe Pereyra. Pereyra, who previously worked at DeepMind and Meta AI, shared Harvey's strategy for competing effectively in the rapidly evolving AI landscape.

The "Unfair Game" and Leveraging the Frontier Ecosystem

Pereyra opened by acknowledging the inherent disadvantage application-layer companies face against well-funded frontier labs, which possess greater financial resources, talent, compute infrastructure, and data. "There are rich teams, there are poor teams, and then there's us in the application layer," he stated. However, he emphasized that by strategically utilizing the existing "frontier ecosystem," companies can indeed compete and build "frontier intelligence."

Harvey's Playbook: Benchmarks, Post-Training, and Production Serving

Pereyra detailed Harvey's three-step playbook for building a research lab:

• Build a benchmark: Pereyra stressed that without a solid benchmark, model training and subsequent production serving are impossible. Harvey has released several datasets, including Legal Agent Bench (LAB) for legal associate tasks, LAB Contracts for negotiation training, and LAB Diligence, an expansive RL environment designed for long-context, complex tasks.
• Use it to post-train models: The core of this step involves using the custom benchmarks to refine open-source models.
• Serve with inference providers: This final stage focuses on deploying these refined models into production environments.

Synthetic Data and Domain Experts

A significant challenge for Harvey, working with highly sensitive and privileged legal data from top law firms, is the inability to train on customer data. The breakthrough came through using domain experts to guide synthetic data generation. Pereyra likened it to how engineers now "vibe code" with coding models. Harvey's legal researchers, including Pereyra's own brother who is a lawyer at the firm, are trained to use AI tools to generate realistic datasets that mimic real-world scenarios. Companies like Mercor and Snorkel are then used to scale this process.

Scaling with Neo Labs and Open Source Models

Pereyra highlighted the increasing competitiveness of open-source models like Kimmi 3, GLM 5.2, and NeMo-Megatron. He recommended leveraging "neo labs", specialized teams or companies with expertise and infrastructure, to help with the post-training process. Harvey has partnered with various providers, including Fireworks, Base10, and Trajectory, to fine-tune these models for specific tasks. He noted that working with multiple neo labs allows Harvey to learn from different research approaches and model bets, and that the ease of post-training is rapidly increasing.

The Model Serving Infrastructure

Deploying models in production is a non-trivial task, especially for a company operating in 60 countries with diverse customer needs and model preferences. Pereyra outlined Harvey's sophisticated model serving infrastructure, which handles multiple model families, fallbacks across providers to meet SLAs, and the integration of open-source models. The decision to put a model into production, and to keep it there, is based on a combination of generic evaluations (like the LAB benchmark), human testing, critical user journeys, automated product tests, and heuristic signals like cost and latency.

The Post-Training Flywheel

Pereyra emphasized the importance of establishing a "post-training flywheel," where serving models in production and collecting feedback (without training on customer data) informs future dataset development and model improvements. He also touched upon the strategy of starting with "naive model swaps" and routing, where simpler tasks can be handled by open-source models, and then gradually incorporating more complex routing strategies.

The Future: Every Company as an AI Company

Pereyra concluded with a powerful analogy from Moneyball, suggesting that by winning on a budget and leveraging the frontier ecosystem, application-layer companies can fundamentally "change the game." He believes that in the future, every company will need to become an AI company and adopt similar playbooks. The key, he asserted, is to be strategic and utilize the available resources effectively.