Artificial Intelligence

LangChain CEO on Building Better AI Agents

LangChain CEO Harrison Chase discusses the essential components of AI agents, the importance of custom harnesses, and the power of evals and observability for continuous improvement.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

4 min read

Harrison Chase, CEO of LangChain, presenting on Evals & Harness at a Sovereign AI event.
Credit: Sequoia Capital
Visual TL;DR
AI Agent ComponentsContext
From the article 9+ mentionsSpeaking at a Sovereign AI event, Chase emphasized that to truly "own your intelligence," developers need to have control over the three core components of an agent: the model, the context, and the harness.
Own IntelligenceEffect
From the article 2 mentionsSpeaking at a Sovereign AI event, Chase emphasized that to truly "own your intelligence," developers need to have control over the three core components of an agent: the model, the context, and the harness.
The HarnessCore
orchestrates agent behavior, central to building intelligent systems
From the article 8 mentionsFinally, the harness, which Chase highlighted as the central focus, is responsible for orchestrating the model and context.
Custom HarnessesEffect
crucial for avoiding lock-in and leveraging best available technology
From the article 6 mentionsChase addressed the common question of when to build a custom harness versus using an off-the-shelf solution.
Evals & ObservabilityCore
power continuous improvement and refine agent behavior over time
From the article 7 mentionsThe second major pillar of Chase's talk was the importance of evals and observability.
Data FlywheelContext
evals and observability create a feedback loop for automated improvement
From the article 2 mentionsChase concluded by outlining a "recipe for continuously improving agents": build a v1 agent, collect traces, curate trace data, and run experiments on that data.
Automate ImprovementOutcome
leverage data flywheel to automatically enhance agent performance and intelligence
From the article 2 mentionsHe also highlighted the potential to automate this process, introducing LangSmith Engine, an agent built on top of traces that can perform data curation and suggest fixes.
Contents(5)

Harrison Chase, co-founder and CEO of LangChain, recently shared his insights on building and refining AI agents, focusing on the critical roles of "harnesses" and "evals" in developing truly intelligent systems. Speaking at a Sovereign AI event, Chase emphasized that to truly "own your intelligence," developers need to have control over the three core components of an agent: the model, the context, and the harness.

LangChain CEO on Building Better AI Agents - Sequoia Capital
LangChain CEO on Building Better AI Agents, Sequoia Capital

Understanding Agent Components

Chase broke down an agent into three main parts. First, the model itself, which can be an open-weight model or a proprietary one, with the ability to switch models being crucial for avoiding lock-in and leveraging the best available technology. Second, the , which includes memory, semantic knowledge retrieved through methods like RAG, and previous conversational history, all of which help personalize and guide the agent's behavior. Finally, the harness, which Chase highlighted as the central focus, is responsible for orchestrating the model and context. The harness's primary job is to deliver the right context to the model at the precise moment it's needed, enabling the agent to perform complex tasks.

The Role of the Harness

The harness acts as the conductor, managing the flow of information and ensuring the agent can interact with external systems, process their outputs, and feed them back into the loop. At its simplest, an agent is an LLM running in a loop, calling tools. However, Chase explained that harnesses allow for significant customization. He illustrated this with LangChain's own minimal harness and compared it to "Deep Agents," a more generalized version. Customizations can be made through "middleware constructs" or "hooks and plugins" that modify the core loop. These allow for actions like running code snippets before agent invocation, wrapping model or tool calls, and managing context through summarization or offloading. These elements enable agents to access sandboxes, file systems, memory, and more, tailoring them to specific domains.

Customization vs. Off-the-Shelf Harnesses

Chase addressed the common question of when to build a custom harness versus using an off-the-shelf solution. He noted that general-purpose harnesses are often sufficient for basic tasks, especially when starting. However, as use cases become more specialized or "out-of-distribution," customizing the harness becomes more important. He used the example of "legal AI," where specific tasks like editing files are well-represented in model training, but the broader task requires a custom harness. LangChain's "Deep Agents" utilizes "model profiles" to switch between different implementations of tools like "edit file" based on the model being used, demonstrating a pragmatic approach to customization.

Evals and Observability: The Data Flywheel

The second major pillar of Chase's talk was the importance of evals and observability. He quoted Satya Nadella, emphasizing the need to "create your private evals, because evals define what 'good' looks like inside the organization." This includes retaining ownership of organizational memory, traces, feedback, decisions, and institutional context. Evals are crucial for defining benchmarks, catching regressions, and "hill climbing" on performance by adjusting models or harnesses.

Chase highlighted Harbor, an open-source eval runner, as an emerging industry standard. Harbor allows users to run agents against datasets, with each task defined by an environment (e.g., a Dockerfile), an instruction (in Markdown), and an evaluation script. The results can then be visualized and compared, providing valuable insights into agent performance. He also touched upon observability, stressing its underrated importance for debugging agents. When agents fail, it's often due to issues with the model or, more commonly, the context provided. Detailed observability into the context window, step execution, and tool usage is vital for effective debugging.

Automating Improvement

Chase concluded by outlining a "recipe for continuously improving agents": build a v1 agent, collect traces, curate trace data, and run experiments on that data. He also highlighted the potential to automate this process, introducing LangSmith Engine, an agent built on top of traces that can perform data curation and suggest fixes. This automated approach aims to streamline the development cycle and accelerate agent improvement, ultimately leading to more capable and reliable AI systems.

Mentioned in this story

TopicsHarrison ChaseLangChainAI AgentsHarness EngineeringAI EvalsObservabilityLLMRAG

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.