AI Agents Need Enterprise-Ready Tech Stacks

The promise of AI agents in enterprise settings is immense, but bridging the gap from proof-of-concept (POC) to production-ready deployment presents significant challenges. Chris Lovejoy, a member of technical staff and forward-deployed engineer at Anterior, and Saul Howard, VP of Engineering at Anterior, recently highlighted these hurdles at the AI Engineer World's Fair, explaining why current enterprise tech stacks often fall short and what architectural primitives are needed to unlock the full potential of AI agents.

The POC Pitfall

Lovejoy and Howard, both with extensive experience in enterprise and healthcare AI deployments, began by illustrating a common scenario: a successful AI POC. This typically involves identifying a use case, scoping the project, defining metrics, and then building a functional prototype with a small engineering team over a few weeks. The POC might connect to various application layers, control planes, and data planes, including LLM APIs and data sources like data lakes.

While these POCs often yield promising results, the speakers emphasized that the real challenge lies in scaling these solutions for production. The transition to production frequently uncovers critical requirements that were overlooked during the initial development phase.

Enterprise Demands: The Unseen Hurdles

The transition from a successful POC to a production-ready application often reveals a series of complex questions from stakeholders. Common concerns include:

• Audit Trails: Security teams require a complete, justifiable record of every action an agent takes, including data accessed and authorizations used. This goes beyond standard developer logs and needs to meet stringent compliance standards like SOC 2, HITRUST, and HIPAA.
• Sensitive Data Handling: Compliance officers need assurance that sensitive data, such as Protected Health Information (PHI), is handled according to strict legal restrictions. Agents, like humans, should only access data they absolutely need for a specific task.
• Decision Approval and Escalation: Mechanisms for human oversight, especially for critical decisions, are necessary. The system must accommodate scenarios where an agent's output needs to be escalated to a human for approval.
• Model Performance and Integrity: Ensuring the agent continues to perform reliably over time and protecting against manipulation by untrusted data are key concerns.
• Integration: Connecting AI agents to existing enterprise systems like Salesforce or healthcare platforms like Epic is vital for real-world utility.

Architectural Primitives for Success

To address these challenges, Lovejoy and Howard proposed several architectural primitives that are essential for building robust AI agent systems in regulated environments:

1. The Immutable Ledger of Actions

Adopting a transaction log, or event sourcing pattern, provides an immutable, append-only, timestamped record of all agent actions. This serves as the single source of truth, ensuring complete auditability. While this pattern can make writes easy, reads might require more effort, but techniques like caching and snapshots can mitigate this. The benefit of this approach is that it allows for easy replay of events, enabling developers to understand agent behavior and compare variations in prompts, models, or code.

2. Orchestration-Adjacent Object Storage

To manage sensitive data, particularly PHI in healthcare, the speakers recommended using segregated object storage. The orchestrator's event log would then contain only metadata and pointers to this data. This separation allows developers to see the shape of the data for debugging purposes without directly accessing the sensitive information. It also facilitates the application of zero-trust principles, where agents use tokens to access data only at the point of use, preventing data from flowing freely throughout the system. This segregation is also key to preventing prompt injection attacks.

3. Human-Agent Equivalency

For dynamic escalation, the concept of defining an "agent" to encompass both LLMs and humans is proposed. This ensures that any action an LLM can perform, a human can also perform. This is beneficial because it allows for seamless handoffs at any point in an agent's workflow. Furthermore, it simplifies context management, enabling shared definitions of context that can be mapped to either agent-friendly prompts or human-friendly UIs.

4. Primitive-Derived Evals

The architectural primitives discussed can lead to effective evaluation as a natural byproduct. The immutable ledger allows for replaying actions and comparing outcomes with modified prompts, models, or code. The human-agent equivalency enables direct comparison of agent and human performance on the same tasks. Orchestration-adjacent object storage allows for running evaluations on production data within customer environments without exposing sensitive information. These capabilities collectively support continuous evaluation and improvement.

The Takeaway: Build for Production from Day One

Lovejoy concluded by stressing the importance of taking the constraints of a production-ready, scaled enterprise system seriously from the beginning. He cautioned against the common mistake of building a POC and then trying to "strap on" enterprise requirements later, which often results in brittle systems. Instead, the focus should be on designing architectures with these requirements in mind from the outset, using existing, proven concepts adapted for the new world of agentic AI.