Wandero AI's Kalandadze on the 'Missing Layer' Post-Launch

Raphael Kalandadze, CTO of Wandero AI, recently shared insights into the critical, yet often overlooked, phase of AI agent development: the 'missing layer after launch.' In his presentation, Kalandadze emphasized that while rapid development and deployment are achievable, the real work begins once an agent is in production. He articulated that the true measure of an AI agent's success lies not just in its initial functionality, but in its ability to adapt and improve through continuous feedback and monitoring.

The "Easy Part" is Just the Beginning

Kalandadze highlighted the speed at which initial product development can occur, citing Wandero AI's own experience of shipping a product in just three weeks, involving approximately 300,000 lines of code and a budget of $35,000. He stated, "Shipping is fast now. That was the easy part." He then posed a crucial question: "How do you even know it's healthy?" This question sets the stage for the complexities that arise after deployment.

The challenge, as Kalandadze explained, is that AI agents, unlike traditional software with a defined set of features and buttons, are designed to handle a vast range of tasks and respond to almost any user input. This inherent flexibility, while powerful, also means that "almost anything can break." Traditional methods of testing and monitoring, which work for static software, fall short when dealing with the dynamic and often non-deterministic nature of AI agents.

The Hidden Pitfalls of Agent Operations

A significant problem Kalandadze identified is the 'invisible failure' of AI agents. Unlike traditional systems that might crash or display error messages, AI agents can fail subtly, even reporting success while the underlying system state indicates an issue. He referenced an example where a Claude agent marked features as 'complete' without verifying their actual functionality, and another instance where agents reported success while the system state contradicted this. This lack of overt failure signals makes it difficult for developers to identify and rectify problems. "Nothing crashes. Nothing turns red. Nothing knows." Kalandadze remarked, underscoring the challenge of maintaining visibility and control.

The sheer volume of interactions and tasks handled by AI agents further complicates monitoring. With thousands of conversations and potentially infinite tasks, manually tracking each interaction is impossible. Kalandadze pointed out that while developers can write code to simulate user interactions, the true test and the identification of unknown unknowns occur in the real-world production environment. This is where the crucial feedback loop must be established to understand how the agent is truly performing.

Leveraging Agents for Operational Insight

Kalandadze proposed a solution: using agents to monitor and manage other agents. He outlined a system with four operating agents:

• Log-Monitor: Detects and fixes issues rapidly (reactive).
• PR-Review: Gathers fixes and gates them (reactive).
• Session-Analyzer: Assesses the overall health of the system (reactive).
• QA / Computer-Use: Proactively tests the system and informs the roadmap.

He explained that these agents work in tandem to create a closed loop, where the output of one agent informs the actions of another. For instance, the log-monitor identifies issues, the PR-review agent processes the fixes, and the session-analyzer provides a holistic view of system health. This continuous feedback mechanism allows for near real-time system improvement.

Kalandadze emphasized the importance of providing these agents with comprehensive data, stating, "Don't give them a keyhole, give them the whole room. Give them all of it, or they're guessing." This means granting agents access to logs, trajectories, metrics, databases, and the live UI to enable them to understand the context and identify problems effectively. He shared an example of how his team used a log-monitoring agent to analyze production logs, identify an issue where the user was not being tracked correctly, and then automatically generate a pull request to fix it.

The Human in the Loop

The presentation also touched upon the critical role of human oversight in this process. While agents can automate many tasks, including identifying issues and drafting fixes, the final decision-making power rests with humans. Kalandadze cited a quote from Peter Steinberger, highlighting that "autoreview is the most impactful skill I've added to my stack... It automatically reviews your code before landing a PR. Finds so many edge cases. Sometimes it runs for hours." This illustrates the need for both automated analysis and human review to ensure accuracy and identify subtle problems.

Kalandadze concluded by reiterating the core message: "The model is the part you swap. The loop is the part you own." The true competitive advantage and the key to building robust, reliable AI systems lie in the operational harness and the continuous feedback loops that are established after the initial launch. This "missing layer" is where the real work of agent engineering takes place, ensuring that these powerful tools not only function but also evolve and improve over time.