A one-line prompt change to require confirmation before any action lifted action safety by 17 points and cut task completion by 17 points, net zero, in the refund-agent demo that AI Engineer featured with forward deployed engineer Sofia Jakovcevic of Arize. She used the trade-off to argue that agent regressions do not show up in a code diff, only in runtime traces.
The point is architectural. Traditional software fails at the code level where a reviewer can see it, Jakovcevic said, while agents fail at runtime across branching trajectories, tool calls and handoffs between sub-agents. The diff showed a single line in a markdown prompt file, the behavior changed in production, and as she put it, the diff tells you what changed but the trace tells you what actually happened.
The fix she outlined is a five-step regression loop built on traces. First, select a dataset pulled from real production traces that mixes representative workflows with known failures, edge cases and successful runs you want to protect. Second, define scoring with evaluators that turn subjective dimensions into numbers. Jakovcevic described LLM-as-judge for non-deterministic checks like hallucination, task completion and step efficiency, alongside code-based evaluators for deterministic checks, often as binary 0/1 or 0 to 5 scores. Third, run a baseline experiment by replaying the dataset through the agent and scoring every output. Fourth, apply the change and rerun to compare. Fifth, automate the comparison as a gate in CI, so a pull request in GitHub Actions does not merge until it meets the regression criteria. All without deploying to production, she said.
In the Arize platform the loop has two surfaces, Jakovcevic showed. Prompt Playground tests a single agent by loading a prompt version from the prompt hub, attaching tools and model config, duplicating to test variants side by side, adding evaluators like task completion and action safety, and running across a 100-row dataset to collect latency, cost, error rate and eval scores. Agent Playground tests multi-agent systems by connecting a remote agent over HTTP, so each dataset record calls the live agent, captures the full trace, and scores it the same way. The same flow also runs programmatically via SDK, REST or CLI, and Arize is built for modern AI applications including chatbots, RAG systems, copilots and agents, according to Arize, which is consistent with the trace-centric workflow shown here.
Her closing turn was about who closes the loop. Takeaway one, she said, is that every failure should become a test, with failed production traces promoted into the regression suite so the suite compounds. Takeaway two is speed. Manual triage of dashboards is the bottleneck, and Arize publishes agent skills that connect coding agents in the IDE to the observability backend so agents can investigate traces, propose fixes and validate them against the same evaluations. That is the shift she described from observability for humans to observability for agents, where the best agents fix themselves.
The numbers also set the limit. The best trade-off in her final comparison reached plus 18 points on action safety and plus six on task completion, Jakovcevic said, versus the flat net zero of the first one-line fix. The data does not show how those scores generalize beyond the curated 100 cases or what thresholds the team would enforce in CI. Change the dataset mix or the judge prompts and the read on ROI for a prompt or model change moves with it. How many teams will let an agent promote its own patch when the gate itself is only as good as the evaluators they wrote?
