AI Engineer host Xander Song, an engineer at Arize AI, introduced Harbor as an open-source framework for testing agents on real tasks with real data in fresh sandboxed environments.
Harbor treats evaluation like an experiment you can rerun. You define a task, a verifier is a script that checks for the intended side effects an agent should leave behind, and a reward is the score it returns, typically a binary pass or fail. Each task runs for a configurable number of trials to account for LLM stochasticity, each trial boots a new Docker container, and a collection of tasks becomes a dataset that maps to a job. There is also an oracle, a script that solves the task without an agent, so you can validate the plumbing first. Song said the framework is horizontally scalable and works with sandbox providers like Daytona, Modal and E2B, with an example of 200 concurrent workers, and verifiers stay agnostic of how the agent is built.
Tracing is where Arize's other open-source project comes in. Phoenix is the company's observability, experimentation and evaluation platform, and Song showed how Harbor's ATIF, a JSON-based trajectory interchange format, converts to the Open Inference Format, an OpenTelemetry-based standard for AI telemetry that Arize recently donated to the Cloud Native and OTel working group. Phoenix ingests trajectories via OTLP, maps Harbor concepts to datasets, experiments, trials and evaluations, and renders traces and metrics in a UI. Harbor also ships its own UI and hosted service, so Phoenix is optional.
The demo was concrete. Arize runs Harbor against its own hybrid system, which pairs coding agents like Claude Code with Opus 5 or Codex with GPT-5 plus its bespoke MCP server, skills, CLI and APIs, and its single-node Phoenix server that is easy to dockerize for tests. One test targeted inefficient aggregate queries. Agents were hammering REST and GraphQL for many tool calls, so the team added a SQL tool and nudged the prompt to favor it. Song showed the second experiment using SQL more frequently and more aggressively with the number of tool calls dropping, and a trace where a bash call that previously hit GraphQL was replaced by a direct SQL query. He summed up the goal as shipping based on evidence rather than vibes.
Evidence here still leans on handcraft. Song said the data is the hard part and the more realistic your data the better the system will run, yet Harbor does not emulate an end user's application, it seeds the database as if the app already created the data, and today that seeding is a one-off process reusing SQLite files or asking Claude to clean files before pushing to cloud storage. Regression testing is in early days and focused on happy paths. Skills, which describe workflows and act like end-to-end tests, create a maintainability trap, changing a skill name can flake tests, and Harbor's contract is just scripts in specific directories with no linters for coding agents. Even the demo's filtering, natural language to a query DSL via a local browser model, and the trick of asking the agent itself to read experiments to bootstrap data, point to how much curation a good verifier and dataset still demand. That is the bet: hold task and verifier constant while you swap the agent, but only if you did the hard work of making the task realistic.
