Arize shows how to grade agents with Phoenix

Laurie Voss, head of developer relations at Arize AI, demoed a full eval loop for a Claude agent using Phoenix in a session carried by AI Engineer that runs locally on localhost:6060 with no remote attacker required.

Phoenix collects OpenTelemetry traces via Open Inference, stores each agent step as a span with inputs, outputs, duration and token cost, and provides a UI, datasets and experiments on a laptop or a server.

Code evals check deterministic outcomes like ticker mentions with TypeScript functions via createEvaluator and logSpanAnnotations, and Voss found 15 passes and 2 failures on 16 reports where he warned "Looks good to me is not a test suite."

LLM judge evals grade meaning against a rubric, and Voss used Claude Sonnet 5.5 to judge an agent running on Claude Haiku, noting a built-in correctness check failed all 16 reports due to a January knowledge cutoff while faithfulness flagged 14 of 16 as unsupported.

Custom judges need role, explicit criteria, XML-wrapped data, labeled examples and constrained tool-call outputs, and Voss tested his actionability judge against 12 human-labeled examples to tune precision and recall before splitting data into dev and test sets.

Experiments rerun the same dataset with the same evals to measure prompt fixes, and an upgrade from Haiku 4.5 to Haiku 5.5, which supports a 1M token context window and up to 128k output tokens with adaptive thinking, improved faithfulness more than three times but kept it terrible while quadrupling cost and pushing the agent to say hold in 41 of 45 calls.

Phoenix locks gains with VEST and GitHub Actions where ticker mention is a guardrail and average actionability above 0.8 is a north-star target, and Voss said teams should start small by instrumenting and reading 12 traces.