AI Engineer brought on SallyAnn DeLucia, Head of Product at Arize, to walk through writing a first AI eval from scratch. DeLucia, four years at Arize and builder of its agent Alex and its evals product, framed evals as the primary signal for agents and used the moment to introduce Decision Judges, a new eval type tied to OpenAI's decision model API released about two weeks prior.
Vibe checks do not scale, she said.
Without evals you cannot catch regressions, compare prompts or models objectively, or put quality gates in CI, DeLucia argued. She drew a hard line between observability and testing. Traces are the logs for AI, evals are the actual tests. A trace is every step to an answer, a root span holding input and final output plus child spans for each unit of work like agent, tool and LLM calls. You read logic from traces, not code, because agents have many valid outputs. Arize instruments that with Open Inference and surfaces it in Prompt Playground, Agent Playground and Signal, its managed agent that finds issues from evals.
The taxonomy is familiar but the new piece is the pitch. DeLucia listed LLM as a Judge, which returns a label and explanation and fits QA or relevance checks, Decision Models which are cheap and fast and return a probability and a true or false with no explanation, code-based checks for deterministic rules like JSON format or Rouge and Blue scores, annotations on golden or synthetic datasets, Agent as a Judge for complex criteria, plus user feedback and business metrics in production. In practice she said to use the first four across the lifecycle. Her rule was judge one quality per eval for LLM as a Judge, because research shows single-criterion judges perform better, while Decision Models and Agent as a Judge can carry multiple. That cheap speed comes with a blind spot for security teams. A gate that fails a deployment but cannot explain why does not help triage prompt injection, data exfiltration or tool misuse. Calibration matters, and outside this talk, a 2026 calibration guide recommends testing judge position and verbosity bias and scoring Cohen's kappa against gold labels before trusting a judge in CI.
The build she demoed was concrete and small by design. Define the task and what success and failure look like in specific bullets, not vague helpfulness. Pick the relevant spans from traces and drop them into a dataset. Write the template in Prompt Playground with an intro, the data, the pass and fail particulars and labels like correct, partially correct or not applicable. Iterate. Her team started Alex with a handful and now runs 32. She said a specific eval beats a generic one, but any eval beats none, and Eval Hub templates are fine to start. DeLucia also noted Arize's Alex can draft and iterate the prompt itself. For teams shipping agents, the workflow breaks at the dataset step. Twenty traces and one dimension is a clean demo. Production needs coverage, human gold and reviewers from product, engineering, support and security to agree on the bullets, plus alignment by comparing judge labels to human annotations.
Metrics tell you something might be broken, DeLucia said. Evals tell you the failure mode. That holds only if the judge you picked can see the mode you care about and you can prove it. Decision Judges without rationales and LLM judges without bias checks leave a gap right where you would want an online eval to block a bad tool call before it ships.
