A single run of a modern benchmark can cost $50,000 to $100,000, Wolfe said on AI Engineer, and still leave you guessing because the task set is small and the agent is non-deterministic. That price is the anchor for his talk on why evaluation is noisy by design, not by accident.
The punch is how quickly that noise doubles. Most public evals Wolfe described rely on a deterministic grader, string matching or test cases for math, where pass or fail is a function. Move to a Netflix-style subjective task like adapting a book into a film, and that grader becomes another non-deterministic LM judge or task-specific rubric, so you are sampling from a model and from an evaluator.
Cameron Wolfe, staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter per daily.dev, laid out a three-part framework to manage it: a calibrated human evaluation as the North Star, offline automatic judges tuned to that human signal, and production monitoring. Human guidelines define a handful of criteria with binary or Likert ratings and extensive examples, then multiple raters are compared and the guidelines are rewritten until agreement rises. Offline judges get the same guidelines, or a checklist rubric like the Dr. Tulu example he cited, and are meta-evaluated for correlation or accuracy against golden human data. Wolfe said Netflix in one project showed judge scores were causally correlated with streaming signals like abandonment, and suggested using human-human agreement as the target for judge-human agreement.
The numbers that would let you price that pipeline were not in the talk. Wolfe gave the $50,000 to $100,000 benchmark cost to explain why eval sets stay small, but did not share Netflix sample sizes, the power analysis inputs he recommends for sizing an eval, or the abandonment correlation coefficient. He described statistical controls, power analysis for sample size, confidence intervals and resampling of both model and judge rollouts to tighten variance, without attaching a figure to how much variance remains after those steps.
What would change the read is a single reported interval. If Netflix published the inter-annotator agreement it achieves on a creative writing or adaptation rubric, the judge-human agreement it considers acceptable for that domain, and the width of the confidence band around a claimed win in offline evals, you could judge whether the judge is a cheap proxy or a noisy mirror. Until then the framework is persuasive, but its economy is opaque, human eval is a bottleneck he acknowledges and the cure inherits the disease it measures.
Does a tighter offline interval survive contact with viewers whose tastes drift, or does production AB testing simply become the only eval that matters?
