Coinbase tests when agents can run alone

Evan Kormos, player coach at Coinbase, told AI Engineer that an agent earns unsupervised work only after it clears a measured policy threshold, not after a demo. Kormos, who said he has spent 20 years building software and 15 in enterprise financial services, framed policy as the original system prompt.

He separated the work into two moments in time, Kormos said. Policy defines how work must, may and should happen for risk and compliance, while the running system evolves constantly with product changes. A compilation event binds the two, but build state and run state stay distinct, so everything expected to remain roughly bound by policy can still be checked in operation.

Agents, in his model, cut across people, process and technology, Kormos said, but the talk focused where agents meet process. Workflows should be mapped as a directed acyclic graph, or DAG, that moves forward without looping. He cited Apache Airflow as open source software that represents those dependencies in Python, and listed policy-driven examples including commercial lending, listing a token in a specific region, and regulated software delivery.

The hard part lives inside the node.

Inside each node the flow is cyclic, Kormos said. Work is done, checked, and refined until it is good enough to pass, a directed cyclic graph or DCG that he said is best managed with state tools like LangChain LangGraph. The outer flow stays deterministic and bounded by policy, while the inner loop is adaptive. He warned that exception flows must exist so work does not get stuck, and that adding agents shifts volume across the DAG, which demands operational visibility more than better authoring tools.

Quality then hinges on the check, not the loop, Kormos argued. Higher quality inputs lower compute cost for better outputs, and injecting DAG state can give an agent trusted context and long horizon goals, but the check must define what good looks like in a reliable, repeatable way. Traditional confidence came from unit tests, he said, while probabilistic agents need representative data sets, repeated evaluations and statistical analysis, plus traces as runtime logs. Without that, more passes do not mean better results.

That check decides where an agent belongs, Kormos said. For each node he asked whether the work can be checked against policy easily, run in a bounded loop, be traced, and fall back to a human. If yes, automation is more plausible. He applied it to lending, with KYC and financials in parallel then credit memo, risk review, documentation and decision in sequence, and said the final decision should stay human-driven as the policy check that is harder to hill climb.

Kormos laid out a three stage turn-on, and gave no numeric error rates, volumes or latency to anchor it. Day one is supervised with humans verifying all outputs and refining data sets with subject matter experts, he said. When measured confidence reaches the policy threshold the team moves to sampling a slice of outputs with systematic evaluation of traces, then to handling exceptions only while humans keep refining data sets. The recipe is deterministic on the outside and adaptive on the inside, he said, and what would change the read is a published threshold, sample size and failover rate from a live Coinbase flow. Until then the open question is what statistical bar actually clears a token listing versus a loan.