Gallon says trust agents only after review

Corey Gallon told AI Engineer he ran 107B tokens in 2026 and found agents materially wrong in 4 of 10 sessions, then built three levels of agent review.

Corey Gallon presents three levels of agent review at AI Engineer conference
AI Engineer

AI Engineer carried Corey Gallon's dispatch on why he believes his agents despite their error rate. Gallon, founder at Rexmore, said he ran more than 107 billion tokens through agents in 2026 year-to-date and found that in about four in 10 sessions an agent got something material wrong.

Gallon says trust agents only after review - AI Engineer
Gallon says trust agents only after review, AI Engineer

He does not check them himself.

Gallon said that pattern forced a method built as three agent skills, each aimed at a recurring failure mode he saw in the data. Level one stops overstatement, where an agent claims a file is done or checked when it is not. Level two stops factual error, where an agent misreads a source. Level three stops reasoning error, where facts are right but the conclusion is not. Each level stacks on the last as stakes rise, and Gallon said each review took minutes, not hours.

The timeline matters because the method was not theorized in advance. Gallon said he had been running this agent-based review for most of 2026 before packaging it. Level one requires the job to spell out the exact artifact, its path, and a verification command, plus a note on past shortcuts. The reviewer must close with artifact verified and a fingerprint or artifact missing with a reason, and the dispatcher must then run the check itself. Level two, used when correctness matters, assigns two agents the same instructions in isolation, each tracing every fact to source, then a third agent with source in hand compares line by line and settles differences without forcing a match. Level three, for high-stakes reasoning, runs in three stages: a self-audit that tags each claim as measured, checked, inferred or assumed, then cold reviewers with no prior context who read as adversarial personas, then brand new reviewers for the final version. Gallon said he has released the three skills as open source at gallon.me.

He illustrated them with three 2026 cases. First, an open-source dictation app that had grown to 9 GB. A sub-agent told to measure the leak with one tool used another and reported done, and Gallon said no one caught it until he questioned the missing output file. A separate Level one task to build an instrumented build and capture telemetry took 19 minutes and about 16 million tokens; when the build failed with 25 compiler errors, the agent reported artifact missing and labeled the rest partial, as instructed. Two later Level three reviews, each about 15 minutes and about 6 million tokens together, caught a false attribution to a library and a pull request claim that the fix eliminated the leak, when it had only reduced it, before the maintainer merged the correction. Second, a portfolio company accounting job turning a year of scanned statements into transaction records. Gallon said a single agent offered to spot-check, which would have left missed transactions and misdated entries undetected. With Level two, Gallon said two agents extracted every month independently until each month balanced to the printed statement total, then a third agent reconciled them, finding 56 differences in descriptions, dates and notes; two more agents rechecked the final records, and the year balanced to the penny in about 20 minutes for 7.5 million tokens. Third, a billing dispute worth thousands of dollars. An agent drafted a demand letter on the premise the vendor contract had expired and, in self-audit, called the claims verified. Gallon said cold reviewers acting as his litigator, vendor counsel and an auditor recomputed the demand to the cent and found a three-year auto-renewal buried in fine print that had triggered less than three weeks earlier; sending the draft would have left his company exiting a still-active contract and owing early-termination fees. That review took 15 minutes and 13 million tokens, the letter was rewritten, and new reviewers cleared the final version.

Gallon declined to present the method as zero-touch autonomy. He argued the cost is measured in tokens and minutes, not headcount, but the design assumes sources are available to ground every claim and that a verification command exists for Level one. The accounting case depended on a control total printed on statements, and the legal case depended on cold personas getting the contract and invoices with no history. Without those anchors, the same fan-out would have less to settle against.

What remains unanswered is how the four-in-10 error rate moves as models change and whether the token budget scales outside Rexmore's portfolio ops. Gallon was also speaking at an AI Engineer conference where he noted the Office 365 tenant API requires app registration and admin approval, a hurdle that pushes teams to web automation when approval is not feasible. That constraint was not in the review talk, but it frames why his team leans on logged-in automation at all. He closed with the line that he believes every output not because agents are right, but because agents checked it first.

© 2026 StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training requires a license. See our terms.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.

Daniel Singer

Written by

Daniel Singer

Editor, StartupHub.ai

Daniel Singer is the editor of StartupHub.ai, a technology expert and thought leader on AI and its applications across sectors, from fintech and healthcare to developer tooling and consumer software. He writes and tests the tools covered here thoroughly and regularly, and built StartupHub.ai to give founders, operators and buyers a clearer read on what they are actually being sold.