Artificial Intelligence

Stop trusting PRs that claim tests passed

Stephen Chin and Cassandra Chin introduced Tracetune plus Relate to prove PR test claims with traces and cut tokens nearly in half.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

PRs that claim tests passed are worthless without proof. Stephen Chin and Cassandra Chin used AI Engineer to introduce Tracetune, an open source writer that rebuilds pull requests from agent traces.

Stop trusting PRs that claim tests passed - AI Engineer
Stop trusting PRs that claim tests passed, AI Engineer

I think they nailed the failure mode.

Cassandra Chin said session logs compress and get more inaccurate as agents work. Stephen Chin pointed to a real PR that needed exact preview links and command attribution. It claimed tests passed while traces showed bun test and bun x tsc running together, which must run sequentially.

The fix adds Relate, a Rust embedded graph database called Archie that builds an evidence packet with zero LLM cost. Chin said it lets a cheaper Haiku reviewer check claims while a Sonnet writer drafts. PR Bench judges scored Tracetune plus Relate 74 to 76 and 82 to 86 on Claude and Codex while using about half as many tokens.

To be fair, simply swapping to cheaper writers cuts spend but Chin found quality drops. Strata takes a different local route with one-click install for Windows and Linux serving Qwen3.8-Flash-Next over localhost OpenAI and Anthropic APIs. Rune stays at the UI layer as a native framework, so neither checks whether tests actually ran.

Tracetune still made occasional mistakes before the graph brief, which is why the provenance step matters. If you ship AI PRs, stop summarizing diffs and ship the trace.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsStephen ChinCassandra ChinTracetuneRelatePR BenchClaudeCodex

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.