# Ramp built evals for months so AI can close books

_Ramp spent 4-6 months building synthetic books and pass-fail evals before Stacks met customers._

**Published:** 2026-11-09
**Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/ramp-built-evals-for-months-so-ai-can-close-books

---

Ryan Stevens told [AI Engineer](https://www.youtube.com/watch?v=WJTQBENv0KI) that Ramp spent four to six months building end-to-end evals before launching Stacks, its AI bookkeeping agent.

They grade only the final ledger, not the path.

To do it without live customers, Stevens said Ramp and [Mercor](https://www.startuphub.ai/startups/mercor) built synthetic businesses with full ERPs, vendor lists and payables, then ran each reconciliation, data entry, variance and accruals task against 10 to 20 pass-fail criteria for accuracy and pass-at-K.

The failures were reasoning, not data gathering, Stevens said, which is why frontier models scored high on accuracy but collapsed on pass-at-one and why auto-tuning skills in the style of [Andrej Karpathy](https://www.startuphub.ai/ai-news/ai-figures/2026/figure-andrej-karpathy-anthropic-pretraining-2026-05-31)'s tuning posts failed while skill ablation and trajectory-distilled memories delivered the biggest gains.

Separate Ramp spend data shared on the All In podcast puts [Google](https://www.startuphub.ai/ai-news/artificial-intelligence/2026/google-blackstone-form-ai-cloud-firm) behind in agentic usage per EconLab, matching Stevens picking Opus.

Now Stacks evals run on schedule and ping Slack on regressions because Stevens said CI gating breaks at 70 to 80 percent pass rates.

---

Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.

© StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.