AI Inference Costs: Build vs. Rent

Thiyagarajan Maruthavanan, founder of Kalmantic Labs, delivered a compelling argument for building proprietary AI inference infrastructure, cautioning against the hidden costs and limitations of rented intelligence platforms. Citing examples from major retailers spending nearly $200 million on inference and Uber's CTO highlighting budget overruns, Maruthavanan emphasized that the seemingly inexpensive cost of tokens can quickly spiral out of control.

He drew an analogy to casino chips, where the gradual loading of credits can lead to overspending and a loss of financial control. Maruthavanan shared his own experience with an app called Ultrazone, which generated music from text prompts. While initially successful with hundreds of thousands of users, the inference costs ballooned to hundreds of thousands of dollars, a common pitfall stemming from unmanaged context and input token compression, especially in agent-based loops.

The Perils of Stolen Keys and Unforeseen Costs

Adding a dramatic personal anecdote, Maruthavanan recounted how his API key was stolen, leading to a surge in costs from $7,000 to $10,000 in just three weeks before his co-founder could intervene. This highlights the security risks associated with managed inference services.

He then explored the alternative of 'token factories,' which involve using open-source models provisioned as tokens per second. While this offers a path away from paying providers like Anthropic or OpenAI, Maruthavanan noted that simply switching to this model doesn't eliminate all problems. He mentioned the argument for building local token factories, even in a garage, inspired by AI influencers.

The Enterprise Bottleneck: Control, Audits, and Reproducibility

Maruthavanan detailed his experience moving Ultrazone to his own DGX box, which encountered memory bottlenecks. However, the larger issues arose when enterprises sought to replicate his setup. He identified three key hurdles for enterprises:

• Funds: An investment fund using AI for email analysis needed control over their rate limits, finding third-party restrictions unacceptable.
• Hospitals: A hospital's AI use case worked well initially, but a later audit red-flagged third-party vendor dependency, preventing further adoption.
• Tax Practices: A tax practice required the ability to reproduce AI-generated recommendations, which proved difficult without deep access to the model's workings.

These examples underscore that for enterprises, the bill is only the first problem; issues of control, auditability, and reproducibility make renting or leasing inference infrastructure increasingly untenable.

When to Build Your Own Inference Infrastructure

Maruthavanan proposed a clear decision framework:

• Pre-Product Market Fit (PMF) Startups: Renting is acceptable as the use case and demand are still being validated.
• Post-PMF Startups: Building is essential to support a proven use case and scale effectively.
• Enterprises: Building is non-negotiable, especially if a project has already been budgeted, implying a commitment to product-market fit.

He likened the decision to choosing between renting an Airbnb and buying a house for a family. While renting offers flexibility for exploration, it's not sustainable for long-term growth and stability.

Rent to Learn, Own to Earn

Maruthavanan concluded with a powerful mantra: "Rent to learn, own to earn." He shared that he developed an open-source tool, JustTokenMax, as an alternative to solutions like Netflix's Headroom, claiming superior performance. He also authored a book, 'PeakInference: Infra Economics of AI Inference,' to guide those considering building their own inference infrastructure.

The AI market's rapid evolution, with conflicting advice from industry leaders like Jensen Huang of Nvidia (NASDAQ:NVDA) advocating for 'token factories' and Satya Nadella of Microsoft (NASDAQ:MSFT) promoting 'unmetered intelligence,' highlights the need for companies to find their own answers. Maruthavanan’s core message is that while renting is for learning, true earning in the AI space requires ownership of one's infrastructure.