AI Inference Costs: Build vs. Rent

Thiyagarajan Maruthavanan, founder of Kalmantic Labs, delivered a compelling argument for building proprietary AI inference infrastructure, cautioning against the hidden costs and limitations of rented intelligence platforms. Citing examples from major retailers spending nearly $200 million on inference and Uber's CTO highlighting budget overruns, Maruthavanan emphasized that the seemingly inexpensive cost of tokens can quickly spiral out of control.

Companies working on this

StartupHub profiles of the companies this article names, with funding and a one-liner from our database.

Microsoft

A global technology leader providing software, cloud services, AI, and devices for individuals and businesses.

OpenAI

$852.0B

OpenAI is an AI research and deployment company dedicated to ensuring that artificial general intelligence benefits all of humanity.

Anthropic

$965.0B

Anthropic is an AI safety and research company building reliable, interpretable, and steerable AI systems, best known for the Claude family of models.

Habana Labs

$105M

Develops AI and Deep Learning processors for data centers and cloud environments.

He drew an analogy to casino chips, where the gradual loading of credits can lead to overspending and a loss of financial control. Maruthavanan shared his own experience with an app called Ultrazone, which generated music from text prompts. While initially successful with hundreds of thousands of users, the inference costs ballooned to hundreds of thousands of dollars, a common pitfall stemming from unmanaged context and input token compression, especially in agent-based loops.

The Perils of Stolen Keys and Unforeseen Costs

Adding a dramatic personal anecdote, Maruthavanan recounted how his API key was stolen, leading to a surge in costs from $7,000 to $10,000 in just three weeks before his co-founder could intervene. This highlights the security risks associated with managed inference services.

He then explored the alternative of 'token factories,' which involve using open-source models provisioned as tokens per second. While this offers a path away from paying providers like Anthropic or OpenAI, Maruthavanan noted that simply switching to this model doesn't eliminate all problems. He mentioned the argument for building local token factories, even in a garage, inspired by AI influencers.

The Enterprise Bottleneck: Control, Audits, and Reproducibility

Maruthavanan detailed his experience moving Ultrazone to his own DGX box, which encountered memory bottlenecks. However, the larger issues arose when enterprises sought to replicate his setup. He identified three key hurdles for enterprises:

• Funds: An investment fund using AI for email analysis needed control over their rate limits, finding third-party restrictions unacceptable.
• Hospitals: A hospital's AI use case worked well initially, but a later audit red-flagged third-party vendor dependency, preventing further adoption.
• Tax Practices: A tax practice required the ability to reproduce AI-generated recommendations, which proved difficult without deep access to the model's workings.

These examples underscore that for enterprises, the bill is only the first problem; issues of control, auditability, and reproducibility make renting or leasing inference infrastructure increasingly untenable.

When to Build Your Own Inference Infrastructure

Maruthavanan proposed a clear decision framework:

• Pre-Product Market Fit (PMF) Startups: Renting is acceptable as the use case and demand are still being validated.
• Post-PMF Startups: Building is essential to support a proven use case and scale effectively.
• Enterprises: Building is non-negotiable, especially if a project has already been budgeted, implying a commitment to product-market fit.

He likened the decision to choosing between renting an Airbnb and buying a house for a family. While renting offers flexibility for exploration, it's not sustainable for long-term growth and stability.

Rent to Learn, Own to Earn

Maruthavanan concluded with a powerful mantra: "Rent to learn, own to earn." He shared that he developed an open-source tool, JustTokenMax, as an alternative to solutions like Netflix's Headroom, claiming superior performance. He also authored a book, 'PeakInference: Infra Economics of AI Inference,' to guide those considering building their own inference infrastructure.

The AI market's rapid evolution, with conflicting advice from industry leaders like Jensen Huang of Nvidia (NASDAQ:NVDA) advocating for 'token factories' and Satya Nadella of Microsoft (NASDAQ:MSFT) promoting 'unmetered intelligence,' highlights the need for companies to find their own answers. Maruthavanan’s core message is that while renting is for learning, true earning in the AI space requires ownership of one's infrastructure.

The 2026 State of AI Inference Economics

The numbers have shifted since this argument was first made. Industry analysis in 2026 estimates that inference now accounts for 55% of enterprise AI cloud spending. API token prices from major providers have fallen 40-60% since mid-2025 as model efficiency improved and competition intensified. Crucially, self-hosted inference has become proportionally cheaper at the same pace, so the economics gap for high-volume workloads has not closed. Open models average $0.23 per million tokens to run, versus $1.86 per million for closed frontier models, an 87% cost difference that accrues to the builder when running on owned infrastructure. The largest enterprises are trending toward a hybrid strategy: own the inference stack for high-volume, data-sensitive workloads; rent frontier closed models for lower-volume or experimental tasks. Dedicated GPU purchases remain justifiable only for constant 24/7 workloads sustained over 18 months or more.

StartupHub.ai tracks 3,095 AI infrastructure startups. Across that set, the build-vs-rent debate Maruthavanan describes is one of the most frequently cited strategic decisions in 2026 fundraising conversations: the window to establish infrastructure moats is narrowing as token prices fall and hyperscalers scale dedicated AI capacity.

Last updated: August 2026

Frequently Asked Questions

Should I build or rent AI inference for my startup?

The decision depends on your stage. Pre-product-market-fit startups should rent: the flexibility and speed outweigh cost concerns when you are still validating a use case. Post-PMF startups with a proven workload should consider building, since the unit economics of owned inference compound over time. The tipping point for high-volume workloads is typically 18 months of sustained 24/7 usage on a dedicated GPU setup.

How much does AI inference cost in 2026?

Inference costs vary by model and provider. Open models run at roughly $0.23 per million tokens on average, versus $1.86 per million for closed frontier models. API prices from major providers dropped 40-60% since mid-2025. For organizations running self-hosted open models, marginal costs are primarily GPU compute, which has also declined with more efficient model architectures and hardware improvements.

What is the "rent to learn, own to earn" framework?

This framework, popularized by Kalmantic Labs founder Thiyagarajan Maruthavanan, argues that renting managed inference from providers is appropriate when a company is still learning its use case and validating demand. Once a use case is proven and the product has reached market fit, building proprietary inference infrastructure delivers compounding cost and control advantages that rental arrangements cannot match long term.

What are the enterprise bottlenecks that make renting AI inference untenable?

Three recurring enterprise blockers: rate limit control (third-party APIs cap throughput), auditability (regulated industries require the ability to explain AI outputs, which is harder with external models), and reproducibility (tax, legal, and financial firms need to reproduce AI recommendations, which requires deep model access). For these sectors, renting inference is not just expensive but structurally incompatible with compliance requirements.