Turbopuffer CEO on AI, Databases, and Startup Growth

Gergely Orosz, author of The Pragmatic Engineer, sat down with Simon Eskildsen, founder and CEO of Turbopuffer, for a deep dive into the world of AI, databases, and the entrepreneurial journey. Eskildsen, a highly technical CEO, shared insights from his early days tinkering with computers via PowerPoint and FrontPage to his impactful tenure at Shopify and the eventual founding of Turbopuffer.

From PowerPoint to Production at Shopify

Eskildsen's fascination with computers began unconventionally with PowerPoint, where he discovered the ability to create complex, Turing-complete games by linking slides. This led him through the Microsoft Office suite and into FrontPage, a tool intended to replace front-end developers, which he found had significant cross-browser compatibility issues. His exploration of HTML and PHP eventually led him to discover the world of algorithmic programming through the International Olympiad in Informatics. He recounted how his article about the benefits of using feature phones over smartphones, which gained traction on Hacker News and was featured by The New York Times, unexpectedly led to a job offer from Shopify while he was still in high school. He moved to Canada to join Shopify in 2013, initially viewing it as a gap year but quickly realizing it was his calling.

During his eight years at Shopify, Eskildsen worked on various infrastructure components, including the storefront that powered a significant portion of the company's traffic. He detailed the engineering challenges faced by a rapidly scaling SaaS company in the 2010s, such as containerizing everything with Docker and preparing for massive traffic spikes like Black Friday. He explained how scaling issues often pushed problems to the database layer, leading him to work on sharding and multi-data center deployments. A key learning from this period was the creation of Toxyproxy, a Layer 7 proxy designed to inject failures and test the resilience of applications, which uncovered numerous issues in the ecosystem.

The Genesis of Turbopuffer: Tackling Vector Storage Costs

After leaving Shopify in 2021, Eskildsen initially didn't have a concrete plan but found himself drawn to the problem of infrastructure costs and performance, particularly for AI applications. The advent of ChatGPT in 2022 highlighted the need for efficient vector storage, especially with the limitations of early context windows. He observed that connecting large document sets to AI models required fast search capabilities, which were proving prohibitively expensive with existing solutions. This led him to his "napkin math" project, analyzing the cost and performance of various storage and memory tiers.

Eskildsen recalled a specific instance with a company called Readwise, where a recommendation engine he built, while effective, would have cost $30,000 per month, a figure unsustainable for the bootstrapped company. This experience solidified his belief that a more cost-effective solution for vector storage was needed. He spent the summer of 2023 developing Turbopuffer, a database built on top of S3, aiming to achieve low latency by optimizing data access and caching strategies. His initial version was a simplified approach to clustering vectors and organizing them into files, a concept he iterated on to achieve the desired performance.

Navigating the Cloud and CPU Scarcity

Turbopuffer's first customer was Cursor, an AI-powered code editor, which reached out after Eskildsen announced his project on Twitter. He noted that Cursor was an early-stage company at the time, and his solution significantly reduced their previous vendor costs by 95%. Today, Turbopuffer serves various AI labs and startups, operating on CPUs rather than GPUs, which Eskildsen believes is a strategic advantage given the ongoing GPU shortage.

He elaborated on the current cloud infrastructure dynamics, pointing out the increased demand for CPUs due to AI workloads like Reinforcement Learning (RL) and the proliferation of AI agents. This demand, he noted, is making CPUs scarce and competitive to acquire, even for large cloud providers. Eskildsen expressed his preference for CPUs due to their availability and cost-effectiveness, even admitting to a humorous encounter with Nvidia CEO Jensen Huang where he emphasized his reliance on CPUs.

Eskildsen also shared his perspective on venture capital, emphasizing his initial reluctance due to the pressure for rapid growth and returns. However, he acknowledged that the success of Turbopuffer, particularly with early customers like Cursor, has validated the market and his approach. His philosophy centers on simplicity and deep understanding, principles he believes are crucial for building software that ages well and for driving innovation in the rapidly evolving AI landscape.