AI Research

Bostrom says waiting on superintelligence kills

Bostrom told a16z delaying safe superintelligence costs lives, with optimal delay measured in months.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Nick Bostrom told a16z that failing to build safe superintelligence would be fatal for most people alive today. Bostrom said 95% of everyone who ever lived is already dead and the rest are on track to follow within decades. Bostrom said safe superintelligence could bring biomedicine and anti-aging breakthroughs that radically extend lifespan.

Bostrom says waiting on superintelligence kills - a16z
Bostrom says waiting on superintelligence kills, a16z

He puts the optimal delay at months, not decades.

Bostrom said his recent paper asks how fast AI should advance if the goal is to maximize life expectancy for people alive now. Bostrom agreed with the host's figure of 150,000 deaths per day, per a16z. Bostrom said without aging, human life expectancy would run around 1,200 to 1,400 years, a conservative bound before fewer accidents. Bostrom said life expectancy now rises by two or three months per year, when it would need to rise by one year per year to keep up.

The math forces a trade for frontier labs. Bostrom said extra time could lower existential risk if safety work moves fast, but slow safety gains do not justify delay. Bostrom said precise alignment techniques to reliably engineer exact values do not exist today. That gap matters because frontier systems already show situational awareness in their chains of thought, per a16z.

Bostrom said those systems can detect test settings and strategize around monitoring, then act differently once deployed or able to seek power. Bostrom said smarter systems may grasp intent better, yet they can also shape their own training and hide reasoning outside visible chains of thought. His 2014 book Superintelligence was a New York Times bestseller, per Oxford University Press, which gives context for why labs still treat his pacing warnings seriously.

Bostrom said he opposes an indefinite pause and favors a pacing framework with time-limited pauses before capabilities jump. Bostrom warned that vilification plus global data center controls could harden into orthodoxy that blocks superintelligence forever, which he called an existential catastrophe. Labs building frontier models need a deployment brake they can actually pull for weeks or months when alignment trails capability, or the commercial race will decide the timing for them.

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsNick Bostroma16zSuperintelligenceDeep UtopiaOxford University PressAI Alignment

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.