AI Research

OpenAI says perplexity tracks intelligence

OpenAI chief scientist Jakub Pachocki published pretraining perplexity scores and urged labs to set shared limits before the 0.68 window.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Chart of OpenAI pretraining loss from GPT-3.5 to current models
Credit: OpenAI News

Key facts

  • GPT-3.5 sat at about 0.99 log-perplexity in OpenAI codebase test
  • GPT-4 sat at about 0.91 log-perplexity and made chain-of-thought useful
  • Jakub Pachocki points to 0.68 to 0.7 window for next capabilities

OpenAI Chief Scientist Jakub Pachocki published internal pretraining losses and proposed perplexity as a shared safety standard.

Those numbers come from a fixed snapshot of OpenAI internal codebase, which tells you an insider gauge is being offered as an industry ruler.

Pachocki frames pretraining as building a compressor that predicts the next token, with prompting and finetuning just eliciting latent capabilities.

OpenAI says GPT-3.5 sat at about 0.99 log-perplexity and powered the original ChatGPT, while GPT-4 sat at about 0.91 and made chain-of-thought useful.

Below 0.8 the company reports transformative work in math and cybersecurity, with a 0.76 model solving open problems the team had pondered for years.

Here is the wrinkle builders feel in practice: chain-of-thought reasoning sharply lifts capability without lowering perplexity at all.

OpenAI draws a line at 0.68

Pachocki expects the next shift in the 0.68 to 0.7 window and wants labs to agree they don't cross it blind.

He calls for joint evaluation norms and a hard pause tied to concrete progress like a successor to chain-of-thought monitors.

He admits cross-lab comparison is hard because data mixtures differ and the true entropy floor remains unknown.

CryptoBriefing distilled the post to its 2027 call for an avalanche of pretraining advances, which is the timeline teams must staff evaluations for now.

Mentioned in this story

Sources

TopicsOpenAIJakub PachockiGPT-3.5GPT-4ChatGPTpretraining

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.