Nicholas Arcolano told AI Engineer that if AI isn't lifting your engineering output, you should not wait for a better model. The problem is you, he argued, and his data says the tool barely moves the needle.
Arcolano, head of AI and research at Jellyfish and a founding team member at the company according to a Tessl profile of his work, came with telemetry not vibes. Jellyfish joins AI coding tool and agent telemetry with workflow data across hundreds of thousands of developers at about 1,300 companies, he said, and a curated cut of about 50,000 engineers at 447 companies using Cursor and Claude Code this year powers the breakdown.
His cleanest test was accidental. When SpaceX announced its acquisition of Cursor, Cursor pushed Grok higher in the UI and changed how auto mode worked, Arcolano said, and Jellyfish saw its fastest ever shift in model share. Engineers who flipped 0 to 100 percent to Grok showed no statistical correlation in change in merged PRs per engineer when compared to teammates at the same company. Dots centered on zero. Output did not move.
That nothingburger repeats when you control for the person, he said. Toggling Opus versus Sonnet, switching to ChatGPT, moving from Cursor to Claude Code, or jumping from autocomplete to agentic mode explains less than 1 percent of the variance in merged PRs per engineer per week. How well you use the harness, skills, cursor rules and customization, explains about 3 percent. How much you use it, token spend and active use days, explains about 9 percent and behaves like rocket fuel. More tokens, more throughput, with diminishing returns. Top 10 percent of consumers double the output of the median, Arcolano said, but it costs about 10x the tokens across roughly 500,000 developer weeks of data.
Skill use is where the gap lives. As of September, almost half of engineer weeks in the dataset had essentially no skill use and only 12 percent hit a level Arcolano would call modern AI engineering, he said. When skill use does hit 30 plus runs per engineer per week, PR throughput rises about 30 percent holding other factors constant, far more than any model family switch he showed. You have used these tools, you know the pattern. The settings are there, the docs say use them, and most weeks they sit idle until someone writes the rules for you.
Place that against the market you actually buy in. Cursor and Claude Code fight over the harness, Anthropic and OpenAI fight over the model, and on Tessl's blog Arcolano has separately warned about the chaperone problem for coding agents as teams scale. Jellyfish found the environment still swamps the tooling choice. Where you work, code base, review process, culture, explains about 12 percent, with about a third team and two thirds company, he said. For a fixed level of AI spend there is a 2x spread between the top 10 percent company and the bottom 10 percent. The individual engineer explains the rest, about three quarters of the model. The spread between the top 10 percent and bottom 10 percent engineer is 3.5x.
That does not mean AI is reshuffling who is good. Among 10,000 engineers ranked longitudinally, about 44 percent cycled out of the top quartile every half year before AI and about 41 percent after, he said. Your best engineers before remain your best after. Arcolano noted the analysis here used merged PRs and did not include a quality metric in that chart, though Jellyfish tracks quality separately. If your review queue is the bottleneck, doubling tokens just doubles waiting. Leaders cannot change the foundation model, he said, but they can change the course, fill the water hazards, and get skills actually used, which is where the throughput is hiding.
