Technology

AI found thousands of bugs. Only 99 mattered

GitHub and Microsoft turned thousands of AI findings into 99 private fix-ready reports, but manual review still did the hard work.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

Maintainer reviewing private AI security fix report
Credit: Github Blog

GitHub and Microsoft piloted MDASH on AutoGPT, Caddy, Express, FastMCP, ImageMagick and Netty, sending 99 private reports with suggested fixes.

All 99 went through private vulnerability reporting, so the pilot did not publish exploits or force a public remote-exploit scramble.

The obvious reading is that AI finally helps maintainers instead of drowning them.

The case against is scale. I do not buy that this result repeats without the same hand-holding.

The claim from the teams is fair on its face. They say maintainers do not need more raw findings, they need checked reports with patches, sent privately with proof of concept.

Caddy backs part of that up. Its maintainers fixed all 22 reports they received in a week or less, according to the teams, and called the reports succinct and obscure in a good way.

ImageMagick is the best vignette. A two-person team got 22 reports and pushed one fix in 15 minutes because the summary pointed to exact lines, versus 30 minutes to one hour normally, its maintainer said.

What is missing is everything needed to copy that outcome.

MDASH, using Microsoft’s MAI Cyber model, flagged thousands of possible issues across the six projects, the teams said. After AI triage by GitHub Security Lab and hand review by seven Microsoft FORGE Lab researchers, only 99 reached maintainers.

Those seven researchers spent about two days per project checking findings by hand, the teams said. That is the pilot.

The sample is also gated. All six projects were alumni of the GitHub Secure Open Source Fund whose maintainers agreed to the scan, set the pace and decided what to fix.

For the obvious reading to hold, triage and verification would have to work without that crew, for projects that did not opt in and do not run CI on private forks.

The teams admit that gap. They say defenders need private reporting, private-fork testing and access to cyber-capable models to verify proofs of concept before and after a fix.

A parallel MDASH run shows both promise and filter burden. Microsoft said its FORGE Lab turned about two weeks of Linux kernel scanning into 59 upstream bug reports, with 33 assigned CVEs and 38 merged fixes, including eight local root privilege-escalation demonstrations.

Christian Grobmeier, a Log4j maintainer who lived through Log4Shell, framed the stakes as the AI defending open source has to be better than the AI attacking it.

He is right, and that is why raw scanners are cheap and fixes are not. As one Netty maintainer put it, finding vulnerabilities is no longer the bottleneck and fixing them depends on maintainers.

If you scan open source, do the triage yourself and bring a patch, or do not send it.

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsGitHubMicrosoftCodename MDASHMAI CyberFORGE LabCaddyExpressNetty

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.