Artificial Intelligence

FAIR data tax costs Europe €10.2bn a year

VAST Data demo writes FAIR metadata at ingest. Europe loses €10.2bn a year to non-FAIR data.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

VAST FAIR data pipeline curating research files at ingest
Credit: VAST Data

€10.2bn a year is the minimum cost to Europe of research data that is not FAIR, per PwC for the European Commission in 2018. VAST Data says its fix is a demonstration pipeline that curates data where it lands, described by engineers Chris Snow and James McClung.

It does not ask researchers to file another form.

A researcher or HPC job writes a folder over S3 or NFSv4 and VAST DataEngine triggers fire on each file, the company said. Events go to Kafka on the VAST Event Broker, files get profiled for format, headers, variables and checksums, and the folder gets a draft record, a FAIR score and a review page. The design matches its broader pitch for DataEngine as a runtime that runs functions, triggers, streams and filters inside the platform.

The loss numbers are stark. The odds a dataset behind 516 papers still existed fell 17% for each year since publication, according to Vines et al. in 2014. Of 1,792 authors who said data was available on request, about 7% sent it, according to Gabelica et al. in 2022. In the 2025 State of Open Data survey, 69% said they get too little credit for sharing.

On six public datasets, the draft got 62 of 64 fields right before human edits, across two runs on different clusters, the company said. The sets spanned NOAA sea surface temperature, a TESS light curve, CMS open data, gnomAD, BBBC039 images and UK traffic counts, plus four renamed dark copies. In all seven cases with no stated licence or creators, it left the fields blank. One renamed physics file drew a wrong guess at 0.7 confidence, which review is meant to catch.

Scores rose after review. Documented sets went from 24 to 52 out of 100 to 72 to 92 on the demo checks, the company said. An export_4.csv with no documentation went from 8 to 76 once a person added licence and creator. Independent assessor F-UJI 3.5.0 scored published sets 83 to 88, capped because test identifiers are not registered with DataCite.

The pipeline is about 3,500 lines of Python, almost all about metadata, with snapshots, Croissant files for ML use, vector search and OAI-PMH feeds, the company said. Search by meaning put the NOAA set first at 0.63 against 0.39 for the next on ocean temperature from satellites. What the test does not show is production behavior on billions of files with real DOIs, and Trino over published tables was not deployed in this demo. Live DOIs and a full archive backlog would change the read. Who pays for curation that mostly benefits the next lab?

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsVAST DataVAST DataEngineVAST DataBaseNOAACMSF-UJIOAI-PMHCroissant

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.