AI Research

Japan's finance test shows open models trail by 12 points

Sakana AI's 321-question JFinBench finds open-weight models trail in Japanese finance on knowledge, not math.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

JFinBench overall score versus cost per question for 12 models
Credit: Sakana

Sakana AI released JFinBench, a 321-question Japanese finance benchmark built from six 2026 certification exams, and the obvious reading is that open-weight models are 12 points behind. I do not buy that as a verdict on reasoning.

To be fair, Sakana AI is right on the problem: US benchmarks do not transfer to Japan's tax and accounting rules, which change often, so it tested 12 models on this year's law with the exam date in the prompt.

The top line still stings.

Claude Fable 5 scored 95.8 and GPT-6 Astra scored 93.6 without tools, per Sakana AI, while the best open-weight model, Kimi K3.1 Pro, scored 83.4. That 10-point deficit matters because the same Kimi model beat GPT-6 Astra on Terminal-Bench 4.0 with 65.7% versus 58.2%, per Kimi disclosures cited by Sakana AI.

Moonshot pitches Kimi K3.1 Pro for in-depth financial research with upgraded agentic coding, per its site, and OpenRouter lists it as a flagship open-weight multimodal reasoning model with a 1M-token window. JFinBench suggests the finance pitch is ahead of the knowledge inside the weights.

Sakana AI classified failures and found knowledge gaps, not arithmetic, drove misses across most models. Calculation errors were rare.

Give the models a terminal and the web and the story shifts, but it does not close.

With shell and web access through Harbor, GLM 5.3 Flash jumped 29.9 points to 87.6 and DeepSeek V4.1 Flash 1017 jumped 18.4 points to 91.5, per Sakana AI, yet Kimi K3.1 Pro at 92.9 still trailed GPT-6 Astra at 97.3 by 4.4 points. Models fetched e-Gov in 67% of trials, which tells you browsing patches missing statutes but does not erase the advantage of already knowing them.

The comparison is messier than the table implies. Sakana AI reported each model's best score across three harnesses, including Terminus-2 and vendor agents like Kimi Code, with six hours per attempt.

That flatters good tool use as much as good finance. Add an LLM judge for 128 written answers and only 12 image questions, and you have a sharp probe, not a full practice exam.

Continuity is also unproven. The human-in-the-loop pipeline needed about two hours of checks per exam, or 12 hours total, per Sakana AI, and the team says the same flow can convert medical or hiring exams. So far it has converted one season, six exams from May to August 2026, with tax accountant work still the hardest at a top score of 79.4.

If you are building for Japanese finance, treat JFinBench as a knowledge audit and retest after every tax revision, not as a moat.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsSakana AIJFinBenchKimi K3.1 ProGPT-6 AstraClaude Fable 5Moonshot AITerminal-BenchHarbor

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.