# Japan's finance test shows open models trail by 12 points _Sakana AI's 321-question JFinBench finds open-weight models trail in Japanese finance on knowledge, not math._ **Published:** 2026-11-16 **Source:** https://www.startuphub.ai/ai-news/ai-research/2026/japan-s-finance-test-shows-open-models-trail-by-12-points --- [Sakana AI](https://sakana.ai/jfinbench/) released JFinBench, a 321-question Japanese finance benchmark built from six 2026 certification exams, and the obvious reading is that open-weight models are 12 points behind. I do not buy that as a verdict on reasoning. To be fair, [Sakana AI](https://www.startuphub.ai/ai-news/technology/2026/sakana-fugu-beta-opens) is right on the problem: US benchmarks do not transfer to Japan's tax and accounting rules, which change often, so it tested 12 models on this year's law with the exam date in the prompt. The top line still stings. Claude Fable 5 scored 95.8 and GPT-6 Astra scored 93.6 without tools, per Sakana AI, while the best open-weight model, [Kimi K3](https://www.startuphub.ai/ai-news/technology/2026/kimi-k3-vs-frontier-china-s-ai-model-race).1 Pro, scored 83.4. That 10-point deficit matters because the same Kimi model beat GPT-6 Astra on Terminal-Bench 4.0 with 65.7% versus 58.2%, per Kimi disclosures cited by Sakana AI. Moonshot pitches Kimi K3.1 Pro for in-depth financial research with upgraded agentic coding, per its site, and [OpenRouter](https://www.startuphub.ai/startups/openrouter) lists it as a flagship open-weight multimodal reasoning model with a 1M-token window. JFinBench suggests the finance pitch is ahead of the knowledge inside the weights. Sakana AI classified failures and found knowledge gaps, not arithmetic, drove misses across most models. Calculation errors were rare. Give the models a terminal and the web and the story shifts, but it does not close. With shell and web access through [Harbor](https://www.startuphub.ai/startups/harbor), GLM 5.3 Flash jumped 29.9 points to 87.6 and DeepSeek V4.1 Flash 1017 jumped 18.4 points to 91.5, per Sakana AI, yet Kimi K3.1 Pro at 92.9 still trailed GPT-6 Astra at 97.3 by 4.4 points. Models fetched e-Gov in 67% of trials, which tells you browsing patches missing statutes but does not erase the advantage of already knowing them. The comparison is messier than the table implies. Sakana AI reported each model's best score across three harnesses, including Terminus-2 and vendor agents like Kimi Code, with six hours per attempt. That flatters good tool use as much as good finance. Add an LLM judge for 128 written answers and only 12 image questions, and you have a sharp probe, not a full practice exam. Continuity is also unproven. The human-in-the-loop pipeline needed about two hours of checks per exam, or 12 hours total, per Sakana AI, and the team says the same flow can convert medical or hiring exams. So far it has converted one season, six exams from May to August 2026, with tax accountant work still the hardest at a top score of 79.4. If you are building for Japanese finance, treat JFinBench as a knowledge audit and retest after every tax revision, not as a moat. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory. © StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.