Vals AI's Finance Agent v2 benchmark shows no frontier model above 58% on real analyst workflows, what that means for deploying AI in finance teams today.
While model launches dominate headlines, the more sobering numbers this month come from Vals AI's Finance Agent v2 benchmark, an independent evaluation built to measure what frontier models can actually do on real financial-analyst workflows. Version 2 was deliberately rebuilt around multi-step agentic tasks: the model must plan, retrieve data, perform calculations and synthesize conclusions across multiple tools, not just answer trivia.
The July leaderboard makes uncomfortable reading for the "AI analyst" narrative. Gemini 3.5 Flash leads at 57.86% accuracy, with Claude Fable 5 close behind at 56.31% and Claude Opus 4.8 at 53.92%; GPT-5.5 answers fewer than 52% of tasks correctly. Even under generous partial-credit scoring, no model clears 58%. Under strict scoring, requiring fully correct answers, every model sits below 46%.
The failure pattern is consistent: models handle definitional questions and simple retrieval well, then degrade sharply on multi-step calculations, cross-document reconciliation, and tasks that depend on industry convention rather than explicit text. These are precisely the tasks that constitute most of a junior analyst's day.
Our Take
the gap between vendor marketing and measured capability is now quantified, and it is wide. That does not argue against deployment, a system that completes half of analyst tasks at near-zero marginal cost is valuable, but it dictates the architecture: AI as a supervised first-draft layer with human verification, not autonomous finance work. Falling per-token prices (Meta's Muse Spark 1.1 at $1.25/$4.25 being the latest) make it cheap to run multiple attempts per task, but sampling more does not fix a reasoning ceiling. Firms should run their own task-level evals before believing any capability claim.