Stanford wins Databricks Grounded Reasoning Cup

Databricks hosted the inaugural Grounded Reasoning Cup to evaluate how well AI agents generalize to new, complex datasets. Eleven academic teams optimized agents on the OfficeQA benchmark, released seven months ago, and then tested them live on OfficeQA Pro V2. This new benchmark, containing roughly 120,000 pages of U.S. Treasury documents, was released just 36 hours before the event. The competition highlighted the difficulty of generalization, as standard, out-of-the-box frontier agents averaged less than 30 percent accuracy offline, and 18.8 percent of the questions went unsolved by every team.

Stanford University won the competition with 63.3 percent accuracy, beating the average team by approximately 22 points and the offline baseline by about 35 points. The Stanford team used a Claude Code agent powered by Claude Opus 4.8, after experimenting with both Opus 4.8 and Fable 5 during development. They prepared a playbook of over 100 skills for tasks like table localization, answering 57 of the 88 questions they attempted. Stanford used an adaptive verification loop, disabling it to gain speed but reactivating it in the final round to correct an answer and secure a 1.75-point victory.

The University of Massachusetts Amherst placed second by prioritizing speed with Claude Opus 4.8 Fast, running three parallel agents and earning 36 speed bonuses worth 0.25 points each. UMass achieved an average correct submission time of four minutes. Yale University secured third place with a multi-arm verification harness that ran four parallel arms using Gemini 3.1 Pro and Gemini 3.5 Flash, answering 49 out of 90 questions correctly. The average team score was about 41 percent, and the gap between the highest and lowest-scoring teams using the identical model was 30.4 points.

For AI practitioners, these results prove that model choice alone does not guarantee performance on complex enterprise tasks. Success relies heavily on the surrounding system architecture, including document parsing, hybrid retrieval, structured tool use, and robust verification. Databricks noted that using tools like its ai_parse to pre-parse documents contributed to Genie's 24.0-point improvement over baseline frontier agents, proving that engineering the data pipeline is just as critical as optimizing the underlying LLM.