Google introduces Multimodal Knowledge Agent benchmark

Google DeepMind, in partnership with Artificial Analysis, launched the Multimodal Knowledge Agent benchmark on November 10, 2026, to evaluate how artificial intelligence systems reason across visually complex files during office work. As detailed in Google Keyword Blog's report, the evaluation suite measures agents on tasks involving blueprints, schematics, spreadsheets, and presentation decks. DeepMind also disclosed results for Gemini 4 Carbon, which achieved second place in overall quality at one-sixth the cost of the top-ranking model.

This analysis examines the task architecture, grading methodology, evaluation harness, and operational efficiency metrics established by the new benchmark. Technical leaders and software engineers evaluating multimodal foundation models can use these findings to understand agent performance across visual business domains. It details how standardized sandboxes, multi-hour professional projects, and automated rubric scoring measure sustained reasoning.

01Ten industry verticals and eighty-three occupations

Traditional knowledge work benchmarks often allow artificial intelligence models to solve tasks using plain text alone without inspecting graphical layouts or images. Modern professional work across enterprises relies heavily on visual information, including architectural blueprints, engineering schematics, spreadsheets, and reports. Google DeepMind and Artificial Analysis constructed the benchmark specifically to evaluate how effectively agents handle visually intricate workflows.

To ground the evaluation in authentic business settings, researchers partnered with specialists across ten industry verticals and eighty-three occupations where daily responsibilities are visual. Participating domain experts, including architects, biologists, insurance underwriters, and data scientists, created a two-hundred-task evaluation set based on their daily work. These assignments reflect realistic corporate challenges where diagrams, formulas, and structured text intersect.

The collaborative design process ensured that each problem captures the genuine friction and multimodal requirements encountered in professional environments. By structuring tasks around actual job duties, the benchmark tests whether agents can assist specialists on complex responsibilities. This foundation prevents models from bypassing visual document inspection during problem resolution.

02Multi-step projects with seventy rubric checks

Every task in the two-hundred-problem evaluation catalog represents a multi-step project that requires a seasoned human professional between three and eight hours to complete. Each challenge combines a realistic starting prompt with a bundle of multimodal input files such as schematics, spreadsheets, and design assets. Candidate models must process these diverse inputs to deliver finished work products including reports and presentation decks.

Grading is governed by expert-authored evaluation rubrics containing roughly seventy discrete yes-or-no verification criteria for each distinct project. The scoring framework prioritizes factual precision, allocating seventy-five percent of the total evaluation weight to accuracy. The remaining twenty-five percent of the score evaluates document design, visual styling, and presentation quality in the deliverable.

This dual grading standard requires autonomous agents to maintain factual rigor while producing professional visual formatting. Models cannot achieve high scores by generating plain unformatted text summaries that disregard formatting specifications. By tying correctness directly to presentation rules, the rubric verifies whether frontier systems can satisfy standard corporate publishing criteria.

03Stirrup harness and the non-Google autorater

Evaluations run inside an identical open-source execution harness named Stirrup to ensure consistent testing conditions across competing artificial intelligence systems. Every candidate model executes in the same isolated sandbox environment, accessing identical tools under a uniform turn budget. Standardizing the execution harness ensures that performance differences reflect model reasoning capabilities rather than variations in computing environments.

Completed project deliverables are graded by an autonomous agentic autorater built on a non-Google foundation model. The autorater opens submitted files, renders visual pages, inspects embedded spreadsheet formulas, and checks each output against the expert rubric. When human domain experts independently graded model deliverables, their determinations matched the autorater scores ninety-six percent of the time.

Relying on an external model architecture for grading eliminates vendor bias and prevents self-preference scoring distortions. The autorater conducts deep structural inspections across complex documents without requiring manual human scoring for every evaluation run. This automated setup enables Artificial Analysis to maintain scalable, objective scoring across diverse frontier models.

04Private evaluations and ten Hugging Face tasks

Artificial Analysis maintains custody of the full two-hundred-task evaluation set and runs tests internally within private infrastructure. Keeping the evaluation problems private prevents the tasks from leaking into public training sets or being indexed by search engines. This quarantine protects benchmark integrity and ensures that published model scores reflect genuine problem-solving capabilities rather than memorization.

Alongside the private evaluation set, the researchers published a separate group of ten public tasks on the Hugging Face platform. This open sample allows developers, academic teams, and enterprise researchers to inspect task formats, input bundles, and grading rubrics. Anyone can review the ten public examples to test their own agent configurations before participating in formal evaluations.

Findings in the technical report highlight stark variations in operational efficiency among frontier models attempting these multi-step projects. Evaluated systems required between thirty and one hundred thirty turns on average to resolve the same workplace assignments. The report demonstrated that robust visual perception is essential, as models failed to compensate when visual inputs were removed.

05Gemini 4 Carbon ranking and pricing

Google DeepMind tested its Gemini 4 Carbon model across the new evaluation set to measure its multimodal reasoning against competing systems. Gemini 4 Carbon earned the number two ranking in overall quality among the tested frontier models. DeepMind reported that the model paired strong task completion quality with notable operational cost efficiency.

Gemini 4 Carbon completed evaluation tasks at six times lower cost on average than the top-scoring model on the benchmark. This combination of high deliverable quality and lower operational expense places the model on the Pareto frontier of price and performance. The data indicates that organizations can deploy capable visual agents without paying top-tier inference rates.

Google made Gemini 4 Carbon available through the Gemini API inside Google AI Studio alongside updated developer guides and documentation. Software engineers and enterprise teams can access the platform to integrate the model into automated workflows. The model enables organizations to build visual agents that interpret technical schematics, analyze complex spreadsheets, and generate formatted reports.