Skip to content
05 of 06Multimodality in Generative AI

Chapter 4 — Evaluation: measuring without fooling ourselves

Library

Series and technical notes.

You are in Multimodality in Generative AI · Evaluation.

Watch video, summary and related content

Estimated reading10 min

This article analyzes why measuring the real capability of a multimodal system is harder than benchmark rankings suggest. By reading it, you will understand grounding and language bias (and why the latter can make a model answer correctly without actually processing the image), how evaluation-data contamination artificially inflates published results, and what benchmarks such as OCRBench v2, MMAU, ZeroBench and HallusionBench reveal about the field's real limits in documents, audio, long video and spatial reasoning. The article is useful both for technical readers evaluating models and for anyone who wants to interpret the comparisons circulating in the field rigorously.

Evaluating whether a language model produces accurate and useful answers is already a complex problem, but adding the visual or auditory dimension multiplies the difficulty in two different ways.

The first is that current multimodality benchmarks have two systematic problems that lead us to overestimate real capabilities: evaluation-data contamination and the dominance of text in benchmarks.

The second is that evaluation has historically been dominated by vision-language tasks, leaving whole capabilities poorly measured: understanding documents with complex layouts, reasoning over audio, temporal coherence in video, or the quality of outputs generated in modalities other than text. OCRBench v2 and MMAU are recent reminders that this space, evaluated only superficially until now, remains difficult terrain for the best current models.


1. What it means to evaluate grounding

In multimodal systems, grounding is the degree to which the model's answer is supported by the actual content of the image or audio, rather than by statistical inferences about what kind of answer is likely given the text of the question. A model can correctly answer "What color is the car in the image?" without actually processing the image if the color most frequent in its training for cars in similar contexts happens to match the correct answer.

That model does not have grounding; it has a strong language bias that produces the right answer for the wrong reasons. The difference remains invisible as long as the statistical bias and the correct answer point in the same direction.

To measure grounding, benchmarks need examples where the correct answer violates statistical expectations. If every question about fruit in images has an answer matching the fruit most represented in training, there is no way to distinguish a model with real visual understanding from one that answers from probability.

The Visual Question Answering Challenge (VQA), historically one of the most widely used benchmarks, has exactly this problem Goyal et al., 2017. A 2017 analysis showed that a model that completely ignored the images and answered only from the distribution of the most frequent responses for each question type still achieved surprisingly high results. Later benchmark improvements introduced balancing techniques to reduce this bias, although they did not remove it completely.

Grounding: what it means for a model to truly "see"
The difference between answering from visual evidence and answering from the statistical distribution learned during training.
The central evaluation question
When a model answers correctly, did it do so because it processed the image, or because the answer was statistically likely regardless of the image?
Without grounding
The image does not determine the answer
image ignored
Statistical prior
distribution learned during training
"White"
most frequent answer in the corpus
vs
With grounding
The image determines the answer
image analyzed
Visual evidence
actual image content processed
"Red"
actual color of the object in the image
Reference question in both columns
"What color is the car in the image?" — The actual car is red, but cars in the training corpus are mostly white or gray.

2. The problem of benchmark contamination

The second systematic problem is contamination. Foundation models are pretrained on massive amounts of internet data, and there is no guarantee that image-description pairs or evaluation datasets do not appear in those data.

Contamination in text is already a documented problem: models that obtain exceptional results on some reasoning benchmarks can recite the correct answers when given the problem identifier, suggesting that the benchmark was present in their training data. In multimodality the problem is potentially larger because benchmark images are often publicly available photographs that may have appeared in pretraining together with their descriptions or labels.

The technical solution is to use benchmarks whose evaluation data did not exist on the internet when the model was pretrained or that are protected from indexing. In practice, however, the most useful recommendation is to interpret results skeptically when the evaluated model was pretrained at internet scale, especially if the benchmark is old.

The most rigorous labs perform contamination analyses before publishing results: they search their training data for images similar to those in the evaluation benchmark and exclude those images from the final analysis. Without that analysis, published results are an upper bound on the model's real capability on that benchmark, not a direct measurement.

Benchmark contamination: when the test has already been seen
Evaluation images appear in the pretraining data. The score reflects memorization, not real generalization capability.
The problem
Foundation models are pretrained on massive amounts of internet data. There is no guarantee that the image-description pairs used for evaluation did not appear in those data. If the model has already seen the benchmark images during training, the score does not measure generalization.
Contamination mechanism
Internet
public photographs with labels · published datasets · image-text pairs
Pretraining
model training data
Benchmark
evaluation images (possibly already seen)
Inflated score
does not reflect real generalization capability
Without contamination analysis, published results are an upper bound on the model's real capability on that benchmark, not a direct measurement.

3. Language bias: answering from probability, not evidence

The language bias/prior is the tendency of models to generate answers that are statistically likely given the text of the question, regardless of the image content. It is the subtlest form of missing grounding and the hardest to detect with standard benchmarks because the errors it causes are invisible when the statistical distribution matches the distribution of correct answers.

Ablation experiments are the standard tool for measuring it: present the model with the question without the image and observe whether the answer distribution changes significantly. When the model without the image obtains results similar to the model with the image, language bias is dominating the answer.

The effect is especially strong in categories where training distributions are skewed: questions about the usual color of certain objects, an animal species when only one animal is visible, or the number of elements in scenes where two or three is the dominant frequency. In all these cases there is a highly skewed answer distribution that the model learns during training and uses as a shortcut, ignoring the image when the bias is strong enough.

Designing benchmarks that resist language bias requires active techniques: counterexamples where an object has an unusual color, scenes where the number of elements violates expectations, and spatial configurations that are uncommon in training. MMStar and SEEDBench are examples of benchmarks designed with explicit attention to this problem.

Language prior: answering from probability, not evidence
The tendency of models to generate statistically likely answers from the text of the question, regardless of the actual content of the image.
What the language prior is
During pretraining, the model learns answer distributions for each type of question. Those distributions act as shortcuts: for questions whose answer is highly predictable, the model generates it directly from the prior without needing the image.
What color is the car?
Learned answer distribution (without seeing the image)
white
42%
gray
26%
black
17%
red
8%
other
7%
Result
The model answers "white" even though the car in the image is red. When the prior is strong enough, the model does not need to consult the image to generate the most likely answer.
Why it is difficult to detect
The error is invisible when the statistical distribution matches the distribution of correct answers. In benchmarks where most cars are white or gray, a model without real vision can score highly using only the prior.

4. Metrics beyond accuracy

Final-answer accuracy does not capture all the relevant information about a multimodal model's capabilities. More rigorous evaluators include three additional dimensions that reveal different aspects of visual understanding.

Consistency. A genuinely capable model should answer paraphrases of the same question consistently. When the answer changes drastically under a semantically equivalent formulation, the model does not have robust understanding of the visual content; it is sensitive to the surface form of the question.

Localization when relevant. For tasks where the answer depends on the location of elements in the image, evaluation should verify not only whether the final answer is correct but also whether the model can indicate where the relevant element is in the image. A model that correctly answers "there are three cars" but cannot delimit where they are has a different kind of understanding from a model that can, and that difference matters in applications where localization is part of the expected result Hu et al., 2024.

Calibration. Models should be able to express uncertainty when visual content is ambiguous or when the question does not have a clear answer given the available content. A model that always generates a high-confidence answer, even for ambiguous images or questions that cannot be answered without additional information, is not properly calibrated. In production, this becomes falsely definitive answers where the system should abstain or ask for clarification.

Four dimensions accuracy does not capture
A model can get benchmark answers right for the wrong reasons. These four metrics reveal whether visual understanding is real or apparent.
Grounding
Is the answer grounded in the actual content of the image?
What it measures
Whether the model's answer depends on the actual image content or on the statistical prior learned during training. A grounded model produces different answers for different images even when the question is the same.
Grounding failure
orange
"white"
answers with the most frequent color in the corpus and ignores the image
With grounding
orange
"orange"
answers from the actual visual evidence
Diagnostic test
Ablation experiment: present the same question without the image. If the answer does not change, the prior dominates and the grounding is apparent rather than real.

5. Domains where evaluation remains difficult

Multimodality evaluation has been dominated by VQA and visual-grounding tasks because they are the easiest to automate and turn into benchmarks with single-choice answers. That has created a systematic blind spot: the domains where evaluation is hardest to automate are precisely those that reveal the most about current model limitations.

Documents with complex layouts. OCRBench v2, published in 2024, evaluated advanced multimodal models on text localization, handwriting recognition and logical reasoning over documents Liu et al., 2024. The results showed that even models with high VQA scores struggle in real-document scenarios: text in non-standard orientations, tables with merged cells, mathematical formulas embedded in text flow, or questions requiring information to be crossed between several regions of the same document. OmniDocBench, presented at CVPR 2025, extended this evaluation to documents with non-standard layouts: multiple columns, floating figures and elements with non-linear alignment. The evaluation of 13 SOTA models showed the same collapse: systems that reach 80–90% accuracy on standard text fall to 36.9% on complex-layout reconstruction, confirming that the limit is not visual recognition but the integration of structure and semantics in scenes that are not linear prose Ouyang et al., 2025.

OCRBench v2: "reading" a document ≠ "reasoning over" it
Models that read clean text accurately still struggle with real document layouts. OCRBench v2 measures the gap between those two regimes.
Solved regime
Standard horizontally oriented text
Input document
Standard font · horizontal orientation · no complex structure
Model output
Text extraction
Text extracted with high fidelity. Occasional minor transcription errors on uncommon characters.
Typical accuracy
~90%
Tasks handled well
Transcribe paragraphs of continuous text
Read numbers and dates in standard formats
Identify titles and sections with clear formatting
Why it works well here
Horizontally oriented text in standard fonts is well represented in model pretraining data. The model can apply pattern recognition directly without needing to reason over structure.

Expert audio. MMAU, published by Adobe Research in 2024, evaluated audio understanding and reasoning in three categories: speech, non-verbal environmental sounds and music Sakshi et al., 2024. The results showed that even the strongest models remain significantly below expert human performance on the hardest tasks in each category, with especially marked degradation when the task requires reasoning about the cause of a sound rather than merely identifying it, inferring context from multiple simultaneous sound sources, or distinguishing musical variants that share superficial structure. These limits are especially relevant for native-audio systems such as Gemini 2.5 or Qwen2.5-Omni, where capability expectations often exceed what available benchmarks can confirm.

Expert reasoning. MMMU, published in 2023, evaluated models' ability to reason over visual content in 30 university subjects grouped into 6 disciplines: Art and Design, Business, Science, Health and Medicine, Humanities and Social Sciences, and Engineering and Technology. Unlike image-description benchmarks, MMMU requires integrating domain knowledge with visual understanding: reading the image correctly is not enough; the model has to know what it sees means. The results showed a persistent gap between the best models and expert human performance, especially in disciplines where the image is not illustration but contains the decisive evidence: circuit diagrams, laboratory graphs and radiographs Yue et al., 2023.

Long-duration video. Video-MME, published in 2024, evaluated video understanding over durations ranging from minutes to hours, with questions requiring temporal tracking, analysis of changes between segments and synthesis of information distributed throughout the video. The evaluation revealed a pronounced quality drop as duration increases: models that understand short videos well fail on long versions of the same tasks because the attention mechanism loses temporal coherence at the scale of minutes or hours, a limitation that image or short-video benchmarks do not capture Fu et al., 2024. ZeroBench, published in February 2025, highlighted another angle of the problem: it evaluated 20 frontier models on one hundred visual-spatial cognition tasks over static images, and all obtained 0.0% accuracy Roberts et al., 2025. This is not temporal coherence but something more basic: pure spatial reasoning in scenes that any three-year-old can solve effortlessly systematically exceeds what any current model can do. LVOmniBench, introduced in 2026, confirmed the pattern for long-duration real-world video between 10 and 90 minutes: every open-source model remains below 35% accuracy, with the best evaluated commercial model reaching only 65%.

Video-MME: video understanding does not scale with duration
Models that understand short videos well fail on longer versions of the same tasks. Attention coherence degrades over time scales of minutes or hours.
Accuracy on temporal questions
78%
best Video-MME result (short segment)
0:00 1:30
active attention window
Temporal attention density
high coherence · entire video in context
Situation
All information fits in the active context
The model can attend to any frame in the video with similar effectiveness. Questions that require linking the beginning and end of the clip remain manageable because the token distance is small.
Tasks handled well
Identify the object that appears first
Count actions in the clip
Describe the sequence of events
Check whether something changes between beginning and end

Visual hallucinations. HallusionBench, published in 2023, was designed to detect hallucinations specific to vision-language systems: cases where the model claims to see absent elements, denies the presence of visible elements, or assigns incorrect spatial relations to objects that it can identify individually. The results showed that visual hallucination is a consistent pattern across all evaluated models rather than a marginal phenomenon, and that frequency varies by task type—counting, spatial reasoning, existence—in such a way that no model is robust across all categories at once Liu et al., 2023.

HallusionBench: three patterns of visual hallucination
Visual hallucination is not a marginal phenomenon. HallusionBench identified three distinct, consistent patterns across all evaluated models, including the most capable ones.
Actual image
glass no objects no objects
There is only one glass in the image
Question
"Is there a book in the image?"
Hallucinating model
"Yes. On the left side of the image there is a red-covered book leaning against the wall."
hallucination · nonexistent element
Correct answer
"No. There is no book in the image. Only a glass appears on the table."
correct · genuine grounding
Underlying mechanism
The model generates an object that is plausible for the described scene (table + glass → book is coherent with a domestic setting) using its prior over typical scenes, without checking the visual evidence. The answer is fluent and convincing.

Multimodal outputs. There are no established benchmarks that adequately measure real-time spoken-response quality, consistency between text and voice generated simultaneously, or the accuracy of images generated conditioned on both text and an input image. This absence of metrics means that we do not know precisely where the current limits of systems operating in that space lie.


Next chapter

Chapter 5 — Risks → — Which risks are specific to multimodality, why poor grounding has different consequences depending on the modality, and how the risk profile changes when perception and action are coupled in the same system.

6. References

Core sources
Key Source Short description
R1 Goyal et al. (2017)Making the V in VQA Matter (arXiv) Analysis of language priors in VQA and VQA v2.
R2 Li et al. (2023)SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (arXiv) Benchmark designed to reduce contamination and language priors.
R3 Chen et al. (2024)MMStar: Are We on the Right Way for Evaluating Large Vision-Language Models? (arXiv) Analysis of leakage in multimodal benchmarks and proposal for more rigorous evaluation.
R4 Hu et al. (2024)Evaluating Visual Grounding in Large Vision-Language Models (arXiv) Review of visual-grounding evaluation metrics for VLMs, emphasizing localization and calibration.
R5 Liu et al. (2024)OCRBench v2 (arXiv) Benchmark for text localization, handwriting and logical reasoning over documents.
R6 Sakshi et al. (2024)MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (Adobe Research) Audio-understanding and reasoning benchmark covering speech, non-verbal sounds and music.
R7 Yue et al. (2023)MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (arXiv) Visual-reasoning benchmark across 57 university disciplines with a persistent gap from expert humans.
R8 Fu et al. (2024)Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (arXiv) Video-understanding benchmark spanning durations from minutes to hours.
R9 Liu et al. (2023)HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (arXiv) Benchmark specifically designed to detect visual hallucinations in vision-language systems.
R10 Ouyang et al. (2024)OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (arXiv) PDF-parsing benchmark with complex layouts; presented at CVPR 2025.
R11 Roberts et al. (2025)ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (arXiv) Impossible benchmark: 20 frontier models evaluated, all at 0.0% on visual-spatial cognition.

Frequently asked questions

Why can a model score highly on a visual benchmark without actually processing the image? Because of language bias: the model answers from the statistical distribution of likely responses given the question text, not from the visual content. The effect is invisible when that statistical bias matches the correct answer because the model is right for the wrong reasons and an accuracy benchmark cannot distinguish it from a model that did process the image.

How can you detect whether a system really reasons over the temporal sequence of a video? With what the article describes as the shuffling test: pass the video frames to the model in random order before asking the question. If the score does not change or even improves after shuffling, the system does not have dynamic temporal reasoning and is answering from semantic cues present in individual frames rather than from the sequence.

What makes MMMU more demanding than image-description benchmarks? MMMU uses university-exam questions across 30 subjects where the image is not an illustration but contains the decisive evidence: circuit diagrams, laboratory graphs and radiographs. Reading the image correctly is not enough; the model has to know what it sees means. The best current models still remain significantly below expert human performance in the most technical categories.

Why does model performance fall so sharply when moving from standard text to documents with complex layouts? Because their representations are optimized for natural photographs, not for integrating spatial structure and semantics at the same time. OCRBench v2 and OmniDocBench show that systems with 80–90% accuracy on standard text fall to 36.9% on reconstruction of layouts with multiple columns, tables with merged cells or formulas embedded in text flow.

Keep learning
Next chapterRisksMultimodality in Generative AI