Evaluating Multimodal Systems
How to evaluate multimodal systems without confusing benchmarks with real capability: OCR, audio, grounding, reasoning and metric failures.
Links containing ?t= open the video at a specific second.
The ideas to retain
1. What it means to evaluate grounding
In multimodal systems, grounding is the degree to which the model's answer is supported by the actual content of the image or audio, rather than by statistical inferences about what kind…
2. The problem of benchmark contamination
The second systematic problem is contamination. Foundation models are pretrained on massive amounts of internet data, and there is no guarantee that image-description pairs or evaluation…
3. Language bias: answering from probability, not evidence
The language bias/prior is the tendency of models to generate answers that are statistically likely given the text of the question, regardless of the image content. It is the subtlest form…
