Visual question answering,
measured.

What the metric actually measures.

Visual question answering asks a model to read an image and answer a natural-language question about it. The archetypal output is a short string — “three”, “red”, “a golden retriever” — graded against human-written references using exact or soft matching.

The classic VQAv2 and GQA benchmarks were designed in 2017–2019 for specialist models trained from scratch. Frontier VLMs trained on internet-scale image-text data now saturate them in zero-shot; the human ceiling on VQAv2 is around 80%, and current frontier models sit at 84–87%. Past that point a two-point delta is noise about dataset quirks, not signal about model quality.

The interesting evaluations in 2026 are MMMU (multi-discipline reasoning), MathVista (visual math), ChartQA, DocVQA and MMBench. Leaderboards on each of these diverge sharply — no single number captures “best VLM”, and your own task should be your primary eval.

What the metric misses: calibration. A model that confidently invents an answer is strictly worse than one that says “I’m not sure.” VLMs systematically over-predict counts and over-affirm presence (“is there a cat?” → “yes”, even with no cat in frame). Leaderboards don’t penalise that; your production eval should.

What VQA-accuracy means here.

The original VQAv2 metric gives full credit when at least three of ten human annotators produced the same answer string (after light normalisation). It is forgiving of synonyms by construction — “sofa” and “couch” both score — and unforgiving of extra words.

Newer suites (MMMU, MMBench) use multiple-choice or short-answer formats evaluated by exact match, which eliminates the ten-annotator quirk but introduces format-sensitivity of its own. DocVQA uses ANLS (Average Normalised Levenshtein Similarity) to tolerate OCR-style near-misses on long strings.

Pricing figures in § 01 are token-derived: each provider prices images as a function of resolution and detail flag, then bills tokens at the model’s text rate. Numbers above assume standard-resolution input; high-detail can cost 2–4× more. Click a price to see the vendor’s own pricing page.