Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models.

Vibe-Eval is an open benchmark for evaluating multimodal chat models. It consists of 269 visual understanding prompts, including 100 of hard difficulty, complete with gold-standard responses authored by experts. The benchmark is designed to be open-ended and challenging with dual objectives: (i) vibe checking multimodal chat models for day-to-day tasks and (ii) rigorously testing and probing the capabilities of present frontier models. Notably, the hard set contains >50% questions that all frontier models answer incorrectly.

Best published scores.

No results indexed yet — be the first to submit a score.

Neighbouring benchmarks.

Other Vision-Language Models datasets on Codesota.

Have a score that beats
this table?

Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.

• 01A public checkpoint or API endpoint
• 02A reproduction script with frozen commit + seed
• 03Declared evaluation environment (Python, deps)
• 04One row per metric declared by this dataset
• 05A contact so we can follow up on discrepancies