SEED-Bench is a large-scale multimodal benchmark for evaluating generative comprehension of Multimodal Large Language Models (MLLMs). Introduced in the paper “SEED-Bench: Benchmarking Multimodal Large Language Models with Generative Comprehension” (arXiv:2307.16125, CVPR 2024), the benchmark contains ~19K multiple-choice questions with human-verified ground-truth answers spanning 12 evaluation dimensions (covering both image and video modalities and a range of capabilities such as scene understanding, instance identity/attribute/location/counting, spatial relations, text recognition, action recognition/prediction, visual reasoning, chart understanding, meme comprehension, etc.). Questions were generated with an automated pipeline followed by manual verification to ensure high-quality human annotations; the format (multiple-choice with gold options) enables objective, automated evaluation without human/GPT intervention. The dataset is distributed under CC BY-NC 4.0 and is available on Hugging Face (author/repo: AILab-CVC/SEED-Bench).
No results indexed yet — be the first to submit a score.
Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.