MuirBench is a benchmark dataset for vision language models. It is designed to evaluate robust multi-image understanding, covering various types of multi-image relations such as temporal, ordered-pages, or narrative relations. It also includes unanswerable questions to fairly assess multimodal LLMs. The dataset aims to identify the gap between current multi-modal language models and humans in understanding multiple image inputs.
No results indexed yet — be the first to submit a score.
Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.