Help Prioritize Research

234 benchmarks need research to determine if they're still relevant. Vote on which ones should be updated first.

234 Undecided
View Saturated/Legacy

How this works

1

Vote for benchmarks

Click the upvote button on benchmarks you want us to research

2

We research top voted

Using Exa and manual review to find latest SOTA results

3

Benchmarks get updated

Fresh data gets added, or benchmark marked as saturated/legacy

Top Voted

OmniDocBench v1.5Computer Vision/Document Parsing
47 resultsNo paper dates
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Computer Code/Code Generation
57 resultsLatest: Oct 2023
inverse-textComputer Vision/Optical Character Recognition
34 resultsLatest: May 2023
X-Ray Weld Defect Detection DatasetIndustrial Inspection/Anomaly Detection
1 resultsNo paper dates
Mathematics Aptitude Test of HeuristicsReasoning/Mathematical Reasoning
34 resultsNo paper dates
coco-textComputer Vision/Scene Text Detection
33 resultsLatest: May 2023
videodb's-ocr-benchmark-public-collectionComputer Vision/Optical Character Recognition
15 resultsLatest: Feb 2025
SuperGLUENatural Language Processing/Text classification
7 resultsLatest: Jul 2024
Open Graph BenchmarkGraphs/Node Classification
0 resultsNo paper dates
Cityscapes DatasetComputer Vision/Semantic Segmentation
0 resultsNo paper dates

Computer Vision161

OmniDocBench v1.5/Document Parsing
47 resultsNo paper dates
inverse-text/Optical Character Recognition
34 resultsLatest: May 2023
coco-text/Scene Text Detection
33 resultsLatest: May 2023
videodb's-ocr-benchmark-public-collection/Optical Character Recognition
15 resultsLatest: Feb 2025
Cityscapes Dataset/Semantic Segmentation
0 resultsNo paper dates
Total-Text/Scene Text Detection
126 resultsLatest: Dec 2024
msra-td500/Scene Text Detection
79 resultsLatest: Oct 2024
icdar-2017-mlt/Scene Text Detection
54 resultsLatest: May 2022
table-recognition-challenge-mini-test/Table Recognition
33 resultsNo paper dates
dart/Optical Character Recognition
32 resultsLatest: Oct 2023
icdar2015/Optical Character Recognition
30 resultsLatest: Nov 2025
tabfact/Optical Character Recognition
30 resultsLatest: Dec 2024
Comprehensive Challenge OCR/General OCR Capabilities
28 resultsNo paper dates
iiit5k/Scene Text Recognition
21 resultsLatest: Aug 2023
sun-rgb-d/Optical Character Recognition
20 resultsLatest: Jun 2021
cute80/Scene Text Recognition
20 resultsLatest: Aug 2023
svtp/Scene Text Recognition
19 resultsLatest: Aug 2023
tobacco-3482/Document Image Classification
18 resultsLatest: Jan 2023
Curved Text in the Wild 1500/Scene Text Detection
18 resultsLatest: Feb 2022
codesearchnet---java/Optical Character Recognition
14 resultsLatest: Apr 2021

+141 more...

Multimodal14

DPG-Bench/Text-to-Image Generation
0 resultsNo paper dates
MMStar/Image-Text-to-Text
0 resultsNo paper dates
MMBench/Image-Text-to-Text
0 resultsNo paper dates
MMMU/Image-Text-to-Text
0 resultsNo paper dates
GenEval/Text-to-Image Generation
0 resultsNo paper dates
DEMON Bench/Any-to-Any
0 resultsNo paper dates
ViDoRe/Cross-Modal Retrieval
0 resultsNo paper dates
InstructPix2Pix/Image-Text-to-Image
0 resultsNo paper dates
MVBench/Video Understanding
0 resultsNo paper dates
VideoBench/Image-Text-to-Video
0 resultsNo paper dates
MJHQ-30K FID/Text-to-Image Generation
0 resultsNo paper dates
AudioBench/Audio-Text-to-Text
0 resultsNo paper dates
Video-MME/Video Understanding
0 resultsNo paper dates
MagicBrush/Image-Text-to-Image
0 resultsNo paper dates

Natural Language Processing12

SuperGLUE/Text classification
7 resultsLatest: Jul 2024
MTEB Leaderboard/Feature Extraction
6 resultsLatest: Sep 2024
MS MARCO/Text Ranking
4 resultsLatest: Oct 2023
BEIR/Text Ranking
4 resultsLatest: Sep 2024
WMT'23/Machine Translation
4 resultsNo paper dates
XNLI/Zero-Shot Classification
3 resultsLatest: Jan 2023
STS Benchmark/Semantic Textual Similarity
3 resultsLatest: Jan 2024
WikiTableQuestions/Table Question Answering
3 resultsLatest: Apr 2020
GLUE/Fill-Mask
3 resultsLatest: Jan 2023
WikiText Perplexity/Language Modeling
0 resultsNo paper dates
FLORES-200/Machine Translation
0 resultsNo paper dates
SQA/Table Question Answering
0 resultsNo paper dates

Reasoning9

Mathematics Aptitude Test of Heuristics/Mathematical Reasoning
34 resultsNo paper dates
AI2 Reasoning Challenge/Commonsense Reasoning
10 resultsNo paper dates
American Invitational Mathematics Examination 2024/Mathematical Reasoning
8 resultsNo paper dates
HellaSwag/Commonsense Reasoning
5 resultsNo paper dates
CommonsenseQA/Commonsense Reasoning
3 resultsNo paper dates
WinoGrande/Commonsense Reasoning
3 resultsNo paper dates
Simple Variations on Arithmetic Math Word Problems/Arithmetic Reasoning
3 resultsNo paper dates
StrategyQA/Multi-step Reasoning
2 resultsNo paper dates
Reading Comprehension Dataset Requiring Logical Reasoning/Logical Reasoning
2 resultsNo paper dates

Medical8

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels/Disease Classification
7 resultsNo paper dates
NIH Clinical Center Chest X-ray Dataset/Disease Classification
4 resultsNo paper dates
MIMIC-CXR: Medical Information Mart for Intensive Care - Chest X-ray/Disease Classification
3 resultsNo paper dates
RSNA Pneumonia Detection Challenge/Disease Classification
3 resultsLatest: Jan 2024
VinDr-CXR: Vietnamese Dataset for Chest Radiograph/Disease Classification
2 resultsNo paper dates
COVID-19 Image Data Collection/Disease Classification
2 resultsNo paper dates
Autism Brain Imaging Data Exchange II/Disease Classification
2 resultsNo paper dates
PadChest: A Large Chest X-ray Image Dataset/Disease Classification
1 resultsNo paper dates

Audio8

The LJ Speech Dataset/Text-to-speech
5 resultsLatest: Jun 2024
AudioCaps/Audio Captioning
3 resultsNo paper dates
MusicCaps/Music Generation
3 resultsNo paper dates
Environmental Sound Classification 50/Audio Classification
0 resultsNo paper dates
DIHARD/Voice Activity Detection
0 resultsNo paper dates
VCTK (Voice Conversion)/Audio-to-Audio
0 resultsNo paper dates
DNS Challenge/Audio-to-Audio
0 resultsNo paper dates
AVA-Speech/Voice Activity Detection
0 resultsNo paper dates

Computer Code7

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?/Code Generation
57 resultsLatest: Oct 2023
SWE-bench Verified Subset/Code Generation
39 resultsNo paper dates
Mostly Basic Python Problems/Code Generation
19 resultsLatest: Sep 2024
MBPP+ Extended Version/Code Generation
0 resultsNo paper dates
Automated Programming Progress Standard/Code Generation
0 resultsNo paper dates
CodeContests Competitive Programming/Code Generation
0 resultsNo paper dates
HumanEval+ Extended Version/Code Generation
0 resultsNo paper dates

Industrial Inspection6

X-Ray Weld Defect Detection Dataset/Anomaly Detection
1 resultsNo paper dates
Kolektor Surface Defect Dataset 2/Anomaly Detection
6 resultsLatest: Aug 2024
MVTec 3D Anomaly Detection Dataset/Anomaly Detection
6 resultsLatest: Mar 2025
Visual Anomaly Dataset/Anomaly Detection
3 resultsNo paper dates
Severstal Steel Defect Detection/Anomaly Detection
1 resultsNo paper dates
NEU Surface Defect Database/Anomaly Detection
1 resultsNo paper dates

Graphs2

Open Graph Benchmark/Node Classification
0 resultsNo paper dates
OGB (Open Graph Benchmark)/Graph Classification
0 resultsNo paper dates

Time Series2

OpenML-CC18/Tabular Classification
5 resultsLatest: Jun 2025
California Housing/Tabular Regression
2 resultsNo paper dates

Other2

RLBench/Robotics
0 resultsNo paper dates
SIMPLER/Robotics
0 resultsNo paper dates

Speech1

Mozilla Common Voice/Speech Recognition
3 resultsLatest: Dec 2022

Methodology1

ImageNet Linear Probe/Self-Supervised Learning
0 resultsNo paper dates

Agentic AI1

Audrey public memory benchmark artifacts/Agent Memory
0 resultsNo paper dates