Codesota · Reasoning · Commonsense Reasoning · ARC-ChallengeTasks/Reasoning/Commonsense Reasoning
Commonsense Reasoning · benchmark dataset · 2018 · EN

AI2 Reasoning Challenge.

7,787 science questions requiring reasoning. Challenge set contains harder questions that retrieval fails on.

Paper ↗Download datasetSubmit a result ↵
§ 01 · Leaderboard

Best published scores.

10 results indexed across 1 metric. Shaded row marks current SOTA; ties broken by submission date.

Primary
accuracy · higher is better
accuracy· primary
10 rows
#ModelOrgSubmittedPaper / codeaccuracy
01o3OpenAIMar 2026openai-simple-evals98.10
02Gemini 2.5 ProAPIGoogleMar 2026google-technical-report97.80
03Llama 4 MaverickOpenMetaMar 2026meta-blog97.40
04o4-miniOpenAIMar 2026openai-simple-evals97.30
05DeepSeek R1OpenDeepSeekMar 2026arxiv97.10
06Llama 3.1 405BOpenMetaMar 2026meta-modelcard96.90
07Claude 3.5 SonnetAPIAnthropicDec 2025anthropic-blog96.70
08GPT-4oAPIOpenAIDec 2025openai-blog96.40
09Gemini 1.5 ProAPIGoogleDec 2025google-blog94.80
10Llama 3 70BOpenMetaDec 2025meta-blog93
Fig 2 · Rows sorted by score within each metric. Shaded row marks SOTA. Dates reflect model or paper release where available, otherwise the date Codesota accessed the source.
Research graph

Research connected to this record.

1 question · 3 claims · 1 experiment

Questions, claims, and experiments linked through CodeSOTA’s public evidence graph.

Research questions · 1

Claims · 3

Experiments · 1

Explore the research graph →
§ 03 · Progress

2 steps
of state of the art.

Each row below marks a model that broke the previous record on accuracy. Intermediate submissions are kept in the leaderboard above; only SOTA-setting entries are re-listed here.

Higher scores win. Each subsequent entry improved upon the previous best.

SOTA line · accuracy
  1. Dec 17, 2025Claude 3.5 SonnetAnthropic96.70
  2. Mar 27, 2026o3OpenAI98.10
Fig 3 · SOTA-setting models only. 2 entries span Dec 2025 → Mar 2026.
§ 06 · Contribute

Have a score that beats
this table?

Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.

Submit a result ↵Read submission guide
What a submission needs
  • 01A public checkpoint or API endpoint
  • 02A reproduction script with frozen commit + seed
  • 03Declared evaluation environment (Python, deps)
  • 04One row per metric declared by this dataset
  • 05A contact so we can follow up on discrepancies