Codesota · Benchmark · HCASTHome/Leaderboards/HCAST
Unknown

HCAST.

90 realistic software engineering tasks calibrated against human performance times. Tests whether agents can complete tasks that take humans 15 minutes to 4 hours. Primary metric: success rate across all tasks.

Paper ↗Leaderboard ↓
§ 01 · SOTA history

Year over year.

§ 02 · Leaderboard

Results by metric.

Found a wrong score or missing run?
Use row edits to send a sourced correction into moderation.
Add / edit result ↗Report issue ↗

Success Rate

Success Rate is the reported evaluation metric for HCAST. Codesota tracks published model scores on this metric so readers can compare state-of-the-art results across sources and model families.

Higher is better

Trust tiers for Success Rateverifiedpapervendorcommunityunverified
RankModelTrustScoreYearLinksFix
01Claude Opus 4
Claude Opus 4 on HCAST. METR evaluation 2025. First model above 50% on HCAST tasks.
verified552025Paper ↗Looks wrong?
02o3
OpenAI o3 on HCAST. METR evaluation, 2025. Near 50% threshold considered significant.
verified492025Paper ↗Looks wrong?
03Claude 3.7 Sonnet
Claude 3.7 Sonnet on HCAST. METR evaluation, Feb 2025.
verified382025Paper ↗Looks wrong?
04o1
OpenAI o1 on HCAST. METR evaluation, late 2024.
verified282025Paper ↗Looks wrong?
05Claude 3.5 Sonnet
Claude 3.5 Sonnet on HCAST. METR evaluation results, 2024.
verified182025Paper ↗Looks wrong?
06GPT-4 Turbo (2024)
GPT-4 Turbo on HCAST. Baseline result from Wijk et al., 2024. 90 software tasks.
verified122023Paper ↗Looks wrong?
§ 04 · Submit a result

Add to the leaderboard.

← Back to Leaderboards