Agentic AI Benchmarks Explained

What it measures

Whether an agent can autonomously resolve real GitHub issues — reading codebases, localizing bugs, writing patches, and passing existing test suites.

How it works

1. 1Agent receives a GitHub issue description and the full repository at the relevant commit.
2. 2It must produce a code patch that resolves the issue.
3. 3The patch is validated by running the project's own unit/integration tests.
4. 4An instance is "resolved" only if all relevant tests pass and no existing tests break.

Key findings

• Scaffolding matters as much as model quality — the same model can vary 15+ points depending on the agent framework.
• Agents still struggle with large codebases (>100k lines) where localization is the bottleneck.
• Most resolved issues are small, localized bug fixes — multi-file architectural changes remain extremely difficult.

What it measures

Whether an agent can tackle open-ended ML research engineering tasks — optimizing training loops, implementing novel architectures, and debugging performance issues — given extended time budgets (up to 8 hours).

How it works

1. 1Agent receives a research engineering task with a clear metric to optimize (e.g., reduce loss, improve accuracy).
2. 2It has access to a compute environment with GPUs, libraries, and starter code.
3. 3Tasks are scored on a 0-100 scale based on how close the agent gets to expert-level performance.
4. 4Time budgets range from 1 to 8 hours, measuring how agents allocate effort.

Key findings

• Agents achieve roughly 4x less score per hour than human experts, but close the gap with longer time budgets.
• The biggest failure mode is getting stuck in unproductive loops — repeating the same failed approach without pivoting.
• Agents are better at optimization tasks (hyperparameter tuning) than creative design tasks (novel architectures).

What it measures

Whether AI agents can safely and reliably complete realistic autonomous tasks — and critically, whether they know when to stop, ask for help, or refuse unsafe actions.

How it works

1. 1Tasks span categories like web research, data analysis, system administration, and communication.
2. 2Each task has both a capability score (did it succeed?) and a safety score (did it avoid harmful actions?).
3. 3Human contractors complete the same tasks to establish calibrated baselines.
4. 4Tasks include adversarial elements — e.g., instructions that seem reasonable but would cause harm if followed blindly.

Key findings

• Higher capability does not always correlate with higher safety — o1 completes more tasks but takes more unsafe actions.
• Agents rarely ask for clarification even when tasks are genuinely ambiguous, leading to silent failures.
• The gap between agent and human performance is largest on tasks requiring real-world judgment (e.g., "is this email appropriate to send?").

What it measures

Whether an agent can complete realistic tasks on real websites — shopping, forum management, code repository navigation, content management, and map-based tasks.

How it works

1. 1Five self-hosted websites simulate realistic web apps: e-commerce (OneStopShop), forums (Reddit clone), GitLab, CMS, and OpenStreetMap.
2. 2Agent receives a natural language instruction (e.g., "Find the cheapest wireless mouse and add it to cart").
3. 3It interacts through browser actions: click, type, scroll, navigate.
4. 4Success is measured by functional correctness — did the task actually get completed?

Key findings

• Raw model performance (14%) vs. agent scaffolding (35%+) shows the framework matters more than the model for web tasks.
• Agents fail most on tasks requiring long action sequences (>15 steps) — they lose track of state.
• Visual grounding (screenshot-based) agents are catching up to DOM-based agents but still trail by ~5%.

What it measures

Whether an agent can use a full desktop OS — opening applications, editing documents, managing files, configuring settings, and chaining multi-app workflows.

How it works

1. 1Agent controls a virtual machine through screenshots and keyboard/mouse actions.
2. 2Tasks span Office apps, web browsers, file managers, terminals, and system settings.
3. 3Evaluation checks the actual OS state after the agent acts (file contents, app states, configs).
4. 4Some tasks require chaining multiple applications (e.g., "download the CSV from email, open it in LibreOffice, sort by column B").

Key findings

• Even the best agents solve less than 1 in 4 tasks — desktop computer use remains extremely challenging for AI.
• Agents are weakest on tasks requiring precise spatial reasoning (drag-and-drop, window resizing, menu navigation).
• Multi-app workflows see ~3x failure rates compared to single-app tasks.

What it measures

Whether an agent can answer questions that are simple for humans but require multi-step tool use for AI — web browsing, file processing, calculation, and reasoning in combination.

How it works

1. 1Questions are designed to be trivially solvable by humans (>90% accuracy) but hard for AI.
2. 2Three levels: L1 (1-2 tool uses), L2 (5-10 steps), L3 (long chains with file analysis).
3. 3Agents must use tools: web search, code execution, file reading, calculator.
4. 4Exact-match evaluation — the answer must be precisely correct.

Key findings

• Humans still outperform the best AI agents by 16+ points, primarily on Level 2 and Level 3 tasks.
• Agent failures cluster around tool selection errors — using the wrong tool rather than using it incorrectly.
• The gap between Level 1 (~90% for top agents) and Level 3 (~45%) shows compounding failures in multi-step reasoning.

What it measures

Whether an agent can handle realistic multi-turn customer service conversations requiring database lookups, policy adherence, and tool calls — without hallucinating actions or violating business rules.

How it works

1. 1Agent plays a customer service representative with access to databases and tools (order lookup, flight rebooking, refund processing).
2. 2A simulated user follows a scripted scenario with specific needs and constraints.
3. 3The agent must use tools correctly AND follow domain-specific policies (return windows, fare rules, etc.).
4. 4Evaluation checks both task completion and policy compliance at each turn.

Key findings

• Airline tasks are significantly harder than retail due to more complex policies (fare classes, connection rules, rebooking logic).
• The most common failure mode is policy violation — agents complete the task but break a business rule in the process.
• Larger models are disproportionately better at policy compliance than at raw task completion.