Recent Papers / arXiv:2605.12925

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

arXiv:2605.12925Submitted Jun 3, 20260 benchmark results

Authors pending

View PDF ↗arXiv page ↗Edit
Abstract

Process-level analysis of 2,614 SWE-agent trajectories; 10.7% of passing runs are 'Lucky Passes' with chaotic behavior.

Tasks
edit
• Agentic AI
Results

No benchmark results recorded yet.

submit

Benchmark results referencing this paper haven't been added to the registry yet. If you have a reproduction, submit it →

CodeSOTA extraction

Benchmark evidence

edit
  • Verify AgentLens-Bench 'Lucky Pass' rate (10.7%) and whether model rankings shift by 5+ positions when using quality score vs. pass rate.
Add or update benchmark results
Logged-in editor · benchmark trail
Read next

Three places to go from here.

Index
All papers
All tracked papers in the registry, with benchmark result, model, and leaderboard linkage where available.
Replacement
Papers with Code is dead — alternatives
What replaced PWC for each use case: LLMs, OCR, speech, vision, robotics.
Top hub
Agentic AI
Every benchmark in Agentic AI.