MultiPL-E is a multi-programming-language benchmark for evaluating natural-language-to-code generation by large language models. It translates unit-test-driven Python benchmarks (OpenAI HumanEval and MBPP) into parallel problems in multiple programming languages, preserving prompts and test harnesses so models can be evaluated via execution-based metrics. The released dataset provides per-language configurations (e.g., humaneval-<lang>, mbpp-<lang>) containing prompts, tests, doctests, stop tokens and related metadata; the original project translated the Python benchmarks into 18 languages (and Hugging Face distributions expose many language-specific configs). Source code and dataset tooling are available from the NuPRL project (GitHub) and the authors published a paper describing the benchmark and methodology (arXiv:2208.08227 / IEEE TSE publication). License: MIT.
1 result indexed across 1 metric. Shaded row marks current SOTA; ties broken by submission date.
| # | Model | Org | Submitted | Paper / code | Pass@1 |
|---|---|---|---|---|---|
| 01 | Qwen2.5-Plus | — | Dec 2024 | Qwen2.5 Technical Report · code | 77 |
Each row below marks a model that broke the previous record on Pass@1. Intermediate submissions are kept in the leaderboard above; only SOTA-setting entries are re-listed here.
Higher scores win. Each subsequent entry improved upon the previous best.
Every paper below corresponds to at least one row in the leaderboard above. Click through for the arXiv preprint and, when available, the reference implementation.
Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.