Mostly Basic Python Problems (MBPP) is a benchmark for function-level Python code generation consisting of short, entry-level programming problems paired with natural language task descriptions, reference solutions, and automated unit tests. The public Hugging Face versions contain 974 problems (with a sanitized subset of 427 examples available) covering basic numeric, list, and string manipulations and common standard-library usage. MBPP was introduced to evaluate the ability of neural models to synthesize short Python programs from natural language prompts (used in few-shot and fine-tuning evaluations); the dataset is commonly used to report pass@k or exact-match test metrics for code generation models. License: CC BY 4.0.
Pass@1 is the reported evaluation metric for MBPP. Codesota tracks published model scores on this metric so readers can compare state-of-the-art results across sources and model families.
Higher is better
Muted rows were not state of the art when published — an earlier or same-year result already scored better.
| Rank | Model | Trust | Score | Year | Links | Fix |
|---|---|---|---|---|---|---|
| 01 | Qwen2.5-72B-Instruct | paper | 88.2 | N/A | Paper ↗Code ↗Source ↗ | Looks wrong? |