Model card
StarCoder2 15B.
BigCodeopen-source15B paramsDense Transformer
15B parameters. Trained on The Stack v2 (900B tokens). Feb 2024.
§ 01 · Benchmarks
Every benchmark StarCoder2 15B has a recorded score for.
| # | Benchmark | Area · Task | Metric | Value | Rank | Date | Source |
|---|---|---|---|---|---|---|---|
| 01 | TransCoder (GeeksForGeeks) | Computer Code · Code Translation | computational-accuracy | 78.4% | #5 | 2024-02-29 | source ↗ |
| 02 | CrossCodeEval | Computer Code · Code Completion | exact-match | 32.1% | #6 | 2024-02-29 | source ↗ |
| 03 | MBPP | Computer Code · Code Generation | pass@1 | 54.4% | #19 | 2024-02-29 | source ↗ |
| 04 | SWE-Bench | Computer Code · Code Generation | resolve-rate | 18.3% | #29 | 2024-10-01 | source ↗ |
| 05 | HumanEval | Computer Code · Code Generation | pass@1 | 46.9% | #41 | 2024-02-29 | source ↗ |
Rank column shows this model’s position vs all other models scored on the same benchmark + metric (competitors after the slash). #1 in red means current SOTA. Sorted by rank, then newest result.
§ 02 · Strengths by area
Where StarCoder2 15B actually performs.
§ 03 · Papers
2 papers with results for StarCoder2 15B.
- 2024-02-29· Computer Code· 4 results
StarCoder2 and The Stack v2: The Next Generation
- 2023-10-10· Computer Code· 1 result
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.
§ 04 · Related models
Other BigCode models scored on Codesota.
§ 05 · Sources & freshness
Where these numbers come from.
arxiv
4
results
swebench-leaderboard
1
result
5 of 5 rows marked verified. · first result 2024-02-29, latest 2024-10-01.