Optical Character Recognition2020en
CodeSearchNet
Benchmark for code summarization (docstring generation) across 6 programming languages: Python, Java, JavaScript, PHP, Ruby, Go. Over 2M (code, docstring) pairs. Primary metric is BLEU-4.
Metrics:accuracy, cer, wer, f1
bleu-4
| # | Model | Score | Paper / Code | Date |
|---|---|---|---|---|
| 1 | GPT-4oAPI OpenAI | 25.3 | Mar 2026 | |
| 2 | Qwen2.5-Coder-32B-InstructOpen Source Alibaba | 23.4 | Sep 2024 | |
| 3 | DeepSeek-Coder-V2-InstructOpen Source DeepSeek | 22.8 | Jun 2024 | |
| 4 | CodeT5+ 2BOpen Source Salesforce | 21.36 | May 2023 | |
| 5 | CodeT5+Open Source Salesforce | 20.01 | May 2023 | |
| 6 | UniXcoderOpen Source Microsoft | 19.06 | Mar 2022 | |
| 7 | CodeBERTOpen Source Microsoft | 17.65 | Feb 2020 |
smoothed-bleu-4
| # | Model | Score | Paper / Code | Date |
|---|---|---|---|---|
| 1 | CodeBERT (MLM+RTD) | 15.99 | Feb 2020 | |
| 2 | CodeBERT (MLM) | 15.55 | Feb 2020 | |
| 3 | pre-train w/ code only | 15.15 | Feb 2020 | |
| 4 | CodeBERT (RTD) | 15.03 | Feb 2020 | |
| 5 | RoBERTa | 14.52 | Feb 2020 | |
| 6 | Transformer | 14.31 | Feb 2020 | |
| 7 | seq2seq | 13.36 | Feb 2020 |
Related Papers6
Qwen2.5-Coder Technical Report
Sep 2024Models: Qwen2.5-Coder-32B-Instruct
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
Jun 2024Models: DeepSeek-Coder-V2-Instruct
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
May 2023Models: CodeT5+ 2B, CodeT5+
UniXcoder: Unified Cross-Modal Pre-Training for Code Representation
Mar 2022Models: UniXcoder
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feb 2020Models: CodeBERT
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feb 2020Models: CodeBERT (MLM+RTD), CodeBERT (MLM), pre-train w/ code only +4 more