Model card
Llama 3.1 405B.
Metaopen-source
Meta Llama 3.1, 405B parameter instruct variant. Released July 2024.
§ 02 · Benchmarks
Every benchmark Llama 3.1 405B has a recorded score for.
| # | Benchmark | Area · Task | Metric | Value | Rank | Date | Source |
|---|---|---|---|---|---|---|---|
| 01 | CNN/DailyMail | Natural Language Processing · Text Summarization | rouge-1 | 45.1% | #4 | 2024-07-31 | source ↗ |
| 02 | CNN/DailyMail | Natural Language Processing · Text Summarization | rouge-l | 42.3% | #4 | 2024-07-31 | source ↗ |
| 03 | CoNLL-2003 | Natural Language Processing · Named Entity Recognition | f1 | 90.6% | #4 | 2024-07-31 | source ↗ |
| 04 | SNLI | Natural Language Processing · Natural Language Inference | accuracy | 91.2% | #5 | 2024-07-31 | source ↗ |
| 05 | HellaSwag | Reasoning · Commonsense Reasoning | accuracy | 89.0% | #5 | — | source ↗ |
| 06 | SuperGLUE | Natural Language Processing · Text classification | average-score | 86.7% | #6 | 2024-07-31 | source ↗ |
| 07 | ARC-Challenge | Reasoning · Commonsense Reasoning | accuracy | 96.9% | #6 | — | source ↗ |
| 08 | BIG-Bench Hard | Reasoning · Multi-step Reasoning | accuracy | 85.9% | #7 | — | source ↗ |
| 09 | SQuAD v2.0 | Natural Language Processing · Question Answering | f1 | 88.7% | #15 | 2024-07-31 | source ↗ |
| 10 | HumanEval | Computer Code · Code Generation | pass@1 | 89.0% | #20 | — | source ↗ |
| 11 | MMLU | Reasoning · Commonsense Reasoning | accuracy | 88.6% | #22 | — | source ↗ |
| 12 | MATH | Reasoning · Mathematical Reasoning | accuracy | 73.8% | #31 | — | source ↗ |
| 13 | GPQA Diamond | Reasoning · Multi-step Reasoning | accuracy | 50.7% | #62 | — | source ↗ |
Rank column shows this model’s position vs all other models scored on the same benchmark + metric (competitors after the slash). #1 in red means current SOTA. Sorted by rank, then newest result.
§ 03 · Strengths by area
Where Llama 3.1 405B actually performs.
§ 04 · Papers
1 paper with results for Llama 3.1 405B.
- 2024-07-31· Natural Language Processing· 6 results
The Llama 3 Herd of Models
§ 05 · Related models
Other Meta models scored on Codesota.
§ 06 · Sources & freshness
Where these numbers come from.
arxiv
6
results
openai-simple-evals
4
results
meta-modelcard
2
results
llm-stats-bbh
1
result
9 of 13 rows marked verified.