LLM/evaluation methodology

CodeSOTA field guide · base models · 2026

Do not ask a small model
questions that are too large.

A practical benchmark ladder for models from 1M to 7B parameters. Each test enters when it produces a signal—after performance separates from chance, but before saturation.

01above chance

The model is no longer behaving randomly.

02below ceiling

The score has not yet saturated.

03measurable movement

Checkpoints differ by more than measurement error.

04base-model first

Scoring does not require instruction tuning.

01 / map

One model family.
A changing measurement instrument.

primary — included in the KPIleading — looks for the first signal
01
1–10Mparameters

Language structure

Text compression and basic sequential dependencies.

PRIMARY
validation BPBsynthetic tests
LEADING
BLiMP
02
10–50Mparameters

Local syntax

The first external language signal, but still no knowledge KPI.

PRIMARY
BPBBLiMPSyntaxGym
LEADING
SciQLAMBADA NLL
03
50–150Mparameters

Simple semantics

The transition from linguistic form to lexical knowledge.

PRIMARY
BLiMPSciQ soft scoreLAMBADA NLL
LEADING
ARC-EasyPIQA
04
150–500Mparameters

Knowledge and common sense

The most informative range for classic small base models.

PRIMARY
SciQARC-EasyLAMBADAPIQA
LEADING
HellaSwagOpenBookQA
05
500M–1Bparameters

Hard completions

Harder distractors take over from simple knowledge tests.

PRIMARY
ARC-EasyLAMBADAPIQAHellaSwag
LEADING
WinoGrandeARC-Challenge
06
1–3Bparameters

Multiple choice

Broader knowledge and common sense; report MMLU by category.

PRIMARY
HellaSwagWinoGrandeARC-ChallengeOpenBookQA
LEADING
MMLUGSM8K
07
3–7Bparameters

Knowledge and reasoning

Math and code only now become stable parts of the suite.

PRIMARY
MMLUARC-ChallengeHellaSwagGSM8K
LEADING
BBHHumanEval
08
7B+parameters

Frontier

General tests give way to harder domain evaluations.

PRIMARY
MMLUGSM8KBBHHumanEval
LEADING
specialist benchmarks
02 / signal before accuracy

Measure the margin first.
Count the hit second.

BPB

Compare tokenizers fairly.

BPB = NLL / (Nbytes · ln 2)

Measure prose, news, dialogue, code, and technical documents separately. BPB remains the primary metric for 1–50M models.

BLiMP

Measure direction and confidence.

m = log P(xgood) − log P(xbad)

Report the mean margin and soft score σ(m) alongside accuracy. A small model can improve its distribution before changing the binary decision.

LAMBADA

Do not wait for exact match.

Ltarget = −log P(wtarget | context)

For 50–150M models, use target-token NLL, MRR, and top-k. Exact match becomes a primary metric only after it stops being nearly zero.

MULTIPLE CHOICE

Normalize answer length.

score(a) = log P(a | q) / |a|

Also report the probability of the correct answer and its margin over the strongest distractor. HellaSwag without normalization favors shorter completions.

03 / dynamic range

An easy benchmark starts earlier.
A hard one remains useful longer.

The Pythia family shows how useful ranges overlap. SciQ produces a strong signal early and approaches its ceiling; ARC-Challenge starts late and improves slowly. ARC-Easy and LAMBADA bridge the middle of the scale.

ModelSciQARC-EasyLAMBADAARC-Challenge
70M60.1%37.4%18.5%18.1%
160M74.1%43.5%32.8%18.8%
410M81.1%52.1%51.6%21.3%
1B84.0%56.9%56.2%24.4%
2.8B88.2%64.4%64.7%29.5%
6.9B89.7%67.3%67.3%31.3%

The Pythia reference values illustrate dynamic range; they do not constitute a new model ranking.

04 / lifecycle

When does a benchmark
move up the ladder?

01LEADING

Finds the onset of capability.

Accuracy is near chance, but the soft score or margin begins to move. Example: SciQ at 28% against a 25% chance baseline for a 50M byte-level model.

02PRIMARY

Separates models and checkpoints.

The score is clearly above chance, differences exceed measurement error, performance remains below 90–95%, and the prompt does not dominate the result.

03CONTROL

Preserves continuity.

The test stays in the report but leaves the aggregate. This applies, for example, to SciQ at several billion parameters or saturated BLiMP categories.

05 / protocol

A minimal suite,
the full history of scale.

  1. 01validation bits-per-byte
  2. 02BLiMP
  3. 03SciQ
  4. 04ARC-Easy
  5. 05LAMBADA
  6. 06PIQA
  7. 07HellaSwag
  8. 08WinoGrande
  9. 09ARC-Challenge
  10. 10MMLU
  11. 11GSM8K
  12. 12HumanEval — code-trained models only

Fixed reporting rules

  • Do not mix base and instruction-tuned results.
  • Break MMLU out by domain; do not begin with one overall mean.
  • Mark GSM8K for a base model as dependent on its prompt and math training data.
  • Run HumanEval and MBPP only for models trained on code.
  • Report confidence intervals for binary PIQA and WinoGrande.
  • Keep at least two shared benchmarks between neighboring scale bands.
OUTSIDE A BASE MODEL'S SCOPE

IFEval, AlpacaEval, MT-Bench, format adherence, and tool use require instruction tuning. They should not lower the assessment of a raw pretraining checkpoint.