LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K.

LV-Eval is a bilingual long-context benchmark designed to evaluate large language models at very large context lengths (up to 256k tokens). It provides controllable evaluation across five length levels (16k, 32k, 64k, 128k, 256k) and includes multiple QA-style tasks (single-hop and multi-hop QA) drawn from several bilingual datasets. The benchmark incorporates techniques to reduce knowledge leakage and increase difficulty and objectivity: confusing facts insertion (CFI), keyword and phrase replacement (KPR), and a keyword-recall-based metric evaluated at multiple lengths. LV-Eval is provided with balanced numbers of instances across lengths and is intended to stress-test long-context capabilities of LLMs.

Best published scores.

1 result indexed across 1 metric. Shaded row marks current SOTA; ties broken by submission date.

Primary

Accuracy · higher is better

1 steps
of state of the art.

Each row below marks a model that broke the previous record on Accuracy. Intermediate submissions are kept in the leaderboard above; only SOTA-setting entries are re-listed here.

Higher scores win. Each subsequent entry improved upon the previous best.

1. Dec 19, 2024Qwen2.5-72B-Instruct60.40

1 paper
tied to this benchmark.

Every paper below corresponds to at least one row in the leaderboard above. Click through for the arXiv preprint and, when available, the reference implementation.

Neighbouring benchmarks.

Other Language Modeling datasets on Codesota.

Have a score that beats
this table?

Submit a checkpoint and a reproduction script. We will run it, publish the score, and — if it takes the top — annotate the step on the progress chart with your name.

• 01A public checkpoint or API endpoint
• 02A reproduction script with frozen commit + seed
• 03Declared evaluation environment (Python, deps)
• 04One row per metric declared by this dataset
• 05A contact so we can follow up on discrepancies