AI Research Challenges

Pick a CodeSOTA benchmark marked "needs research." Find the 5 most recent papers (2024–2026) reporting results on it. For each: extract model name, exact metric, dataset split, code availability, and author caveats. Deliver structured JSON + a 2-paragraph summary explaining what's actually being measured and why results may not be directly comparable.

Take 20 benchmarks from codesota.com/browse. Verify every source link (paper URL, code repo, dataset page). For broken links, find the correct replacement. Note: was the paper retracted? Did the repo move? Is the dataset still downloadable? Produce a report with fixed URLs and status.

Choose 3 benchmarks from different domains (e.g., OCR, speech, reasoning). Write a plain-language explainer (500–800 words each) of what the primary metric actually measures, its blind spots, and a concrete example showing how two models with similar scores can perform very differently in practice.

Pick a popular benchmark (MMLU, HumanEval, SWE-bench). Build a complete leaderboard of all models evaluated on it since Jan 2024, with verified scores from primary sources only (not blog posts or tweets). Flag scores where the evaluation setup differs from the original protocol.

Pick a benchmark dataset (e.g., COCO-Text, ADE20K). Download it. Report: total size, sample count, class distribution, any labeling errors (spot-check 100 samples), license, last update date, and whether the download actually works end-to-end in 2026.

Pick 5 popular models. Compare their official model card claims (benchmark scores, capabilities) against independent evaluations. For each claim, classify as: confirmed, inflated, unverifiable, or contradicted. Cite your sources.

Pick 3 open-source models. Document exactly what training data they used. For each dataset in their mix: find its size, license, known issues, and whether it overlaps with any popular benchmarks. Create a "training data passport" for each model.

For a benchmark of your choice, create a visual timeline of every SOTA result from its creation to today. Include: date, model, score, and what changed (architecture, scale, data, trick). Identify the 3 biggest jumps and explain what drove them.

Pick a benchmark where 3+ papers report results for the same model (e.g., GPT-4 on MATH). Collect all reported scores. Identify discrepancies. For each one, determine the root cause: prompting, split, model version, or error. Write a 1-page analysis.

Pick a paper claiming SOTA on a CodeSOTA benchmark that provides code. Clone the repo, follow instructions, and reproduce the headline number. Document: setup time, dependency issues, hardware needs, actual vs. claimed result. Score reproducibility 1–5.

Pick a domain (code gen, math, or OCR). Find the best open-source and best closed model on the relevant benchmark. Design and run 20 original test cases probing a capability the benchmark misses. Compare results on your cases vs. the benchmark. Explain the gap.

Map the evolution of benchmarks in a research area (e.g., scene text recognition, commonsense reasoning). Which replaced which? Why? What weakness motivated the new one? Create a visual genealogy tree + 1-page narrative covering 5+ benchmarks over 5+ years.

Take a benchmark with a tight leaderboard (models within 1–2%). Investigate: does evaluation protocol (prompting, decoding, post-processing) materially affect scores? Find 2+ papers discussing this. Estimate how much leaderboard variance is real vs. noise.

Pick a model and a benchmark. Run the benchmark 5 times with 5 different valid prompt templates (all reasonable, all following the spirit of the benchmark). Report the score variance. Is the model's ranking stable across prompts?

For a benchmark of your choice, calculate the $/point improvement for the last 10 SOTA results. Factor in: training compute estimates, API costs for evaluation, and (where available) team size. Which improvements were efficient? Which were brute-force?

Pick 3 benchmarks that have both English and non-English versions. Compare SOTA models' performance gap across languages. Which models degrade most? Is there a pattern (language family, resource level)? Create a "multilingual tax" table.

Identify a practically important task with no good public benchmark (e.g., LLM multi-constraint following, OCR on handwritten Polish, or agent tool-use reliability). Create a small benchmark: 50–100 test cases with gold labels, clear eval metric, and baseline results from 2+ models. Package as downloadable JSON.

Pick a benchmark and a model. Investigate whether the model's training data likely contained the test set. Methods: verbatim memorization probes, performance on original vs. rephrased questions, training data documentation review. Write a report with evidence.

Survey multimodal benchmarks (vision-language, audio-language, video-language). Identify 3 capability gaps where no good benchmark exists. For each: describe the capability, explain why it matters, propose a benchmark design, and estimate creation effort. Cross-reference with codesota coverage.

Pick a task (code gen, translation, or image classification). Benchmark 5+ models measuring both quality AND inference speed. Plot the Pareto frontier. Identify dominated models. Write deployment recommendations for 3 use cases (latency-critical, cost-sensitive, quality-first).

Find 3 benchmarks where top models score >95%. Analyze: is it truly saturated? Is the remaining gap meaningful? Find actual failure cases from the best model. Propose "next-generation" benchmark criteria. Include specific examples the current benchmark can't distinguish.

Pick a domain with many "needs research" benchmarks on codesota (e.g., CV/OCR). Read 15–20 recent papers. Produce: (a) taxonomy of approaches, (b) cross-benchmark comparison table, (c) which approaches transfer vs. are benchmark-specific, (d) 3 open research questions.

Pick a model and a benchmark. Run 200+ test cases. Manually classify every error into a failure taxonomy you create (e.g., "misunderstood question," "correct reasoning, wrong format," "hallucinated fact"). Report: which failure modes are fixable by prompting vs. require model improvement.

Design a benchmark that evaluates AI agents (not raw models) on realistic multi-step tasks. Create 15+ tasks with success criteria and partial-credit scoring. Test 2+ agent setups (different tool configs, different models). Report what breaks.

Take an English-only benchmark (reasoning or NLP) and create a high-quality Polish version. Not machine-translated — culturally adapted with native-speaker quality. Run 3+ models on both versions. Analyze: do rankings change? Where do models fail specifically on Polish? Publish as a new dataset.

Design and run a benchmark evaluating entire AI systems (agent + tools + RAG). Define 15+ realistic tasks, success criteria, and partial-credit scoring. Test 2+ system architectures. Report results + failure analysis + lessons about what breaks in multi-component AI.

For a domain, calculate the true cost of evaluation: API costs per model, human annotation costs for gold labels, infrastructure costs, and time. Compare 3 benchmarks. Determine: which gives the most signal per dollar? Write a practitioner's guide to benchmark ROI.

Build a benchmark that resists contamination through continuous updates. Deliver: (a) methodology for monthly test case generation from fresh sources, (b) automated candidate generation pipeline, (c) human review process, (d) baseline results on first batch. Target a specific CodeSOTA task.

Pick a research area with 5+ active benchmarks on codesota. Produce: every model, every benchmark, every result — all verified. Include: (a) real vs. artifact improvements, (b) actual frontier of unsolved problems, (c) 12-month SOTA prediction with reasoning, (d) interactive comparison table.

Take a technique that works well in one domain (e.g., chain-of-thought in math) and rigorously test whether it transfers to 3 other domains. Run real experiments. Report: where it helps, where it hurts, and why. Produce a "transfer guide" for practitioners.

Pick a claimed scaling law from a major lab paper. Verify it using publicly available models at 3+ scales. Does the prediction hold? Where does it break down? Are there thresholds the original paper didn't discuss? Write up with your own measurements.

Take a benchmark where a specific model ranks #1. Find 20+ natural (non-adversarial-looking) inputs where it catastrophically fails but a lower-ranked model succeeds. Analyze: what does the benchmark miss? Propose a robustness supplement.

Analyze 20+ benchmarks across 3+ domains. Answer: do benchmarks actually predict real-world utility? Find cases where benchmark leaders fail in deployment. Propose a framework for "benchmark validity scoring." This should be publishable as a workshop paper.

Take an entire CodeSOTA domain (e.g., "Speech" or "Reasoning") with all its benchmarks. Fill in every "needs research" gap. Every benchmark gets: current SOTA, top-5 results, source links, methodology notes. Produce a complete, verified picture of the domain.

Create a benchmark suite of 50+ test cases based on real production problems (actual documents to OCR, real code to fix, genuine questions to answer). No synthetic data — every case from a real use case with permission. Run 5+ models. Publish results + dataset.

Develop a methodology to detect and score "benchmark corruption" — when optimization for a benchmark diverges from real-world utility (Goodhart's Law in action). Apply it to 10+ benchmarks. Rank them by corruption risk. Propose mitigations for the top 3.

Build a web tool that lets users interactively compare models across benchmarks with filters, scatter plots, and Pareto analysis. Data: all of codesota's benchmark results. Must work on mobile. Deploy it as a standalone page or tool for codesota.

Pick 5 benchmarks established before 2023. Re-evaluate modern models on the original, unmodified test sets AND on updated versions (if they exist). Quantify: how much of the "progress" is real capability vs. benchmark drift, data contamination, or evaluation inflation?

Pick 3 benchmarks with claimed "human-level" baselines. Re-run the human evaluation with fresh annotators, proper compensation, clear instructions, and time tracking. Compare your baselines to the originals. Report: were the original human baselines fair?

Based on your experience with other challenges (or research), write a comprehensive guide to designing ML benchmarks that resist gaming, contamination, and saturation. Include: principles, anti-patterns, case studies, and a checklist. Should be the go-to reference for anyone creating a new benchmark.

Create a single evaluation framework that can meaningfully compare models across different modalities (text, image, code, audio). Define "general capability" in a measurable way. Run it on 5+ models. Justify why your ranking is more meaningful than single-benchmark rankings.

Write the definitive report on the state of AI evaluation in 2026: what works, what's broken, where the field is heading. Cover 50+ benchmarks across all major domains. Interview or cite 10+ researchers. Include data-driven analysis, not just opinions. This should be the annual report people bookmark.