Skip to main content
Parameter count alone doesn’t tell you whether a model is good. A 7B model trained on the right data can outperform a sloppy 13B model at coding, and a well-tuned 8B model can beat a 70B model at following instructions. Without a way to measure capability, you’re guessing. Evaluation benchmarks — “evals” — are standardized tests that give you comparable numbers across models. They’re imperfect, sometimes misleading, and occasionally gamed, but they’re the best tool available for comparing models before you commit hardware to one. This page explains what the major benchmarks measure, how to read their scores, and which ones actually matter for your use case.

​
Why evals exist

You can’t judge a model by its parameter count, training data size, or architecture alone. Those numbers describe capacity — what the model could theoretically do — not capability, what it actually does when you give it a prompt. Benchmarks exist to bridge that gap. A good benchmark:
  • Standardizes comparison — every model answers the same questions under the same conditions.
  • Reveals strengths and weaknesses — a model that scores well on math might be mediocre at code.
  • Tracks progress — you can see whether a new generation of models actually improved or just got bigger.
But every benchmark has blindspots. A high MMLU score doesn’t guarantee the model writes good emails. A low HumanEval score doesn’t mean the model can’t help you debug. Benchmarks are a starting point, not a verdict.

​
The benchmark landscape

Here are the major benchmarks you’ll encounter on model cards and leaderboards:
MMLU (Massive Multitask Language Understanding) is the most widely cited benchmark. It covers 57 subjects from STEM, humanities, and social sciences at the high-school to professional level. Each question is multiple-choice with 4 options. A score of 70 means the model gets 70% of these questions right. MMLU is useful as a broad capability signal but doesn’t test open-ended generation, creativity, or instruction following. Most model cards report the 5-shot score.
These 2 benchmarks measure code generation. HumanEval has 164 hand-written Python problems where the model generates a function from a docstring, then tests are run to check correctness. MBPP is larger (~1000 problems) but covers simpler algorithmic tasks. Both report pass@1 — the percentage of problems solved correctly on the first attempt. They’re good for comparing coding capability but cover a narrow slice of real-world programming.
MT-Bench evaluates conversational quality through 80 multi-turn prompts rated by GPT-4 or human judges on a 1–10 scale. Chatbot Arena (LMSYS) is different: it pits 2 anonymous models against each other and lets users vote, producing an ELO ranking like chess ratings. Arena scores reflect real-world user preference but update slowly and favor models that produce polished, confident-sounding answers over models that are correct but terse.
SWE-Bench tests whether a model can solve real GitHub issues end-to-end: read the issue, find the bug in a large codebase, and generate a correct pull request. Top models resolve 30–50% of issues on the verified subset. This is the closest benchmark to actual software engineering work, but it’s extremely difficult and results vary heavily by language and framework.
LiveBench refreshes its test set monthly using new data, which makes it resistant to benchmark contamination — models can’t have memorized answers they haven’t seen during training. It covers math, coding, reasoning, language, data analysis, and instruction following. It’s newer than MMLU or HumanEval but increasingly cited as a more trustworthy signal.

​
How to read a benchmark table

Scores look authoritative, but they need context to be useful.

​
What the numbers mean

Most benchmarks report a score from 0 to 100, which is the percentage of questions the model answered correctly. An MMLU score of 70 means the model gets 70% of college-level multiple-choice questions right. ELO ratings (used by Chatbot Arena) are different — they’re relative rankings where a 100-point gap means the higher-rated model wins about 64% of head-to-head matchups.

​
Margin of error

A 1–2 point difference on any single benchmark is noise. Scoring methodology, prompt formatting, and even random seed variation can shift scores by that much. If Model A scores 72 and Model B scores 74 on MMLU, treat them as equivalent and look at a benchmark that tests the specific capability you care about.

​
Contamination

Benchmark contamination happens when test questions leak into a model’s training data. The model scores artificially high because it has already seen the answers. This is a growing problem as benchmarks become widely published. LiveBench and some newer benchmarks combat this by refreshing their test sets regularly.
When a model’s benchmark score seems suspiciously high compared to its real-world performance, contamination is a likely explanation. Check whether the benchmark was published before the model’s training cutoff date.

​
Cherry-picking

Never judge a model by a single benchmark. Publishers sometimes highlight the 1 score where their model looks best while burying weaker results. Look for a spread of benchmarks that cover different capabilities — knowledge, code, math, instruction following — before drawing conclusions.

​
Which benchmarks matter for your use case

Not every benchmark matters for every task. Focus on the ones that test what you actually need:

Coding assistant

Look at: HumanEval, MBPP, SWE-Bench. These directly measure code generation and software engineering. Ignore MMLU for this decision — a model can score high on trivia and still write bad code.

Chatbot and general conversation

Look at: MT-Bench, Chatbot Arena ELO, IFEval. Conversational quality is subjective, so human-rated benchmarks matter most. IFEval adds a measure of whether the model follows formatting and constraint instructions reliably.

Reasoning and analysis

Look at: GSM8K, MATH, GPQA, ARC-Challenge, BigBench Hard. These test multi-step logic at different difficulty levels. GPQA is the hardest — useful for separating frontier models from merely good ones.

General assistant

Look at: MMLU + MT-Bench + IFEval as a composite. No single number captures “general assistant” quality. Check all 3 to get a picture of knowledge breadth, conversational skill, and instruction compliance.
When comparing 2 similar models, skip the averages and look at the specific benchmark that matches your task. A 7B coding model and a 7B chat model might have similar MMLU scores but very different HumanEval results.

​
Benchmark scores at different quantizations

Benchmark scores are usually reported at full precision (FP16 or BF16). When you run a quantized model locally, expect some score degradation.

​
How much does quantization cost?

The impact depends on the benchmark, the model size, and the quantization level:

​
When quantization hurts more

  • Small models lose more than large models. A 7B model quantized to Q4_K_M drops more than a 70B model at the same quant level, because the smaller model has less redundancy to absorb rounding errors.
  • Precise tasks suffer more. Math and code generation are more sensitive to weight precision than general chat. If you’re running a coding assistant, consider Q5_K_M or higher.
  • I-quants can help at low bit-widths. Imatrix quantization (IQ4_XS, IQ3_XXS) preserves critical weights more carefully and may recover 1–2 points compared to standard K-quants at the same size.
A 70B model at Q4_K_M almost always outperforms a 7B model at full precision on most benchmarks. Model size matters more than quantization level when the size gap is large.

​
Running your own evals

Published benchmarks are a starting point. If you need to know how a model performs on your tasks with your setup, run your own evaluation.

​
When to run your own

  • You have a domain-specific use case (legal, medical, finance) that standard benchmarks don’t cover.
  • You’re comparing 2 models that score similarly on public benchmarks.
  • You’re testing a specific quantization or fine-tune and want to measure the real impact.
  • You need to validate that a model handles your edge cases before deploying it.

​
Tools for running evals

lm-evaluation-harness (by EleutherAI) is the standard open-source tool. It supports 60+ academic benchmarks out of the box and runs against any model served through llama.cpp, vLLM, or an OpenAI-compatible API:
OpenAI Evals and HELM (Stanford) are alternatives with different task libraries and reporting formats.

​
A simple custom eval

You don’t need a framework to test domain-specific quality:
  1. Write 50 questions from your actual use case.
  2. Send each question to the model and record the response.
  3. Score each response on a 1–5 scale (or pass/fail for factual questions).
  4. Repeat for each model you’re comparing.
50 questions is enough to spot large differences. Use the same questions for every model, and score blind (hide which model produced which response) to avoid bias.

​
When to trust published scores

Trust published benchmark scores when:
  • The model publisher has a track record of honest reporting (Meta, Google, Mistral, Qwen, Anthropic).
  • Multiple independent sources confirm similar numbers.
  • The benchmark was published after the model’s training cutoff (reducing contamination risk).
Run your own eval when:
  • You’re using a third-party quantization and want to verify it hasn’t degraded quality significantly.
  • Your task is narrow enough that general benchmarks don’t apply.
  • You’re making a production decision and need confidence beyond a leaderboard number.

​
Red flags in benchmark reporting

Not every benchmark claim is trustworthy. Watch for these patterns:
  • No quantization disclosed. “Our model scores 78 on MMLU” — at what precision? If they don’t say, they might be reporting full-precision scores for a model most users will run quantized.
  • Only reporting the best benchmark. A model page that highlights 1 score and omits others is cherry-picking. Look for a full benchmark table.
  • Comparing different model sizes. “Our 7B model beats a 70B model on X” — yes, on a narrow benchmark that favors the 7B’s training data. This isn’t a fair comparison.
  • Contaminated benchmarks without disclosure. If a model’s training data likely included the benchmark’s test set and the publisher doesn’t address it, treat the score skeptically.
  • “Beats GPT-4 on…” Almost always on a narrow, favorable benchmark. GPT-4 is a large proprietary model — beating it on 1 specific task doesn’t mean the model is generally better.
  • No methodology described. If the score doesn’t include details like shot count, prompt format, and scoring method, you can’t compare it to other published numbers.

​
Where to find benchmark data

Cross-reference at least 2 sources before trusting a benchmark number. If the Open LLM Leaderboard and the model’s own paper report different scores, investigate why — it’s usually a difference in evaluation settings (shot count, prompt format, or quantization).

​
Go deeper

Read a model name

Decode model names, quantization labels, and architecture suffixes before you look at benchmarks.

Read a model card

Find the parameters, architecture, context length, and license that benchmarks don’t cover.

Quantization

Understand how quantization affects the benchmark scores you just learned to read.

Which local model for which task?

Use benchmark insights to pick a model matched to your specific use case.