Skip to content

Chapter 2.11 — LLM evaluation

🎯 Objective

Show that evaluating LLMs is different and harder than evaluating classic models. Traditional ML has ground truth, stable metrics and established benchmarks. An LLM has often ambiguous ground truth, several equally valid answers, high sensitivity to the prompt and context window, and plenty of room for silent failure.

LLM evaluation is not the project's final step; it is a system that runs alongside, with its own datasets, gatekeepers and metrics.

🧠 Dataset types

Dataset What it is What for
Golden Curated real cases, with input + expected answer Practical truth for regression
Synthetic Automatically generated cases Cover distributions and edge cases
Adversarial Hostile cases (injection, OOD, ambiguity) Test defense in depth
Regression Cases that must keep passing Block a release that breaks previously correct behavior
Production traces (redacted) Anonymized replica of real traffic Ensure the eval reflects the world

Dataset best practices:

  • Version with semver: golden_support_v3, adversarial_pi_v2.
  • Document origin, inclusion criteria, known bias.
  • Do not train with the dataset you will evaluate against (valid for FT and for LLM-as-judge).
  • Update with new incidents: each incident becomes an eval case so that it never happens silently again.

🧪 Test types

  • Unit — tools, parsers, formatters.
  • Contract — schemas, function calling, structured outputs.
  • Integration — end-to-end pipeline with real dependencies.
  • End-to-end — the user's task, from prompt to final answer.
  • Prompt regression — compare version N vs N-1 on stable cases.
  • Policy — positive and negative tests on policy-as-code (see Ch. 4.7).
  • Security — direct and indirect prompt injection, DLP, tenant isolation.
  • Cost / latency — guardrails on cost per task, p95.
  • Trace replay — re-run historical traffic against the candidate version and measure the difference.

🤖 LLM-as-a-judge (with caution)

Using an LLM to judge another LLM's output works in several contexts, mainly for subjective evaluation (summary quality, fidelity to tone, faithfulness). But it is not magic:

  • Judge biases propagate into the metric (preference for long answers, for a confident style, for answers with lists, etc.).
  • Cost. At scale, the judge can be as expensive as the evaluated model.
  • Self-favoring. Models tend to prefer answers similar to those they would produce themselves; avoid using the same model to generate and to judge.
  • Clear rubric. Without a written rubric, the judge becomes a calibrated random-number generator.

Recommended pattern:

  1. Define a rubric in text with objective criteria.
  2. Use a model different from the evaluated one when possible.
  3. Calibrate the judge against human annotation on a sample (kappa, correlation).
  4. Use the judge for scale, but reserve human review for critical samples and for high-disagreement cases.

👥 Human evaluation and golden-set governance

Automatic evaluation does not replace human review in sensitive domains (legal, financial, medical, support with operational impact). Principles for doing this evaluation right:

  • Trained labelers. Provide a rubric and examples. Without it, each annotator invents the criterion.
  • Inter-annotator agreement. Measure agreement (Cohen's κ, Fleiss' κ) and investigate disagreement cases: either the rubric is ambiguous, or the case is genuinely hard — both teach.
  • A sufficient, not absurd, sample. Small, well-labeled samples beat large, noisy ones.
  • Disagreement as a signal. Cases where annotators diverge are usually exactly where the model also errs.
  • Golden-set updating. Evaluation datasets age: new verticals, new rules, new violations. Treat it as a living artifact, with an owner, a review cycle and a changelog.
  • Overfitting to the internal benchmark. The more the team iterates against the same dataset, the higher the chance of improving on it without improving in production. Reserve a holdout fraction, never used to iterate, only to validate before a release.
  • Governance of production traces in eval. If you reuse traces as a golden set:
  • ensure PII redaction;
  • document the legal basis for retention;
  • declare that customer traffic may appear aggregated in eval;
  • provide opt-out paths when applicable.

In high-risk domains, the rule is simple: automatic evaluation prepares, human review decides. Continuous eval filters; human review validates what goes to release.

📊 Frequent metrics

  • Task success rate.
  • Groundedness / faithfulness.
  • Hallucination rate.
  • Tool call accuracy.
  • Schema validity.
  • Citation coverage.
  • Retrieval precision/recall, NDCG, MRR.
  • Cost per task / per success.
  • Latency p95/p99.
  • Policy violation rate.
  • Escalation rate (HITL), fallback rate.

📌 Checklist

  • [ ] Are golden, adversarial and regression sets versioned?
  • [ ] Is eval running in CI before promoting a prompt/model/tool?
  • [ ] Is there human review on a sample for sensitive domains?
  • [ ] Does LLM-as-judge use a different model and a documented rubric?
  • [ ] Is there a holdout that is never used to iterate?

📚 References