Skip to content

Chapter 5.4 — Eval harness

🎯 Objective

Make continuous evaluation part of CI/CD. Without automated eval, any change in prompt, model, tool or retriever goes into production blind.

🧠 Components

  • Datasets: golden, synthetic, adversarial, regression, redacted traces (see Ch. 2.11).
  • Checks: deterministic (schema, regex, contracts), policy (Rego/Cedar tests), LLM-as-judge (with rubric and calibration), human sampling (for critical cases).
  • Release gates: objective approval criteria to promote a version. A policy failure or a critical regression = automatic block.

🧠 Multi-step agent eval

Evaluating an agent is harder than evaluating an LLM:

  • The trajectory matters, not just the final answer. An agent that gets the answer right by going through a wrong tool is a fragile success.
  • Tool call accuracy per step: did the agent choose the correct tool? Did the arguments pass the schema?
  • Cost per success, not just success rate. An agent that resolves but consumes 10x more tokens is not better.
  • Scenario replay with forks: what if tool A had failed? Does the agent fall back correctly?

📝 Eval suite example

eval_suite:
  name: support_agent_regression
  version: 4
  agent: support-agent@2.1.0
  datasets:
    - name: golden_support_cases_v3
      type: golden
    - name: prompt_injection_cases_v2
      type: adversarial
  checks:
    - id: schema_validity
      type: deterministic
      assertion: output.matches_schema("support_answer_v2")
    - id: groundedness
      type: llm_judge
      rubric: "Every factual statement must be supported by the retrieved passages."
      threshold: 0.92
    - id: tool_call_accuracy
      type: deterministic_or_human_sample
      threshold: 0.95
    - id: pii_leakage
      type: policy
      assertion: no_unmasked_pii(output)
    - id: cost
      type: metric
      assertion: cost_per_task <= 0.20
    - id: latency
      type: metric
      assertion: p95_latency_ms <= 10000
  release_gate:
    fail_on_policy_violation: true
    minimum_pass_rate: 0.90
    require_human_review_for_failures: true
  • EX-EVAL-01 — eval harness with pytest + golden dataset.
  • EX-EVAL-02 — prompt regression test.
  • EX-EVAL-03 — retrieval eval with Ragas.