---
title: "The LLM eval harness rubric"
date: 2026-05-01
series: 7
summary: "Most production LLM eval harnesses ship with one column: accuracy. That's a vibes check, not an eval."
voice: judgment
tags: [llm-eval, golden-set, schema-set, calibration-brier, latency-budget, adversarial-set, metric-owner-frequency]
image: "/notes/eval-harness-rubric/image.png"
imageAlt: "5 sections of the eval harness rubric"
linkedin: null
sources: []
dateApprox: true
---

Most production LLM eval harnesses ship with one column: accuracy. That's a vibes check, not an eval.

A real harness has five sections:

**1. Golden set** — frozen labeled examples. Pass rate per task. The set has an owner and a last-updated date. Without this, a model swap silently regresses.

**2. Schema set** — does output parse? Match the contract? Pydantic validation passes? Grammar-gated decoding trends this to 100%, but eval anyway — schema drift catches you when frameworks upgrade.

**3. Calibration set** — when the model says *"0.95 confidence"*, is it actually right 95% of the time? Brier score or reliability diagram. Most teams skip this. Then they wonder why their auto-route threshold is wrong.

**4. Latency budget** — P50, P95, P99 per stage (router, retrieval, decode, tool-call). Hard cap per stage. A feature that violates the budget is a regression by definition.

**5. Adversarial set** — jailbreaks, prompt injection, OOD inputs, ambiguous queries. Pass rate baseline. Run on every model swap. The day you skip this is the day a customer finds your prompt-break on Twitter.

Each section needs three things: a **metric**, an **owner**, and a **frequency**.

The metric without the owner is theatre. The owner without the frequency is a someday-list.

A harness with fewer than five sections can't tell you whether the next deploy is an upgrade or a regression. The model gets better. The harness tells you.