The LLM eval harness rubric

Most production LLM eval harnesses ship with one column: accuracy. That's a vibes check, not an eval.

A real harness has five sections:

1. Golden set — frozen labeled examples. Pass rate per task. The set has an owner and a last-updated date. Without this, a model swap silently regresses.

2. Schema set — does output parse? Match the contract? Pydantic validation passes? Grammar-gated decoding trends this to 100%, but eval anyway — schema drift catches you when frameworks upgrade.

3. Calibration set — when the model says "0.95 confidence", is it actually right 95% of the time? Brier score or reliability diagram. Most teams skip this. Then they wonder why their auto-route threshold is wrong.

4. Latency budget — P50, P95, P99 per stage (router, retrieval, decode, tool-call). Hard cap per stage. A feature that violates the budget is a regression by definition.

5. Adversarial set — jailbreaks, prompt injection, OOD inputs, ambiguous queries. Pass rate baseline. Run on every model swap. The day you skip this is the day a customer finds your prompt-break on Twitter.

Each section needs three things: a metric, an owner, and a frequency.

The metric without the owner is theatre. The owner without the frequency is a someday-list.

A harness with fewer than five sections can't tell you whether the next deploy is an upgrade or a regression. The model gets better. The harness tells you.