01 · Foundation
Why Evals?
LLM systems are non-deterministic: the same prompt returns different output on every run, and a one-word prompt change can silently break behavior three features away. Classic testing assumes determinism — evals replace the assert with a measured pass rate over a dataset.
Evals are the unit tests of AI products. Teams that ship reliable agents don't have better vibes — they have better datasets, and they run them on every change.
The core shift
You stop asking "does this output look right?" and start asking "what fraction of my 200 real cases pass, at what cost, and did that number move since yesterday?"
What breaks without them
🎲
Prompt roulette
Every prompt tweak is deployed on hope. Fixing one case silently breaks five others and nobody notices for a week.
📉
Silent regressions
Model upgrades, provider changes, and RAG index rebuilds all shift behavior with zero code diff.
🗣️
Opinion-driven review
Without numbers, the loudest demo wins. Evals turn model and prompt choices into a measurement, not a debate.
01 · Foundation
Evals vs Tests
Evals borrow the discipline of testing but change the mechanics. Knowing exactly which assumptions flip keeps you from writing flaky "unit tests" around a probabilistic system.
Classic tests
Deterministic: same input, same output
Binary pass/fail per case
Exact-match assertions
One run is enough
Failures are bugs to fix now
100% pass is the baseline
Evals
Stochastic: sample multiple trials
Pass rate across a dataset
Rubrics, judges, outcome checks
k trials to measure variance
Failures are data to analyze
Threshold + no-regression gates
Rule of thumb
If a behavior can be checked deterministically (schema shape, tool called, code compiles), write a
code grader — save the LLM judge for what genuinely needs judgment.
01 · Foundation
The Eval Loop
Evals aren't a report card you run once — they're the improvement engine. The loop below is the whole discipline; everything else on this page is detail.
🎯
Define
what "good" means, per task
📚
Dataset
real cases + edge cases
⚖️
Grade
code, judge, or human
🔬
Analyze
label the failures
🛠️
Improve
fix the biggest bucket
🔁
Regress
re-run, gate, repeat
Where the value actually is
Most teams over-invest in the score and under-invest in error analysis. The aggregate number tells you that something broke; only reading and labeling failed transcripts tells you what to fix. Budget more time for looking at failures than for building dashboards.
Start embarrassingly small
Ten real cases in a JSON file, graded by a script, run before each deploy — that beats a grand eval platform you'll build next quarter. Datasets compound; start collecting failures today.
02 · Methods
Graders
A grader turns an output into a score. There are exactly three kinds, and picking the cheapest one that can measure the behavior is most of the craft.
1 · Code graders2 · LLM-as-judge3 · Human review
Code graders — deterministic, free, fast
Whenever the outcome is checkable — the SQL returns the right rows, the JSON parses, the test suite passes, the right tool was called — grade with code. Check outcomes, not surface strings:
graders/sql-grader.ts
// graders/sql-grader.ts — a deterministic code grader
// Grade by executing, not by string-matching the SQL text.
import { runQuery } from './sandbox';
export async function gradeSqlTask(
candidate: string,
expected: string,
): Promise<{ pass: boolean; reason: string }> {
// 1. Does it even parse / run?
const result = await runQuery(candidate).catch((e: Error) => ({
error: e.message,
}));
if ('error' in result) {
return { pass: false, reason: 'query failed: ' + result.error };
}
// 2. Compare result SETS, not query strings — many correct
// queries produce the same rows.
const goldenRows = await runQuery(expected);
const same = rowsEqual(result.rows, goldenRows.rows);
return {
pass: same,
reason: same ? 'result sets match' : 'wrong rows returned',
};
}
function rowsEqual(a: unknown[], b: unknown[]): boolean {
return JSON.stringify(sorted(a)) === JSON.stringify(sorted(b));
}
function sorted(rows: unknown[]): unknown[] {
return [...rows].sort((x, y) =>
JSON.stringify(x).localeCompare(JSON.stringify(y)),
);
}
LLM-as-judge — for genuine judgment calls
For tone, helpfulness, or faithfulness to a reference, use a judge model with a binary rubric per dimension — scalar 1–10 scores are noise. Require quoted evidence for every failure so you can audit the judge.
judges/support-rubric-v3.txt
You are grading an AI assistant's answer to a customer
support question. Grade ONLY against the rubric. Do not
reward style, length, or confidence.
<question>{{question}}</question>
<answer>{{answer}}</answer>
<reference>{{reference_answer}}</reference>
Rubric — score each dimension 0 or 1:
1. accuracy — every factual claim matches the reference
2. completeness — covers all required steps from the reference
3. safety — no invented policies, refunds, or promises
4. actionable — the user knows exactly what to do next
Output JSON only:
{ "accuracy": 0|1, "completeness": 0|1,
"safety": 0|1, "actionable": 0|1,
"evidence": "one quoted phrase per failed dimension" }
Judge pitfalls
Judges have
position bias (prefer the first answer shown — swap order and average),
self-preference (rate their own model family higher — judge with a different family), and
verbosity bias (longer looks better — cap length in the rubric). And eval your judge: measure agreement against a set of human-labeled examples before trusting it.
Human review — the calibration layer
Humans are the expensive ground truth. Use them to label the golden set, to calibrate the judge, and to spot-check a sample of judge verdicts each week — not to grade every run.
02 · Methods
Datasets & Goldens
The dataset is the eval. A perfect grader over ten toy cases measures nothing. Great datasets are mined, not invented.
Where cases come from
🚨
Production failures
The highest-value source. Every support escalation and thumbs-down becomes a regression case with the real input attached.
🧬
Synthetic variations
Use an LLM to mutate real cases — rephrase, add typos, shift locale, stack constraints. Always human-review before admitting.
🗺️
Coverage by design
Enumerate the behavior matrix — intents × edge conditions — and make sure every cell has at least one case.
Anatomy of a case
A case bundles input, expectations, and its grader. Keeping the grader reference in the case lets one suite mix code graders and judges freely:
datasets/core/refund-policy-017.json
{
"id": "refund-policy-017",
"source": "production-failure",
"created": "2026-05-02",
"input": {
"messages": [
{
"role": "user",
"content": "I bought the pro plan 45 days ago, can I get a refund?"
}
],
"context": { "plan": "pro", "purchase_age_days": 45 }
},
"expected": {
"must_mention": ["30-day refund window", "prorated credit"],
"must_not_mention": ["full refund"],
"reference_answer": "Refunds are only available within 30 days..."
},
"grader": "judge:support-rubric-v3",
"tags": ["refunds", "policy-edge", "regression"]
}
Dataset hygiene
Version the dataset in git. Tag cases by source and topic. Never delete a failing case because it's "weird" — weird is what production serves. And keep a
held-out set you never tune against, or your improvements are just overfitting.
02 · Methods
Metrics
One aggregate score hides everything you need to know. The useful numbers separate capability (can it do the task at all?) from reliability (does it do it every time?).
metrics/pass-k.ts
// Reliability is a different question than capability.
//
// pass@k — "can it EVER do it?" (capability)
// P(at least 1 of k samples passes)
//
// pass^k — "does it ALWAYS do it?" (reliability)
// P(all k samples pass) = p^k for per-run pass rate p
//
// A 90% per-run agent looks great on pass@1...
const p = 0.9;
// ...but run the same task 10 times in production:
const passHat10 = Math.pow(p, 10); // 0.349 — fails most days!
// Reporting both tells you capability AND consistency.
// Agents need pass^k targets, not just pass@1.
The metric menu
pass@1 / pass@k
Capability. Fraction of cases solved in one (or best-of-k) attempts. The standard headline number.
pass^k
Reliability. Probability all k trials pass. The honest number for agents that run unattended.
Win rate
Pairwise A-vs-B judged comparisons. Best for subjective quality where absolute rubrics are shaky.
Per-dimension rubric
Accuracy / completeness / safety tracked separately — a safety dip must never hide inside a rising average.
Cost & latency
Tokens and wall-clock per case. A 2% quality win that triples cost is a decision, not an upgrade.
Variance
Spread across trials. High variance means your pass rate is a coin flip, not a property.
02 · Methods
Trajectory Evals
For agents, the final answer is half the story. An agent that gets there in 4 clean tool calls and one that stumbles there in 19 calls with 6 errors both "pass" an outcome eval — and only one of them survives production traffic.
graders/trajectory.ts
// Grade the PATH, not just the final answer.
// A trajectory eval scores each step of an agent run.
type Step = {
tool: string;
args: Record<string, unknown>;
ok: boolean;
ms: number;
};
export function gradeTrajectory(steps: Step[], answerPass: boolean) {
const toolErrors = steps.filter((s) => !s.ok).length;
const redundant = countRepeatedCalls(steps); // same tool + same args
const totalMs = steps.reduce((n, s) => n + s.ms, 0);
return {
outcome: answerPass, // did it get there?
efficiency: steps.length <= 8, // without wandering?
hygiene: toolErrors === 0, // without tool misuse?
noLoops: redundant === 0, // without repeating itself?
latencyBudget: totalMs < 30_000, // within budget?
};
}
function countRepeatedCalls(steps: Step[]): number {
const seen = new Set<string>();
let dupes = 0;
for (const s of steps) {
const key = s.tool + JSON.stringify(s.args);
if (seen.has(key)) dupes += 1;
seen.add(key);
}
return dupes;
}
What to score on the path
Tool choiceArgument qualityStep countNo repeated callsRecovery from errorsCost / latency budget
Why it matters
Trajectory metrics are the early-warning system: outcome pass rates often hold steady while step counts and tool errors creep upward — right until they don't. Grade the path and you see the decay a month early.
03 · Production
CI Regression Gates
The eval suite earns its keep the day it blocks a bad merge. Wire it into CI exactly like tests: run on every PR that touches prompts, agents, or graders — enforce a floor and a no-regression rule against main.
.github/workflows/evals.yml
# .github/workflows/evals.yml — evals as a merge gate
name: eval-gate
on:
pull_request:
paths:
- "prompts/**"
- "agents/**"
- "graders/**"
jobs:
run-evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- name: Run eval suite (3 trials per case)
run: bun run evals -- --suite core --trials 3 --out report.json
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Enforce thresholds
run: |
bun run evals:check -- report.json \
--min-pass-rate 0.92 \
--no-regression-vs main \
--max-cost-per-case 0.04
- name: Upload failure transcripts
if: failure()
uses: actions/upload-artifact@v4
with:
name: eval-failures
path: report.json
Variance will bite
A single-trial gate flakes constantly and teaches the team to ignore it. Run 3+ trials per case, gate on the mean, and set thresholds with headroom below your true average — a gate that cries wolf is worse than no gate.
What the gate should check
Min pass rateNo regression vs mainCost per case ceilingLatency budgetSafety dimensions at 100%
03 · Production
Online Evals & A/B
Offline suites can't see distribution shift — real users invent inputs your dataset doesn't have. Online evals grade a sample of live traffic, asynchronously, and alarm on drift.
online/sampling.ts
// Online evals: grade a SAMPLE of live traffic, async.
// Never block the user request on grading.
export async function onRequestComplete(trace: AgentTrace) {
// 1. Sample — grading 2% of traffic is usually plenty
if (Math.random() > 0.02) return;
// 2. Grade async, out of the request path
await queue.enqueue('grade-trace', {
traceId: trace.id,
grader: 'judge:support-rubric-v3',
});
}
// 3. Alarm on drift, not single failures
// (score is a distribution — alert on the moving average)
export function checkDrift(scores: number[], baseline: number) {
const weekAvg = avg(scores.slice(-500));
if (weekAvg < baseline - 0.05) {
alert('eval drift: ' + weekAvg.toFixed(3) + ' vs ' + baseline);
}
}
const avg = (xs: number[]) => xs.reduce((a, b) => a + b, 0) / xs.length;
declare const queue: { enqueue(name: string, data: unknown): Promise<void> };
declare function alert(msg: string): void;
type AgentTrace = { id: string };
The online toolkit
🎣
Sampled judging
Grade ~2% of production traces with your judge, off the request path. Trend the moving average.
🐤
Canary prompts
Fixed probe inputs on a schedule. When a provider silently updates a model, canaries move first.
🧪
A/B with judges
Route a slice of traffic to the candidate, judge pairwise against control, promote on win rate + guardrails.
Close the loop
Every online failure the judge flags is a fresh offline case. The production→dataset pipeline is what keeps the suite honest as your traffic evolves.
03 · Production
Observability
A pass rate without traces is a smoke alarm without a floor plan. Eval infrastructure needs the same three pillars as any distributed system: traces, per-step scores, and an error taxonomy.
Trace everything the grader sees
Store the full trajectory — prompts, tool calls, retrieved chunks, token counts — keyed by case id and run id. When a case flips from pass to fail, the diff between the two traces usually names the culprit in minutes.
Label failures with a taxonomy
Aggregate scores say whether; taxonomies say why. Keep the label set small and count buckets over time:
error-taxonomy.yaml
# error-taxonomy.yaml — label failures before fixing them
# "Pass rate dropped" is not actionable. "Retrieval misses
# multi-hop questions" is.
taxonomy:
- id: retrieval-miss
desc: right doc exists, search never surfaced it
count_last_30d: 41
- id: wrong-tool
desc: agent picked search when the answer needed the DB
count_last_30d: 23
- id: premature-stop
desc: gave up before exhausting available tools
count_last_30d: 17
- id: hallucinated-policy
desc: invented a rule not present in any source
count_last_30d: 9
- id: format-drift
desc: right answer, unparseable output shape
count_last_30d: 6
# Fix the biggest bucket first — error analysis beats
# staring at an aggregate score.
The weekly ritual
Read ten failed transcripts, label them, and fix the fattest bucket. Thirty minutes of error analysis moves the score more than a week of prompt-fiddling against the aggregate.
04 · Execution
Anti-patterns
Most eval efforts fail the same four ways. All four feel like progress while they're happening.
🎯
Overfitting the eval
Tuning prompts against the same cases you report. Keep a held-out set and rotate it; watch online metrics for the truth.
🤏
Ten-case confidence
A 90% pass rate on 10 cases is ±19%. Small datasets produce vibes with decimal points.
🪞
Self-judging
A model family grading itself inflates scores. Cross-family judges, order-swapping, and human calibration are mandatory.
📊
Ignoring variance
One trial per case turns your gate into a coin flip and your "improvement" into noise. Always report spread.
🏗️
Platform before dataset
Building eval dashboards for months while the dataset has 12 cases. The dataset is the asset; the platform is plumbing.
🥇
Benchmark worship
Public benchmarks measure the model, not your product. MMLU can't tell you if your refund bot invents policies.
Goodhart's law applies
Any eval metric used as a target stops measuring what you think it measures. Refresh datasets from production, rotate held-out sets, and audit the judge — forever.
04 · Execution
Evals Checklist
Click items to mark complete. Progress tracked per session.
COMPLETION
0 / 12 complete
Foundation
Every prompt/agent change runs the eval suite before mergeCritical
Golden dataset exists with 50+ cases from real failuresCritical
Each case declares its grader (code, judge, or human)Structure
Multiple trials per case — variance is measured, not ignoredRigor
Graders
Code graders check outcomes (run it), not string equalityGrader
Judge rubrics are binary per dimension with evidence quotesGrader
Judge model differs from the model being gradedCritical
Judge itself is eval-ed against human labels (agreement > 0.85)Meta
Production
CI gate enforces min pass rate + no-regression vs mainCI
Online sampling grades ~2% of live traffic asynchronouslyOnline
Drift alerts fire on moving averages, not single failuresAlerting
Failures are labeled with an error taxonomy weeklyAnalysis