Week 5 · Lesson 16

Evals suite

Grade quality and policy. Pair with Evals & observability. Re-run when you change models.

Time · 1–2 days Quality bar
Learn
  • Golden sets, schema checks, trajectory/policy checks
  • LLM-as-judge vs human labels (when each fits)
  • Regression discipline on prompt/model swaps
Build
  1. evals/ with 10–20 fixtures (topics + expectations)
  2. Runner: schema + “never publish without approve”
  3. Optional rubric judge for hook quality
  4. Script: pnpm eval against LM Studio
Checkpoint

Change model → eval report shows pass/fail deltas. Failures are explainable via run traces.

Four notes
  • What problem does this layer solve?
  • What is allowed to call it?
  • What must it never do?
  • How do I observe and test it?