Concept
Evals & observability
Observability answers “what happened?” Evals answer “was that any good?” You need both before trusting a model swap (like moving to LM Studio).
Observability
Reconstruct a run: inputs, model calls, tool calls, decisions, errors, latency, cost. For agents this means traces, not only “HTTP 200” logs.
run_id: abc123
goal: "Draft IG post about morning routines"
step 1 model plan → use content_planning
step 2 tool content_planning(...) → {hook, angle}
step 3 model plan → write caption
step 4 tool caption_writing(...) → {caption}
step 5 model finish → pending_approval
status: succeeded
tokens: 4.2k latency: 8.1s model: local-qwen| Signal | Why |
|---|---|
| Traces / spans | Reconstruct the decision loop |
| Tool I/O | Bad args, bad results, retries |
| Prompts + completions | Debug drift (redact secrets) |
| Status transitions | draft → pending_approval → … |
| Latency / tokens | Budgets and regressions |
| User actions | Approve / request-changes |
Evals
Systematic quality tests — unit/integration tests for behavior, not only “didn’t throw.”
- Skill / schema evals — fixed input → valid structured output
- Trajectory evals — right tools, no forbidden publish
- End-to-end evals — topic → draft scored by rubric
- Golden / regression set — re-run when prompts or models change
- Online evals — sample production; thumbs / request-changes rate
How they relate
Observability → traces explain a run
↓
sample / golden cases
↓
Evals → pass/fail + regressions
↓
improve prompts, tools, policy