AI product evals
A model benchmark helps you choose a starting model. A product eval checks whether your actual application meets a user’s goal, including its prompts, retrieval, tools, permissions, and application code. A fluent answer is not evidence that the requested action succeeded. This workflow synthesizes Hamel Husain’s evals flashcards and FAQ with OpenAI’s evaluation guidance. It complements Agent harness: the harness controls execution; evals provide evidence about its behavior.Start with failures, not a dashboard
A trace records the path from a user request to its outcome: messages, retrieved context, tool arguments and results, and relevant state changes. For multi-agent work, preserve a shared trace ID and the source and ordering of events.- Collect diverse traces. Mix random samples with different workflows, negative feedback, retries, and unusually slow runs. Keep some random sampling even after you know which failures to search for.
- Write open-ended notes first. Have someone who understands the product inspect examples before an automated reviewer suggests labels. Record what went wrong for the user, including ordinary code or design bugs, not just model errors.
- Group recurring failures. Turn notes into specific categories such as “booked an unavailable slot” or “claimed success after a tool failed.” Then label pass/fail for each relevant criterion and retain the evidence.
- Prioritize and repeat. Consider frequency, impact, and remediation cost. Continue until new reviews stop revealing or changing failure categories; revisit after meaningful product changes.
Turn an observed failure into a regression case
For a long conversation, locate the first upstream failure before counting its downstream consequences as separate problems. Preserve the full trace for investigation, then minimize the reproduction. Consider an illustrative scheduling agent:
Run replay tests in a sandbox with mocked tools or isolated test accounts, never against live booking, payment, or deletion endpoints. If the workflow or tool schema changes substantially, rebuild stale fixtures. Keep end-to-end tests too: passing an isolated next-step test does not prove the whole session succeeds.
Use the cheapest trustworthy check
Fix obvious prompt omissions and ordinary bugs first. Add cheap regression tests where useful; invest in an LLM judge when an important semantic failure needs repeated measurement. Do not wait for a real incident to test known safety requirements, such as authorization or irreversible actions.
Prefer a narrow pass/fail criterion over an undefined “quality: 4/5.” For the scheduling example, separate “did not book an unavailable slot” from “did not claim an unconfirmed booking succeeded.” Each needs different evidence. Numeric scores or pairwise comparisons can still be useful when their meaning is explicit and validated; binary labels are a practical default, not a universal law.
A semantic grader can return a small structured result such as:
Check the path, conversation, and final state
LangChain’s agent-evals guide distinguishes several evaluation scopes. Pick the scope that can actually reveal the failure:
A sequence of individually reasonable turns can still fail the overall task. Conversely, different tool sequences can achieve the same valid result. Assert necessary constraints—such as approval before a write or no duplicate booking—rather than requiring one exact trajectory unless that sequence is itself a requirement.
For state-changing tasks, start every evaluation from a clean, known fixture and inspect the state afterward. A coding task can be scored with the repository’s tests and the resulting diff instead of the agent’s “done” message. Keep the trusted grading checks outside the agent’s editable work products, and do not let success depend only on tests the agent wrote or changed itself.
Containers are one way to isolate such tasks, not the definition of every benchmark and not a complete security boundary by themselves. Limit credentials, network access, mounts, and run time to what the test needs. Record the fixture version, tool configuration, grading result, and trace. Repeat stochastic cases when needed and report passes over attempts; record infrastructure errors and timeouts separately rather than silently excluding them or counting them as success.
Calibrate the judge against people
A second model’s opinion is not ground truth. Build a human-labeled set containing both passes and failures for the same criterion, and resolve unclear definitions before tuning the judge. Separate examples by purpose:- Train/examples: choose few-shot examples for the judge prompt.
- Development: revise the rubric and inspect disagreements.
- Held-out test: check generalization without tuning on these labels. Keep related traces and paraphrases together so they do not leak across splits.
- TPR / failure recall =
TP / (TP + FN): how many human-labeled failures the judge catches. - TNR / specificity =
TN / (TN + FP): how many human-labeled passes the judge correctly leaves unflagged.
Separate regression tests, monitoring, and guardrails
A check after the response can discover a bad purchase; it cannot prevent that purchase. Enforce permissions before the side effect, outside the model. Guardrails also need latency and false-alarm budgets. Do not retry payments or other writes blindly when a check fails; see purchase approval boundaries.
CI pass rates are not estimates of user satisfaction: regression suites deliberately overrepresent known failures. Likewise, a green suite proves only what it covers. Keep adding independently reviewed production failures and deliberate edge cases, and rerun relevant evals when code, prompts, models, retrieval, or tools change.
A minimal starting checklist
- Review a small, diverse batch with a product/domain expert.
- Name one important failure and retain its trace evidence.
- Reproduce it with a minimal safe fixture and a clear assertion or rubric.
- Validate any automated judge against both human-labeled passes and failures.
- Compare the current and candidate systems on the same versioned cases.
- Assign an owner for failures and feed new production cases back into the suite.
References
- meng shao: Common agent-eval questions — Folo intake, October 15, 2026
- LangChain: The Most Common Questions About Agent Evals, Answered — step, conversation, trajectory, and environment-state evaluation
- meng shao: Pocket Guide to AI Evals — Folo intake, October 12, 2026; points to Hamel’s original cards
- Hamel Husain: Evals flashcards — especially error analysis, judge validation, trace-based cases, and deployment
- Hamel Husain: AI Evals FAQ — sampling, evaluator selection, calibration, and multi-turn debugging
- Hamel Husain: Your AI Product Needs Evals — scoped tests and human review as the foundation
- OpenAI: Evaluation best practices — task-specific evals, human calibration, edge cases, and continuous evaluation