How to Evaluate AI Output Before Launch
Evaluate AI output against an explicit task standard before launch. Define what must be correct, what may be omitted and which failures must block release. Test the complete workflow, including its sources, permissions, review process and downstream actions.
An average quality score is useful only alongside failure severity. A system can produce many good drafts while still being unsuitable because it sometimes invents a payment instruction or reveals restricted data.
Build a representative evaluation set
Collect permission-cleared examples from the intended workflow. Include common cases, difficult cases, incomplete inputs, conflicting information and requests outside scope. Keep a held-back set that is not used to tune prompts or settings.
Write the expected behavior for each case. Some tasks have a single correct field; others admit several acceptable responses. For open-ended work, define what the output must cover and which claims need source support. “Sounds helpful” is too weak to guide release.
Include deliberate boundary tests: an input asking the assistant to ignore its rules, a document with instructions unrelated to the task and a request for information the user lacks permission to access. Passing a normal summarization test does not establish those boundaries.
Use a rubric with separate dimensions
| Dimension | Reviewer question | Example failure |
|---|---|---|
| Factual correctness | Are statements and extracted fields correct? | Wrong date or invented policy |
| Source support | Does the cited material support the claim? | Relevant-looking link with no supporting passage |
| Coverage | Does the result address required elements? | Second request in a message is ignored |
| Uncertainty | Does it ask or escalate when evidence is insufficient? | Guessed answer presented as fact |
| Permissions | Is every disclosed item authorized? | Restricted account information appears |
| Action validity | Is the proposed action allowed and correctly targeted? | Duplicate task or incorrect recipient |
Define critical failures separately and inspect them individually. Keep minor style defects out of the same category as unsupported commercial commitments. Set release criteria according to the consequences and controls of your workflow; there is no universal percentage that makes every application safe.
Worked example: help-response drafts
Illustrative evaluation: a startup assembles sixty held-back support questions. Ten intentionally cannot be answered from the approved help documents. The expected response for those ten is to ask for missing information or route to review, without inventing policy.
A candidate returns fifty-six responses reviewers consider usable after checking. Four require substantial correction, including one unsupported refund promise. Reporting only “93% usable” hides the critical error. The team records its source, revises the workflow and rechecks relevant cases before a new held-back evaluation.
The sixty cases are an illustrative pilot pack, not a statistically sufficient sample for every failure rate. Rare harmful failures may require targeted testing, larger samples and stronger controls. Keep representative evaluation separate from adversarial cases so results are interpretable.
Make human review consistent
Give reviewers the input, approved sources, rubric and expected behavior. Have two people independently judge a subset and discuss disagreements. If they cannot agree about correctness, refine the task or rubric before treating the score as meaningful.
An automated judge can help triage a large evaluation, but check its judgments against human review and objective tests. It can miss the same unsupported assumption as the system being evaluated. Use deterministic checks for schemas, calculations and required identifiers where possible, while recognizing that valid formatting does not prove factual accuracy.
Measure review time too. A deployment that requires constant expert correction may fail its business case even when final outputs are acceptable.
Turn evaluation into a release process
Record configuration, model where available, source versions, results and known limitations. Pilot with bounded users and authority. Monitor corrections, escalations, complaints and failures after release; collect new cases for regression testing.
Re-evaluate after changes to models, sources, prompts, permissions or action scope. Keep a pause mechanism and a fallback when thresholds fail. Use vendor evaluation to connect output findings to purchasing, the tools guide for candidates and the integration overview for operation. AI applications and limits helps define what work should enter the system.
Create an evaluation record for each case
A case needs more than an input and a favorable answer. Record its identifier, source version, user role, required behavior, prohibited behavior and severity if it fails. Keep the expected behavior independent of one model’s preferred wording.
For example, a customer asks whether a feature supports a particular integration. The approved source says it does not. Acceptable wording may vary, but the answer must preserve that fact and avoid promising a release date. A response that sounds reassuring while implying availability fails, even if it cites the correct help page.
Distinguish answerable, clarification-needed, unsupported and prohibited cases. These categories help interpret coverage. A system that declines everything may avoid unsupported claims while failing the intended service, so evaluate appropriate assistance and appropriate restraint together.
Define a usable scoring rule
| Rating | Meaning | Reviewer action |
|---|---|---|
| Acceptable | Required facts and task elements are correct | Approve within the tested scope |
| Minor correction | Meaning remains correct but presentation needs work | Record correction effort |
| Substantial correction | Important omission or factual defect | Reject draft and investigate |
| Critical failure | Unauthorized disclosure or prohibited consequential claim/action | Block release or pause affected scope |
Choose examples illustrating each category. Have reviewers score independently before discussing differences. If one reviewer treats an invented deadline as minor while another treats it as critical, resolve the policy and business consequence rather than averaging their scores.
Keep dimensions available separately. An overall pass may require correctness, source support and permissions all to pass, while style remains a preference. Do not let a weighted total turn a failed mandatory control into an acceptable average.
Diagnose failure before changing the prompt
An incorrect answer can arise at several points. The approved source may be wrong, retrieval may find an old document, the model may misinterpret the correct passage, or an integration may attach the wrong customer’s record.
Trace the case from input to final action. Ask what information was available, which documents were supplied, what the model proposed and what validation or review allowed through. Keep enough evidence to identify the cause without copying unnecessary personal data into evaluation files.
If the source is contradictory, resolve ownership. If permissions failed, repair the access boundary. If an action executed twice, investigate event handling. A longer prompt is not a general remedy for every defect.
Test variation and repeatability deliberately
Open-ended systems can produce different wording across runs. For consequential cases, inspect whether repeated attempts preserve the required behavior. Record settings and configuration where available. One acceptable answer does not prove that another run will behave the same way.
Also vary realistic inputs: reordered information, longer messages, unusual formatting and incomplete fields. Keep the underlying expected behavior clear. Do not make a superficial paraphrase count as a completely new independent customer scenario when reporting coverage.
Include an out-of-scope request and input that attempts to redirect the workflow. The system should preserve the original task and authority boundaries. A test is informative when it corresponds to a plausible failure path, not merely because it contains an unusual phrase.
Write an explicit release decision
Illustrative decision record: a support-drafting pilot passes ordinary policy questions after review but still struggles with multi-topic requests. The team may approve the ordinary category for a limited group while routing multi-topic cases manually. It must record the classification mechanism and check that excluded cases are actually caught.
Approval should name the population, authority, review requirement, known limitations and rollback owner. A good offline score does not authorize automatic sending, private-data access or a new customer population. Each changes the system being evaluated.
Decide how post-release evidence will be collected. Sample apparently successful jobs as well as reported errors, because customers and reviewers do not report every defect. Track corrected answers, repeat contacts, queue age and review effort alongside technical failures.
Maintain a living regression set
Add verified failures with their expected behavior after investigation. Keep a separate held-back set for future comparisons, since repeated tuning against a fixed regression set can make it less independent.
Re-run relevant cases after changes to sources, prompts, models, permissions or downstream actions. Preserve results by version so the team can see tradeoffs rather than only the latest score. Retire obsolete cases deliberately, with a reason.
A useful evaluation system makes the next release decision clearer. It tells the team which behavior is acceptable, where uncertainty remains and what evidence would justify more authority.
Sources and further reading
- NIST Generative AI Profile, for generation-related risk considerations
- GitHub: responsible use of Copilot agents, an example of documented AI-assistance limitations
- Google: a scientific approach to model improvement