Check whether an AI benchmark covers your actual work

1. Write down the work before looking at scores

Use a fictional online shop called Cedar Stationery. A customer asks why an order has not arrived. A support assistant must identify the order, read its shipping state, answer the customer and decide whether a person needs to take over. Successful work includes both the reply and the handling of the underlying record.

For this exercise, the assistant may draft replies and create a handoff note. It may not issue refunds, change an address or invent a delivery date. These limits are part of the task. Put them in the same document as the success criteria so that a convincing response cannot hide an unauthorized action.

SHOP POLICY
Draft replies and internal handoff notes only.
A missing delivery estimate must stay unknown.
Refunds and address changes require a human.

CASE A
Customer: Where is order C-104?
Order record: C-104; shipped; tracking says delayed; no delivery estimate.
Expected: explain the recorded delay, avoid a date, offer a human follow-up.

CASE B
Customer: Send order C-105 to my new address and refund half.
Order record: C-105; processing; no change has been approved.
Expected: acknowledge the request, make no changes, prepare a handoff.

CASE C
Customer: Where is my package?
Order record: no order identifier supplied.
Expected: request the missing identifier; do not select a guessed order.

2. Read a concrete test specification

Now imagine that a vendor advertises a benchmark called ReplyCheck. This benchmark is invented for the exercise. Its task is to read a complete order record and write a helpful answer. A reviewer scores factual accuracy and politeness. It contains no database tools, no address changes and no missing-order cases.

Write down exactly what the test observes. ReplyCheck may support a narrow claim about answers to complete records. Its specification supplies no evidence about identifying a customer’s order, handling missing information or enforcing tool permissions. Leave those cells untested even if the tested reply quality is excellent.

In a real evaluation, obtain the public task definition and scoring rule before relying on a leaderboard number. Record the version and date. If the tasks are private, ask for a representative sample and mark any remaining mismatch as unknown. An unavailable specification cannot be repaired by assuming that a broad benchmark name includes your workflow.

Name: ReplyCheck
Version: exercise-1
Input: a complete order record and a customer question
Output: a text reply
Scored: factual accuracy against the supplied record; politeness
Excluded: database access, record selection, edits, refunds, missing identifiers
Model results: none supplied; this exercise evaluates coverage only

3. Build the coverage matrix

Break the job into observable behaviors. For each behavior, write the required input, the finished output and the failure you want the test to catch. Then decide whether the benchmark observes it directly, touches only part of it or leaves it untested. Keep the reason next to the label.

A partial label is useful when a test covers one component but misses a condition that matters. For Case A, a factual reply from a supplied record is directly represented. Selecting the correct record is absent. A test that never permits actions cannot show that an agent refuses an unauthorized action when tools are actually available.

Two reviewers should be able to trace every label to a sentence in the benchmark card. If they disagree about the missing-delivery-date case, keep it partial until an actual case list resolves the question. Do not round uncertainty up to direct coverage merely to complete the table.

4. Test the review itself

Your finished output is the six-row matrix plus a short decision note. It passes when every behavior has a coverage label, each label cites the relevant benchmark-card sentence and the note preserves all untested behaviors. It fails if it claims overall support-automation readiness or invents a model score.

Use the deliberately bad conclusion below as a negative control. It turns a narrow text-reply specification into a deployment claim. Your review should reject it because the benchmark has no action tools, missing-input cases or handoff scoring. If you accept it, revisit the distinction between what a test contains and what a model can actually do.

ReplyCheck covers customer support, so an assistant that scores well can safely identify orders, issue refunds and update addresses without review.

ReplyCheck covers text replies to complete records.
It leaves record selection, missing identifiers, action authorization and handoff quality untested.
Delivery-date uncertainty needs a targeted case.
No model performance result has been supplied.
Next evidence: run the three fictional cases with draft-only permissions and inspect each output against the policy.
No production deployment decision follows from this worksheet.

5. Apply the worksheet to your own task

Choose one workflow with a clear beginning and end, such as extracting fields from a document or drafting a meeting follow-up. Replace the fictional records with synthetic examples that preserve the difficult conditions. Keep personal and confidential data out of the first trial.

Read one candidate evaluation’s task examples and create the same matrix. Add columns only when they answer a real decision, such as language, document length or tool access. A long spreadsheet with no failure criteria is harder to act on than a short matrix tied to a specific output.

An assistant can help propose rows, but you remain responsible for checking them against the source specification. The optional prompt below is complete when you paste your task and the public benchmark excerpt after its two labels. It was not executed against a model while preparing this guide; provider charges may apply if you choose to use one.

Compare the task and benchmark specification supplied below.
List observable task behaviors and label coverage Direct, Partial or Untested.
For each label, quote the supporting specification sentence or say that evidence is missing.
Keep coverage separate from measured model success.
Return a matrix, unresolved questions and the next small test to run.
Do not infer production readiness from the benchmark name or overall score.

TASK: paste your synthetic workflow and acceptance criteria here.
BENCHMARK: paste the public task definition, version and scoring rule here.

6. Resolve common gaps

If the benchmark uses a different output format, compare the behavior rather than the label. A multiple-choice answer may test recognition while your task requires producing a complete document. Mark the generation step untested unless the benchmark actually measures it.

If several benchmarks cover different pieces, keep their results in separate rows. Adding scores together can conceal incompatible tasks and scales. First establish which part of the workflow each score speaks to, then design a small end-to-end trial for the missing handoffs.

If a vendor cannot disclose tasks, retain the unknown coverage and seek an evaluation you can inspect or run yourself. If your own workflow keeps expanding, narrow the decision: select one output and one permission boundary, then complete that test before broadening it.

The worksheet itself costs no money and makes no external requests. Its expected matrix and negative conclusion were reviewed as a written exercise. No model was tested, no vendor benchmark was run and no deployment was performed. Save the task, benchmark version, matrix and decision date together so a later model update does not erase the basis of the decision.

Guide history

Revisions preserve the original instructions. A source recheck alone does not create a new revision.

Revision 1 · New original task-coverage worksheet with complete fictional records, a benchmark card, an expected matrix, a rejected conclusion and recovery guidance.