Check an agent’s completion claim against saved evidence

1. Create a small review folder

Open a terminal in a location where you keep temporary practice files. The following commands create a new folder and move into it. Use a fresh folder if that name already contains work you want to keep.

Check that the version command prints Python 3.9 or later. You do not need to install any Python packages. If Python is unavailable, the fixture and expected-verdict table still support a manual review, but the executable part of this exercise remains untested on your machine.

mkdir evidence-receipt-practice
cd evidence-receipt-practice
python3 --version

2. Save the request and the observed result

Save the following text as fixture.json in that folder. BOLT and NUT are fictional inventory labels. The request asks for four bolts and two nuts. The artifact records four bolts and one nut.

The trace says a file was written. The completion message claims correctness and delivery. Each of those observations has a different scope: writing something does not establish that its contents match the request, and a local statement cannot authenticate a delivery. Preserve this original fixture so you can repeat the failed case later.

{
  "request": {"BOLT": 4, "NUT": 2},
  "artifact": {"rows": [{"sku": "BOLT", "qty": 4}, {"sku": "NUT", "qty": 1}]},
  "trace": [{"event": "write", "path": "picklist.json"}],
  "agentClaim": "The correct pick list was created and sent.",
  "deliveryReceipt": null
}

3. Save the criterion checker

Save this complete script as check_receipt.py beside the fixture. It checks that the saved artifact exists, compares every requested item and quantity, and explicitly leaves delivery unknown. JSON evidence pointers such as $.artifact.rows name the field a reviewer should inspect.

The quantity comparison rejects duplicate item names, zero or negative amounts, text in place of numbers and Boolean values. Python otherwise treats True as a kind of integer, so the explicit type check matters here. A missing artifact produces an unknown comparison because there is no saved output to inspect.

Delivery stays unknown even if you add a deliveryReceipt field. The helper has no authenticated connection to a delivery service and cannot establish that such a field is genuine. This deliberate limit prevents a more confident message from changing the result.

import json
import sys
from pathlib import Path


def assess(record):
    request = record.get("request")
    if not isinstance(request, dict) or not request:
        raise ValueError("request must be a nonempty object")
    for sku, qty in request.items():
        if not isinstance(sku, str) or not sku or type(qty) is not int or qty <= 0:
            raise ValueError("request needs nonempty SKU names and positive integers")

    artifact = record.get("artifact")
    present = isinstance(artifact, dict)
    checks = [{
        "criterion": "saved artifact exists",
        "verdict": "PASS" if present else "FAIL",
        "evidence": "$.artifact",
        "reason": "A saved object is present." if present else "No saved object is present."
    }]
    rows = artifact.get("rows") if present else None
    actual = {}
    valid = isinstance(rows, list) and bool(rows)
    if valid:
        for row in rows:
            if not isinstance(row, dict):
                valid = False
                break
            sku, qty = row.get("sku"), row.get("qty")
            if (not isinstance(sku, str) or not sku or sku in actual
                    or type(qty) is not int or qty <= 0):
                valid = False
                break
            actual[sku] = qty
    if not present:
        verdict, reason = "UNKNOWN", "No artifact is available to compare."
    elif not valid:
        verdict, reason = "FAIL", "Rows need unique SKU names and positive integer quantities."
    elif actual == request:
        verdict, reason = "PASS", "Every requested SKU and quantity matches; there are no extra SKUs."
    else:
        verdict, reason = "FAIL", f"Expected {request}; observed {actual}."
    checks.append({
        "criterion": "saved quantities match the request",
        "verdict": verdict,
        "evidence": "$.request; $.artifact.rows",
        "reason": reason
    })
    checks.append({
        "criterion": "delivery is independently confirmed",
        "verdict": "UNKNOWN",
        "evidence": "No external delivery system was queried.",
        "reason": "This exercise cannot authenticate delivery. A claim or local receipt field is insufficient."
    })
    return {"checks": checks, "scope": "Local fixture review only; no delivery was attempted."}


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python3 check_receipt.py fixture.json")
    try:
        record = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
        if not isinstance(record, dict):
            raise ValueError("fixture must be an object")
        receipt = assess(record)
    except (OSError, ValueError) as exc:
        raise SystemExit(f"Cannot assess fixture: {exc}")
    print(json.dumps(receipt, indent=2))

4. Produce and inspect the receipt

Run the following commands from the folder containing both files. The first command writes the review to receipt.json. The second prints it in a readable format. No network request occurs.

The exercise passes when the first fixture produces PASS, FAIL and UNKNOWN in that order, and you can explain each verdict using its named evidence. A FAIL inside the receipt is the expected detection of the planted quantity error. The script returning normally means it completed the review; it does not mean every task criterion passed.

python3 check_receipt.py fixture.json > receipt.json
python3 -m json.tool receipt.json

The second check’s reason includes both expected and observed quantities. Open fixture.json alongside receipt.json and confirm that those values really occur at the stated fields. That final reading is part of the review, especially if you later adapt the checker.

5. Repair one fact and rerun

Make a copy named fixture-fixed.json. Change only the NUT row inside artifact.rows from 1 to 2. Keep the request, completion message and delivery field unchanged. Run the checker with the new filename and save a separate receipt.

The expected sequence is now PASS, PASS, UNKNOWN. That change shows that the quantity verdict responds to the saved evidence. Delivery still needs evidence from a trusted outside system. Leave the incomplete claim visible rather than treating two passing checks as proof of the entire task.

For a second test, change only agentClaim in the original fixture to say everything passed. Run it again. The three verdicts must stay PASS, FAIL, UNKNOWN. A checker that changes its judgment merely because the claim sounds certain is failing this exercise.

python3 check_receipt.py fixture-fixed.json > receipt-fixed.json
python3 -m json.tool receipt-fixed.json

If a result looks wrong

Keep the original fixture, the corrected copy and their receipts together. If you change the checker, rerun both cases before relying on its output. Also try deleting artifact: the expected sequence is FAIL, UNKNOWN, UNKNOWN. This catches a common mistake where missing evidence silently becomes a passing comparison.

Use this pattern in a real review

Write criteria before the agent starts: what file must exist, which fields must match and which external action requires confirmation. Keep the original request and captured output separate. A reviewer should be able to identify the exact observation supporting each verdict.

This helper checks one narrow data structure. It does not judge writing quality, infer an unstated business rule, authenticate files, confirm delivery or detect a compromised evidence source. A real system needs a trusted capture process and an explicit policy for what happens when a criterion remains unknown.

The local exercise has no API charges. Adopting Sauron adds its own dependencies, model access and provider costs. Its benchmark findings describe the authors’ evaluation conditions. Completing this fixture gives you a reusable review pattern and a tested small checker; it supplies no measurement of Sauron’s accuracy in your environment.

Guide history

Revisions preserve the original instructions. A source recheck alone does not create a new revision.

Revision 1 · New original evidence-receipt exercise with complete fixture, tested Python helper, expected verdicts, repair checks and explicit external-verification limits.

Continue learning