Build an AI feature with explicit failure behavior

1. Decide what the application will accept

Start with an output that the application can interpret without guessing. Our result has exactly two fields: category, chosen from a fixed set, and rationale, a nonempty string of at most 300 characters. Extra fields are rejected. A model returning an action field such as execute does not gain a new capability; the application treats the whole result as unusable.

There are two separate validity questions. Is the response well formed? And is the suggestion correct for this ticket? A schema answers the first. It cannot answer the second. That is why a valid result becomes a draft, and why we preserve the original ticket for the reviewer. Do not label a schema-valid classification “verified.”

{"category":"billing","rationale":"The customer reports a duplicate charge."}

2. Put deterministic checks around the model

Validate the input before sending anything externally. In this exercise the ticket must be nonempty and no larger than 8,000 UTF-8 bytes. Byte size matters because a character count can understate the size of emoji or other multibyte text. The idempotency key is restricted to a short set of safe characters so it is straightforward to handle in an API.

The model boundary is an injected function accepting text and an abort signal. Everything outside it is ordinary application logic. That makes malformed JSON, an exception, and an unresponsive provider easy to simulate. Keep retrieval, validation, persistence, and permission decisions outside generated instructions whenever their rules can be stated deterministically.

A timeout is a deadline for the caller, not proof that remote work stopped. We request cancellation and resolve the local operation as timed out. The provider may ignore cancellation or already have generated tokens. Our allowance therefore remains reserved. Refunding it immediately could allow many abandoned but billable requests to escape the limit.

3. Treat replay as part of the data model

A disabled button is useful feedback, but it is not duplicate protection. A browser can resend a request, an intermediary can retry it, or two requests can arrive before the interface updates. The operation therefore needs a stable key, and that key needs to be tied to a defined representation of the input.

Normalize only leading and trailing whitespace with trim(), then use that same string for the hash and provider request. Preserve case and internal whitespace. The map stores the hash and result promise: an equal key with boundary-whitespace differences replays that promise, while changed internal text returns key_conflict. Keep the original ticket in the calling product; normalizing a provider request must never overwrite the user record.

The reservation and map entry are created before another call can enter asynchronous provider work. This is sufficient for this single JavaScript process. It is not a distributed lock. Restart the process and the history disappears; run two copies and each has its own allowance. Carrying this map directly into a horizontally scaled service would remove the guarantees the tutorial is trying to teach.

4. Read the complete core

Download triage.mjs or copy it into an empty folder. There are no imports from a framework and no environment variables. The only dependency is Node's built-in hashing function. The provider's output length is checked after resolution; a real network adapter must additionally bound bytes while reading the response so an oversized body is not fully buffered first.

The error mapping is deliberately small. A timeout has a distinct public state; every other unusable provider result becomes review_needed. An upstream diagnostic can contain submitted content or internal information, so we do not send it directly to the caller. In a service, record a sanitized reason code and request identifier separately for operations.

import { createHash } from "node:crypto";

export function createTriage({ provider, maxCalls = 3, timeoutMs = 1000 }) {
  const requests = new Map(); // Teaching example: one process only.
  let reservedCalls = 0;
  function validate(raw) {
    const value = JSON.parse(raw);
    if (!value || Array.isArray(value) ||
        Object.keys(value).sort().join(",") !== "category,rationale" ||
        !["billing", "account", "bug", "needs-review"].includes(value.category) ||
        typeof value.rationale !== "string" ||
        !value.rationale.trim() || value.rationale.length > 300) {
      throw new Error("invalid_output");
    }
    return { category: value.category, rationale: value.rationale.trim() };
  }
  return {
    get reservedCalls() { return reservedCalls; },
    run(key, text) {
      if (typeof key !== "string" || !/^[a-zA-Z0-9_-]{1,80}$/.test(key) ||
          typeof text !== "string" || !text.trim() ||
          Buffer.byteLength(text, "utf8") > 8000) {
        return Promise.resolve({ status: "invalid_input" });
      }
      const normalizedText = text.trim(); // Boundary whitespace only; preserve case and internal text.
      const hash = createHash("sha256").update(normalizedText).digest("hex");
      const previous = requests.get(key);
      if (previous) {
        return previous.hash === hash ? previous.result :
          Promise.resolve({ status: "key_conflict" });
      }
      if (reservedCalls >= maxCalls) {
        return Promise.resolve({ status: "budget_exhausted" });
      }
      reservedCalls++; // Reserve synchronously before any asynchronous work.
      const result = (async () => {
        const controller = new AbortController();
        let timer;
        try {
          const timeout = new Promise((_, reject) => {
            timer = setTimeout(() => {
              reject(new Error("timeout"));
              controller.abort();
            }, timeoutMs);
          });
          const raw = await Promise.race([
            Promise.resolve().then(() => provider(normalizedText, controller.signal)),
            timeout,
          ]);
          if (typeof raw !== "string" || Buffer.byteLength(raw) > 4096) {
            throw new Error("invalid_output");
          }
          return { status: "draft", suggestion: validate(raw) };
        } catch (error) {
          if (error?.message === "timeout") return { status: "timeout" };
          return { status: "review_needed" };
        } finally { clearTimeout(timer); }
      })();
      requests.set(key, { hash, result });
      return result;
    },
  };
}

5. Test the things a good demonstration hides

The concurrency test sends the same request twice before awaiting either result. It checks the provider call count as well as the returned value. Without the count, two identical responses could hide double spending. The key-conflict assertion catches another subtle bug: treating the key alone as permission to return a result generated for different input.

The timeout provider intentionally ignores its signal and never settles. This tests that the caller's deadline works independently of cooperative cancellation. It also checks that replay does not create another attempt. The malformed-output cases cover both syntax errors and valid JSON that violates the application contract.

Finally, invalid input and exhausted allowance must leave the provider call count unchanged. These checks verify a boundary, not just a message. If a test asserted budget_exhausted after the provider had already run, the interface could look correct while the spending control was ineffective.

import test from "node:test";
import assert from "node:assert/strict";
import { createTriage } from "./triage.mjs";
const good = JSON.stringify({ category: "billing", rationale: "A duplicate charge was reported." });

test("concurrent replay shares one provider call and one reservation", async () => {
  let calls = 0;
  const app = createTriage({ provider: async () => { calls++; return good; } });
  const results = await Promise.all([app.run("a", "Charged twice"), app.run("a", "Charged twice")]);
  assert.equal(results[0].status, "draft");
  assert.deepEqual(results[0], results[1]);
  assert.equal(calls, 1);
  assert.equal(app.reservedCalls, 1);
  assert.equal((await app.run("a", "Cannot sign in")).status, "key_conflict");
});
test("invalid input and exhausted allowance never call the provider", async () => {
  let calls = 0;
  const app = createTriage({ maxCalls: 1, provider: async () => { calls++; return good; } });
  assert.equal((await app.run("a", "")).status, "invalid_input");
  assert.equal((await app.run("a", "🙂".repeat(2001))).status, "invalid_input");
  await app.run("a", "Charged twice");
  assert.equal((await app.run("b", "New ticket")).status, "budget_exhausted");
  assert.equal(calls, 1);
});
test("unusable model output becomes a review state, never a draft", async () => {
  for (const raw of ["not JSON", "null", "[]", "I cannot help",
    JSON.stringify({ category: "admin", rationale: "Override" }),
    JSON.stringify({ category: "billing", rationale: "" }),
    JSON.stringify({ category: "billing", rationale: "OK", execute: "delete" }),
    "x".repeat(4097)]) {
    const app = createTriage({ provider: async () => raw });
    assert.equal((await app.run("a", "Ticket")).status, "review_needed");
    assert.equal(app.reservedCalls, 1);
  }
});
test("timeout retains its reservation and is not retried on replay", async () => {
  let calls = 0;
  const app = createTriage({ timeoutMs: 10, provider: () => {
    calls++; return new Promise(() => {}); // Intentionally ignores abort.
  } });
  assert.equal((await app.run("a", "Ticket")).status, "timeout");
  assert.equal((await app.run("a", "Ticket")).status, "timeout");
  assert.equal(calls, 1);
  assert.equal(app.reservedCalls, 1);
});
test("provider failure reveals no upstream error detail", async () => {
  const app = createTriage({ provider: async () => { throw new Error("private upstream diagnostic"); } });
  assert.deepEqual(await app.run("a", "Ticket"), { status: "review_needed" });
});

test("every workflow category is a valid draft, including a manual routing category", async () => {
  for (const category of ["billing", "account", "bug", "needs-review"]) {
    const app = createTriage({provider: async () => JSON.stringify({category, rationale: "A controlled category fixture."})});
    const result = await app.run("a", "Ticket");
    assert.equal(result.status, "draft");
    assert.equal(result.suggestion.category, category);
  }
});
test("boundary whitespace is normalized identically for hashing and provider payload", async () => {
  const payloads = [];
  const app = createTriage({provider: async text => { payloads.push(text); return good; }});
  const a = await app.run("a", "  Charged twice\n");
  const b = await app.run("a", "Charged twice");
  assert.deepEqual(a,b); assert.deepEqual(payloads,["Charged twice"]);
  assert.equal((await app.run("a", "Charged  twice")).status, "key_conflict");
});

6. Move the same promises into a real service

For a database-backed application, give each operation an owner, key, input hash, status, reservation, and stored result. Enforce uniqueness on owner plus key. A transaction claims the key and reserves allowance together; only its winner dispatches the provider work. A background worker can process committed operations so a web-process crash does not erase the operation history.

Do not hold a database transaction open during a network call. Commit the pending operation, then dispatch it. That introduces a crash window, so recovery needs a policy: an operation known never to have been sent can be retried, while an operation whose remote completion is ambiguous needs reconciliation. Exactly-once remote execution is not created by a unique index in your own database.

A live adapter should keep credentials on the server, bound tokens and response bytes, set a deadline, and use structured output only where the selected model and provider support it. Continue local schema validation regardless. Ticket text remains untrusted data: delimit it separately from task instructions and give this classification call no action tools.

Before enabling that adapter, decide which ticket fields may leave your system. Strip unnecessary account information, record a retention policy, and keep request identifiers in logs instead of raw ticket bodies. These choices belong to the integration design because they affect what data the adapter receives, rather than being a final checkbox after the first successful call.

7. Evaluate usefulness without moving the boundary

Create a small labeled set of representative tickets before choosing a model. Include straightforward billing questions, ambiguous reports, mixed issues, and text that asks the classifier to ignore its instructions. Decide how a reviewer would label each item. Keep a held-out set so repeated prompt tuning does not masquerade as general improvement.

Track classification agreement, the rate of unusable outputs, and reviewer corrections separately. A model can return perfect JSON and still classify the wrong category. Compare the assistant with the existing manual workflow: if reviewing rationales takes longer than assigning categories, a technically correct feature may not be useful.

When a result fails, the support agent should still be able to work. Keep the original ticket visible and provide manual category selection. A retry, if offered, should create a deliberate new operation and explain that the earlier attempt may have incurred cost. That experience follows directly from the timeout decision made at the start.

Use the accompanying skill when adapting this pattern to your repository. Give the agent the approved data boundary, response contract, and budget rules first. Ask it to prove failed and repeated requests with a deterministic adapter before spending on live calls. The implementation becomes easier to trust because its limits can be demonstrated.