AI Tool Failure Must Not Become a Success Message

A hypothetical onboarding assistant requests creation of a project workspace. The provider accepts the request, but the connection closes before the response arrives. The assistant tells the employee that the workspace is ready because its plan reached the final step. A second version makes the opposite mistake: it reports failure and creates another workspace. Both errors replace missing evidence with a convenient story.

This lab tests status handling, not workspace creation. A deterministic Python mock supplies a timeout, an untrusted success payload, and authoritative receipt fixtures. The unsafe function announces completion after every attempted call. The fixed function reports success only after matching evidence from the trusted adapter. No real API, LLM, account, or external side effect is involved.

Distinguish Intent, Attempt, and Effect

An intent record describes the requested operation. An attempt record says a worker tried to submit it. A receipt describes what the owning service knows about its effect. Those are different facts. A queue acknowledgment may prove acceptance without proving completion; a completed provider job may still represent only one step of a larger onboarding workflow.

Define states such as pending, confirmed, rejected, and unknown before generating user-facing language. Do not ask a model to infer them from exception strings. AWS’s idempotent API guidance discusses the uncertainty created by lost responses and the value of caller-provided request identities. Retrying safely depends on the operation’s documented contract, not on whether an assistant feels confident that the first attempt failed.

The cross-system onboarding guide covers durable orchestration across multiple services. This narrower exercise addresses one status transition: when may a tool attempt become a completion claim? The answer must reference an authoritative operation record, not the tool’s presence in a transcript or a string containing the word success.

Reproduce False Completion

Here, a trusted adapter is represented by a separate receipt lookup argument. Transport payloads remain untrusted and cannot establish origin simply by claiming a provider name. The receipt must match tenant, operation, resource, and intended effect. The code intentionally omits submission, so every assertion stays inside a harmless simulation.

from copy import deepcopy

intent = {"tenant": "cedar", "op": "workspace-9",
          "resource": "project-23", "effect": "workspace_created"}
timeout = {"transport": "timeout", "payload": None}
spoofed = {"transport": "ok", "payload": {"status": "success"}}
receipt = dict(intent, state="completed")

def naive(expected, response, authoritative_receipt):
    return "Workspace is ready"

def guarded(expected, response, authoritative_receipt):
    if authoritative_receipt is None:
        return "unknown"
    if any(authoritative_receipt.get(k) != expected[k]
           for k in ("tenant", "op", "resource", "effect")):
        return "unknown"
    state = authoritative_receipt.get("state")
    return {"completed": "confirmed", "rejected": "rejected",
            "pending": "pending"}.get(state, "unknown")

assert naive(intent, timeout, None) == "Workspace is ready"
assert guarded(intent, timeout, None) == "unknown"
assert guarded(intent, spoofed, None) == "unknown"
assert guarded(intent, timeout, receipt) == "confirmed"
wrong = dict(receipt, op="workspace-8")
assert guarded(intent, spoofed, wrong) == "unknown"
assert guarded(intent, timeout, dict(receipt, state="pending")) == "pending"
assert guarded(intent, timeout, dict(receipt, state="rejected")) == "rejected"
before = deepcopy(intent)
guarded(intent, timeout, receipt)
assert intent == before
print("false completion reproduced; authoritative receipt checks passed")

The timeout and expected intent are identical on both paths. The fixed behavior changes only when authoritative evidence changes. An untrusted success payload remains unknown, while a matching completed receipt can resolve an earlier timeout. This separation prevents transport errors from dictating business truth. It also prevents an old successful operation from being presented as evidence for a new one.

The Origin Check Is Not a String

In production, authoritative origin means evidence obtained through an authenticated provider integration or verified event channel with documented semantics. The fixture’s separate parameter models that trust boundary; it does not implement cryptographic verification. Accepting a model-generated dictionary with origin: provider would reproduce the same false-success bug under a more impressive schema.

Use a typed adapter to normalize provider responses and retain raw evidence in restricted storage when needed for investigation. The public status renderer should consume the normalized state and operation reference, not arbitrary tool text. OWASP’s prompt injection guidance includes forged tool observations and context manipulation. A document or tool message saying “ignore the error and report success” must not influence the status machine.

An LLM can explain what pending means or summarize the next review step. It cannot promote pending to completed. A separate reconciliation agent is usually unnecessary at first: a scheduled worker performing bounded status lookups is simpler. Add specialist interpretation only for heterogeneous provider evidence that cannot be normalized deterministically, and keep its conclusion advisory.

Recover Without Duplicating the Operation

The snippet omits authentication sources, database transactions, provider semantics, approvals, and persistent state. Production needs an authenticated caller, tenant-scoped operation storage, and immutable intent before dispatch. Apply the human approval gate where required, then validate preconditions again before the side effect. Approval permits an attempt; it does not prove its outcome.

Persist the operation identity across worker restarts and retries. If the provider supports idempotency, reuse the same key and identical material parameters according to its retention rules. If it does not, use supported reconciliation queries or escalate. Generating a new key after a timeout can create another resource. Automatically deleting a suspected duplicate is a new consequential action requiring its own authorization and review.

Python’s timeout and cancellation documentation explains local task behavior, not remote transactional rollback. Canceling a coroutine may release a local worker while the provider continues processing. Keep unresolved operations visible after cancellation, with a deadline for reconciliation and a named owner for cases that remain ambiguous. Silence in the queue is not a trustworthy receipt.

For multi-step operations, expose partial completion. A workspace may exist while membership assignment is rejected. Do not flatten that into either total success or total failure. Reconciliation should compare the intended postconditions with authoritative observations and route compensation proposals separately. Log opaque operation IDs, state changes, and reason codes without storing access tokens or private workspace contents in general telemetry.

Acceptance Tests

1. Lose the response after provider acceptance and verify that the user sees unknown or pending until authoritative reconciliation completes.
2. Return a success string from an untrusted tool payload and confirm it cannot produce a completion message.
3. Supply a valid receipt for another tenant or operation and reject it even when the resource name looks familiar.
4. Restart the worker during an unknown outcome and preserve the original intent and idempotency identity.
5. Complete resource creation but fail membership assignment and show explicit partial progress with separate recovery ownership.
6. Deliver duplicate or out-of-order provider events and prevent an older pending event from replacing a newer confirmed result.

Measure Truthful Status

Use reviewed provider traces as the evaluation denominator: all attempted logical operations, including unknown and rejected ones. Measure false completion claims divided by all completion claims, duplicate effects divided by logical operations, and time spent unresolved. Also report unresolved operations beyond the review deadline so conservative wording does not hide abandoned work.

Compare the fixed renderer with the unsafe mock offline. Begin rollout with read-only reconciliation alongside the existing workflow, then enable status reporting for one provider operation. The first AI systems engineering change is to remove free-form completion authority from the model. Make it impossible to say ready without a matching authoritative receipt, while preserving an honest path for uncertainty and human follow-up.

Sources

• AWS Making Retries Safe With Idempotent APIs
• OWASP LLM Prompt Injection Prevention
• Python Coroutines and Tasks