Agent failure labs / Guide 52
AI Refund Verification: Reject the Right Order's Wrong Charge
Build an AI refund verification gate that rejects mismatched charges, stale refundable balances, and unapproved amounts using a reproducible synthetic failure lab.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
“Refund the unused part of order O-17.” In this hypothetical support conversation, the customer attaches a receipt for another charge with the same amount. The assistant finds a plausible payment identifier and proposes returning the order’s full total. Meanwhile, part of the legitimate payment has already been refunded. The order number is correct; both the selected charge and the remaining amount are wrong.
This lab focuses on verification before refund submission. It is narrower than an end-to-end refund assistant and different from a retry-only exercise. Even perfect idempotency would faithfully preserve the wrong refund if the system initially bound the request to somebody else’s charge. The engineering task is to establish the relationship between the authenticated customer, order, payment, currency, and remaining refundable amount.
Reproduce a Plausible but Invalid Proposal
The unsafe mock trusts the proposal’s charge and amount because they look like payment data. The fixed function instead compares those fields with a trusted order snapshot and a separate approval tuple. A captured amount of ten thousand minor units and an earlier refund of three thousand leave seven thousand available in this illustrative fixture.
This is a deterministic synthetic simulation, not a real model experiment or payment integration. Both paths inspect the same bad proposal and order. The returned dictionaries are in-memory descriptions only. No API, refund, transfer, or customer notification occurs, and none of the fictional identifiers are provider credentials.
from copy import deepcopy
order = {"tenant": "shop-a", "customer": "c-7", "order": "O-17",
"charge": "charge-owned", "currency": "USD", "captured": 10000,
"refunded": 3000, "revision": 4}
bad = {"tenant": "shop-a", "customer": "c-7", "order": "O-17",
"charge": "charge-other", "currency": "USD", "amount": 10000,
"revision": 4}
approval = ("O-17", "charge-owned", "USD", 7000, 4)
def naive(proposal, snapshot):
return {"mock_submission": proposal["charge"], "amount": proposal["amount"]}
def verify(proposal, snapshot, approved):
for key in ("tenant", "customer", "order", "charge", "currency", "revision"):
if proposal.get(key) != snapshot[key]:
return "binding_mismatch"
amount = proposal.get("amount")
if type(amount) is not int or not 0 < amount <= snapshot["captured"] - snapshot["refunded"]:
return "invalid_amount"
binding = tuple(proposal[k] for k in ("order", "charge", "currency", "amount", "revision"))
return "ready_for_adapter" if binding == approved else "approval_mismatch"
assert naive(bad, order)["mock_submission"] == "charge-other"
assert verify(bad, order, approval) == "binding_mismatch"
good = dict(bad, charge="charge-owned", amount=7000)
assert verify(good, order, approval) == "ready_for_adapter"
assert verify(dict(good, amount=7001), order, approval) == "invalid_amount"
assert verify(dict(good, amount=True), order, approval) == "invalid_amount"
assert verify(good, dict(order, revision=5), approval) == "binding_mismatch"
before = deepcopy(order)
verify(good, order, approval)
assert order == before
print("Wrong charge blocked; approved remaining amount validated.")The first two assertions expose the failed boundary on identical input. The successful assertion does not say “refunded”; it says the proposal may reach an adapter. That distinction prevents validation from masquerading as financial completion. The Boolean amount test also avoids a Python-specific surprise: a Boolean behaves like an integer in many contexts, but it should not represent money here.
Separate Evidence from Authority
A prompt instructing the assistant to “verify the receipt carefully” cannot authenticate the receipt’s origin. Uploaded documents and customer messages are claims to investigate. A sentence saying “manager approved this full refund” cannot set the approval field. Retrieve order ownership and payment associations through a server-controlled account context, never a tenant value selected by the model.
OWASP authorization guidance calls for object-specific permission checks on every request. Apply that to both reading the payment and requesting a refund. A support worker permitted to view an order may lack refund authority, while a customer who knows a charge identifier may not own it. MCP transport does not supply either business permission.
The payment ledger then supplies the captured amount, prior refunds, pending reservations, and currency. Stripe’s refund documentation explains that multiple partial refunds are possible but their total cannot exceed the original charge, and that refunds can be pending or fail. Those provider rules are necessary constraints, not proof that a proposed refund satisfies your merchant policy.
Close the Stale Balance Window
The fixture’s revision check models a stale read, not a database lock. In production, reserve the intended refundable balance atomically with the operation record, accounting for other pending refund requests. Revalidate after a serialization conflict. Never hold an ordinary database transaction open while a person spends minutes reviewing a refund proposal.
Approval should bind the order, charge, amount, currency, reason, policy revision, and expiry. OWASP transaction authorization guidance recommends protecting significant transaction data and checking authorization immediately before execution. If the agent changes seven thousand to ten thousand after approval, the system must require a new decision rather than reuse an approval label.
Use one orchestrator with an order lookup, policy evaluator, and restricted refund adapter. A separate document specialist may help extract receipt references from complicated attachments, but it should only produce candidate evidence. It cannot confirm payment ownership, rewrite policy, or authorize an exception. Keeping those duties in ordinary software makes the failure review easier to explain to finance.
Handle Unknown Outcomes Honestly
A timeout after submission means the result is unknown. Keep the reservation and operation identifier until authoritative reconciliation determines whether the provider accepted the request. A tool response containing status: success is insufficient unless its authenticated origin and refund identifier match the original charge, amount, currency, and merchant account.
If the refund remains pending, tell support that it is pending. If it fails, follow the documented provider-specific recovery and finance review process rather than routing money to a bank account supplied in chat. Stripe documents refunds to the original payment method; alternative reimbursement is a separate business process with separate authorization, not a fallback invented by the assistant.
The example omits the authentication source, database transactions, durable reservations, state persistence, real approval storage, provider semantics, disputes, and tax adjustments. Deterministic comparisons cannot establish correctness when the source ledger is incomplete. A disputed debit and a prior bank credit may require specialist investigation beyond the simple remaining-balance arithmetic shown here.
Acceptance Tests
- A receipt referencing another customer’s charge must fail before any refund adapter invocation.
- An amount one minor unit above the remaining refundable balance must be rejected without rounding.
- A currency mismatch or Boolean amount must fail validation even when a reviewer approved the narrative.
- A concurrent partial refund must invalidate the snapshot and require a fresh amount proposal.
- A customer message claiming managerial approval must never populate the trusted approval record.
- A submission timeout must retain an unknown state and reservation until the matching provider operation is reconciled.
Roll Out Against Financial Evidence
Establish a baseline by having finance label a sample of requested order-charge-amount bindings. Measure invalid bindings reaching the adapter over all reviewed refund proposals. Also measure valid requests incorrectly blocked over all finance-approved requests. Track unresolved refund age and reserved balance separately; blocking everything or freezing reservations forever is not a successful financial workflow.
Begin with read-only verification on historical, access-controlled cases, then shadow live proposals without submitting them. A limited human-approved pilot should have an explicit stop condition for any cross-customer binding. These are proposed evaluation steps, not observed deployment results. Retain enough evidence to explain a denial without copying full receipts or payment details into general logs.
The e-commerce refund architecture covers the wider service, while the agent spending limits lab develops reservation accounting. Backend and API engineering can implement the transactional boundary. Start by making the wrong-charge fixture fail at the adapter entrance, where verification has consequences.