Agent failure labs / Guide 69
A Payment Timeout Must Not Create a New Payment Intent
Reproduce an AI payment retry idempotency failure after a lost response, then preserve operation identity, reject payload drift, and reconcile unknown outcomes.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.

“Pay this approved invoice once.” A hypothetical assistant submits the instruction, but the connection closes before the response arrives. The assistant interprets the timeout as failure and retries with a newly generated operation key. The provider accepts both requests. A transport error has become a duplicate business operation because retry identity followed the attempt rather than the user’s intent.
This lab isolates the lost-response window. It does not implement payment processing or a full invoice workflow. The mock provider records a local ledger entry and deliberately raises a timeout after that entry exists. The naive caller changes its key; the fixed caller reuses the same key and payload. Nothing in the exercise contacts a provider or transfers money.
Define the Retry Contract Precisely
Stripe’s idempotent request documentation describes saving the first request’s status and body for an idempotency key, comparing parameters on reuse, and returning the same result for subsequent requests. It also documents limits: keys can be pruned after they are at least twenty-four hours old, and results are saved only after endpoint execution begins. These are provider-specific semantics, not a universal exactly-once guarantee.
Create the business operation identifier before submission and persist it with the approved payload. The identifier must survive model retries, process restarts, queue redelivery, and browser refreshes. Do not generate a new key simply because a new HTTP request is being made. A deliberate second payment is a new business request requiring its own authorization, not a recovery shortcut.
The mock below supports permanent in-memory replay and exact tuple matching. That is intentionally simpler than Stripe. The first call records the fixture and loses its response. Both branches experience the same fault; only their retry-key policy differs. Assertions also show that changing amount under the original key is rejected.
from copy import deepcopy
fixture = {"invoice": "INV-9", "account": "A", "amount_minor": 12500,
"currency": "EUR"}
class MockProvider:
def __init__(self):
self.saved = {}
self.ledger = []
self.drop_next_response = True
def submit(self, key, payload):
fingerprint = tuple(payload[k] for k in
("invoice", "account", "amount_minor", "currency"))
if key in self.saved:
previous, receipt = self.saved[key]
if previous != fingerprint:
raise ValueError("same_key_different_payload")
return deepcopy(receipt)
receipt = {"origin": "mock_provider", "operation": key,
"payload": fingerprint, "status": "recorded_mock"}
self.ledger.append(fingerprint)
self.saved[key] = (fingerprint, receipt)
if self.drop_next_response:
self.drop_next_response = False
raise TimeoutError("response_lost_after_recording")
return deepcopy(receipt)
def run(reuse_key):
provider = MockProvider()
try:
provider.submit("op-9", fixture)
except TimeoutError:
outcome = "unknown"
assert outcome == "unknown" and len(provider.ledger) == 1
receipt = provider.submit("op-9" if reuse_key else "op-10", fixture)
return provider, receipt
naive_provider, naive_receipt = run(False)
fixed_provider, fixed_receipt = run(True)
assert len(naive_provider.ledger) == 2
assert len(fixed_provider.ledger) == 1
assert fixed_receipt["operation"] == "op-9"
assert fixed_receipt["payload"] == ("INV-9", "A", 12500, "EUR")
assert fixed_provider.submit("op-9", fixture) == fixed_receipt
try:
fixed_provider.submit("op-9", dict(fixture, amount_minor=13000))
except ValueError as error:
assert str(error) == "same_key_different_payload"
else:
raise AssertionError("payload drift accepted")
assert len(fixed_provider.ledger) == 1
This is an explicitly synthetic deterministic simulation of a possible failure, not a real model experiment, provider sandbox test, or payment. The local ledger is a list. Its returned status means only that the mock recorded a tuple. There is no authorization, capture, settlement, banking network, or customer balance in this example.
Prompt Instructions Do Not Persist Identity
“Never pay twice” does not tell a restarted worker which request might already exist. A model may generate a fresh UUID because its context lacks the previous attempt, or decide that a timeout warrants another tool call. Keep operation identity outside the conversation and inject it through a server-controlled executor. The model can request progress, but it cannot invent replacement keys for unresolved operations.
Separate candidate invoice selection from authorization. The server proves which tenant owns the invoice and whether the actor can initiate the approved operation. Preconditions confirm the invoice version, amount, currency, destination, and existing payment state. A named approval binds to those terms. Only then does the provider adapter submit the request and begin reconciliation.
The approved payload must remain immutable across retries. If the invoice changes, stop recovery and determine the original outcome before proposing a new operation. Reusing a key with a changed amount is an error, not a convenient adjustment. Replacing the key without resolving the first submission is equally dangerous. A specialist agent may explain the discrepancy, but should not decide whether to transfer additional funds.
Reconciliation Is More Than a Success Flag
Stripe’s webhook guidance describes signature verification, duplicate events, and the absence of guaranteed event ordering. Use authoritative provider events or authenticated reads to update operation state, matching the provider account, resource, amount, currency, and relevant operation identifiers. A model-generated status: success or an unsigned webhook-shaped object is not a payment receipt.
Distinguish submission accepted, processing, succeeded, and the relevant downstream settlement states according to the actual provider and payment method. Do not infer finality from a generic transport response. If an event arrives before the request response, reconcile both observations to the same operation rather than creating another ledger entry. Keep the raw event under restricted retention with a deduplicated processing record.
The snippet omits authentication sources, database transactions, provider semantics beyond the toy behavior, approval persistence, and durable state storage. PostgreSQL’s transaction isolation documentation explains concurrency behavior relevant to claiming a local operation. A local transaction cannot atomically commit a remote provider effect; the crash window requires provider idempotency plus reconciliation, not optimistic wording.
When a provider key expires, do not automatically replay an unresolved old operation with a new key. Escalate for authoritative lookup and review. If a duplicate is confirmed, remediation is a separately authorized financial operation; never issue a speculative refund based on a timeout. Limit worker permissions to the required provider account and operation types. Invoice notes containing payment instructions cannot override a verified destination.
Acceptance Tests
- A lost response after mock recording must create two entries under the naive new-key retry and one under the fixed same-key retry.
- Replaying the same key and payload must return the original mock receipt without adding another ledger entry.
- Reusing a key with a changed amount, currency, account, or invoice must fail as payload drift.
- Restart recovery must load the persisted operation key rather than request a replacement key from the model.
- Duplicate, reordered, unsigned, or mismatched provider events must not create false completion or duplicate local postings.
- An unresolved operation beyond the provider’s replay-retention window must enter human reconciliation, not a fresh automatic submission.
Measure Unknown Outcomes Honestly
Use the new-key retry as the synthetic baseline and inject failures before submission, after acceptance, and during reconciliation. Count duplicate provider effects over distinct intended business operations, not over HTTP attempts. Track unresolved operations over all submitted operations and their age. A system that suppresses retries forever may avoid duplicates while leaving customers and finance staff with unacceptable uncertainty.
Start with a nonmutating adapter contract and reviewed recovery runbooks, then use an authorized provider test environment in a separate integration project. Proposed gates include stable identity across restarts and a documented owner for every unresolved operation. The local assertions are evidence about this mock only; no real payment reliability result is claimed.
The invoice processing guide covers upstream evidence, while the subscription confirmation lab binds consent to exact terms. Backend API engineering should begin by persisting operation identity before the first submission. Recovery becomes tractable when retries continue one operation instead of inventing another.