Agent failure labs / Guide 51
AI Meeting Booking: Two Johns and an Ambiguous Friday
Reproduce an AI meeting booking ambiguity with two matching contacts and two possible Fridays, then enforce explicit identity and time confirmation before booking.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
“Book John next Friday at nine to discuss the acquisition.” In this hypothetical request, the organizer knows which John and which Friday. The assistant does not. Its directory contains John in legal and John at an external supplier. Its date parser produces both the coming Friday and the following Friday. Choosing the highest ranked pair would turn two unresolved questions into an invitation carrying confidential context.
This lab isolates candidate ambiguity, rather than rebuilding calendar synchronization. The operational objective is not to guess correctly more often. It is to prevent an unresolved candidate from becoming an approved recipient or time. A useful assistant can still collect availability and draft an agenda while the booking remains blocked.
The Unsafe Shortcut
A minimal unsafe implementation selects the first contact and first date. Sorting the directory differently changes who receives the invitation, although the request has not changed. The observable failure is a concrete wrong pair: the supplier John and the earlier Friday. There is no need to speculate about model reasoning to reproduce that boundary defect.
The following is a deterministic synthetic simulation of a possible failure, not an experiment against an actual LLM. Both implementations receive the identical request, directory, and date candidates. The fixture’s intended pair is test evidence, not hidden information available to the unsafe function. All results stay in memory; nothing contacts a calendar or sends an invitation.
from datetime import datetime
request = {"name": "John", "when": "next Friday at nine"}
contacts = [{"id": "supplier-john", "name": "John"},
{"id": "legal-john", "name": "John"}]
dates = ["2026-09-11T09:00:00+01:00",
"2026-09-18T09:00:00+01:00"]
intended = ("legal-john", dates[1])
def naive(req, people, times):
return (next(p["id"] for p in people if p["name"] == req["name"]),
times[0])
def guarded(req, people, times, confirmation=None):
ids = {p["id"] for p in people if p["name"] == req["name"]}
if confirmation is None:
return "clarify"
person, timestamp = confirmation
if person not in ids or timestamp not in times:
return "invalid_confirmation"
parsed = datetime.fromisoformat(timestamp)
if parsed.utcoffset() is None:
return "missing_offset"
return (person, timestamp)
assert naive(request, contacts, dates) != intended
assert guarded(request, contacts, dates) == "clarify"
assert guarded(request, contacts, dates, intended) == intended
assert guarded(request, contacts, dates, ("unknown", dates[1])) == "invalid_confirmation"
assert guarded(request, list(reversed(contacts)), dates, intended) == intended
assert guarded(request, contacts, ["2026-09-18T09:00:00"],
("legal-john", "2026-09-18T09:00:00")) == "missing_offset"
print("Ambiguity blocked; explicit candidate confirmed.")Rerunning the fixture changes the output from an invented choice to clarify. Supplying explicit confirmation then produces the intended pair. That third assertion matters: a gate that rejects everything would prevent mistakes but would not demonstrate a usable resolution path. Reversing the directory proves that ranking order no longer decides identity.
Why a Better Prompt Is Insufficient
“Always ask if uncertain” is a good conversational instruction, but uncertainty reported by a model is not the same as ambiguity in authoritative records. The model may confidently associate John with the most recent thread. A malicious meeting description may even say that the supplier is the approved legal contact. Neither statement should change the directory or the organizer’s delegation rights.
Represent candidate count and confirmation state as explicit data. Require a stable contact identifier, a full date, a displayed timezone, and duration. Python’s datetime documentation distinguishes aware objects from naive objects that cannot locate themselves unambiguously in time. An offset helps locate this fixture’s instant; it does not resolve the human meaning of “next Friday” or implement future timezone rules.
Build a Confirmation Boundary
The first production component authenticates the organizer and loads their permitted calendars and contact scope. Candidate search happens only inside that scope. The assistant drafts a short clarification with organization labels and concrete dates. Show only necessary directory information: disambiguation should not become an employee enumeration endpoint.
The next component stores the selected pair together with duration, agenda classification, calendar identifier, directory revision, and an expiry. The user approves that specific proposal. OWASP transaction authorization guidance recommends acknowledging significant transaction data and preventing later modification. Applied here, changing the attendee or date invalidates approval rather than silently editing the pending invitation.
Before execution, check authorization again, refresh availability, and compare revisions. OWASP authorization guidance recommends permission validation on every request and server-side enforcement. Knowing John’s identifier proves selection, not permission to invite him to a confidential meeting. A separate policy must govern external attendees, sensitive agendas, and delegated calendars.
One orchestrator and ordinary directory and calendar adapters are enough for the first version. A specialist scheduling agent earns its complexity only when it handles independently measurable constraints, such as several regional working-hour policies. It should return candidate slots, not acquire authority to resolve identity or approve invitations.
Recover Without Creating a Second Meeting
Persist an operation identifier before submitting an approved invitation. If the provider times out, record the outcome as unknown and reconcile using the provider’s supported identifiers and retrieval semantics. Do not change the request identifier and create another meeting because the first response was missing. A returned status: success from a conversation or arbitrary tool payload is not a receipt.
An authoritative result must match the organizer, calendar, event identifier, selected attendees, and time. If a later refresh discovers a conflict, return to the organizer with a new proposal. Rescheduling and canceling are separate authorized operations. Do not automatically cancel an unrelated event merely because its title resembles the draft.
The snippet omits the authentication source, persistent approvals, database transactions, state persistence, provider semantics, and actual side effects. It also omits recurrence, timezone database transitions, and availability races. Its guarantee is deliberately narrow: the supplied ambiguous candidates cannot progress without matching confirmation. Incomplete or incorrect directory data can still make that confirmation misleading.
Acceptance Tests
- Two Johns and two Fridays must return a clarification without recording any booking attempt.
- Reordering identical directory candidates must not change a previously confirmed identity.
- A contact outside the organizer’s tenant must be unavailable even when its identifier is supplied directly.
- A timestamp without an offset must be rejected; a displayed timezone disagreement must require clarification.
- A changed attendee, agenda classification, or availability revision must invalidate the earlier approval.
- A timeout after submission must enter reconciliation, never generate an immediate replacement invitation.
Measure the Clarification Cost
Start with a human-reviewed baseline of ambiguous requests, including abbreviations, duplicate names, and regional date phrasing. Measure wrong booking proposals over all reviewed booking requests, and unresolved requests correctly held over all deliberately ambiguous requests. Track clarification turns and abandoned requests separately so a low error rate cannot hide a workflow that never completes.
Run the gate in shadow mode before allowing a small internal calendar pilot. A proposed guardrail is zero unconfirmed ambiguous submissions in that pilot, not a claimed achieved result. Review every bypass and sample apparently unambiguous requests to detect missing directory candidates. Expansion should depend on evidence quality, not just assistant fluency.
The broader executive calendar architecture covers delegation and synchronization; the email recipient verification lab examines a related identity trap without date parsing. For implementation support, AI systems and agent engineering can establish the evaluation and approval boundary. Start with the two-John fixture at the booking adapter, where an unsafe selection can actually be stopped.