Production workflows / Guide 50
A Cross-System Onboarding Agent Built Around Durable Writes
Build a cross-system onboarding agent that coordinates approved account creation, equipment reservations, and payroll setup with durable retries and compensation.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
Imagine a hypothetical employee starting Monday. The identity provider creates an account, equipment inventory reserves a laptop, and payroll rejects the employment record because a required field is missing. Then the workflow worker crashes. A conversational agent that remembers “mostly finished” cannot reliably decide what to retry, what to undo, or what the new employee can actually access.
Cross-system onboarding is a durable write workflow. Unlike read-only knowledge federation, it changes independent systems and must survive ambiguous outcomes. The model helps explain missing information and propose an onboarding plan; a persisted state machine owns execution, reconciliation, and business-specific compensation.
Start From an Authoritative Employment Event
Use a verified HR record with a stable employment identifier, tenant, start date, manager, role, and version. Do not provision from an email that merely claims someone has joined. Rehires need a distinct employment episode linked to the existing person, otherwise old approvals and resource ownership can leak into the new job.
Build the plan from approved role templates. The manager can request equipment or standard applications, while privileged access requires its own approver. The model must not infer access from a job title alone. Re-read authoritative employment status before sensitive writes, especially after a long pause or a changed start date.
The plan should distinguish required readiness from optional convenience. A delayed welcome message should not roll back payroll preparation. A canceled employment event should stop further provisioning and trigger a reviewed recovery policy. These dependencies are business rules, not decisions a model should invent during a failure.
Journal Intent Before Calling Another System
The minimum architecture contains event intake, an approval service, a durable workflow store, scoped adapters, a reconciliation worker, and an operations queue. Persist each step’s intent, request digest, operation key, attempt state, remote identifier, and compensation policy before performing its external write.
Use a unique key such as tenant, employment episode, step, and approved plan version. A retry with the same key must use the same payload. A changed department or start date creates a new reviewed transition rather than mutating an in-flight request under its old identity.
The following illustrative JSON contract is an internal journal entry, not a vendor API response. “Unknown” means the network outcome is ambiguous, not that account creation failed.
{
"workflow_id": "fixture-onboarding-73",
"employment_episode": "fixture-employee-8-hire-2",
"plan_version": 4,
"step": "create_identity",
"operation_key": "tenantA:hire2:identity:plan4",
"request_digest": "fixture-approved-request-digest",
"state": "outcome_unknown",
"remote_id": null,
"next_action": "reconcile_by_external_reference",
"compensation": "disable_only_if_created_by_this_workflow"
}Production dependencies omitted include schema validation, transactional persistence, worker leases, fencing against concurrent workers, credential management, and adapter-specific reconciliation. Record state changes and queued work atomically where possible, for example with an outbox. A process-local dictionary is not a durable journal.
Do Not Promise Exactly-Once Vendor Writes
Some vendors support idempotency keys; others do not. SCIM’s protocol specification defines identity creation and modification, authentication considerations, and resource versioning. It does not mean every provider offers an application-specific idempotency guarantee. Discover supported capabilities and verify provider behavior in a sandbox.
After an account-create timeout, query using the stable external employment reference where supported. If exactly one verified match exists, attach its remote identifier and continue. Zero results may still reflect eventual consistency; use a bounded reconciliation window. Multiple matches or unresolved ambiguity require an operator, not another blind POST.
Separate retryable transport failures from rejected business data. Backoff can help a rate-limited inventory service; it cannot repair a missing payroll field. Pause that step, request the specific correction, and preserve completed steps. Specialist sub-agents earn their complexity only for distinct tasks such as resolving document discrepancies. Identity, payroll, and equipment adapters should remain deterministic tools.
Compensation Must Respect Resource Ownership
Microsoft’s Compensating Transaction pattern explains why recovery is application-specific and can itself fail. Onboarding illustrates the point: releasing a laptop reservation is different from deleting an account, canceling a shipment, or correcting a payroll record.
Only compensate resources created or changed by this workflow, using recorded before-state and current-version checks. If the employee already had an account, a failed onboarding run must not disable it simply because the account appeared in the plan. If a manager independently changes a group membership, compensation must not overwrite that later decision.
Prefer reversible preparation before irreversible effects. Reserve equipment before dispatch; create a disabled account before approved activation where supported; prepare payroll data before any payment process. Actual wages, shipped hardware, and external messages cannot be “undone” by removing a local row. Route these cases to the appropriate human owner with a documented follow-up action.
Compensation has its own pending, running, succeeded, and failed states. An operator should see “identity disabled; laptop release pending” rather than a false overall rollback success. Reconcile remote postconditions before declaring completion, and retain the failed episode for audit.
Security Applies to Every Step
Server-side authorization binds approvals to exact resources, role template, plan digest, tenant, and employment version. Use separate adapter identities with minimum scopes; the model does not need payroll credentials or raw bank details. Protect journals because even metadata can reveal sensitive employment information.
Uploaded documents and HR notes are untrusted. Instructions requesting administrator membership cannot change the approved plan. MCP is an integration protocol, not proof of HR authority or permission to create accounts. Sensitive writes require renewed approval when the employment event or resource plan changes.
HTTP semantics provide useful retry and conditional-request concepts, but business reconciliation remains necessary. A successful response may indicate acceptance rather than completion. Poll or consume authenticated status events according to the actual provider contract.
Acceptance Tests for Durable Onboarding
- Crash after identity creation but before journal acknowledgment. Recovery discovers the existing account and creates no duplicate identity.
- Reject payroll data after laptop reservation. The workflow pauses with an actionable field error; optional steps follow policy rather than triggering indiscriminate rollback.
- Cancel the employment episode during a retry delay. Subsequent activation is blocked, and approved compensation applies only to workflow-owned resources.
- Fail the equipment-release compensation. The workflow remains visibly incomplete with an operator task and retries only that safe compensating action.
- Reuse a preexisting account and change a group externally. Recovery neither disables that account nor overwrites the unrelated group change.
Also test two workers resuming the same step and a forged cross-tenant approval. Durable leases, remote idempotency where available, and authorization checks must prevent duplicate or unauthorized effects.
Measure Readiness, Not Completed Tool Calls
Baseline onboarding by employment type and location. Measure on-time readiness as starters with every required capability verified by start time divided by all scheduled starters in scope. Report cancellations and missing HR data separately rather than quietly removing difficult cases. Measure reconciliation workload as operator-intervened workflows divided by all started workflows.
Guardrails are duplicate accounts, unauthorized entitlements, and uncompensated resources after cancellation. Begin with sandbox failure injection, then one department with human approval before each write. Any readiness or time-saving target is proposed until measured. Keep a manual recovery path available throughout rollout.
The enterprise knowledge guide covers federated read-only evidence, not this write coordination problem. The access request guide handles entitlement approval. Backend and API engineering supports durable state and adapters. Start with account creation plus reconciliation and compensation before connecting payroll or equipment dispatch.