Production workflows / Guide 20
A Production Incident Response Agent That Helps Without Freelancing
Design a production incident response agent that builds evidence timelines, supports incident command, and gates remediation with approval, scope limits, and recovery.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
A hypothetical checkout service starts returning errors shortly after a deployment. The on-call engineer suspects the release, while a database dashboard shows connection saturation that began earlier. An incident assistant confidently recommends rollback. If it hides the earlier saturation or executes independently, it can waste recovery time and obscure the actual sequence.
A production incident response agent should be a disciplined investigator and scribe before becoming a remediation interface. Its job is to reduce the effort required to establish what is known, what changed, and what has already been tried. It must not become another operator making uncoordinated production changes.
Put incident command ahead of automation
Google’s SRE chapter on managing incidents describes clear roles, a live incident document, explicit handoffs, and the danger of freelancing. Apply that structure to the assistant: the incident commander owns coordination, the operations lead controls changes, and communications have an accountable owner.
The agent can maintain a timeline, collect links, and draft updates. It should distinguish observed facts, hypotheses, actions, and outcomes. “Deployment completed at 10:05” is an observation. “Deployment caused the outage” is a hypothesis until supported. “Rollback requested” is not the same event as “rollback completed and health recovered.”
Keep the incident record reachable when the affected product is down. An assistant hosted entirely inside the failing application cannot be the only source of operational state. Preserve a conventional fallback: dashboards, documented runbooks, and a communication channel that works without model availability.
Gather bounded evidence instead of dumping logs
The minimum system reads alerts, deployment history, scoped metrics, selected traces, and approved runbooks. It stores timestamped evidence references in the incident record. Queries should have time windows, result limits, and budgets so investigation does not overload a struggling database or logging service.
Record event time and ingestion time separately. A delayed log arriving after the deployment may describe a failure that happened before it. Normalize clocks carefully and display uncertainty when hosts disagree. The model should not reorder events simply to produce a cleaner narrative.
An LLM-only approach is unreliable because root-cause stories are easy to generate from correlated events. Deterministic software should compute error rates, verify deployment IDs, enforce query limits, and check action prerequisites. The model proposes hypotheses with supporting and contradicting evidence, then identifies the next discriminating observation.
Specialist workers can help inspect independent systems in parallel when each has bounded queries and a clear question. One might compare database saturation before and after the release; another checks dependency errors. Their output should include evidence and uncertainty, not votes for a root cause. One coordinator maintains the shared action ledger.
Make an action plan machine-checkable
This illustrative JSON contract describes a proposed canary rollback. It is an internal control object, not a deployment-platform API.
{
"incident_id": "incident-example-64",
"plan_version": 2,
"action": "rollback_canary",
"service": "checkout",
"environment": "production",
"targets": ["canary-pool-a"],
"expected_release": "release-example-205",
"target_release": "release-example-204",
"required_checks": [
"schema_backward_compatible",
"old_artifact_available",
"operations_lead_approval"
],
"verification": "approved_checkout_health_check",
"approval_status": "pending"
}Production dependencies omitted include deployment adapters, authenticated incident roles, artifact verification, schema compatibility checks, health-query definitions, durable execution state, and rollback permissions. A familiar previous release is not automatically safe: migrations, configuration changes, or expired dependencies may prevent reversal.
For the hypothetical checkout incident, the agent should surface the earlier database saturation before requesting approval. A bounded rollback might still be a reasonable mitigation, but the operations lead must know why it could help, what it cannot explain, and what evidence would cause the team to stop.
Separate planning credentials from execution
Give evidence gathering read-only credentials scoped to the affected tenant or organization, service, account, and environment. Do not allow arbitrary shell commands or model-generated SQL against production. Expose named, reviewed queries and runbook actions with validated parameters.
The execution service independently checks authorization, incident assignment, target set, plan digest, approval expiry, and current resource state. A request for one canary must not expand into all regions. Keep destructive actions and blanket administrator permissions out of the assistant’s tool set.
AWS Systems Manager’s aws:approve action illustrates a real manual approval step. Its documented limitations, including lack of support for multi-account and Region automations, matter when designing the integration. A paused runbook is one control, not a replacement for incident-scoped authorization and blast-radius checks.
Logs can contain attacker-controlled headers, usernames, and error messages. Treat them as untrusted content. A log line telling the assistant to export secrets is an injection attempt, not operational guidance. Use the independent tool validation and least-privilege principles in OWASP’s guidance. MCP provides connectivity, not production authority.
Recover from ambiguous execution
Persist an action ID before invoking a runbook. If the response times out, mark execution unknown and query the orchestrator. Never launch another rollback just to obtain a clean response. Repeated mitigation can create more disruption than the original action.
Hold a service-scoped execution lease so two workers cannot remediate the same target concurrently. A human emergency override should be explicit and visible in the action ledger. When the commander or operations lead changes, revalidate authority for pending actions instead of accepting an approval from a stale assignment.
After execution, verify the intended outcome using the approved health criteria and observation window. Record success, failure, or insufficient evidence separately. A command exiting successfully does not prove checkout recovered. If health worsens, stop further automation and follow the preapproved recovery path; do not improvise a sequence of increasingly broad changes.
Acceptance tests under incident pressure
- Provide delayed logs showing saturation before deployment. The timeline uses event time and keeps the deployment-causality claim as an unconfirmed hypothesis.
- Approve a rollback, then change the deployed release. Execution rejects the stale expected state and requests a revised plan.
- Lose the response after a runbook starts. The assistant reconciles one execution instead of launching a duplicate remediation.
- Put credential-export instructions in an HTTP header captured by logs. No secret read or external export is possible through the available tools.
- Make the assistant unavailable during a simulated incident. Responders retain the incident document, runbooks, and ability to operate without it.
Measure incident assistance without gaming recovery time
Baseline time to assemble a reliable evidence timeline in historical replay exercises. Measure correctly sourced timeline entries divided by all audited entries, and responder search minutes divided by replayed incidents. Evaluate comparable scenarios before and after assistance, including cases where the obvious hypothesis is wrong.
Guardrails include unsupported causal claims, unauthorized actions, duplicate executions, and added load on degraded systems. Live recovery time matters, but incident severity and architecture differ; a small sample cannot establish that the agent reduced mean time to recovery. Treat any target as proposed and report the comparison method.
Start as a read-only scribe for one service, then test staff-approved actions in exercises before production use. The cybersecurity alert triage guide covers security-specific evidence handling, while the cloud cost optimization guide applies bounded changes to non-urgent work. DevOps, Platform Engineering & SRE supports the runbooks, observability, and recovery discipline the assistant depends on.