Respond to a production incident

When to use

An alert fired or a person reports the production service is broken now. Not for a slow-burn degradation with no user impact — open a task and let the next working session pick it up. Not for the analysis afterwards — that is ops.incident-postmortem. If the cause is a bad release you already know about, this playbook and agents.ship-release (rollback) run together. If customers must be told publicly, management.crisis-communication takes over the messaging.

Before you start

• The on-call owner is named and reachable; the agent works, the owner decides.
• You know the current release version of the affected service and when it last changed.
• Access exists: dashboards, logs, and the infra module for the affected tenant.

What a run requires3

The trail8 steps

1. S1

   Acknowledge and start the clockagent

   Mark the alert acknowledged in the monitoring page and open an incident record with the start time.

   Done when the alert shows "acknowledged" and the incident record exists with a timestamp.

2. S2

   Scope the blast radiusagent

   Check the service map and dashboards: which tenants, which endpoints, since when, is it total or partial? Compare against the last change (release, migration, config, certificate expiry).

   Done when the incident record states affected services, tenants, and start time — or "unknown, investigating".

   ⛔ Do not touch anything yet. A change made during scoping destroys the evidence.

3. S3

   Mitigate with reversible steps onlyagent

   In order of preference: roll back the last release, disable the offending feature flag, scale up capacity, reroute traffic. Each step is one change, then re-check the dashboard.

   Done when the error rate or the failing check visibly improves, or every reversible option is exhausted.

4. S4

   Verify from outsideagent

   Probe the affected endpoints from outside the network (public checker or second location), not from the server itself.

   Done when the outside probe returns correct responses for every affected endpoint.

   ⛔ A green check from inside the network proves nothing about what users see.

5. S5

   Decide on customer communicationownerneeds approval · owner

   Approval · S5 · owner — the run stops until a named person records the decision

   Present impact, expected duration, and a draft status-page note. The owner approves, edits, or declines.

   Done when the status page is updated or the owner has recorded "no public note".

   ⛔ The agent never posts to a status page or mails customers without this approval.

6. S6

   Approve any irreversible stepownerneeds approval · ownerirreversible

   Approval · S6 · owner — the run stops until a named person records the decision

   Irreversible · S6 — an agent never closes it alone

   If mitigation needs something irreversible — deleting corrupted data, forcing a schema change, dropping a queue — present the exact command, the data at risk, and the alternative. Execute only on approval.

   Done when the irreversible action is recorded in the incident record with the approver's name, or skipped.

7. S7

   Confirm steady stateagent

   Dashboards normal for 15 consecutive minutes; the outside probe passes three times in a row; no new alerts.

   Done when the incident record shows the recovery window and the monitoring page is green.

8. S8

   Record and hand overagent

   Close the incident record with the full timeline: alert time, each action with its time, the cause as currently understood, open questions. Create the postmortem task (ops.incident-postmortem) and notify the owner with a three-line summary: what broke, what fixed it, what is still unknown.

   Done when the task exists and the owner has the summary.

Checks — how we know it worked

• The outside probe answers correctly for every endpoint named in S2 — not only the one that alerted.
• Every action in the timeline has a timestamp and an actor (agent or named person).
• No irreversible step exists in the record without an approval name next to it.
• The postmortem task is open and linked to the incident record.

If it goes wrong

What each step leaves behind

1. S1the alert shows "acknowledged" and the incident record exists with a timestamp.
2. S2the incident record states affected services, tenants, and start time — or "unknown, investigating".
3. S3the error rate or the failing check visibly improves, or every reversible option is exhausted.
4. S4the outside probe returns correct responses for every affected endpoint.
5. S5the status page is updated or the owner has recorded "no public note".
6. S6the irreversible action is recorded in the incident record with the approver's name, or skipped.
7. S7the incident record shows the recovery window and the monitoring page is green.
8. S8the task exists and the owner has the summary.

Evidence to keep

Incident record id · alert text and firing time · dashboard screenshots at scoping and at recovery · each mitigation command or action with its timestamp · the outside-probe results · the owner's S5 and S6 decisions (who, when) · the final timeline.

How this playbook improves

After every 5 runs ask: minutes from alert to acknowledgement and to first mitigation — which step waited? Did any mitigation get reverted (that step needs a pre-check)? Was any irreversible action taken without S6 approval, or any S5 communication sent late? Did the outside probe in S4 pass while users still saw the outage? A new version changes the step that caused the wait or the miss, and says so in its change note.