Respond when the company website is down

When to use

The company website or web app is unreachable or broken for visitors. Not for a slow site that still answers — that is a performance task, not an outage. Not for a compromised or defaced site — stop, do not "fix" it, escalate to the owner and use tech.security-audit; restoring a compromised site can destroy evidence. Not for mail-only failures — use tech.dmarc-review or the mail runbooks. After the site is back, the follow-up run is ops.incident-postmortem.

Before you start

• The domain and the expected behaviour are known (what should answer, on which paths).
• Access exists: DNS control in Brain Club, the hosting/admin panel, and the deploy history.
• Last 24 h of changes are listed before anything is touched.

What a run requires3

The trail8 steps

1. S1

   Confirm the outage from outsideagent

   Open the site from at least two independent vantage points (public checker + mobile network, not the office Wi-Fi). Check DNS resolution (bc dns lookup of the domain) and note whether the name resolves at all.

   Done when the outage is confirmed with two observations, or refuted — if it only fails from one vantage point, record that and stop; it is not an outage.

2. S2

   Name the failing layeragent

   Work down the stack in order: DNS (does the name resolve to the expected address?) → certificate (is TLS valid, does tech.ssl-expiry-check logic apply?) → hosting (is the server up, does the provider status page report an incident?) → application (does the server answer with a 5xx or a broken page?).

   Done when one layer is named as the fault with the evidence next to it, and the other three are marked checked with their result.

3. S3

   Report to the owneragent

   Send the owner three lines: what is down since when, which layer is at fault, what the fix will be.

   Done when the owner has acknowledged and said whether customers must be told.

4. S4

   Decide on customer communicationownerneeds approval · owner

   Approval · S4 · owner — the run stops until a named person records the decision

   If the outage is visible to customers and will last longer than ~15 minutes, approve a short status note (channel and wording are the owner's call). The agent never posts it unilaterally.

   Done when a decision is recorded: note sent, or deliberately not sent, and why.

5. S5

   Apply the reversible fixagent

   One change at a time, in the layer named in S2: correct a DNS record via bc dns; renew or reissue the certificate through the hosting panel; restart or redeploy the app from the deploy history. Wait out the DNS TTL before judging a DNS fix.

   Done when the site answers 200 over HTTPS from an outside vantage point.

   ⛔ Do not change two things at once — if the second change breaks something, you no longer know which one fixed or broke it.

6. S6

   Rollback if the fix fails or the cause is a bad deployownerneeds approval · ownerirreversible

   Approval · S6 · owner — the run stops until a named person records the decision

   Irreversible · S6 — an agent never closes it alone

   Rolling back a deploy discards the changes it carried; the owner decides whether the release ships later. Restore the last known-good version from the deploy history.

   Done when the last known-good version is live and the site answers from outside.

   ⛔ A rollback past the point of a data migration can lose data — if a migration ran, stop and escalate to agents.database-migration instead of rolling back blind.

7. S7

   Verify from outside, fullyagent

   Two vantage points again: HTTPS 200 on the homepage and on a critical path (login, checkout, contact form — a homepage that answers while checkout 500s is not "restored"). Certificate valid. DNS answers consistent across resolvers. Monitoring green.

   Done when every check passes and the answers match what the zone says.

8. S8

   Record and hand overagent

   Write the incident record: timeline with timestamps, failing layer, evidence, every change made and when, the owner's S4 decision. Create the follow-up with bc tasks add linking to ops.incident-postmortem.

   Done when the record exists and the postmortem task points at it.

Checks — how we know it worked

• HTTPS 200 from two independent vantage points, on the homepage and on one critical path.
• DNS answers identical across at least two public resolvers.
• Certificate valid for the name, not expiring within the next 14 days.
• Monitoring shows the recovery (not just the human looking at a browser).
• The incident record exists with a timeline, not just "fixed it".

If it goes wrong

What each step leaves behind

1. S1the outage is confirmed with two observations, or refuted — if it only fails from one vantage point, record that and stop; it is not an outage.
2. S2one layer is named as the fault with the evidence next to it, and the other three are marked checked with their result.
3. S3the owner has acknowledged and said whether customers must be told.
4. S4a decision is recorded: note sent, or deliberately not sent, and why.
5. S5the site answers 200 over HTTPS from an outside vantage point.
6. S6the last known-good version is live and the site answers from outside.
7. S7every check passes and the answers match what the zone says.
8. S8the record exists and the postmortem task points at it.

Evidence to keep

S1 confirmation observations with timestamps · S2 layer verdict with evidence · every change made, in order, with time · the owner's S4 and S6 decisions (who, when) · S7 verification outputs · the incident record id and the postmortem task link.

How this playbook improves

After every 3 runs ask: how long from alert to restored, and which step waited longest? Was the failing layer named correctly on the first try, or did a step get wasted on the wrong layer? Did any rollback lose data or discard work that then had to be rebuilt? Did any customer notice the outage before the S4 note went out? A new version changes the step that caused the wait or the miss, and says so in its change note.