Incident report

Write it while it’s fresh — a Mermaid sequence of the actual failure path.

Use this templateDocs template
Incident reportWrite it while it’s fresh — a Mermaid sequence of the actual failure path.

Incident <n> — <one line, no jargon>

Severity: <n> · Detected: <time> · Resolved: <time> · Author: <name>

Caution

Blast radius — <who was affected, how many, and for how long>. Write this first: it's the only part some readers get to.

What happened

Two sentences. <The trigger, and what it cascaded into.>

How it failed

sequenceDiagram
  participant D as Deploy
  participant C as Cache
  participant B as Database
  participant U as Users
  D->>C: ship config change
  C--xC: keys invalidated
  C->>B: every read falls through
  B--xU: saturated, 5xx

Timeline

TimeEvent
<time><what happened, or what someone noticed>
14:02First 5xx alert fires
14:19Config rolled back, error rate falls

Contributing factors

  • <The thing that made this possible, not the person who typed it>
  • <The thing that made it take as long as it did to notice>

What we're changing

  • <who> — <the change that makes this class of failure impossible>, due <date>
  • <who> — <the change that makes it visible in one minute instead of seventeen>, due <date>

What went well

  • <Say at least one true thing here. Postmortems people dread don't get written.>