Writing Blameless Postmortems

DevOpsSREIncident ResponseCulture
Share on LinkedIn

The first postmortem I wrote named an engineer who typo'd a config. It never got shared widely. The second, rewritten blamelessly, identified that prod config wasn't validated in CI and had no diff review — we fixed that in a week. Same incident, different outcome. Blameless postmortems aren't about being nice; they're about getting honest timelines so fixes address why the system allowed failure.

Anatomy of a useful postmortem

  1. Summary — one paragraph: what broke, user impact, duration
  2. Impact — metrics: error rate, revenue, customers affected, SLA burn
  3. Timeline — UTC timestamps, detection → mitigation → resolution
  4. Root cause — technical chain, not person chain
  5. Contributing factors — why detection was slow, why rollback failed
  6. What went well — reinforce good instincts
  7. Action items — owned, prioritized, tracked
  8. Lessons learned — optional narrative for humans

Skip lengthy background unless context is non-obvious.

Timeline that tells the truth

2026-04-08 14:02 UTC — Deploy v2.4.1 completes (Git SHA abc123)
2026-04-08 14:07 UTC — PagerDuty: checkout error rate > 5%
2026-04-08 14:09 UTC — On-call acknowledges, opens war room
2026-04-08 14:15 UTC — Hypothesis: DB migration lock; ruled out
2026-04-08 14:22 UTC — Identified feature flag default change in v2.4.1
2026-04-08 14:24 UTC — Flag disabled; error rate normalizes
2026-04-08 14:30 UTC — Incident resolved; monitoring green

Link Grafana snapshots, deploy records, Slack thread exports. Future you won't remember.

Blameless language examples

Blameful Blameless
"Jane forgot to run migration" "Migration didn't run automatically; manual step undocumented"
"Ops clicked wrong button" "Runbook ambiguous between staging and prod endpoints"
"Developer shipped buggy code" "Test suite lacked case for empty cart edge condition"

Assume everyone acted rationally given information and tools available.

Action items that don't rot

Bad: "Improve monitoring" — unowned, unmeasurable.

Good:

| Action | Owner | Priority | Due | Ticket |
| Add checkout error rate SLO alert | @alex | P1 | Apr 15 | INC-442 |
| Feature flags default-off in CI test | @sam | P2 | Apr 22 | INC-443 |
| Runbook: flag kill switch steps | @platform | P2 | Apr 20 | INC-444 |

Review open postmortem actions in weekly platform meeting. Close or re-prioritize — stale actions erode trust.

Severity-based depth

Severity Postmortem required Review meeting
SEV1 — major outage Yes, 5 business days Required, cross-team
SEV2 — degraded Yes, 10 business days Team + stakeholders
SEV3 — minor Optional short form Team async
Near miss Encouraged Blameless learning

Don't skip postmortems on near misses — cheapest learning available.

Publishing and privacy

Share widely internally — security incidents redact exploit details until patched. Customer-facing status page summary separate from internal technical doc.

Google's public postmortem culture sets expectation: incidents are learning events, not career events.

Anti-patterns

Template starter

# Incident: [Title] — [Date]

## Summary
[1 paragraph]

## Impact
- Duration:
- Users affected:
- Error budget consumed:

## Timeline (UTC)
| Time | Event |
|------|-------|

## Root Cause
[Technical description]

## Contributing Factors
- 

## What Went Well
- 

## Action Items
| Item | Owner | Due | Status |
|------|-------|-----|--------|

Executive summaries

Attach one-page executive summary for SEV1 incidents: business impact in dollars, customer segments affected, confidence in root cause, top three preventive actions with dates. Executives read this; engineers read full doc.

Operational notes

Store postmortems in searchable repository linked from service catalog — new engineers onboarding to a service read past incidents first week. Institutional memory prevents repeating rollback mistakes.

Archive postmortems where new hires will find them during service onboarding.

Action item tracking

Postmortems fail when action items die in spreadsheets:

# action-items.yaml in repo — CI checks due dates
- id: PM-2026-014-01
  incident: PM-2026-014
  description: Add circuit breaker to payment gateway client
  owner: @payments-team
  due: 2026-02-15
  status: open
  verified_by: null  # link to PR or test when done

Review open items in weekly eng sync — not the full postmortem, just status. Close items with evidence (merged PR, deployed flag, new alert). "Done" without verification repeats the incident.

Severity classification

Consistent severity drives response expectations:

Level Definition Postmortem required
SEV1 Customer-facing outage, data loss Yes, within 5 business days
SEV2 Degraded service, workaround exists Yes, within 10 days
SEV3 Internal-only impact Optional short form
SEV4 Near-miss, no user impact Optional learning note

Near-miss postmortems are undervalued — the deployment that almost took down production teaches as much as the one that did.

Facilitation tips

Blameless doesn't mean unstructured. Facilitator responsibilities:

Invite customer support for user-visible incidents — they see impact engineers miss.

Pair with career running effective meetings for postmortem meeting structure.

Production checklist

Common production mistakes

Teams get postmortems blameless culture wrong in predictable ways:

Production implementations of postmortems blameless culture fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Resources

Frequently asked questions

What does blameless mean in a postmortem?

Focus on systemic causes and process gaps, not punishing individuals for mistakes. 'The engineer deployed without review' becomes 'Our deploy pipeline allowed single-approver merges to production.' People act within system constraints; fix the constraints.

Who should write the postmortem?

An incident commander or rotating scribe who participated but maintains neutrality. The on-call engineer who fixed the incident contributes timeline facts; a facilitator prevents defensive framing. Distribute writing within the team over time.

How many action items should a postmortem produce?

Three to seven prioritized, owned, dated items. Fifty unchecked boxes mean the postmortem failed. Each item maps to a ticket with an engineer owner and severity. If the same root cause repeats, previous action items weren't effective — escalate priority.

Hiring a senior Android / Flutter engineer?

I architect and ship production mobile software — Kotlin, Jetpack Compose, Flutter — for robotics, EV infrastructure, fintech, and real-time systems. Open to remote roles in Europe and the US.

Get in touch →