Runbooks in version control, severity matrices that decide who gets paged, and postmortems that commit their own follow-ups.
55 min 240 XPSkills: Incident response, Runbooks, Severity
Brief
At 2am, an incident is run by whoever is awake, out of whatever document they can find. If the runbook lives in a wiki last edited during the previous incident, the response is archaeology. If severity is decided by how panicked the caller sounds, every incident is the worst incident.
Incident response as code treats the response itself as an artefact: runbooks in version control, severity declared from a matrix, the on-call rota as data, and the postmortem as a commit that changes the system.
The incident will not read your runbook. The responder will: if the runbook is findable, current, and has been executed before it mattered.
The runbook is a repo
A runbook in version control gains everything code gains: review on change, blame when a step is wrong, and a render pipeline that publishes the current version to the place responders actually look. The on-call rota sits beside it as YAML, so 'who do I page' is a lookup, not a rumour. And because every incident ends in a postmortem commit, the runbook improves under the same workflow as everything else you ship.
The test of a runbook is not whether it reads well. It is whether someone who has never run it can execute it at 2am and produce the same decisions as the person who wrote it.
The postmortem is a commit. If it does not change the repo, it changed nothing.
Severity is declared, not felt
Severity inflation is the default state of an undeclared process: the person reporting the incident is having a bad day, so it is a SEV1, and the org learns to ignore the word. A severity matrix replaces adrenaline with two axes, scope of impact and data sensitivity, and a lookup that says who gets paged and how fast.
Declared severity also gives you the audit trail: why this incident paged the CISO and that one waited until morning is a question an auditor, a regulator, or your own postmortem will eventually ask.
Two axes, one lookup. Severity is a decision you can defend later.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: The 2am treasure hunt
Scenario sim+80 XP
Read the brief and answer both questions.
Incident brief
Production is down. The on-call engineer finds three runbooks: one in the team wiki (last edited 14 months ago), one in a Google Doc titled 'Runbook FINAL v2 USE THIS', and one pinned in Slack as a screenshot. They contradict each other on the rollback step. She guesses. The guess is wrong, and the outage doubles.
In the postmortem, the engineering manager says: 'The documentation failed us. Everyone should try harder to keep it updated.'
1.1Where should runbooks live so that every change is reviewed and the current version is unambiguous? (two words)
1.2What is the name for the per-system document a responder executes step by step during an incident?
Task 2: Everything is a SEV1
Scenario sim+80 XP
Read the brief and answer both questions.
Incident brief
Last quarter your org declared eleven SEV1 incidents. Reviewing them: three were single-team tooling outages, one was a typo on the marketing site. Meanwhile a genuine restricted-data exposure was declared SEV3 because the reporter 'didn't want to make a fuss', and the regulatory clock started two days late.
The VP says the answer is clear communication: 'Just tell people to be honest about severity.'
Term bank, not all of these are right
honestyseverity matrixdeclared severityescalation
2.1What tool replaces gut-feel severity with a lookup over impact scope and data sensitivity? (two words)
2.2What is the property called when severity comes from a defined table rather than the reporter's confidence? (two words)
Task 3: The postmortem that changed nothing
Scenario sim+80 XP
Read the brief and answer both questions.
Incident brief
The postmortem for the breach-response incident is excellent: forty pages, honest timeline, root cause nailed. It ends with 'lessons learned' and a list of ten recommendations. Six months later, the same failure recurs. The recommendations were never assigned; the document was never linked from the runbook; the 'blameless culture' was cited as the reason nobody owned the follow-ups.
'Blameless means we don't point fingers,' the director explains, 'so we deliberately kept recommendations impersonal.'
Term bank, not all of these are right
impersonal recommendationsaction itemblamelessroot cause
3.1What should every postmortem recommendation become, with an owner and a due date? (two words)
3.2What is the name for the culture that examines failure without punishing the people involved, which this company is using as an excuse?
Labs
90 bonus XP
Hands-on work. You edit the artefact, the server runs the checks: the expectations and the reference solution never reach your browser.
Three runbooks contradict each other, severity is decided by whoever sounds most alarmed, and the last postmortem's recommendations were 'kept impersonal', so nobody owned them. The on-call rota is a screenshot in the wiki.
Write the specification for response-as-code: versioned runbooks rendered to the incident channel, a declared severity matrix, the rota as data, and postmortems that land as tracked changes.
Current state (from the postmortem)
runbooks: wiki + google docs + slack screenshots (3 versions, contradictory)
severity: decided on the call, "based on impact"
on-call: wiki screenshot of a rota, updated manually
postmortems: 40-page docs; recommendations "impersonal", unowned
regulatory_clock: started 2 days late on the last restricted-data event
1
2
3
4
5
6
7
8
9
10
11
12
13
Checks
Runbooks live in version control with review
Current runbook is rendered where responders look
Severity comes from a declared matrix on two axes
On-call rota is machine-readable data
Postmortem recommendations become owned action items
The postmortem updates the runbook it exercised
Severity that triggers a regulatory clock is declared, not felt
Eight lines came out of the quarterly incident review. The responder rotation wants the systemic failures ranked for the next engineering cycle, not the one-off bad days.
Separate the failures of the system from the failures of the week.
List only the lines that qualify, worst first. Severity is one of: critical | high | medium | low. Add or delete rows as needed.
incident review notes
I-1 restricted customer data exposed; severity felt 'low-key' on the call, so the regulatory clock started two days late
I-2 vendor breach notice sat in an unmonitored inbox for three days before anyone read it
I-3 two runbooks contradicted each other on rollback; the wrong guess doubled the outage
I-4 eleven SEV1s last quarter, three of them single-team tooling outages and one a marketing typo
I-5 a new engineer asked excellent questions during the SEV2 bridge
I-6 the incident wiki got record page views this quarter
I-7 the facilitator called the last exercise 'the most engaged session yet'
I-8 the coffee machine on floor four was broken during the SEV1
1
2
3
4
5
6
7
8
9
10
11
12
Checks
The right findings are raised
Benign lines are left out
Severity matches the actual exposure
Ranked worst first
Next on your path · AI & Advanced
Tabletop to Pipeline
An exercise finding without an owner never ships. Turn tabletop outcomes into tracked work that lands as controls, alerts and runbook edits.
