Most engineering teams have on-call runbooks. Very few have on-call runbooks that anyone opens during an actual incident. The document exists, it was written with good intentions during a quiet sprint, it is linked from a wiki page called "Operations", and when the pager fires at 3 in the morning the responder ignores it completely and starts guessing in a terminal instead. That gap between the runbook that exists and the runbook that gets used is one of the cheapest reliability problems to fix, and one of the most consistently neglected.

The failure is rarely effort. Teams write a lot of documentation. The failure is that runbooks are usually written as explanations when what a responder needs is a decision aid. Explanations are for daylight. At 3am, under adrenaline, with a customer escalation building in a Slack channel, a responder needs to know three things fast: what is broken, what is the safest thing to do about it right now, and who to wake up if that does not work. Everything else is furniture.

This post is about how to write runbooks that survive contact with a real incident, how to keep them from rotting, and how to place them so the responder finds them without looking. It is opinionated in places, because vague advice about documentation is exactly how teams ended up here.

Why Most On-Call Runbooks Go Unread

Before fixing the format, it helps to be honest about the specific ways runbooks fail. In practice there are three, and they compound.

They were written for the author, not the responder

The person who writes a runbook is almost always the person who knows the system best. They write from a position of context they cannot see themselves holding. They say things like "restart the consumer group" without saying which consumer group, on which cluster, with which command, or what happens to in flight messages when you do. To the author this is obvious. To the generalist who is on call this week and has never touched that service, it is a dead end.

A good test: hand the runbook to an engineer who has been at the company for six weeks and ask them to walk through it out loud. Every place they hesitate is a defect. Not a nitpick, a defect. That hesitation will cost real minutes during a real outage.

They rot silently

Runbooks describe systems, and systems change constantly. A service gets renamed, a dashboard moves, a script is deprecated, a database is migrated to a new cluster. None of these changes trigger a documentation update, because nothing in the deploy pipeline knows the runbook exists. Six months later the runbook is a set of confident instructions pointing at things that are no longer there.

This is worse than having no runbook at all. A missing runbook makes the responder careful. A wrong runbook makes them confident in the wrong direction, and they lose ten minutes discovering that the command they were told to run does not exist anymore.

They live where nobody looks at 3am

The average runbook is four clicks and one search away from the alert that should have surfaced it. The responder gets a page on their phone, opens Slack, sees an alert payload with a metric name and a threshold, and has no link. So they search the wiki for the service name, find three pages with similar titles, two of which are drafts from a reorg, and by the time they find the right one they have already started debugging by instinct. The runbook loses to the terminal every time because the terminal is closer.

What a Runbook Is Actually For

It is worth being precise about the job, because the job determines the format.

A runbook is not a system design document. It is not a training guide. It is not a place to explain the architecture, justify past decisions, or teach someone how Kafka works. Those are all valuable documents and they should exist somewhere else. A runbook has exactly one purpose: to compress the time between "an alert fired" and "the right mitigation is in progress."

The runbook is a decision aid, not a manual

Framing it as a decision aid changes what you write. A manual tries to be complete. A decision aid tries to be fast and to fail safe. It accepts that it will not cover every case, and instead makes sure that the common cases are handled in seconds and the uncommon cases route quickly to a human who knows more.

The best runbooks read like a checklist crossed with a flowchart. They are boring, specific, and slightly repetitive. That is the correct aesthetic. Elegance in a runbook is a warning sign that someone was writing to impress rather than to be used.

A Runbook Structure That Survives Contact With an Incident

Here is a structure that consistently works. It is deliberately rigid, because consistency across runbooks means a responder does not have to reorient every time they open a new one. When every runbook has the same seven sections in the same order, muscle memory does half the work.

1. The trigger line

One sentence at the very top stating exactly which alert or symptom brings someone here. Not a category, the actual alert name. If the runbook covers three alerts, list all three. A responder should be able to confirm in two seconds that they are in the right document. Getting this wrong is expensive: a responder who follows the wrong runbook does not just waste time, they may apply a mitigation that makes things worse.

2. Blast radius and user impact

What is actually broken for customers when this fires, and what is not. This section governs urgency. A responder needs to know quickly whether this is "checkout is down for everyone" or "the nightly report will be late." Those deserve very different responses at 3am, and without this section every alert feels equally urgent, which is how alert fatigue starts.

Be specific about who is affected. "Some users may see errors" is useless. "Users in the EU region cannot complete password resets, roughly 4 percent of daily active sessions" gives the responder something to act on and something to tell stakeholders.

3. First five minutes

A short numbered list of the first things to do, in order, before any diagnosis. Usually this is: acknowledge the page, open the incident channel, check the two dashboards that matter, and post a one line status. Nothing clever. The value here is that it removes decision making from the most panicked part of the incident and gives the responder momentum.

4. Diagnosis tree

This is the heart of the runbook and the part most often written badly. Do not write prose. Write conditionals. "If queue depth is rising and consumer lag is flat, the consumers are stuck: go to section 5a. If both are rising, this is upstream load: go to 5b. If neither, this is probably a false alarm from the metrics pipeline: verify with X, then resolve."

Each branch should name the exact dashboard, query, or command that distinguishes it. Include the command verbatim, in a form that can be copied and pasted. The responder should never have to reconstruct a query from a description of a query.

5. Mitigations, ranked by reversibility

List mitigations from safest to most drastic, and label each with what it costs. Restarting a stateless service is cheap and reversible. Failing over a database is not. Scaling up costs money. Rolling back a deploy may lose a feature that another team just launched.

Ranking by reversibility rather than by likely effectiveness is the important choice. Under pressure, people reach for the thing they believe will work, and sometimes that is a one way door. A runbook that puts the reversible options first nudges responders toward decisions they can walk back if they were wrong about the cause, which at minute four of an incident they very often are.

6. Escalation path with names and hours

Not "escalate to the platform team." That is not an escalation path, that is a hope. The runbook should name who owns this system, how to reach the current on-call for that team, what the expected response time is, and what to do if nobody answers within it. If the escalation depends on time of day or region, say so explicitly.

The single most common failure mode in a bad night is a responder who knows they are out of depth but does not know who to wake, and spends fifteen minutes being polite in a channel instead of paging someone. Make paging the next person a documented, explicitly sanctioned step, so nobody has to decide whether it is socially acceptable at 3am. It is.

7. Verification and exit criteria

How do you know it is actually fixed. This section prevents two opposite mistakes: declaring victory while the error rate is still elevated, and staying up for another hour watching a graph that recovered twenty minutes ago. State the specific metric, the specific threshold, and the specific duration it must hold. Then state what to do next: what to note for the postmortem, what to hand over, what to leave monitoring.

The essentials to carry away from the structure:

Put the trigger alert name at the very top so responders can confirm they are in the right place in two seconds.
State customer impact explicitly, because impact governs urgency and prevents every alert from feeling like a crisis.
Write the diagnosis section as conditionals with copy pasteable commands, never as prose.
Rank mitigations by reversibility, not by how likely they are to work.
Name the human to escalate to, with hours and expected response time, and make paging them an explicitly approved step.
Define exit criteria as a metric, a threshold, and a duration, so nobody guesses when it is over.

Writing the Runbook: Practical Rules

Structure is half the problem. The other half is line by line writing discipline, which is where most runbooks quietly degrade.

Write commands, not descriptions

"Check the consumer lag" is a description. It requires the responder to know where lag is displayed, what tool shows it, and what number is bad. The command, the dashboard link, and the threshold are the actual content. If a step cannot be reduced to something copy pasteable or clickable, it usually means the step is underspecified and needs to be broken down further.

One page, or split it

If a runbook does not fit on roughly one screen plus a scroll, it is covering more than one failure mode and should be split. Long runbooks are a symptom of trying to be complete. Completeness is the enemy here. It is much better to have six short, sharp runbooks than one comprehensive document that nobody reads past the third heading.

Name the human, not just the team

Teams are abstractions and abstractions do not answer phones. Where possible the runbook should resolve to whoever is currently on call for the owning team, dynamically, rather than to a static name that goes stale the moment someone changes roles. This is one of the places where tooling genuinely helps: a runbook that says "page the current on-call for payments" is only useful if there is a reliable, low friction way to do exactly that from wherever the responder is standing.

Say what not to do

Underrated section. Every mature system has a few actions that look reasonable and are actually catastrophic. Do not restart this service during a backfill. Do not scale this deployment past this number or you will exhaust the connection pool. Do not clear this cache during business hours. These warnings are usually held only in the heads of two or three senior engineers, and they are exactly the knowledge that disappears when those engineers are asleep or on vacation. Write them down, in bold, next to the action they apply to.

Keeping Runbooks Alive

A runbook that is not maintained becomes a liability within about two quarters. Maintenance cannot depend on goodwill, because goodwill loses to sprint deadlines. It needs to be structurally attached to things that already happen.

Attach the runbook to the alert

This is the highest leverage change available, and it is usually a config field. Nearly every monitoring and alerting system supports attaching a runbook URL to an alert definition, and nearly every team leaves it empty. When the link travels with the page, the responder does not search, and the runbook actually gets read. As a bonus, a broken link becomes immediately visible to whoever is on call rather than staying quietly wrong for a year.

Make it a rule: no alert ships to production without a runbook link. An alert without a documented response is not really an alert, it is just a notification that something is unusual, and those belong in a dashboard rather than on a pager.

Review during the postmortem, not the quarter

Quarterly documentation reviews do not work. Nobody wants to do them and the person doing them lacks the context to know what is wrong. The moment of maximum context is immediately after an incident, when someone has just discovered which parts of the runbook were wrong the hard way.

Add a standing question to the postmortem template: did the runbook help, and what specific line in it needs to change. Then make that edit an action item with an owner, exactly like any other remediation. Ten minutes of editing while the memory is fresh is worth more than a whole afternoon of scheduled review.

Track whether runbooks get used

If your documentation platform gives you page views, look at them next to your alert volume. A runbook for an alert that fires weekly and has eleven views in a year is telling you something clear: either the alert is being silently ignored, or the runbook is useless. Both are worth knowing. Views are a crude metric, but crude beats absent.

Delete aggressively

Every stale runbook makes the useful ones harder to find and lowers trust in the whole collection. If a runbook refers to a system that no longer exists, delete it the same day you notice. If nobody can say whether a runbook is current, it is not current. A small set of trusted documents beats a large set of uncertain ones, and the uncertainty is contagious: once a responder has been burned by one out of date runbook, they start treating all of them as suspect.

Runbooks and Slack: Closing the Last Few Feet

There is a practical reason so much incident response has moved into chat, and it is not fashion. During an incident the responder is already in Slack: that is where the alert landed, where the incident channel is, where the stakeholders are asking questions, and where the handover will happen. Every context switch out of that surface costs attention that is already scarce.

So the distance between the alert and the runbook, and between the runbook and the next human, is worth minimizing deliberately. In practice this means the alert arrives in the incident channel with the runbook link already attached, the runbook names an escalation target that resolves to whoever is genuinely on call right now rather than a stale name, and paging that person is a single action from the same channel rather than a trip through another tool with another login.

This is the design philosophy behind a Slack-native on-call tool like Pagerly: keep the schedule, the current on-call resolution, the escalation, and the incident conversation in the place where the work is already happening, so a runbook step that says "page the payments on-call" is something the responder can do in one motion instead of four. The runbook is only as good as the actions it points to, and actions that require leaving the incident to perform tend not to get performed.

A 30 Day Plan to Fix Your Runbooks

If your runbooks are in poor shape, a full rewrite is not realistic and not necessary. Sequence the work by pain instead.

Week 1: List the ten alerts that paged most often in the last quarter. That list, not your service catalog, defines the scope of the work that matters.
Week 2: For each of those ten, confirm whether a runbook exists and whether it is accurate. Most teams find that two or three are missing entirely and half are stale.
Week 3: Rewrite those ten into the seven section structure above. Ask someone who did not write them to walk each one aloud, and fix every point of hesitation.
Week 4: Attach each runbook URL to its alert definition, and add the runbook question to your postmortem template so the maintenance loop closes itself from now on.

Ten good runbooks covering your noisiest alerts will do more for your response times than eighty mediocre ones covering everything. Coverage is the wrong goal. Reliability of the documents you actually reach for is the right one.

Common Mistakes Worth Naming

A few patterns show up often enough to call out directly. Runbooks written as postmortem summaries, describing what happened once rather than what to do now. Runbooks that open with three paragraphs of architecture context before the first actionable step, which is exactly backwards for a document read under stress. Runbooks that link to other runbooks in a chain, so following them requires holding a stack in your head at the worst possible moment.

And the most common of all: runbooks that assume the responder is the person who wrote them. Almost every problem in this post reduces to that assumption. The reader is tired, unfamiliar with the system, and working against a clock. Write for that person specifically, and most of the other rules follow on their own.

The Takeaway

Good on-call runbooks are not a documentation exercise, they are a latency optimization on human decision making. The measure of one is not how thorough it is but how many minutes it removes between the page firing and the right mitigation starting. That reframing tends to make the writing decisions obvious: cut the context, name the commands, rank the mitigations by how easily you can undo them, and always, always name the next human.

The maintenance question answers itself the same way. Attach the runbook to the alert so it gets found, and review it in the postmortem so it gets corrected, and the document stays alive without anyone scheduling a review meeting about it. Do those two things and most of the rot problem disappears.

Start with your ten noisiest alerts this week. If even half of them get a sharp, current, one page runbook with a working escalation path, the next person woken up at 3am will have a materially better night, and so will everyone they would otherwise have had to wake.