Capacity failures are quietly becoming the most expensive class of outage in production software, and most on-call runbooks are not written for them. On August 20, 2026, GitHub CTO Vlad Fedorov published an update on the platform's August 17 outage, a 7 hour and 47 minute event that took down github.com, authentication, GitHub Actions, the REST and GraphQL APIs, pull requests, issues, and Copilot. Buried in that update is a sentence every on-call engineer should read twice: "Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core."

That single line invalidates the first question almost every incident responder asks. If nothing changed, "what did we ship?" is a dead end, and the twenty minutes you spend scrolling the deploy log is twenty minutes of customer impact you will never get back. This post breaks down what capacity failures look like from inside an incident channel, why they run so much longer than change failures, and how to restructure your on-call practice so your team does not lose the first half hour to the wrong hypothesis.

What GitHub's August 2026 outages reveal about capacity failures

GitHub had two significant incidents in August 2026. The first, on August 6, hit GitHub Actions and ran roughly ten hours from first impact to resolution. The second, on August 17, lasted 7 hours and 47 minutes and spread across essentially the whole platform. Both were public, both were painful, and both had the same underlying shape.

Growth outran the capacity plan

According to GitHub's own writeup, the August 17 incident began when traffic reached a new peak and a critical infrastructure component in the Central US data center failed to scale with it. Capacity pressure propagated outward, authentication started failing, and dependent services fell over in sequence.

The context makes the failure legible. GitHub reports that monthly commits grew from 1.4 billion in April to 2.9 billion by August. That is not linear growth you can absorb with a quarterly capacity review. It is a doubling in four months, driven in large part by AI coding agents that commit, open pull requests, and trigger CI at machine speed rather than human speed. GitHub says it has added more than 3 million CPU cores and 120 petabytes of high speed storage, and that Azure now serves roughly 58% of platform load, up from 12% in May. Even with that investment, demand found the one component that had not been scaled.

The industry data points the other way, and that is the problem

Here is the tension worth sitting with. The Uptime Institute Annual Outage Analysis 2026 reports that configuration and change management failures are now the leading driver of network related outages, ahead of third party provider issues and hardware faults. The report also finds a sharp rise in human error tied to staff skipping established procedures.

So the industry data says change is the top cause, and your runbook is correctly optimized for that. But frequency and duration are different distributions. Change failures are common and usually short, because the mitigation is obvious: revert. Capacity failures are rarer and much longer, because there is nothing to revert and the fix requires provisioning, rerouting, or shedding load. If you rank your incidents by customer minutes lost rather than by count, the picture inverts. The long tail is where your worst quarters come from.

Why the "what changed?" playbook fails on capacity failures

Most incident response training, tooling, and muscle memory assumes a change caused the problem. That assumption is usually right and occasionally catastrophic.

The rollback reflex burns your first thirty minutes

Walk through a typical opening. Alerts fire. Someone opens the incident channel. The first three messages are some version of "who deployed?", "is there a feature flag we can flip?", and "can we roll back to the last known good?". Someone pulls up the deploy dashboard. Someone else checks the flag audit log. Both come back clean.

Now what? In a change-caused incident, that check is the fastest path to mitigation. In a capacity failure, it is a null result that tells you almost nothing, and the team has no prepared second branch. What typically follows is unstructured exploration: several engineers each pulling a different dashboard, nobody assigned to any specific hypothesis, and the incident commander summarizing the same "still investigating" three times in a row. The customer impact clock is running the whole time.

Detection was never the bottleneck

It is worth being precise about where the time actually goes. In GitHub's July 2026 availability report, the July 8 incident (7 hours and 4 minutes) was detected by automated monitoring within about 19 minutes. The July 19 DNS control plane incident (2 hours and 11 minutes) was detected within roughly one minute. Detection worked fine in both cases. Recovery is what took hours, because the correct configuration had to be restored and validated across many machines in one case, and caches had to repopulate in the other.

The same holds for capacity failures, only more so. Saturation is not subtle. Your dashboards will show it. The expensive part is deciding what to do, getting authorization to do it, and coordinating the sequence of actions across several teams while the system is actively degrading. That is a coordination problem, not an observability problem, and buying another monitoring tool will not touch it.

Recovery is its own failure mode

The most instructive detail in GitHub's August 17 writeup is what happened during recovery. Most services came back earlier in the day, but errors in Copilot services triggered a client side retry loop that increased traffic while the platform was trying to stabilize. GitHub had to mitigate that retry behavior before it could safely restore traffic at all.

This is the thundering herd problem, and it is specific to capacity incidents. When a change breaks something, reverting it returns the system to a known state. When capacity breaks something, the queued and retrying work that piled up during the outage becomes a second load spike aimed directly at the component you just brought back. Teams that plan only for the failure and not for the recovery get to have the incident twice.

GitHub's two immediate remediations speak directly to this: applying consistent retry limits, retry budgets, and variable timeouts across service to service calls, and reviewing lower priority CPU and memory alerts to find components that could fail during sudden traffic spikes. Both are worth stealing.

The signals that tell you a capacity failure is underway

If you want your team to reach the right hypothesis faster, you need to make capacity signals as prominent in triage as change signals.

Lead with saturation, not errors

Error rate tells you something is broken. Saturation tells you why. Connection pool utilization, queue depth, thread pool occupancy, disk and memory headroom on the specific component under stress: these are the metrics that separate a capacity failure from a code failure in the first two minutes.

A practical test for your current setup: when an incident opens, can a responder answer "which resource is closest to its limit right now?" in under sixty seconds, without knowing in advance which service is involved? If the answer requires knowing which dashboard to open, you do not have a triage signal. You have a debugging tool.

Audit the alerts you deliberately deprioritized

Almost every team has a category of alerts that got downgraded because they were noisy. CPU above 80%. Memory above 85%. Queue depth climbing. These fired constantly, nobody acted on them, and eventually somebody routed them to a channel nobody reads.

That was a reasonable decision under the old cost model, when those alerts were mostly false positives and the real incidents came from deploys. It is a less reasonable decision when your traffic can double in four months. GitHub explicitly listed reviewing lower priority CPU and memory alerts as a post incident action, which is a polite way of saying they had signals that could have provided lead time and were not being read.

The fix is not to page on every threshold crossing. It is to distinguish two different questions. "Is this resource above a static threshold?" is noisy and mostly useless. "Is this resource's headroom trending toward zero faster than we can provision?" is a genuinely useful, low frequency signal, and it belongs somewhere a human will see it during business hours.

Watch the components nobody owns

Capacity failures concentrate in shared infrastructure: service discovery, authentication, connection brokers, the component that bridges two systems that used to be one system. These have two properties that make them dangerous. They scale differently from the services on top of them, and they frequently have no clear owner, because they were built once and then worked for years.

GitHub's July 9 incident is a clean example. A backend data service responsible for provisioning hosted runners entered an unhealthy state, one shard serving the highest volume workload became overloaded and could not synchronize across regions, and the result was 9 hours and 18 minutes of delayed and failed job starts. The user visible symptom was Actions. The saturated resource was several layers down.

How to restructure on-call for capacity driven incidents

None of this requires a new platform. It requires a few explicit changes to how your rotation operates.

Add a second branch to your triage tree

Make it structural, not cultural. Your incident template should force a branch at the top: recent change, or no recent change. If the change check comes back clean within the first few minutes, the incident commander moves to the capacity branch rather than improvising. That branch should name specific things to check, in order, with an owner for each.

The point is not that the checklist will always be right. The point is that it removes the dead air between "nothing was deployed" and "okay, so what now?", which is where long incidents get their length.

Pre-authorize load shedding

The single largest source of delay in capacity incidents is waiting for permission. Shedding non-critical traffic, disabling an expensive feature, rate limiting a noisy client, and failing over a region are all actions that responders hesitate to take alone at 3am, because they have visible customer consequences and no obvious blast radius.

Decide in advance. Write down which degradation levers the on-call engineer may pull without escalation, and which require a second approver. Then rehearse pulling them. A lever nobody has ever used in production is not a mitigation, it is a hope.

Rehearse recovery, not just failure

Game days almost always test the break. Very few test the restore. Given how often the recovery surge causes a second incident, run at least one exercise per quarter that starts from a fully degraded state and requires the team to bring traffic back safely: throttled, staged, with the retry storm accounted for.

Track duration distribution, not just average MTTR

Average mean time to resolution is a misleading number when your incident population is bimodal. Twenty incidents of eight minutes and one incident of eight hours produce a flattering average and a terrible quarter. Report your p90 and your worst incident alongside the mean, and make sure the review focuses on the tail. Your users experienced the tail.

Cutting coordination overhead when the fix takes hours

Once you accept that capacity incidents run long, the question stops being "how do we detect faster?" and becomes "how do we keep twelve people coordinated for eight hours without losing context?" That is a fundamentally different design problem.

Why the incident channel is the actual control plane

In a short incident, tooling barely matters. One person finds the bad deploy, reverts it, and posts a summary. In a long incident, the coordination cost dominates everything. Responders hand off across time zones. Executives ask for updates. Customer support needs language they can send out. Three different people independently investigate the same theory because nobody wrote down that it was already ruled out.

This is why Slack-native on-call tooling earns its keep specifically on the incidents that hurt most. When paging, acknowledgment, escalation, status page updates, and the running timeline all live in the same channel where the humans are already talking, the context does not fragment across four tools. Nobody has to reconstruct what happened during the previous shift from a web console with a different mental model. With Pagerly, the rotation, the escalation path, the incident channel, and the action items live where the conversation already is, which is the whole point: during a multi hour incident, every context switch out of Slack is a small tax paid dozens of times.

A concrete shape for a long incident channel

Whatever tooling you use, these practices measurably shorten long incidents:

Pin a live status block at the top of the channel covering current impact, current hypothesis, current action, and who owns it. Update it every fifteen minutes. Newcomers read the pin instead of asking, which alone saves the incident commander enormous overhead.
Separate the coordination channel from the debugging channel once more than about eight people are involved. Deep technical work generates message volume that drowns out decisions.
Name a scribe as a distinct role from the incident commander. On an eight hour incident, the timeline is not a nice to have, it is what makes the handoff and the postmortem possible at all.
Record ruled out hypotheses explicitly. "Not a deploy, confirmed 14:20 UTC" written in the pin prevents the fourth person who joins from starting there.
Set a hard handoff cadence. Nobody should command an incident for more than four hours. Fatigue driven decisions during hour six are how recovery surges get triggered.
Draft external comms early, even if you do not send them. Writing the status page update forces clarity about what you actually know, and it is much harder to do well at hour seven.

The connection to on-call sustainability

There is a staffing dimension here that is easy to skip. Long incidents are the ones that burn people out. A short page at midnight is annoying. An eight hour incident that consumes an engineer's entire night, followed by a normal workday, is the thing that makes people leave. Surveys of DevOps and platform teams in 2026 consistently report chronic exhaustion among a large share of SREs and on-call engineers, with response pressure outside working hours as a named driver.

Reducing the number of multi hour incidents is therefore a retention intervention, not just a reliability one. So is having enough people in the rotation that a long incident can be handed off rather than absorbed by whoever happened to be paged first.

What to take into your next on-call review

The takeaway from GitHub's August 2026 outages is not that GitHub is unreliable. It is that a well resourced team with strong monitoring, a serious availability program, and a public commitment to reliability still lost nearly eight hours to a failure mode their playbook was not shaped around. Growth exposed a component that had not scaled, and the response took hours because there was nothing to revert.

The odds are good that your systems are also growing faster than your capacity plan assumes, particularly if AI agents have started generating traffic on your CI, your APIs, or your databases. The odds are also good that your incident response starts with "what changed?" and has no prepared second step.

Three things to do this week. Add an explicit capacity branch to your incident triage, with named checks and named owners. Go find the CPU, memory, and queue depth alerts you downgraded for noise and decide which ones deserve a trend based version that a human reads. And pick your longest incident from the last two quarters and ask, honestly, how much of that duration was diagnosis and how much was coordination overhead. If the second number is larger, the fix is not a new observability vendor. It is putting your on-call workflow where your team already communicates, and rehearsing the recovery as carefully as you rehearse the failure.