The GitHub outage postmortem published on August 19 is one of the most useful incident writeups of the year, because the failure was not exotic. A routine traffic peak met an autoscaling policy that was watching the wrong metric, and the resulting gap was amplified roughly tenfold by clients retrying too aggressively. Impact ran from 13:28 to 21:15 UTC on August 17, a total of 7 hours and 47 minutes. Almost none of that time was spent on the original trigger. It was spent on the retry storm the trigger set off. If you run anything with a client library and an autoscaler, the same shape is probably sitting in your stack right now.

What GitHub's Postmortem Actually Says

Worth stating up front that GitHub disclosed a lot here, including the embarrassing parts. That is unusual and it deserves credit. It also means the rest of us get a rare look at how a very well run platform fails.

The timeline, 13:28 to 21:15 UTC

Customer impact began at 13:28 UTC. GitHub's first public status update landed at 13:40 UTC, twelve minutes later. At peak, roughly 20 percent of web and API requests failed, with archive and raw content downloads failing at closer to 50 percent.

The affected surface was wide: Issues, Pull Requests, the REST and GraphQL APIs, Actions, Webhooks, Git Operations, Pages, and Copilot. The authentication path went too, which is the detail most people missed. SAML and OIDC authentication, SCIM provisioning, and Team Sync all saw delays and failures, so organization user management broke alongside everything else. Actions workflows on GitHub Enterprise Cloud with data residency were also hit, because they resolve public workflow step definitions hosted on GitHub.com. A dedicated environment does not save you when the reference source is down.

Recovery was staggered. Most services came back by 16:36 UTC. Actions stayed degraded until roughly 18:03 UTC. The Copilot Token Service did not fully recover until 21:02 UTC, four and a half hours after the first broad recovery. That long tail is the retry storm, and we will come back to it.

The trigger: an autoscaling blind spot

The immediate cause was network saturation at load balancers in the Central US data center, set off by a new traffic peak. The interesting part is why capacity did not grow to meet it.

Istio sidecar pods hit their concurrency limits. The autoscaling policy was monitoring the host service, not the sidecar. So from the autoscaler's point of view, everything looked fine. The host service was not the bottleneck. The proxy sitting next to it was, and nothing was watching that number.

This is a specific and very common class of bug: the thing that saturates first is not the thing you scale on. Service mesh sidecars are a textbook case because they are injected automatically, they are frequently invisible in the service owner's dashboards, and their limits are set by a platform team that may not know the traffic profile of every workload they inject into.

The cascade: four HAProxy nodes and the auth path

Once the sidecars were saturated, the failure spread outward. Flow limits were exhausted on four HAProxy nodes. Delays and failures propagated through the gateway's authentication path, which is why SAML, OIDC, and SCIM went down alongside the API.

Then, in GitHub's own framing, overly optimistic retry logic overloaded internal load balancers and made everything worse. The retries were not a side effect of the incident. They were a load-bearing part of it.

The resolution detail is striking: once all four HAProxy nodes were paused simultaneously, broad service recovery followed immediately. That is the signature of a metastable failure, where the system cannot recover under its own load even after the original trigger is gone, and only a hard interruption of the traffic breaks the loop.

The Retry Storm That Turned Minutes Into Hours

The single most transferable lesson in this GitHub outage postmortem is about retries, so it is worth sitting with the numbers.

Tenfold amplification on the Copilot Token Service

The Copilot Token Service normally handles somewhere between 7,000 and 9,000 requests per second. During the incident it saw 70,000 to 100,000 requests per second. GitHub characterized this as roughly a tenfold amplification.

Nothing about user demand changed by 10x. Developers did not suddenly want ten times more Copilot completions at 2pm UTC on a Monday. The traffic was manufactured by the system itself: failed token operations generated additional requests and retry loops, and each failure produced more load, which produced more failures.

Think about what that means for capacity planning. You could have provisioned 3x headroom, a genuinely generous margin that would sail through most capacity reviews, and it would have made no difference at all. You cannot buy your way out of a 10x self-inflicted amplification. The only fix is to stop generating the traffic.

Why the client was part of the outage

GitHub cited a potential retry bug in VS Code, triggered by delayed responses from a single internal endpoint, as a factor that increased traffic and delayed recovery.

This deserves emphasis because it breaks a boundary most teams treat as fixed. The outage was not contained to GitHub's infrastructure. Millions of installed VS Code clients, running on other people's laptops and entirely outside GitHub's control, became an active participant in prolonging it. GitHub could not deploy a fix to them. They could only defend against them.

Your client library is part of your production system. If you ship an SDK, a CLI, a mobile app, or a desktop editor extension, its retry behavior is your retry behavior, and it is the part you cannot hotfix. A bad backoff policy shipped to a million installs is a latent outage amplifier with a multi-month remediation timeline, because you have to wait for people to upgrade.

Metastable failure, explained plainly

A metastable failure is one where the system has two stable states: working, and stuck. A trigger pushes it from the first into the second, and then removing the trigger does not bring it back, because the stuck state sustains itself.

The mechanism is almost always a feedback loop involving retries. Requests slow down. Clients time out and retry. Retries add load. Added load slows things further. More timeouts, more retries. The system is now generating enough of its own traffic to stay saturated indefinitely, even at zero organic demand.

This is why GitHub had to pause all four HAProxy nodes at once, and why recovery was immediate afterward. You cannot ease out of a metastable state gradually, because partial capacity gets instantly consumed by the backlog of retrying clients. You have to break the loop, then let traffic back in deliberately.

GitHub's mitigations follow exactly this logic. They temporarily reduced the gateway's retry logic via a pull request. They had the load balancer block incoming token requests to the Copilot Token Service with 403 responses, then restored traffic gradually on a per-site basis. Deliberately returning errors to your own users, faster, is the correct move when the alternative is a queue that never drains.

Reading This Postmortem as an On-Call Engineer

Here is how to convert someone else's bad day into concrete work on your own systems.

Your autoscaling probably has the same blind spot

Go look at what your autoscaling policies actually key on. In most organizations the answer is CPU, sometimes memory, occasionally request rate. Then ask what else sits in the request path that has its own independent limit:

Service mesh sidecars with their own concurrency and connection ceilings, injected by a platform team.
Connection pools to databases and caches, sized in config and usually not scaled with replica count.
Thread and worker pools in the application runtime, frequently left at a default nobody has revisited.
File descriptor and ephemeral port limits on the host.
Sidecar log shippers and agents competing for the same CPU you are scaling on.
Downstream rate limits that do not move when you add capacity, so scaling up just fails faster.

For each one, the question is simple: if this saturates, will my autoscaler notice? If the answer is no, you have GitHub's bug. The fix is to emit that saturation as a metric and either scale on it or alert on it.

Audit your retry budgets before you need them

Most retry configuration is written once, early, by someone optimizing for a flaky dependency on a good day. It is almost never revisited with a total outage in mind. Ask these questions about every client you own:

Is there exponential backoff, and is it actually exponential? A fixed 100ms retry three times is not backoff, it is a burst.
Is there jitter? Without it, every client that failed at the same moment retries at the same moment, forever, in waves.
Is there a retry budget? A cap on the fraction of total requests that may be retries, typically 10 to 20 percent, is the single most effective guard against amplification.
Do retries stack across layers? Three retries in the SDK, times three in the gateway, times three in the service mesh, is 27 requests for one logical call. Layered retries multiply.
Does the circuit breaker actually open? Test it under sustained failure, not just a blip.
Do you retry non-idempotent operations? Retrying a write without an idempotency key is a correctness bug hiding inside a reliability feature.

The layered retry question catches people out most often. Do the multiplication for your own stack. The number is usually much larger than anyone expects.

The twelve minute detection gap

Impact started at 13:28. The first status update went out at 13:40. Twelve minutes is genuinely good for a platform of GitHub's size, and it is not a criticism. But it is a useful benchmark, because it is the number your own customers experience against you.

During those twelve minutes, every team depending on GitHub was in the worst phase of a vendor incident: something is broken, the status page is green, and you do not know whether it is you. Teams with dependency-level monitoring knew inside a minute. Teams relying on the status page waited twelve, then longer, because seeing a status update requires someone to be looking at it.

That gap is where the cost of a vendor outage is decided, and it is entirely within your control.

What GitHub Committed To, and What Is Still Missing

The remediation list

GitHub's stated follow-ups map cleanly onto the failure chain: revise autoscaling configuration to account for sidecar concurrency capacity, audit Istio request count, concurrency, and scaling limits, review retry limits and backoff settings for both gateways and clients, address the amplification behavior in VS Code, and improve load balancer capacity monitoring and regional failover protection.

That is a good list. It addresses the trigger, the amplifier, and the detection gap separately, which is the mark of a postmortem that actually understood its own incident rather than stopping at the first cause it found.

What remains undisclosed

In fairness to readers, several things were not published. GitHub did not disclose the capacity or flow limits of the HAProxy nodes, the specific Istio sidecar concurrency limit, or the misconfigured value. The VS Code fix, its affected versions, and its release timeline remain unstated. Multiple scraping attacks against the codeload endpoint were mentioned as complicating recovery, but GitHub did not identify them as a direct cause and did not break down what share of the traffic peak came from normal usage, retries, and scraping respectively.

That last omission matters more than it might seem. Without the breakdown, it is impossible to say whether this was fundamentally a capacity planning failure or fundamentally a retry design failure. GitHub's own remediation list suggests they believe it was mostly the latter, and the 10x amplification number supports that, but the data to confirm it is not public.

Also worth noting: the traffic redirect from Central US to North Virginia was explicitly partial, covering only a portion of the failing traffic, not a full regional failover. Regional failover protection appearing on the remediation list suggests GitHub sees room to improve there too.

A Retry Design Checklist You Can Run This Week

None of this is a quarter-long project. Most of it is an afternoon of reading configuration files.

Map every retry layer in one request path and multiply the counts. Write the number down. If it is above 5, you have work to do.
Add jitter everywhere you have backoff without it. This is usually a one-line change with an outsized payoff.
Set a retry budget at the gateway, capped as a percentage of total request volume, so amplification has a hard ceiling regardless of client behavior.
Emit saturation metrics for sidecars and pools, not just CPU, and put them on the same dashboard as your scaling metrics.
Add a retry rate alert. It moves earlier than error rate, because clients absorb the first wave before users see it.
Build a load shedding switch you can flip deliberately: a way to return fast 429s or 403s at the edge for a specific service. Test that it works before you need it at 3am.
Document the gradual restore. After shedding, how do you let traffic back in without immediately re-saturating? Per-site, per-region, or percentage ramp, decided in advance.
Review your shipped clients for retry behavior, and treat a bad backoff policy in a released SDK as a production incident waiting to happen.

Running the Drill on Your Own Stack

Reading a postmortem feels productive but changes nothing by itself. The version that actually works is a one hour exercise with the people who carry the pager.

Pick your most heavily used internal service. Ask the room: if this service started returning responses 10 seconds slower than normal, but did not fail outright, what happens? Trace it out loud. Which clients time out first? What do they do next? How many times? Does anything upstream of them also retry? What does total inbound request volume look like after two minutes of that?

Most teams discover two things in that hour. First, nobody in the room knows all the retry settings, because they live in four different config systems owned by three different teams. Second, the multiplication is worse than anyone guessed. Both discoveries are the point.

Then ask the follow up: if this happened right now, who would notice, how, and how fast? If the honest answer involves a customer support ticket, you have found the more urgent problem.

The Takeaway for Your Next Incident

The GitHub outage postmortem is not a story about a company that got sloppy. It is a story about a well engineered system where a metric was measured one layer away from where saturation actually happened, and where clients behaved reasonably by their own local logic while behaving catastrophically in aggregate. Both of those conditions are extremely easy to reproduce accidentally.

The practical summary is short. Scale on the thing that saturates, not the thing that is convenient to measure. Cap your retries with a budget, because backoff alone does not prevent amplification. Treat your shipped clients as production infrastructure you cannot hotfix. And build the ability to shed load deliberately, because in a metastable failure, returning errors quickly is what lets you recover at all.

The teams that got through August 17 with the least damage were not the ones with the cleverest architecture. They were the ones who knew within two minutes that GitHub was the problem, communicated it clearly, and stopped their own automation from hammering a service that was already on the floor. That is a low bar, it is achievable this quarter, and it is worth more than most reliability projects on your roadmap.