Third-Party Outage Response Lessons From August 2026
Third-party outage response is now core on-call work. Learn to detect, triage, and communicate when GitHub, AWS, or Cloudflare goes down.

Third-party outage response has quietly become one of the most important skills on any on-call rotation, and the middle two weeks of August 2026 made the case better than any conference talk could. GitHub went down worldwide on August 17. AWS had another connectivity problem in us-west-2. Cloudflare logged more than a dozen incidents in eight days across R2, Workers KV, and Durable Objects. Slack had a lingering issue with historical message access. None of these were your code, and all of them were your problem. This post breaks down what actually happened, why vendor failures now dominate incident volume for most teams, and how to build a practical playbook so the next one costs you an hour instead of a day.
What Actually Broke During the Middle of August 2026
It is worth being specific, because the pattern only becomes obvious when you line the incidents up next to each other. Individually each of these looks like bad luck. Together they look like the shape of modern software.
GitHub went down worldwide on August 17
Customer impact began at 13:28 UTC and ran until 21:15 UTC, a total of 7 hours and 47 minutes. GitHub posted its first status update at 13:40 UTC, twelve minutes after impact started. The blast radius covered the website, the REST and GraphQL APIs, Actions, Webhooks, Issues, Pull Requests, and Copilot, plus SAML and OIDC authentication, SCIM, and Team Sync. Error rates ran around 20 percent across web and API traffic, and closer to 50 percent on archive downloads and raw repository content. Most services recovered by 16:36 UTC, Actions stayed degraded until roughly 18:03 UTC, and the Copilot Token Service did not fully recover until 21:02 UTC. Reporting from DevOps.com and others tracked the disruption as it unfolded.
Two details matter for on-call teams. First, GitHub published a detailed root cause analysis on August 19, and it is worth reading. Istio sidecar pods hit their concurrency limits while the autoscaling policy watched only the host service, so capacity never grew when it needed to. That cascaded into flow limit exhaustion across four HAProxy nodes, and optimistic retry logic then made the load worse. Traffic to the Copilot Token Service went from a normal 7,000 to 9,000 requests per second up to 70,000 to 100,000, roughly tenfold amplification, with a suspected retry bug in VS Code contributing. The Register covered the findings in detail. Second, the partial error rate is the real story. A 20 percent error rate is far more operationally hostile than a clean 100 percent failure, because retries succeed often enough that automation keeps trying, humans keep second-guessing whether it is really broken, and dashboards look merely degraded rather than red.
AWS us-west-2 had another connectivity problem
AWS logged a regional connectivity issue in us-west-2 in mid August, traced to networking devices routing traffic toward the Seattle metro area, and resolved inside a roughly 20 minute window. That short duration is easy to shrug off until you notice it follows a similar us-west-2 event on July 24, 2026, which cost most customers about 20 minutes and hit some AWS Direct Connect users for an hour and seventeen minutes. Add the thermal event and the network disruption earlier in the year and you get a cluster of independent failures in the same region across a few months, each landing on a different layer of the stack.
Short outages train teams into complacency. A 20 minute regional blip is under most alerting thresholds for paging a human, over most thresholds for breaking a batch job, and squarely inside the window where a retry storm can turn a vendor problem into a self-inflicted one.
Cloudflare logged 13 incidents in eight days
Cloudflare's status history recorded 13 separate incidents between August 7 and August 14, 2026, touching R2 object storage, Durable Objects, Workers KV, Workers AI, and regional network performance across four continents. Twelve carried a minor severity label. The cluster began with R2 write failures in the Eastern North America region on August 7, with a small set of buckets unable to accept writes between 14:52 and 17:02 UTC, and mitigation confirmed later that evening. Customers elsewhere saw 503s on Magic Transit, elevated Workers KV errors, and regional 5xx spikes in Kuwait, Bangkok, Jakarta, and Dammam. You can review the raw timeline yourself on the Cloudflare status history page.
The severity labels are the trap here. Minor to the provider means a small percentage of a very large customer base. If your object storage lives in the affected region, a minor incident is a total outage for you.
The wider picture
Zoom out and the same week also included a major Anthropic service disruption starting around 21:58 UTC on August 16 that affected Claude across multiple surfaces, an ongoing Slack incident from August 13 affecting access to historical messages for workspaces with custom retention policies, and 502 global network outage events logged in a single week by ThousandEyes, as summarized in Network World's running outage report. This is not an unusual month. This is a normal month that happened to be well documented.
Why Third-Party Outage Response Is Now Core On-Call Work
For most engineering organizations, the majority of user-visible incidents no longer originate in code the organization wrote. They originate in a dependency. Understanding why changes how you staff and run on-call.
Your dependency graph is much bigger than your architecture diagram
Ask a team to draw their system and you get services, databases, queues, and a load balancer. Ask what has to be working for a customer request to succeed and the list gets uncomfortable fast: a DNS provider, a CDN, an identity provider, a payments processor, an email relay, a feature flag service, an object store, an observability vendor, and increasingly a model provider. Most of those have no representation in the architecture diagram and no owner on the on-call rotation.
The asymmetry is the problem. You have deep instrumentation on the twelve services you built and almost none on the thirty you rent. When one of the rented ones degrades, your dashboards show a symptom with no cause attached, and the first twenty minutes of the incident go to answering a question that should have been answered by a graph.
The CI and CD blast radius nobody models
The GitHub outage is instructive precisely because GitHub is not in the request path for most production systems. Nothing customer-facing broke because Pull Requests were unavailable. What broke was the organization's ability to respond to anything else.
Think about what a source control and CI outage actually removes. You cannot ship a fix. You cannot roll back through your normal pipeline if the pipeline lives in Actions. You cannot read the runbook if the runbook is a markdown file in a private repo. You cannot see the diff that probably caused the incident you are already in. Automated dependency updates, security scanning, and release automation all stop. For several hours on August 17, a large share of the industry lost its remediation capability while its production systems stayed up. That is a strange and dangerous state, and it is the single best argument for keeping a copy of your critical runbooks somewhere that does not depend on your source control provider.
The data backs up the pressure on-call teams feel
The SRE Report 2026, based on 418 responses from reliability practitioners worldwide, found that 67 percent of respondents agree performance degradations are as damaging as full outages, and that median toil sits at roughly 34 percent of working time. Both numbers describe the vendor outage experience exactly. Partial degradation is the common failure mode, and the manual work of triaging something you cannot fix is textbook toil.
The Detection Problem: Knowing Quickly That It Is Not You
The single largest cost in a vendor outage is the time spent establishing that it is a vendor outage. Teams routinely burn 15 to 40 minutes here, and almost all of it is avoidable.
Instrument the dependency, not just the symptom
Most teams alert on the symptom: checkout error rate is up, p99 latency crossed the threshold, the queue is growing. Those alerts are necessary but they are ambiguous by design. They fire identically whether the cause is your last deploy or your payment provider.
The fix is unglamorous. For every critical external dependency, emit three things as first class metrics: request rate, error rate broken out by the dependency's own error codes, and latency distribution. Put them on one dashboard, ordered by how much of your revenue path they sit on. When an ambiguous alert fires, the responder opens one panel and the answer is visible in seconds rather than inferred from a chain of guesses.
Add synthetic checks that exercise the dependency directly rather than through your full stack. A tiny periodic job that writes and reads one object from your object store, or requests one token from your identity provider, gives you a clean signal that is not contaminated by your own application logic. When the synthetic fails and your internal checks pass, you have your answer immediately.
Status pages are a lagging indicator, so treat them that way
Every one of the August incidents was visible in customer telemetry before it appeared on a provider status page. That is structural, not malicious. Providers need to confirm scope before posting, and confirmation takes time. The gap is typically 10 to 30 minutes and can be longer for regional or partial impact.
So use status pages, but use them as confirmation rather than detection. Subscribe to the machine-readable feeds for your top dependencies and pipe them into the same channel where your alerts land. When the status update finally arrives, it should show up next to the alert your own monitoring fired 20 minutes earlier, which turns the update into useful confirmation rather than an embarrassing surprise. Also worth noting: some providers only publish incidents that cross a global threshold, so a regional failure that is total for you may never appear at all.
Watch the second-order signals
Some of the best early indicators are indirect. A sudden spike in retries, a connection pool saturating, a queue depth climbing without a matching traffic increase, a jump in timeouts concentrated on one hostname. Alert on retry rate specifically. It moves earlier than error rate because your clients are absorbing the first wave of failures before they surface as user-visible errors.
A Practical Third-Party Outage Response Playbook
Here is a structure that works, built around what a responder can realistically accomplish under time pressure.
Minutes zero to five: classify
The only goal in the first five minutes is answering one question: is this ours or theirs? Open the dependency dashboard. Check the synthetic probes. Check whether the failures cluster on a single external hostname, region, or error code. Check whether a deploy went out in the last hour. That last check should be automatic, and the deploy timeline should be pinned in the incident channel without anyone asking.
Resist the urge to start debugging before classifying. A responder who spends 30 minutes reading their own application logs during a provider outage has not just wasted 30 minutes, they have also delayed the customer communication that actually matters.
Minutes five to twenty: contain
Once classified as external, your job shifts from fixing to limiting. The realistic options are narrower than people expect, which is exactly why they should be decided in advance rather than invented live.
- Shed the dependency. If the feature can degrade gracefully, turn it off behind a flag. A checkout that hides one payment method beats a checkout that times out.
- Stop the retry storm. Confirm your circuit breakers actually opened. If they did not, cut retry budgets manually. Aggressive retries against a struggling provider make their recovery slower and your latency worse.
- Fail over if failover is real. Multi-region or multi-provider failover only counts if it has been exercised. If it has not been tested in the last quarter, attempting it mid-incident is usually a second incident.
- Queue instead of dropping. For asynchronous work, buffer and replay once the provider recovers. Verify your buffer has the capacity for a multi-hour outage, not a five minute one.
- Freeze risky changes. Deploying into an ongoing external degradation makes attribution nearly impossible if something else breaks.
Minutes twenty to sixty: communicate
Vendor outages are disproportionately communication problems, because the engineering work is mostly waiting. Your customers do not care whose fault it is. They care whether you know, whether you are on it, and when to check back.
Post an initial customer update as soon as you have classified, not when you have a resolution. Name the impact in user terms rather than infrastructure terms. "Uploads are failing" is useful. "Elevated error rates in the storage layer" is not. Commit to a next-update time and hit it, even if the update is that nothing has changed. Internally, keep a single running timeline in one channel so that anyone joining the incident 40 minutes late can catch up by scrolling rather than by asking five people the same question.
After recovery: the postmortem you still owe
Teams routinely skip the postmortem for vendor outages because there is no action item that fixes the vendor. That is the wrong conclusion. You do not control the provider, but you control everything about how you detected, contained, and communicated. Those are all reviewable.
Useful questions: how long between first customer impact and first internal detection? How long between detection and correct classification? Did the circuit breakers open? Did the graceful degradation path work, and had anyone tested it since it was written? Was the customer update posted before customers started reporting the issue? Did the responder have access to the runbook, given that the runbook may have been in a system that was also down?
Designing Alerts That Survive a Vendor Outage
Suppress downstream noise, keep the upstream signal
A single provider failure typically fires dozens of alerts, because everything that depends on the failing thing also fails. The responder gets buried, and the one alert that would have identified the cause is lost in the pile. This is where alert dependency modeling earns its keep. If the storage dependency alert is firing, the seven service-level alerts that all depend on storage should be grouped underneath it rather than paging independently.
If you cannot model dependencies formally, a cruder version still helps: group alerts by the external dependency they touch, and set aggregation windows so that a burst collapses into one notification with a count rather than 40 separate ones.
Route to the human who can actually act
During an external outage, the useful responder is often not the service owner. It may be whoever owns the vendor relationship, or whoever can flip the feature flag, or whoever writes the customer update. Encoding that in your escalation policy in advance saves the round of "who do we even page for this" that otherwise eats ten minutes.
It also helps to have named a vendor owner per critical dependency ahead of time. Someone should know the support tier, the escalation contact, and where the contractual commitments live, before an incident makes those questions urgent.
Where a Slack-Native On-Call Tool Fits
Most vendor incidents get resolved socially before they get resolved technically. Someone notices something odd, asks in a channel, three people confirm, and a picture assembles. That conversation is the incident response, and it already happens in Slack.
The reason we built Pagerly as Slack-native is that context switching costs the most exactly when it costs the most. During a third-party outage, the responder is already juggling a provider status page, a dashboard, a customer support queue, and a vendor support ticket. Making them open a separate incident tool to acknowledge a page, check who is on call, or file a follow-up action item adds friction at the worst possible moment.
Concretely, that means paging, acknowledgment, escalation, and schedule overrides happen in the channel where the discussion is already occurring. The incident timeline builds itself from that conversation rather than being reconstructed afterward from memory. Action items get created inline while the lesson is fresh, which is the only time anyone actually writes them. And when your source control provider is the thing that is down, an on-call workflow that lives in chat keeps working, which is not a trivial property.
Whatever tool you use, the principle holds: the fewer tabs between noticing a problem and acting on it, the shorter your incidents.
A Checklist to Run Before Your Next Vendor Outage
None of this requires a large project. Most of it is a single afternoon of preparation that pays for itself the first time a provider has a bad day.
- List your critical external dependencies. Everything that has to work for a paying customer to get value. Rank by blast radius.
- Add a dependency health dashboard. Request rate, error rate, and latency per dependency, on one screen, linked from every alert.
- Add synthetic probes. One per critical dependency, exercising it directly outside your application path.
- Subscribe to status feeds. Machine-readable where available, routed into the same channel as your alerts.
- Verify your circuit breakers. Test that they open under sustained failure, and that they close cleanly on recovery.
- Test one graceful degradation path this quarter. Pick the highest blast radius dependency and actually turn it off in staging.
- Mirror your runbooks somewhere independent. If your incident documentation lives only in the system that might be down, you do not have incident documentation.
- Name a vendor owner per dependency. Support tier, escalation path, and contractual commitments documented before you need them.
- Draft customer communication templates. One for "we are investigating," one for "the cause is an upstream provider," one for resolution.
- Write vendor outages into your postmortem policy. External cause is not an exemption from review.
What to Carry Into Your Next On-Call Rotation
The lesson of mid August 2026 is not that GitHub, AWS, and Cloudflare are unreliable. By almost any historical measure they are extraordinarily reliable, and the very fact that a few hours of GitHub downtime became global news is evidence of how rare it is. The lesson is that reliability at the provider level does not translate automatically into reliability at your level, because your availability is the product of every dependency in your path, and that product gets smaller as the list gets longer.
You cannot fix someone else's infrastructure. You can control how fast you notice, how accurately you classify, how much you contain, and how honestly you communicate. Those four things are almost entirely within your control, they are cheap to improve, and they are what separates a vendor outage that costs you an hour from one that costs you a day and a handful of customers.
Start with detection, because everything downstream depends on it. Build the dependency dashboard this week. Add the synthetic probes. Then the next time your alerting fires at 3am and the cause is three companies away from your code, your on-call engineer will know that within two minutes instead of forty, and will spend the rest of the incident doing something useful.
