Cloud Outage Response: Lessons From July's Bad Week

Category
Falit Jain
July 28, 2026
5 min read
Cloud Outage Response: Lessons From July's Bad Week
Table of Content

In a single week, two of the largest cloud providers on earth failed at almost the same time, and a good chunk of the internet went with them. Effective cloud outage response stopped being a theoretical exercise and became the difference between a calm 30 minutes and a chaotic afternoon for thousands of on-call engineers. On July 23, 2026, a maintenance bug inside Microsoft Azure pulled IP routes off more devices than intended in the West US region, cutting Microsoft 365 access for millions. Less than 24 hours later, on July 24, an AWS networking fault in us-west-2 knocked Apple Pay, Reddit, Hulu, DoorDash, and PlayStation Network offline for the better part of an hour. Neither failure was in your code. Both became your problem.

This post breaks down what actually happened, why cloud concentration keeps turning a single provider's bad day into an industry-wide event, and, most importantly, what on-call teams should change now. The payoff is a practical playbook you can adapt this week, because the uncomfortable truth is that the next one is already scheduled. It just has not told you when.

A Week of Cloud Outages: What Actually Happened

Two big incidents landed inside 24 hours, and they rhymed in a way that should make every reliability team pay attention. Neither was a compute failure. Neither was a storage corruption. Both were about the network edge, the boundary where a region meets the rest of the internet, and both were triggered by routine changes rather than exotic hardware faults.

Azure West US and the maintenance bug that wiped routes

On July 23, 2026, Microsoft traced a major outage to a defect in how a maintenance request was translated into device changes. The system incorrectly marked additional network devices as part of a maintenance event, which caused IP routes to be removed from far more devices than intended between the datacenter and the wide-area network. The result: traffic trying to enter or leave the West US region had nowhere to go. Anything that stayed entirely inside the region was fine, but the moment a request needed to cross that boundary, it failed.

Customer impact ran from 14:44 UTC to 19:41 UTC, roughly four hours and 57 minutes. Microsoft began rolling back the maintenance changes at 17:45 UTC and completed the rollback at 18:26 UTC, after which the network stabilized. Downstream, thousands of users reported problems with Outlook, Teams, SharePoint, OneDrive, Copilot, Xbox Live, and the Microsoft Store, according to reporting on the incident by BleepingComputer. The full timeline and route-removal mechanics were captured in a detailed IncidentHub writeup.

AWS us-west-2 and the 80-minute internet blip

The next morning, July 24, outage trackers lit up again. Around 7:40 a.m. ET, Apple Pay, DoorDash, Reddit, Hulu, and PlayStation Network stopped responding almost simultaneously. AWS attributed the fault to networking hardware that carries routing between the us-west-2 region in Oregon and the Seattle metro area. The failure sat at the boundary between the Oregon data centers and the public internet, not inside the compute or storage layers that most engineers instinctively suspect first.

Engineers worked several mitigation paths in parallel, and the first restored connectivity at 11:15 UTC. As the network settled, routes reconverged between 11:47 and 11:59 UTC, and connectivity flapped again for some customers while the new paths stabilized. For most customers the initial impact window was around 20 minutes, but a subset relying on AWS Direct Connect through a specific Seattle facility saw closer to an hour and 17 minutes of disruption. This was the third distinct AWS reliability incident in eleven weeks, following separate cooling and network provider failures earlier in the season, as documented by Tech Times and analyzed in an IncidentHub breakdown.

The common thread: the edge, not the servers

Look past the two logos and the pattern is striking. In both cases the servers were healthy, the disks were spinning, and the applications were ready to serve. What broke was the ability to reach them. A routing change removed paths in one case and a hardware fault severed a link in the other. When the network edge fails, every service behind it goes dark at once, regardless of how well engineered it is.

That matters for on-call teams because it changes the shape of the problem. You are not debugging a memory leak or a slow query. You are staring at a wall of red where nothing you own is actually broken, and the fix lives inside a provider you cannot page. Understanding that distinction early is the first skill of modern cloud outage response.

Why Cloud Concentration Turns Local Faults into Global Events

A routing bug in one datacenter should be a footnote. Instead it takes down payment systems, food delivery, social media, and gaming networks across an entire continent. The reason is concentration. A handful of regions from a handful of providers now sit underneath an enormous share of the consumer and enterprise internet. When one of those regions loses its edge, the blast radius is not one company. It is every company that placed a critical dependency there and assumed the region would always be reachable.

Two structural facts make this worse. First, default architectures pull teams toward a single primary region because it is cheaper, simpler, and faster to ship. Second, hidden dependencies accumulate quietly. Your app might run in three regions, but your authentication provider, your payment processor, your feature flag service, and your observability vendor might all funnel through the same one you just lost. You inherit their concentration risk whether you chose it or not.

Industry guidance has been consistent on this point for years: resilience has to be designed before the incident, not improvised during it. Analyses like this one from CloudZero on building cloud resilience before the next AWS or Azure outage keep landing on the same conclusion. The teams that ride out these events are the ones that mapped their dependencies and decided, in advance, what they would sacrifice to stay partially up.

The On-Call Reality When You Do Not Own the Root Cause

There is a particular flavor of dread that comes with a provider outage. Your dashboards are screaming, your customers are furious, and the root cause is a router you will never touch. This section is about operating well inside that constraint, because it is where a lot of teams lose the plot.

You cannot fix someone else's router

The instinct during an incident is to find the thing you broke and unbreak it. During a provider outage there is no such thing, and the sooner your responders internalize that, the sooner they stop burning energy on dead-end debugging. The job shifts from repair to mitigation and communication. Can you fail traffic to another region? Can you degrade gracefully? Can you shed nonessential load so the parts that still work stay healthy? Those are the levers you actually hold.

Detection and confirmation are your job, not your provider's

Provider status pages are famously slow to turn red. They are updated by humans who need confirmation before they alarm millions of customers, which means your monitoring will almost always see the problem first. Do not wait for an official acknowledgment to declare an incident. If your synthetic checks, error rates, and customer reports all point the same direction, that is your signal. Treat the provider status page as corroboration, not as the trigger.

This is also where blameless thinking pays off. When the fault is external, there is genuinely no one on your team to blame, which makes it a perfect opportunity to practice the calm, factual incident posture you want during every event. The Google SRE guidance on blameless postmortems applies just as cleanly to outages you did not cause as to ones you did.

Building a Cloud Outage Response Playbook

Improvisation is expensive at 3 a.m. A written cloud outage response playbook turns a panic into a procedure. Here is a structure that holds up whether the culprit is AWS, Azure, Google Cloud, or a critical SaaS vendor two layers down your stack.

Detect and confirm scope fast

The first five minutes are about answering one question: how big is this? Is it one region, one service, one dependency, or everything? Pull up the signals that tell you scope quickly, error rates by region, synthetic checks from multiple locations, and a fast look at whether the failing calls share a provider or region. The goal is not root cause yet. The goal is a clear, shared statement of blast radius that the whole response can align around.

Declare, assign roles, and open one channel

Declare the incident the moment scope is confirmed, even if the fix is out of your hands. Assign an incident commander, a communications lead, and an operations lead. Crucially, collapse everything into one conversation. Nothing wastes an outage like three parallel threads, a video call, and a DM chain all holding different pieces of the truth. A single source of coordination keeps everyone reading from the same timeline.

Decide on mitigations you actually control

With scope confirmed and roles assigned, work only the levers you own. Fail over to a healthy region if you architected for it. Serve cached or read-only responses if writes depend on the dead region. Disable nonessential features to protect core flows. Put up an honest status message for your own customers rather than waiting for the provider. Each of these is a decision you can make without a single reply from your provider, and each one shortens the felt duration of the outage for the people who depend on you.

  • Confirm blast radius before root cause. Know whether it is one region, one service, or everything before you theorize about why.
  • Do not wait for the status page. Your monitoring will see it first, so let your own signals trigger the incident.
  • Coordinate in one place. A single channel with clear roles beats scattered threads every time.
  • Work only your own levers. Failover, graceful degradation, and load shedding are yours; the provider's router is not.
  • Communicate early and honestly. A plain status update to your customers buys more goodwill than silence and a fast recovery.
  • Capture the timeline live. The best postmortem is written from notes taken during the event, not reconstructed from memory a week later.

Architecture Lessons: Reduce the Blast Radius Before the Next Outage

Playbooks help you survive an outage. Architecture decides how much of it you even feel. The July incidents are a useful forcing function to revisit a few choices that are easy to defer until they become emergencies.

Multi-region and multi-cloud, honestly

Multi-region is often claimed and rarely tested. Running in two regions does you no good if failover has never been exercised, if your database only replicates one direction, or if your DNS takes 20 minutes to shift traffic. Be honest about your actual recovery posture. If you cannot fail a region out in a game day without a scramble, you are single-region with extra steps. Multi-cloud raises the ceiling further but carries real cost and complexity, so reserve it for the dependencies that genuinely warrant it rather than adopting it as a blanket policy.

Graceful degradation over hard dependencies

The services that came through July looking best were the ones that could lose a dependency and keep serving something. A read-only mode, a cached response, a queued write that reconciles later, these are the difference between a degraded experience and a blank page. Audit your critical user journeys and ask what each one does when a downstream dependency disappears. If the answer is a hard error, that is a design decision waiting to be revisited before the next edge failure makes it for you.

Map your hidden dependencies

Most teams cannot list, from memory, every third party in the critical path of a login or a checkout. That gap is exactly where outages hurt most. Build a dependency map that names each external service, the region it lives in, and what your product does when it is unavailable. The exercise alone routinely surfaces surprises, like three supposedly independent vendors that all terminate in the same region you just watched go dark.

Where a Slack-Native On-Call Tool Fits

When the network edge fails, coordination becomes the bottleneck. The technical fix is out of your hands, so the quality of your response comes down to how fast you detect, declare, align, and communicate. That is exactly the ground a Slack-native on-call and incident management tool like Pagerly is built to cover.

Because Pagerly lives inside Slack, there is no context switch at the worst possible moment. Alerts land where your team already works, the on-call schedule and escalation policy are a slash command away, and declaring an incident spins up a dedicated channel with roles, a running timeline, and stakeholder updates in the same place the conversation is already happening. During a provider outage, when three teams are trying to figure out whether it is them or AWS, that single coordinated channel is worth more than any dashboard.

The other quiet win is the timeline. Every action, acknowledgment, and status change is captured as it happens, which means the postmortem practically writes itself. Instead of reconstructing a four-hour Azure event from fuzzy memory, you export a factual record of who did what and when. For teams that want to shorten mean time to acknowledge and keep coordination in one calm place, keeping on-call in Slack removes a whole category of friction on the days it matters most.

Turning a Rough Week Into a Sharper Practice

The July 2026 outages were not caused by anything most teams could have prevented. A maintenance bug in Azure and a networking fault in AWS are, from the outside, weather. What you can control is how prepared you are when the weather turns. Every one of these events is a free rehearsal for the next one, and the teams that treat it that way pull ahead.

Run a blameless review even though the fault was external. Ask how long it took you to know, how long to declare, how cleanly you coordinated, and whether your customers heard from you before they heard from a news site. Those questions have nothing to do with AWS or Microsoft and everything to do with your own maturity. The point of studying someone else's outage is to fix your own response before you need it.

Then do the unglamorous work. Write the playbook. Map the dependencies. Schedule the game day. Make sure your on-call rotation is clear enough that the right person is reachable in the first five minutes, because in a provider outage those five minutes are most of the game. None of this is exotic. It is just the difference between a team that scrambles and a team that executes.

Key Takeaways

The week two clouds broke at once is a reminder that reliability is a shared responsibility, and the part you own is the response. Here is what to carry forward:

  • Edge failures are the new normal. Both July incidents hit the network boundary, not compute or storage, so watch reachability, not just server health.
  • Concentration is your risk too. You inherit the blast radius of every region your dependencies live in, whether you chose it or not.
  • Detection beats the status page. Your monitoring sees provider outages first, so let your own signals trigger the incident.
  • Response is what you control. Failover, graceful degradation, and clear communication are yours even when the root cause is not.
  • Coordination wins outages. One channel, clear roles, and a live timeline shorten the felt duration for everyone who depends on you.
  • Rehearse on other people's outages. Run the blameless review, write the playbook, and test failover before the next edge failure writes the plan for you.

The next major cloud outage is a matter of when, not if. The teams that come through it calmly will be the ones that decided, on an ordinary day, exactly how they would respond on an extraordinary one.

View all
Design
Product
Software Engineering
Customer Success

Latest blogs

Cloud Outage Response: AWS us-west-2 Lessons
July 27, 2026

Cloud Outage Response: AWS us-west-2 Lessons

A breakdown of the July 24, 2026 AWS us-west-2 outage and what it teaches on-call teams about detecting and responding to cloud outages faster.
What the July 2026 AWS Outage Teaches On-Call Teams
July 27, 2026

What the July 2026 AWS Outage Teaches On-Call Teams

A breakdown of the July 2026 AWS us-west-2 outage and what on-call teams should change before the next dependency failure.
On-Call Lessons From Meta's Repeat Outage
July 26, 2026

On-Call Lessons From Meta's Repeat Outage

Meta's second outage in 72 hours shows why on-call incident response is judged by the recurrence, not the first page.