On August 3, 2026, Snowflake customers in the AWS Asia Pacific (Tokyo) region started seeing login failures, slow page loads, and HTTP 503 errors when they tried to reach Snowsight. The window ran from roughly 00:53 UTC to 06:15 UTC, and the preliminary cause was not a hardware failure or a hostile attack. It was a change. An update meant to improve the efficiency of connection handling at the network routing layer had to be rolled back to restore service. If that sounds familiar, it should. Months earlier the same platform lost query and ingestion service across ten of its twenty three regions for roughly thirteen hours after a backwards incompatible schema update went out. Two different incidents, one recurring theme, and a clear lesson for anyone who carries a pager: on-call incident response is mostly about how well your team handles the changes you ship yourselves.
This post is not a pile-on. Snowflake ran a fast rollback, scoped the blast radius, and committed to a public root cause analysis, which is exactly what mature reliability looks like. The point is broader. Most outages are self inflicted, they arrive during ordinary work, and the difference between a five minute blip and an all night war room comes down to detection, escalation, and recovery muscle that you build before the incident, not during it. Below we unpack what happened, why routine changes cause the majority of production incidents, and a concrete on-call playbook you can adopt this week.
Start with the facts that are publicly known. In the August 2026 event, the trigger was a change to connection handling in the network routing layer. Initial signals suggested multiple deployments were affected, and only after investigation did the team confirm the impact was limited to a single region. Users saw intermittent failures rather than a hard down, with messages like "Failed to fetch" and "Execution failed," and retries often succeeded. The fix was a rollback, and a formal RCA was promised on the status page within five business days.
In the earlier multi region event, a new release introduced a backwards incompatible database schema update. Older release packages still referenced the previous field definitions, which produced version mismatch errors. Queries failed with internal error messages, Snowpipe and Snowpipe Streaming ingestion stalled, and clustering looked unhealthy. Ten of twenty three regions were affected, including deployments on AWS, Azure, and Google Cloud, and full recovery took longer than the first estimate because one large region was slow to come back.
When you line these up next to the wider industry data, three truths stand out. First, the change itself is usually the incident. Neither event was caused by a data center losing power or a novel exploit. They were caused by shipping something. Second, the initial blast radius estimate is often wrong, and usually wrong in the scary direction, so early minutes are noisy and your process has to tolerate uncertainty. Third, recovery time is dominated by human coordination, not by the technical rollback itself. Rolling back a release is a known operation. Deciding to roll back, getting the right people in the room, and confirming safety across regions is where the clock runs.
The DevOps research community has measured this for years. Change failure rate is one of the four core delivery metrics precisely because a meaningful slice of deployments cause a degradation that needs remediation. Elite performers keep that rate low, but no one drives it to zero. When you combine a high volume of deploys with distributed systems that span regions and clouds, you get a steady stream of change induced incidents. This is not a sign of a broken engineering culture. It is the normal operating condition of any team shipping quickly.
The reliability data from 2026 makes the on-call cost of this pattern concrete. A survey of more than a thousand SRE, DevOps, and IT operations professionals found that toil rose sharply even as teams adopted more automation and AI tooling. Nearly seventy percent of SREs cite on-call stress as a direct contributor to burnout, and around sixty five percent reported experiencing burnout in the past year. Roughly forty six percent of respondents handled more than five incidents in the last thirty days. Teams receive thousands of alerts per week, and only a tiny fraction demand immediate action. Alert fatigue has become a reliability risk in its own right, because when almost everything pages, the one page that matters gets lost.
Consider what happens during a change induced incident when the alerting layer is already noisy. A deploy goes out. Latency creeps up in one region. A dozen downstream checks fire because they all depend on the same routing path. The on-call engineer now has to separate the root signal from the symptomatic noise while half awake at two in the morning. If the alerts are not deduplicated and correlated, the responder spends the first fifteen minutes just building a mental model of what is actually wrong. Those fifteen minutes are the most expensive part of the entire outage, because the failure is still spreading while the human is still orienting.
Noise is only half of the problem. The same research found that a large majority of teams experienced at least one incident where no alert fired at all, and a significant share suffered an outage tied directly to suppressed or ignored alerts. In other words, teams are simultaneously drowning in alerts they do not need and missing the ones they do. For change induced outages this is especially painful, because the symptoms often show up as elevated error rates or slow responses rather than a clean binary of up versus down. If your monitoring only pages on hard failures, an intermittent degradation like the Tokyo event can run for an hour before a human even looks.
Every incident has the same three phases, and change induced incidents stress each of them in specific ways. Understanding the phases helps you design a response that holds up under pressure.
The goal of detection is to notice real user impact quickly and route it to a human who can act. For change related failures, the most reliable early signal is almost always a correlation with a recent deploy. If your incident tooling can automatically surface "a release went out eleven minutes ago in this region" next to the first alert, you have handed the responder the likely cause before they even open a dashboard. Detection also means watching the right symptoms. Error rate, tail latency, and saturation catch degradations that a simple health check misses. The Snowflake events were both degradations before they were outages, which is exactly the pattern good detection is built to catch.
Escalation is where minutes are won or lost. The responder needs to reach the person who owns the changed component fast, and if that person does not answer, the path to the next person has to be automatic. A surprising number of long incidents are long not because the fix was hard but because the first responder could not find the right owner and burned twenty minutes paging into the void. Clear ownership, an up to date schedule, and reliable escalation policies turn that twenty minutes into two. When the change that broke production came from a specific team, the fastest recovery routes the incident straight to that team rather than bouncing it through a general queue.
Recovery for a change induced incident usually means one of three moves: roll back the release, disable the offending feature, or roll forward with a fix. Rollback is the default because it is the fastest way to a known good state, which is exactly what Snowflake did in both events. The hard part is rarely the mechanics. It is the decision. Someone with authority has to say "we are rolling back" and someone has to confirm the rollback is safe across every affected region. This is a coordination problem, and coordination problems are solved by having the right people, the right context, and the right controls in one place.
Here is a concrete sequence you can adopt. It assumes nothing fancy and works whether you run five services or five hundred. The goal is to compress the time from first symptom to stable recovery.
A playbook only helps if the team has muscle memory for it. Run game days where you deliberately ship a bad change into a staging environment and practice the full loop of detect, escalate, and roll back. Rotate who plays incident commander so the skill is not concentrated in one or two heroes. Teams that rehearse recover faster for the same reason pilots train in simulators: the first time you practice an emergency should never be during the real one.
Both Snowflake events ended with a commitment to a public root cause analysis, and that instinct is worth copying. A good postmortem is blameless, which means it treats the failure as a property of the system rather than a mistake by a person. The engineer who shipped the connection handling change did not cause the outage any more than the last person to touch a shared codebase causes the next bug. The system allowed a change to reach production without a guardrail that would have caught it, and that is what you fix.
A postmortem earns its keep when it produces action, not narrative. Capture a precise timeline with timestamps, the user facing impact in plain language, the technical root cause, the factors that made detection or recovery slow, and a short list of concrete follow ups with owners and due dates. For change induced incidents, the most valuable follow ups usually strengthen the release pipeline itself: better pre production checks for schema compatibility, staged rollouts that limit blast radius to one region before going wide, automated rollback triggers tied to error rate, and clearer ownership so escalation is instant.
Action items that never ship are how the same outage happens twice. Track postmortem follow ups like any other work, with an owner and a deadline, and review them until they are done. A reliability program is only as strong as its willingness to actually complete the boring preventive work that no incident is forcing it to do today.
Most engineering teams already live in Slack, and that matters more than it sounds. During a change induced incident the bottleneck is coordination, and coordination happens in chat. A Slack-native on-call and incident management approach like Pagerly keeps the entire response loop in the place the team already works, which removes the friction of hopping between a paging tool, a separate incident app, a dashboard, and a chat window while the clock runs.
In practice that means a few things line up naturally. On-call schedules and escalation policies live next to the conversation, so paging the right owner and finding the next person up happens without leaving the channel. When an alert fires, the incident channel becomes the single source of truth: the timeline, current status, and who is doing what are all in one thread. Because the context is already in Slack, the record you need for the postmortem is captured as the incident unfolds rather than reconstructed afterward. And when the fix is a rollback, the decision and the confirmation happen where everyone can see them, which is exactly the coordination clarity that turns a thirteen hour incident into a short one.
None of this replaces good engineering practices like staged rollouts and compatibility checks. It complements them. The change pipeline reduces how often bad changes reach production, and a tight on-call loop reduces how long they hurt when they inevitably slip through. You want both, because the reliability data is unambiguous that changes will keep causing incidents no matter how good your testing gets.
The Snowflake incidents are a useful mirror because they are so ordinary. A routine update, a rollback, a scoped blast radius, and a public RCA. That is the daily texture of running production at scale, and it is the texture your on-call process has to be built for. Here is what to carry away.
Reliability is not the absence of incidents. It is the practiced ability to detect a change gone wrong, get the right people together fast, and return to a known good state before most users notice. The teams that treat on-call as a designed system rather than an afterthought are the ones that turn a bad deploy into a footnote instead of a headline. Build that system now, rehearse it, and keep the whole loop close to where your team already works.
