At about 2 a.m. Eastern on Sunday, September 6, 2026, reports that Google services were failing started trickling into Downdetector. They did not spike. They climbed. By roughly 9 a.m. the volume was running about ten times higher than normal, with users saying that Google Search, Gmail, YouTube and YouTube TV were failing to load. That is a seven hour ramp, and it is the single hardest incident shape for an on-call team to catch.

Slow degradation incident response is a different discipline from responding to a hard down. When a service falls off a cliff, everything you own is designed to notice: synthetic checks fail, error rates jump, dashboards go red, and somebody pages within ninety seconds. When a service degrades over seven hours, every individual measurement along the way is indistinguishable from a bad sample. The alert never fires, because no threshold was ever crossed cleanly. The incident is real the whole time. It just never announces itself.

This post is not about Google. Google will publish whatever it publishes, and the specifics will matter to Google. What matters to everyone else is the shape, because that shape recurs constantly and most on-call setups are quietly blind to it. Below is what the ramp does to detection, what it does to escalation, and what you can change this week so the next slow burn does not run for seven hours before a human looks at it.

What the September 6 ramp actually looked like

The publicly reported facts are worth stating plainly, because the details are the lesson. According to reporting on the incident, the first excessive reports began appearing around 2 a.m. and increased steadily through the morning, reaching roughly ten times the normal rate by about 9 a.m. Users described Google.com and Gmail failing to load both in browsers and in apps. On some streaming devices, YouTube showed no thumbnails at all, but videos played normally if you clicked where a thumbnail should have been. Google's own Workspace Status Dashboard showed no confirmed incident while this was being reported.

The shape is a ramp, not a step

Take the timeline seriously for a moment. Seven hours is longer than most incidents run end to end. If you plotted the failure rate, you would not see a vertical line. You would see a gradient that, at any given fifteen minute window, looks like a modest and entirely plausible increase over the previous fifteen minutes. Each window is unremarkable. The sum of the windows is a major outage.

This is why point-in-time thresholds fail here. A rule that says "page if error rate exceeds five percent over five minutes" is a step detector. It is excellent at catching a deploy that broke everything at 14:32. It is close to useless against a curve that takes four hours to reach five percent, because by the time it fires, the incident has already been happening for most of a workday and your customers have already formed an opinion about your reliability.

Partial failure is the normal case

The YouTube thumbnail detail is the most instructive part of the whole event. Thumbnails did not render. Video playback worked. That is a service that is simultaneously down and up, depending entirely on which part of it you measure.

Every binary health check in the world reports that service as healthy. A synthetic that loads the page and asserts a 200 passes. A synthetic that starts playback passes. A synthetic that asserts a specific thumbnail image loaded is the only one that fails, and almost nobody writes that check, because it feels absurdly specific right up until the day it is the only thing standing between you and a seven hour blind spot.

Partial failure is not an exotic edge case. In a system built from dozens of services behind a shared edge, partial failure is the statistically likely failure. One asset pipeline degrades. One region's cache is cold. One dependency starts timing out on a subset of requests. The user-visible result is a product that is subtly broken, and subtly broken does not trip binary alarms.

Why slow degradation incident response is genuinely harder

It is tempting to file this under "our monitoring needs work" and move on. That undersells the problem. There are three distinct failures at work, and only one of them is technical.

Threshold alerting is built for cliffs

Most alerting configurations are inherited. Somebody set a static threshold at some point based on what felt bad, and it has been tuned downward every time it made noise and never tuned for sensitivity to slope. The result is a detector that is well calibrated for the failure mode it has seen most often, which is a sudden break, and blind to the one it sees less often but which lasts far longer.

Static thresholds also have a nasty property during a ramp: they encourage a false sense of safety. Responders look at a dashboard, see the metric is at three percent against a five percent threshold, and conclude things are fine. Nobody notices that it was at one percent two hours ago and is doubling every ninety minutes.

The first hour of a ramp is indistinguishable from noise

Be fair to the on-call engineer here. At 3 a.m., with error rates slightly elevated and no correlated deploy, the correct prior really is "this is noise." Elevated errors at low volume happen constantly. Retries clean most of them up. If you paged on every mild elevation you would be paging several times a night and you would burn your rotation out inside a month.

So the responder is being asked to distinguish, in real time and half asleep, between the ordinary noise floor and the first fifteen minutes of a seven hour degradation. Those two things look identical when you only have the current value. They look completely different when you have the trend. That distinction is the whole ballgame, and it is a tooling problem, not a judgment problem.

The social cost of a false alarm is asymmetric

There is a real and rarely discussed pressure against escalating early. Waking six people at 4 a.m. on a Sunday for something that turns out to be nothing has a social cost that the engineer feels personally. Letting a ramp run for another hour has a cost too, but it is diffuse, borne by customers, and nobody will trace it back to that specific decision.

Teams get the behaviour they make cheap. If declaring an incident requires a Zoom bridge, a conference number, a status page update and a message to leadership, people will not do it on a hunch. If declaring an incident is one Slack command that opens a channel and pulls in the current on-call, people will do it on a hunch, and hunches turn out to be right often enough that this is a straightforwardly good trade.

The weekend multiplier

This one started at 2 a.m. on a Sunday. That is not incidental. It is the specific window where every mitigating factor you normally rely on is at its weakest.

Rosters are thinnest and paths are longest

On a Tuesday afternoon, a slow ramp gets caught by ambient attention. Somebody notices a graph looks odd. A support agent mentions a cluster of similar tickets. An engineer working on something unrelated sees a spike in their own dashboard. None of that is on the incident response plan, and all of it is doing enormous unacknowledged work.

At 2 a.m. on a Sunday, ambient attention is zero. There is one primary on-call and a secondary who has not been contacted in six weeks. Nobody is idly looking at graphs. Support is closed. The only detection path that exists is the automated one, which is exactly the path that a ramp defeats.

Weekend baselines are different, and most alerts do not know that

Traffic on a Sunday morning is a fraction of a weekday peak. That matters twice over. First, low volume means percentage-based metrics get noisy, so any sensible team has already widened tolerances for that window, which widens the blind spot. Second, absolute-count alerts almost never fire on a weekend, because the absolute counts are low even when the percentage is catastrophic.

If your alerting compares against a flat baseline rather than a same-time-last-week baseline, your weekend sensitivity is materially worse than your weekday sensitivity and probably nobody has ever measured by how much. That is a number worth knowing.

How to actually detect a ramp

The good news is that the fixes are well understood and mostly cheap. None of this requires a new observability vendor.

Alert on the rate of change, not only the level

Add a companion alert to your important service level indicators that fires on slope rather than absolute value. Something like "error rate has more than doubled compared to the same window last week, sustained over thirty minutes" will catch a ramp at hour one that a five percent static threshold will not catch until hour four.

Route it differently from your hard-down page. A slope alert is a warning, not a wake-up. Send it to a channel with a clear rule that the on-call should acknowledge it within a defined window during business hours and should look at it, not necessarily act on it, overnight. The point is to put the trend in front of a human early, not to add another siren.

Use multi window, multi burn rate error budget alerts

Burn rate alerting exists precisely for this problem. Instead of asking "is the error rate high right now," it asks "at the current rate, how fast are we consuming the error budget for this period." A slow degradation burns budget steadily, and a well configured burn rate alert catches it because the integral is what triggers, not the instantaneous value.

The standard pattern uses a fast window paired with a slow window: a high burn rate over a short window pages immediately, while a lower burn rate sustained over a long window opens a ticket. The long window arm is the one that catches ramps. If you have service level objectives defined but no burn rate alerts wired to them, that is the single highest leverage change available to most teams.

Instrument the user journey, not the endpoint

The thumbnail case says everything. Health checks that assert a service is reachable will not tell you that the product is broken. Write a small number of synthetic checks that walk an actual critical journey end to end and assert on the things a user would notice: the search results contain results, the dashboard renders the chart, the image actually loaded and is not a broken placeholder.

You do not need many. Three to five journey checks covering your highest value paths, running from outside your own network, will catch a category of failure that no amount of internal instrumentation will surface. Run them from more than one region so you can tell a regional edge problem from a global one.

Treat outside reports as a legitimate signal

On September 6, third party report volume was the leading indicator. It moved hours before there was any official acknowledgement. Teams routinely dismiss this class of signal as unreliable, and individually it is. In aggregate, a tenfold increase in independent user reports is one of the highest quality signals available, because it measures the thing you actually care about, which is whether real people can use the product.

Wire the signals you can into the same place your alerts land:

Support ticket volume and, more usefully, ticket clustering. Five tickets mentioning the same symptom in twenty minutes is an alert, even if each one individually reads as user error.
A dedicated channel where anyone in the company can report "something looks wrong," with an explicit norm that reporting is free and never punished.
Third party outage aggregators for your critical vendors, pulled in automatically rather than checked manually when somebody remembers.
Social and community mentions for your own product, which frequently move before your support queue does.
Vendor status feeds, understood clearly as a lagging confirmation rather than a detection mechanism.

That last point deserves emphasis. Google's dashboard was still clean while report volume was ten times normal. This is not unusual and it is not a scandal. Status pages update after an incident is confirmed and scoped, which is properly a deliberate process. The failure is only ever on the consumer side, when a team treats a green status page as evidence that nothing is wrong. It is evidence that the vendor has not yet finished confirming that something is wrong.

How to respond when the ramp is real

Detection is half the problem. The other half is what happens in the twenty minutes after somebody suspects a slow degradation is underway.

Declare early, cheaply, and reversibly

The correct response to "I think something might be slowly breaking" is to open an incident channel, note the suspicion with a timestamp, and keep watching. Not to escalate to a full severity one. Not to wake the VP. Just to create a place where the observation is written down and other people can see it.

This costs almost nothing and pays off enormously. If the ramp is real, you now have a timeline that starts at hour one rather than hour seven, which is worth more during the postmortem than any amount of reconstruction. If it turns out to be nothing, you close the channel and the total cost was one message. Make the cost of being wrong so low that nobody hesitates.

Separate the "is it us" question from the "fix it" question

During a ramp there is a strong pull toward diagnosis before anyone has established scope. Resist it. The first question is always whether the degradation is inside your blast radius or upstream of it, because the answer completely changes what the team should be doing.

If it is upstream, your job is not to fix it. Your job is to establish which of your user journeys are affected, apply whatever degradation you have designed for, communicate to customers, and stop your engineers from spending four hours debugging code that is working correctly. That last one is a real and expensive failure mode. Teams routinely burn a full weekend debugging their own service during someone else's outage.

Communicate on the ramp's timeline, not on your confidence timeline

The instinct is to wait until you understand the problem before saying anything. During a slow degradation this is the wrong instinct, because the degradation is going to last hours and your customers are already noticing. Say what you know, including that you do not yet know the cause, and commit to an update interval you can actually keep.

A useful default: post an update within thirty minutes of declaring, then every hour on the hour whether or not there is news. "No change, still investigating, next update at 10:00" is a genuinely valuable message. Silence during a seven hour event is what turns a reliability problem into a trust problem.

Where a Slack native on-call setup changes the math

Everything above depends on one thing being true: that the cost of raising a weak signal is near zero. If it is expensive to say "this looks slightly off," nobody says it at 3 a.m., and the ramp runs unopposed.

That is a workflow problem more than a tooling problem, but tooling determines the friction. If your alerts land in one system, your on-call schedule lives in a second, your incident channel is a third, and your status page is a fourth, then every escalation involves context switching between four tools at the worst possible hour. People will not do that on a hunch. They will wait for certainty, and certainty during a ramp arrives around hour five.

This is the case for running on-call where the team already is. With Pagerly, the schedule, the current on-call, the alert routing and the incident channel all live inside Slack, so raising a suspicion is a message and paging the right person is a command in the same window. The person who noticed the odd graph does not need to know who is on call, or open another tool to find out. They mention the on-call group and the right human gets notified. Escalation policies handle the case where the first person is asleep through their phone alert.

The specific capability that matters most for slow degradation is the ability to convert an informal observation into a tracked incident without ceremony. Somebody posts "search results look sparse this morning" in a channel, and it becomes an incident with a timeline, an owner and a running record, in one action, without anybody deciding first whether it is severe enough to deserve that. Severity gets assigned later, when you know. The record starts now, when you do not.

A checklist you can run this week

None of this needs a quarter. Most of it is an afternoon.

Find your slowest historical incident. Pull the last twelve months and identify the one with the longest gap between first customer impact and first human acknowledgement. That gap, not your mean time to resolve, is the number this post is about.
Test your weekend sensitivity. Take a real weekday alert and check whether the same conditions at Sunday morning traffic levels would still fire it. If the answer is no, you have a scheduled blind spot.
Add one slope alert. Pick your single most important service level indicator and add a week-over-week comparison alert routed to a channel rather than a phone. One is enough to start.
Write three journey synthetics. Cover your top user paths, assert on user-visible outcomes rather than status codes, and run them externally from at least two regions.
Wire support signal into engineering. Make ticket clustering visible in the same channel as your alerts. Support usually knows first and is usually structurally unable to tell you fast.
Cut the cost of declaring. Time how long it takes, right now, from suspicion to an incident channel with the on-call in it. If it is more than a minute, that is your real detection latency, added on top of everything else.
Write down the "not ours" branch. Give the on-call an explicit runbook step for confirming an upstream dependency is degraded and for what to do next, so nobody spends the morning debugging working code.

The takeaway

Hard down outages are loud, well tooled, and comparatively easy. The industry has spent fifteen years getting good at them. The failure mode that still routinely runs for hours before anyone reacts is the slow one: a gradient that starts in the dead of night, never crosses a threshold cleanly, breaks the product partially rather than completely, and does not appear on any status page until it is already well established.

Slow degradation incident response comes down to two changes. Give your alerting some notion of trend so that a curve triggers something before a threshold does. And make it socially and mechanically free for a human to say "this looks off" at 3 a.m. on a Sunday, so the one signal that a ramp cannot defeat, which is a person noticing, actually reaches somebody who can act on it.

The seven hour ramp is not a rare event. It is the common event that your tooling is not shaped to see. Reporting on the September 6 disruption is available from the Houston Chronicle if you want the contemporaneous account. The more useful exercise is to open your own incident history and look for the ramps you already had, and did not name.