On September 3, 2026, engineering teams across the industry watched something that is supposed to be statistically unlikely: OpenAI's ChatGPT, Anthropic's Claude, and xAI's Grok all degraded at roughly the same time. Axios reported the simultaneous disruption mid morning US Eastern time, noting that while outages are common, three independent providers failing at once is not. For anyone responsible for AI outage incident response, it was one of the most instructive mornings of the year. 9to5Mac confirmed the impact spanned web, mobile, and desktop clients, and that Codex users were caught in it too.

If your product calls a model API, or your developers live inside an AI coding agent, or your support queue is triaged by an LLM, that morning was an incident for you whether or not you had an alert for it. This post is a practical guide to AI outage incident response: what made this event different, why most monitoring setups missed it, and what to change in your on-call practice before the next correlated failure.

What actually happened on September 3

The shape of the event matters more than the vendor names. Around 10:58 AM Eastern, reports spiked simultaneously across three consumer facing AI products. OpenAI acknowledged elevated errors. Anthropic reported errors across multiple models and said it had identified a cause. xAI confirmed a service issue with Grok. Google's Gemini drew user reports as well, though the company did not post a confirmed outage notice.

The timeline as responders saw it

For most teams, the sequence went like this. First, a trickle of user complaints or internal Slack messages saying a feature "feels broken." Then a spike in latency or 5xx rates on whatever internal service wraps the model call. Then, several minutes later, someone finds a news article and the picture snaps into focus. That gap, between "our thing is broken" and "the thing our thing depends on is broken," is the expensive part. It is where responders burn time rolling back deploys that were fine, restarting pods that were healthy, and paging people who could not have helped.

Downstream tools felt it too. Coding assistants and agent platforms that proxy to these providers reported their own degradation, which is the classic second order effect: your vendor's vendor is the actual root cause, and the status page you are refreshing belongs to neither of them.

What we do not know yet

Be careful here, because the temptation to over explain is strong. As of this writing there is no confirmed shared root cause. Coverage noted that Microsoft Azure serves as cloud infrastructure for multiple frontier model providers, and that Azure had reported regional issues the same day, but no vendor has published a postmortem establishing a causal link. It is entirely possible the correlation is partly coincidental and partly structural.

The honest position for an engineering team is this: you do not need to know the root cause to act on the lesson. The lesson is that three vendors you chose partly for redundancy failed inside the same window, and your architecture should not have assumed they could not.

Why correlated failures break AI outage incident response

Most incident response playbooks were written for a world where third party dependencies fail one at a time. That assumption is doing a lot of quiet work in your architecture, and events like this one expose it.

Vendor diversity is not the same as failure independence

Teams that added a second model provider generally did so for one of three reasons: cost arbitrage, capability differences, or resilience. The resilience argument only holds if the two providers fail independently. In practice, frontier model providers share a surprisingly narrow substrate. They rent capacity from a small number of hyperscalers. They sit behind a small number of CDN and DNS providers. They depend on the same certificate authorities, the same BGP fabric, the same handful of GPU regions. Two vendors on the same underlying cloud region are not two vendors from a reliability standpoint. They are one vendor with two logos.

This is the same concentration risk that made the cloud outages of the past two years so painful, just pushed one layer up the stack. The practical implication is that your failover target needs to differ in substrate, not just in brand. A local model, a cached response path, or a deterministic non AI fallback is a genuinely independent failure domain. A second API endpoint in the same region is not.

The blast radius is wider than the chat window

When a model API degrades, the visible symptom is usually a chat feature returning errors. The invisible symptoms are worse and they surface later:

Internal tooling stalls. If your engineers use AI coding agents, your team's throughput drops during the incident, including the throughput of the people responding to the incident.
Async pipelines back up. Enrichment jobs, summarisation workers, classification queues, and embedding pipelines silently accumulate backlog. The pager fires hours later when a queue depth threshold trips.
Retry storms amplify the outage. Naive exponential backoff without jitter across thousands of workers turns a provider blip into a self inflicted denial of service against a recovering API.
Cost spikes. Failover to a more expensive model or a higher tier can quietly multiply spend for the duration, and nobody notices until the invoice.
Data quality degrades. If a classifier falls back to a weaker model or a default label, you may be writing bad data that outlives the outage by weeks.

That last one is the sleeper. An outage that lasts ninety minutes can seed a data problem that takes a month to find. Any AI outage incident response process worth the name includes a data integrity check after recovery, not just a green dashboard.

Mapping your real AI dependency graph

You cannot respond well to a dependency you have never written down. Most teams underestimate their AI surface area by a factor of two or three, because model calls tend to arrive through product features rather than through architecture review.

Start with the user journeys, not the vendor list

Asking "which model APIs do we call" produces an incomplete answer, because it misses the SaaS tools that call models on your behalf. Instead, walk the top ten user journeys in your product and the top five internal workflows your team depends on, and at each step ask whether an inference call happens. You will find things like a support tool that auto summarises tickets, a CRM that scores leads, a security product that explains alerts, and a docs search that embeds queries. None of those show up in your own dependency manifest.

Classify by degradation mode

Once you have the list, sort each dependency into one of four buckets. This classification is what makes the incident tractable at 3 AM, because it tells the responder what to do without thinking.

Hard dependency, user visible. The feature is unusable without inference. These need an explicit user facing degraded state and a status page component of their own.
Soft dependency, user visible. The feature works worse but works. These need a fallback path and a way to verify the fallback is actually live.
Async, deferrable. Batch and queue work that can wait. These need backpressure and a replay mechanism, not a page.
Internal only. Developer tooling. These need to be known so that responders account for reduced team capacity during the incident.

Write this classification into the runbook, not into a wiki page nobody opens. The distinction matters: the runbook is what the responder reads while the pager is buzzing.

Detection: why your monitors probably missed it

Many teams learned about the September 3 outage from a news site or a customer, not from their own telemetry. That is a detection failure, and it is fixable.

Status pages are a trailing indicator

We have written before about why vendor status pages should not be your primary signal, and this event is another data point. Status pages are updated by humans after internal confirmation, which means they lag customer impact by anywhere from five minutes to an hour. Worse, they are scoped to the vendor's definition of an incident, which may not include the specific model, region, or endpoint you use. Polling a status page is fine as context. It is not detection.

What to alert on instead

Instrument the call site, not the vendor. Every outbound inference call should emit latency, status code, token counts, and a provider label. From that you get alerts that fire before any status page updates:

Error rate by provider and model. A jump in 429s and 5xx on a single provider is the earliest reliable signal you will get.
Latency percentile shift. Providers often degrade before they fail. A p95 that doubles is a warning you can act on while the API is still technically up.
Fallback activation rate. If your code silently falls back to a secondary model, alert on that. Silent fallbacks are how outages hide for hours.
Queue depth and consumer lag. For async inference work, backlog growth is the symptom that actually matters to the business.
Cross provider correlation. This is the new one. Alert specifically when two or more independent providers degrade inside the same short window, because that pattern means the problem is upstream of all of them and your normal failover will not help.

That last alert is cheap to build and it would have compressed the September 3 diagnosis from twenty minutes to two. It is also the alert that tells you to stop debugging your own code immediately.

The first thirty minutes: a practical response sequence

Here is a sequence that works for third party AI degradation. It assumes the incident channel is already open and the on-call responder is engaged.

Minute 0 to 3: confirm scope. Check error rates by provider. If more than one provider is affected, declare it an upstream event and say so in the channel. This single sentence prevents the most common waste of time, which is three engineers independently checking whether the last deploy caused it.
Minute 3 to 8: pull in the right people. Page the owner of the affected surface, not the whole platform team. Correlated vendor outages generate a lot of curious spectators and very few useful actors. Keep the responder set small and the observer channel separate.
Minute 8 to 15: reduce load and stop amplification. Cap retries, widen backoff, and shed non essential inference traffic. Turn off the batch enrichment job before it drains your rate limit budget for the user facing path.
Minute 15 to 20: choose a degraded state deliberately. Either fail fast with a clear message or queue for later. Do not let requests hang. A ten second spinner that ends in an error is a worse experience than an immediate honest message.
Minute 20 to 30: communicate. Post to your status page with the affected component and the fact that it is an upstream provider issue. Tell support what to say. Give a next update time and keep it.

Notice how little of this is technical. Most of the value in a third party outage comes from coordination speed, which is exactly why the tooling you coordinate in matters. A Slack-native on-call tool like Pagerly helps here because the escalation, the incident channel, the responder acknowledgement, and the status page update all happen in the place your team is already sitting. When the outage is upstream and there is no code to fix, the entire job is routing attention correctly, and every context switch to another browser tab is pure overhead.

Designing for graceful degradation before the next one

The architectural work is more interesting than the response work, and this is where a correlated outage should actually change your roadmap.

Fallback chains that actually get exercised

Most fallback code is written once, tested manually, and then rots. If your secondary path has not run in production in the last month, assume it is broken. The fix is to route a small percentage of live traffic through the fallback continuously, so that it is always warm and always monitored. This costs a little money and saves an enormous amount of incident time.

Design the chain to cross substrates. A reasonable chain looks like: primary hosted model, then a different provider on a different cloud, then a small self hosted or on device model, then a deterministic rules based path, then an explicit degraded message. Each step should be independently testable and independently observable.

Queue and defer instead of failing

For anything that is not synchronously user facing, the correct behaviour during a provider outage is to enqueue with a durable store and replay on recovery. This turns a hard failure into a latency problem. It requires idempotency and a replay worker, which is real engineering work, but it converts the majority of your AI surface area from "outage" to "delay." Teams that had this in place on September 3 lost some freshness. Teams that did not lost data.

Budget and circuit breaker hygiene

Put a circuit breaker in front of every provider, with a trip threshold based on error rate over a short window and a half open probe on recovery. Pair it with a spend guardrail so that failover to a pricier model cannot run unbounded. Both of these are boring, well understood patterns, and both are routinely missing from AI integration code because that code was written fast to ship a feature.

On-call practice changes worth making this quarter

Own the dependency, not just the service

Assign a named owner to each significant AI dependency, the same way you would for a database or a payment processor. That owner is responsible for knowing the rate limits, the degradation modes, the fallback path, and the contract terms. Without an owner, a vendor dependency is everyone's problem and therefore nobody's runbook.

Run the game day

Schedule a two hour exercise where you block outbound traffic to your primary model provider in a staging environment carrying realistic load, and let the on-call rotation respond for real. You will discover, reliably, that the fallback does not work, that the alert does not fire, and that nobody knows who updates the status page. Those three findings alone justify the exercise. Do it again after you fix them.

Keep the coordination layer independent

One uncomfortable lesson from the past year of outages is that incident tooling shares infrastructure with the things it monitors. If your paging provider, your chat, and your production stack all sit on the same cloud region, a bad day becomes a very bad day. Know where your incident tooling runs, know your out of band path, and make sure at least one person can reach the response team through a channel that does not depend on the systems currently on fire. A phone number written down somewhere is not an anachronism, it is a control.

What to put in the postmortem even if it was not your fault

Teams often skip the postmortem for third party outages on the grounds that there was nothing they could have done. That is almost never true, and the review is where the compounding value lives. Cover these points:

Time to correct attribution. How long from first symptom to the moment someone said "this is upstream"? This is your single best improvement metric for vendor incidents.
What the user actually saw. Pull real examples. Spinners, generic errors, and silent wrong answers are three very different failure experiences and you should know which one you shipped.
Fallback behaviour. Did it activate? Did anyone notice? Did it produce acceptable output?
Data integrity. What was written during the degraded window, and does any of it need to be recomputed or invalidated?
Amplification. Did your retry behaviour make the provider's recovery harder or exhaust your own quota?
Communication. How quickly did support and the status page reflect reality, and did the two agree with each other?

Convert each finding into a single owned action item with a date. A postmortem with six observations and no owners is a document. A postmortem with two owned actions is an improvement.

The takeaway for on-call teams

The September 3 event is worth remembering not because any one provider had a bad morning, but because it falsified an assumption a lot of architectures quietly rely on. Multi vendor does not mean multi failure domain. When your redundancy strategy is three companies renting from the same substrate, you have concentration risk wearing a diversification costume.

The response is not to abandon AI dependencies, which would be absurd, and it is not to build elaborate multi cloud inference infrastructure, which is expensive and mostly unnecessary. It is more modest and more achievable: know your dependency graph, instrument the call site instead of trusting status pages, alert on cross provider correlation, make degradation an explicit designed state rather than an accident, and keep your coordination layer fast and independent. Teams that do those five things will treat the next correlated AI outage as a forty minute inconvenience rather than a half day scramble.

The next one is coming. The only real question is whether your team finds out from your own telemetry or from a news article someone pastes into Slack.