Opsgenie shuts down April 2027 - migrate to Pagerly in one click
PagerlyPagerly
← All postsEngineering

When Your Incident Response Tooling Is the Outage

Atlassian's August 27 outage took Jira, JSM and Confluence offline. Make your incident response tooling degrade instead of disappearing.

Pagerly blog cover: When Your Incident Response Tooling Is the Outage

On August 27, 2026, two unrelated companies had a bad day within hours of each other, and together they exposed a gap that most reliability programs never audit: incident response tooling is a production dependency, and almost nobody treats it like one. Atlassian confirmed that Jira Cloud, Jira Service Management and Confluence were unavailable to affected users. A few hours earlier, Proton went globally dark after a cooling failure at its Frankfurt data center. One of those is a ticketing and documentation stack. The other is a consumer service. The common lesson is the same: the systems you respond with live inside the same blast radius as the systems you respond to.

Most on-call programs are built on an unstated assumption. When the pager fires, the responder will have a working chat client, a reachable runbook, a ticket system that accepts writes, a dashboard that renders, and a status page that publishes. Remove any one of those and the response does not simply slow down. It changes shape. Responders fall back to memory, to direct messages, to screenshots, to whatever they can improvise. That improvisation is where minutes turn into hours.

This post walks through what happened, why response tooling fails in correlated ways, and how to build a response path that degrades gracefully instead of disappearing. It is not a pitch for redundancy theater. It is a practical argument that you should know, on paper, exactly what your team can still do when half the toolchain is unavailable.

What happened on August 27, 2026

Atlassian: the incident tool was the incident

Atlassian reported a disruption affecting Jira Cloud, Jira Service Management and Confluence, with services unavailable to affected users. Australian teams noticed it during their working afternoon, and Atlassian moved the incident into recovery roughly an hour after identifying it, marking it resolved with services operating normally. A public postmortem was promised once the full investigation completes, which is the correct sequence and worth acknowledging.

The duration is not the interesting part. The interesting part is what those three products do for a typical engineering organization. Jira Service Management is, for many teams, the on-call and alerting layer. Confluence holds the runbooks, the escalation matrix, the postmortem archive, and the architecture diagrams. Jira holds the follow-up actions from every previous incident. For an organization that consolidated on that stack, an Atlassian outage does not just block ticket updates. It removes the paging system, the instructions, and the institutional memory at the same moment.

If that outage had overlapped with a real production incident of your own, your response would have been running on human recall alone.

Proton: redundancy that failed together

Proton's own incident report describes a global outage that took Mail, VPN, Calendar, Drive, Pass, SimpleLogin, Wallet and Lumo offline. Engineers began investigating at 00:09 CEST, identified a critical cooling failure at the Frankfurt facility at 00:38 CEST, started shifting traffic to backup sites, and had the outage largely resolved by 02:27 CEST. No user data was lost. The root cause was traced to an air filter replacement performed on both of the redundant air compressors that power the cooling system.

Read that last sentence again, because it is the most instructive detail of the month. The redundancy existed. It was documented. It was defeated by a routine maintenance action applied to both halves of the redundant pair. This is a physical-world instance of a failure mode software teams reproduce constantly: rolling both replicas in the same window, upgrading both database nodes from the same automation run, rotating both certificates from the same expiring root, or running your primary and failover alerting integrations through the same third-party webhook provider.

The pattern behind both

August 2026 was not short of examples. Earlier in the month, a storm-related event took the building management system and chilled-water loop offline at a facility hosting a cloud region, and the data halls overheated within roughly two hours as servers, switches and rack power began shutting down on thermal protection. Different vendors, different layers, same shape: a dependency nobody modeled failed in a way nobody rehearsed.

The response-tooling version of this pattern is simply less visible, because it only bites when two things go wrong at once. That is exactly why it deserves deliberate attention rather than luck.

Why incident response tooling fails in correlated ways

Consolidation buys convenience and sells independence

There are good reasons teams consolidate. One vendor means one bill, one identity integration, one audit trail, one support relationship. The cost is that alerting, documentation, ticketing and postmortems now share a control plane, an authentication path, and an availability fate. When they were four separate tools, an outage in one degraded the response. When they are one suite, an outage takes all four.

You do not have to reverse consolidation to manage this. You do have to know which capabilities collapse together, and have a plan for the ones that matter in the first fifteen minutes.

Your identity provider is a shared root

Single sign-on is the most common hidden coupling in a modern response path. Chat, paging, dashboards, cloud console, ticketing and the status page publisher frequently authenticate through the same identity provider. If that provider is degraded, the responder who was successfully paged cannot log into anything they were paged about. Teams that have thought this through keep a small number of break-glass credentials, stored somewhere that does not itself depend on the same login, with usage that is logged and reviewed.

Runbooks live where the outage is

A runbook stored only in a cloud wiki is a runbook that is available exactly when you do not need it most. The same applies to architecture diagrams, escalation contact lists, vendor support phone numbers and account identifiers. These are small documents. There is no good reason for them to have a single hosting dependency.

Detection has the same problem

Response tooling gaps compound detection gaps that are already widespread. Survey work published in the 2026 State of Production Reliability and AI Adoption report, drawn from more than 1,000 SRE, DevOps and IT operations professionals, found that 78% of organizations had experienced at least one incident where no alert fired and customers discovered the problem first, and 44% had incidents tied to suppressed or ignored alerts. The same research puts roughly 40% of engineering time into incident management rather than product work, and 34% of organizations report downtime costs above $100,000 per hour. If detection already leaks that badly, a degraded response path is not a small additional risk.

Map your response path as a dependency graph

Before you buy anything or change anything, spend an hour writing down the actual chain of systems a responder touches between the alert firing and the customer-facing update going out. Most teams have never drawn this, and the exercise is usually uncomfortable in a productive way.

For each step, record the vendor, the authentication path, and what a responder can still do if that step is unavailable:

  • Signal: the monitoring or observability system that generates the alert, and the webhook or integration path it uses to reach your on-call tool.
  • Routing: the on-call schedule, escalation policy and the notification channels, including SMS and voice carriers, which are their own third parties.
  • Acknowledgement: where the responder confirms they have it, and whether anyone else can tell that acknowledgement happened.
  • Coordination: the chat workspace, the incident channel, and the video bridge if you use one.
  • Instruction: runbooks, dashboards, query consoles and the cloud provider console itself.
  • Action: deployment tooling, feature flags, infrastructure automation and the source control host they depend on.
  • Communication: the status page, the customer support tool and any internal stakeholder broadcast.
  • Record: the timeline capture, the ticket, and where the postmortem will be written.

Now mark every place the same vendor appears more than once, and every place the same identity provider appears. Those are your correlated failure points. In most organizations the graph collapses to three or four vendors doing almost everything, which is a fact worth knowing before an incident rather than during one.

Design for degradation, not for perfect redundancy

Full redundancy across your response toolchain is neither affordable nor sensible. Two paging vendors running in parallel means two sets of schedules to keep in sync, two integration surfaces to maintain, and a reliable source of confusion about which one is authoritative. The realistic goal is graceful degradation: for each capability, know the fallback, make sure the fallback is reachable without the failed system, and make sure people have used it at least once.

Keep an offline copy of the small critical documents

Export the on-call roster, the escalation matrix, the top ten runbooks, vendor support contacts with account numbers, and a one-page incident command checklist. Store them somewhere that does not share a failure domain with your wiki. A generated PDF in a shared drive on a different provider, refreshed weekly by a scheduled job, is enough. The test is simple: can a responder on a phone, with the wiki down, reach the escalation path in under two minutes?

Establish a named fallback coordination channel

Decide in advance where the team gathers if the primary chat workspace is unavailable, and make sure everyone knows the answer without looking it up. A standing bridge line, a secondary workspace, or a group SMS thread all work. What does not work is deciding this at 02:00 while people are already scattered. Write it on the same one-page checklist as the escalation matrix.

Separate paging from the rest of the stack

Of every capability in the graph, paging is the one worth deliberately decoupling. Detection can be slow and you will survive. Documentation can be stale and you will survive. But if the alert never reaches a human, nothing else in the chain matters. Ideally your paging path does not share a vendor with your ticketing system, and your notification channels include at least one that does not depend on data connectivity, which in practice means voice or SMS as a genuine last resort rather than a checkbox.

Practice publishing a status update by hand

Status pages are frequently hosted by the same vendor as the ticketing suite, or gated behind the same login. Know how to publish a customer-facing update without the automated path: who has credentials, what the fallback channel is, and what the first message says. Draft that first message now, as a template, so nobody is writing prose under pressure.

Reduce the number of steps that require a login

Every authenticated hop is a place the response can stall. Alerts that carry enough context to start diagnosis, dashboards with shareable read-only links, and runbook steps written so a responder knows what they are looking for before they log in all shorten the path. This is the same reason mature teams put the relevant graph directly in the alert rather than a link to a dashboard.

Run a tooling blackout game day

The exercise that finds these problems is cheap and takes ninety minutes. Pick a normal business hour, gather the on-call rotation, and declare a scenario: a moderate production incident is underway, and one named response tool is unavailable. Do not simulate the production incident in production. Simulate the response.

Run it three times over a quarter with a different tool removed each round:

  • Round one, the wiki is down. Can responders find the runbook, the escalation path and the vendor support number? Time it.
  • Round two, chat is down. Where does the team assemble, how does the incident commander establish who is doing what, and how does a status update reach customers?
  • Round three, the identity provider is degraded. Who can still reach the cloud console, the deployment pipeline and the status page publisher? Does break-glass access work, and does using it generate an audit record?

Capture the failures in the same tracker you use for real postmortem actions, and give them the same priority. The output is usually three or four small, boring fixes: a document that needs a second home, a phone number nobody had, a break-glass account that expired. Boring fixes are the point.

Where a Slack-native approach helps, and where it does not

Running on-call and incident response inside Slack or Microsoft Teams removes a meaningful number of steps from the graph above. Paging, acknowledgement, incident channel creation, timeline capture and stakeholder updates happen in the tool responders already have open, which means fewer logins, fewer context switches and fewer places for the response to stall. It also means the escalation policy and the schedule are visible to the whole team rather than buried in a console that only the on-call lead opens.

This is the design principle behind Pagerly: keep the coordination surface where the humans already are, so that the response path has fewer independent systems in it. Fewer hops is a real reliability property, not a convenience feature.

It would be dishonest to stop there, though. Consolidating into chat creates its own coupling, and a chat outage is a real scenario with real precedent. The right posture is the same one this whole post argues for: know the dependency, and have a fallback for the capability that matters most. Concretely, that means notification paths that do not require the chat client to be working, an on-call roster that is readable outside chat, and a documented answer to the question of where the team gathers when chat is unavailable. Any vendor that tells you their tool has no failure mode is selling something. The useful question is whether the failure mode is small, known and rehearsed.

What to do this week

None of this requires a budget cycle or a platform migration. It requires an hour of honesty about what your team actually depends on, followed by a handful of small changes.

  • Draw the response dependency graph from alert to customer update, and circle every vendor that appears more than once.
  • Export the small critical documents to a second home on a different provider, and set a weekly job to refresh them.
  • Name the fallback coordination channel and tell everyone, in the channel they read, before they need it.
  • Check that paging does not share a vendor with your ticketing and documentation stack, and that at least one notification path does not need data connectivity.
  • Verify break-glass access works today, and confirm using it produces an audit trail.
  • Schedule one tooling blackout game day this quarter, and treat its findings like postmortem actions.

The August 27 outages will get their postmortems, and those postmortems will be about cooling loops and control planes. They will be worth reading. But the lesson that transfers to your team is not about anyone else's chilled water. It is that a response plan which quietly assumes every tool will be available is not a plan, it is a hope. Redundancy that fails together, whether that is two air compressors serviced in the same maintenance window or four capabilities riding the same control plane, is not redundancy at all.

Find the shared roots in your own response path. Write down what a responder can still do without each one. Then go make sure somebody has actually tried it.

Sources referenced in this post: Proton's August 27 outage incident report, the Atlassian status page, Data Center Dynamics coverage of the Frankfurt cooling failure, the us-central1 post-mortem analysis, and the 2026 State of Production Reliability and AI Adoption report.