A datacenter cooling failure used to be one of the more forgiving physical incidents an on-call engineer could catch. Cooling dies, thermal mass in the room absorbs the heat, and you have a few hours to get someone on site before anything actually breaks. That assumption is now wrong, and the August 27, 2026 Proton outage is the clearest public evidence yet. According to Proton's own incident report, the room temperature in its Frankfurt facility went from a nominal 21.8 degrees Celsius to 51.9 degrees Celsius in under half an hour, with some probes reading 60 degrees. Servers began dying one at a time. As Proton put it, what used to take three to four hours to go critical went critical in twenty minutes.

That single sentence is the most important reliability finding published this month, and it has nothing to do with software. Rising server power density, driven largely by higher wattage CPUs and GPUs bought for AI workloads, has quietly compressed the physical response window in every facility running modern hardware. If your escalation policy, your failover runbook, or your vendor communication path was designed around hours, it was designed around a number that no longer holds. This post walks through what happened, why the physics changed, the three decisions Proton's on-call engineers had to make under extreme pressure, and what your team should change before you face the same night.

What Actually Happened in the Frankfurt Datacenter

The timeline is worth reading closely, because it is a textbook example of a failure mode that most incident response playbooks handle badly.

Just after 11 p.m. Central European Time on Wednesday, August 26, the cooling system in the main room of Proton's Frankfurt datacenter failed completely. By roughly 11:15 p.m., temperatures started climbing. Within thirty minutes the room had passed 51 degrees Celsius. Server and networking equipment started failing individually rather than all at once, which matters enormously for how automated failover behaves.

The user-facing incident began around midnight on August 27, when the accumulating hardware failures finally crossed a threshold: both the primary and the backup network switch on one critical rack died. That rack happened to contain several primary database copies. Proton notes that almost all of its systems fail over automatically and immediately, but primary database failovers are deliberately kept under human supervision to avoid split brain scenarios where replicas silently de-sync and become painful to reconcile later.

The Maintenance Nobody Was Told About

The root cause investigation on August 27 traced the cooling failure to an air filter replacement performed on both of the redundant air compressors powering the cooling system. Both. At the same time. In the middle of the night. Proton reports that the datacenter operator carried out the work without prior notice and did not communicate the resulting cooling failure when it happened, which cut down the time available to respond.

This is the part that should make every reliability engineer uncomfortable, because it is not unique to one provider or one facility. Your redundancy calculations assume independent failure of redundant components. A maintenance procedure applied simultaneously to both halves of a redundant pair converts N plus one redundancy into a single point of failure, and it does so invisibly, on someone else's change calendar, with no entry in your own change log.

Why Redundancy Did Not Save the Room

Proton had redundant cooling. It had redundant network switches. It had database replicas in a second facility in Zurich. It had enough capacity to survive a complete datacenter failure. None of that prevented a user-facing outage, and the reason is instructive.

Redundancy protects against the failure modes you modeled. Proton's failover logic was built for a clean, total site loss: Frankfurt goes dark, traffic shifts to Zurich, done. What it got instead was a slow, partial, correlated decay in which random servers died one by one over the course of an hour while the rest of the room kept serving traffic. As the report states plainly, a situation where random servers are dying one by one is not handled well by their failover logic. That is an honest sentence, and it describes the failover logic at most companies.

Why a Datacenter Cooling Failure Now Goes Critical in Minutes

The compression from hours to minutes is not a Proton-specific problem. It is a consequence of an industry-wide change in what sits inside a rack.

The Thermal Math Changed Under Everyone

A room full of servers is a heat engine with a fixed cooling budget. When cooling stops, the rate at which air temperature climbs is roughly proportional to the power the equipment is dissipating divided by the thermal mass of the room. A decade ago a typical rack might draw somewhere in the range of five to ten kilowatts. Modern racks packed with high wattage CPUs and accelerators can draw many multiples of that. The thermal mass of the room did not change. The heat being poured into it did.

The result is exactly what Proton measured: a temperature curve steep enough to cross the point of equipment damage in twenty minutes rather than three or four hours. Network cards in the Frankfurt infrastructure reached 105 degrees Celsius against a normal operating temperature of 45 degrees, which triggered a protection mode that disabled them until a cold system reset. Note that this is a failure that survives the fix. Restoring cooling did not restore the hardware, because the cards stayed latched off.

Your Incident Response Window Is a Physical Constant

Most on-call programs treat response time as a policy choice. You pick a five minute acknowledgement target, a fifteen minute escalation, a thirty minute engagement of the secondary. Those numbers are usually chosen for human sustainability and for the blast radius of software failures.

Physical infrastructure does not negotiate. If your facility crosses the damage threshold twenty minutes after cooling stops, then a fifteen minute escalation ladder plus a five minute page acknowledgement plus the time to reach a human at the datacenter operator adds up to a response that arrives after the equipment is already dead. The window is not a target you set. It is a property of the building and the hardware inside it, and you should measure it rather than assume it.

Three Decisions Under Extreme Pressure

Proton's report is unusually candid about the decisions its on-call engineers faced around midnight. There were three, and each one had no good answer.

First: save the service or save the hardware? Restoring service would mean leaving servers powered on, drawing power, and generating heat in a room that was already past 51 degrees. Saving the hardware meant powering off as many machines as possible and extending the outage. Proton chose the hardware, and the reason is worth noting: a server equipment shortage tied to the AI boom meant that lost equipment could not be replaced on short timelines. The right call was driven by supply chain reality, not by an SLO.

Second: fail over locally or remotely? Failing over primary databases to replicas inside Frankfurt was faster and less disruptive, but risked promoting a replica that was about to die of the same heat. Failing over to Zurich was safer but rerouted traffic flows and risked introducing new instability. Proton eventually chose a hybrid: Frankfurt where a replica was still alive, Zurich where none was.

Third: fail over everything or only what is currently down? Full failover is well rehearsed and mostly automatic. Partial failover, driven by which specific machines happened to die in the last five minutes, is a manual judgment call repeated dozens of times while the ground keeps shifting.

Decide the Tradeoffs Before the Night They Matter

Every one of those three questions could have been answered in advance, in daylight, by people who were not simultaneously reading temperature probes and phoning a datacenter operator. That is the practical lesson. An incident response plan that only tells you how to execute a failover is incomplete. It also needs to tell you when a failover is the wrong move, and who is allowed to decide that.

Write down, ahead of time, the answer to questions like these:

Under what conditions do we accept a longer outage to protect physical assets? Who has the authority to make that call at 00:15, and does the on-call engineer know it is their call to make?
When a site is degrading rather than dead, do we fail over to the same site or a different one? What signal tells us the local replica is unsafe?
What is our threshold for converting a partial failover into a full site evacuation, and can we execute that in one command rather than fifty?
Who at the facility operator can be reached at 3 a.m., through what channel, and is that channel tested?
If out-of-band management access requires additional staff or credentials, who are they and how fast can they be woken?

That last one is not hypothetical. Proton's security posture limits access to out-of-band controllers, which meant recovery required waking additional staff. That is a defensible security decision and probably the right one. It is also a recovery dependency, and recovery dependencies belong in the runbook with names and phone numbers attached, not discovered during the incident.

Failover Logic Assumes Clean Failures

The deeper technical lesson here is about the shape of the failure, not the cause. Almost all automated failover is designed around a binary: the thing is up, or the thing is down. Health checks return green or red. Circuit breakers open or closed. Site evacuation triggers when a region is declared lost.

What Frankfurt produced was neither. It was an hour of gray failure in which the site was mostly healthy, degrading unpredictably, and getting worse in a way that was correlated across every rack in the room but expressed as uncorrelated individual machine deaths. Automation built for the binary case handles this badly in a specific way: it keeps making locally reasonable decisions that are globally wrong. Promote this replica. Now promote that one. Route around this switch. The system spends its budget on a series of small corrections instead of the one large correction that would have actually helped.

Detect the Correlation, Not Just the Symptoms

If you take one engineering action from this incident, make it this: build a signal that answers the question are these individual failures correlated by location? Most alerting stacks page per service, per host, or per check. Very few group by rack, room, power feed, or facility. When six unrelated services degrade in twenty minutes and the only thing they share is a physical location, that shared location is the incident, and no amount of per-service triage will find it.

Concretely, that means tagging hosts with physical topology metadata (facility, room, rack, power domain) and having your alerting layer surface that grouping to the responder. It also means ingesting environmental telemetry into the same pipeline as everything else. Room temperature, inlet temperature, and network card temperature are as much a production signal as p99 latency, and on the night they matter they are the leading indicator while everything else is lagging.

What On-Call Teams Should Change This Quarter

Here is the practical checklist. None of these require a datacenter of your own; most apply equally if you rent capacity from a colocation provider or run in a cloud region with a single availability zone footprint.

Measure your actual thermal window. Ask your facility operator, or your colocation account manager, how long the room stays under the damage threshold with cooling fully offline at current power draw. If the answer is a decade old estimate, it is wrong. If nobody knows, that is your finding.
Alert on the rate of change, not just the threshold. A room at 30 degrees is fine. A room going from 22 to 30 degrees in four minutes is an emergency, and a static threshold alert will not fire for another ten minutes you cannot spare.
Put physical topology in your alert metadata. Rack, room, power domain, facility. Then build one view that groups active alerts by that topology.
Add your vendors' maintenance windows to your change calendar. If a provider is touching redundant infrastructure, that is a change to your system even though you did not make it. Ask for notice in your contract and treat unannounced work as an incident finding.
Rehearse partial failure, not just total failure. Your game day probably kills a region. Try killing seven percent of hosts in one rack over twenty minutes and see whether your automation helps or thrashes.
Document the decision authority, not just the procedure. Name who can trade uptime for hardware, who can declare a site evacuation, and how the on-call engineer reaches them in under three minutes.
Test the out-of-band path. Whatever your recovery mechanism is when the primary access path is dead, exercise it quarterly. Security controls that block recovery are a reliability risk that needs an explicit, rehearsed exception process.
Plan for the abnormal steady state after recovery. Proton restored service by 01:30 but spent the rest of the night and the next day with primaries split across two cities and reduced redundancy. Recovery is not the end of the incident. It is the start of a different one.

Vendor Maintenance Belongs on Your Change Calendar

There is a persistent gap in how most teams think about change management. We instrument our own deploys heavily. We correlate incidents with our own releases. We require change tickets, approvals, and rollback plans for our own infrastructure. And then a third party performs simultaneous maintenance on both halves of a redundant physical system at 11 p.m. with no notification, and it appears nowhere in any of that machinery.

You cannot control a provider's maintenance practices, but you can make the invisible visible. Push for contractual notice requirements on any work that touches redundant components. Log vendor maintenance windows into the same timeline your responders look at during an incident, so that "did anything change?" has a complete answer rather than a partial one. And when notice does not arrive, record it as a finding in your postmortem with the same weight you would give an internal process failure, because the impact is identical.

Where Coordination Actually Breaks

Read the Proton timeline again and count the coordination load. On-call engineers were simultaneously talking to on-site datacenter operations about restoring cooling, deciding which servers to power off, evaluating which database replicas were still alive, choosing failover targets per database, waking additional staff for out-of-band access, and publishing status updates to users. That is at least four parallel workstreams under a hard deadline set by a rising temperature curve.

This is the part of incident response that tooling either helps with or actively obstructs. When the coordination surface is a separate incident tool, a separate video call, a separate status page editor, and a chat channel that someone has to remember to create, every context switch costs time drawn from a twenty minute budget. When the coordination happens where the team already is, the cost drops.

That is the design bet behind a Slack-native approach like Pagerly. Paging, acknowledgement, escalation, the incident channel, the responder roster, and the running timeline all live in the place the engineers are already typing. Nobody logs into a second system at 00:15 to find out who owns the database rotation, and the timeline of who decided what and when is captured as a byproduct of the conversation rather than reconstructed afterward from memory. For an incident like this one, where the postmortem hinges on which tradeoffs were made and why, having that record write itself is worth more than any dashboard.

Escalation Design for Compressed Windows

If your physical response window is genuinely twenty minutes, the escalation ladder needs to reflect that. A few concrete adjustments:

For environmental and facility alerts specifically, skip the gradual ladder. Page primary and secondary simultaneously, and notify the incident commander rotation immediately rather than after a timeout.
Attach the facility operator's emergency contact directly to the alert payload so nobody searches a wiki for it.
Pre-authorize the destructive action. If powering off half a rack is the correct response, the on-call engineer should not need to find a director at midnight to approve it.

The Honest Takeaway

Proton's report ends by noting that the database resilience work needed to address this failure mode was already underway, planned for completion by the end of the year, and that the incident simply arrived before the improvements did. That is the most relatable sentence in the entire document. Every engineering organization has a known weakness with a funded fix and a date, and the only question is whether the failure shows up before or after the fix lands.

What you can do in the meantime is make sure your response is calibrated to the real physics rather than the physics of ten years ago. The three to four hour cushion that made cooling failures a manageable class of incident is gone in any facility running dense modern hardware, and it went away quietly, without an announcement, as a side effect of buying faster chips. Nobody sent a change notification for that either.

Go find out what your actual window is. Then go look at whether your escalation policy, your failover automation, and your vendor communication path can all complete inside it. If they cannot, you now know what to fix, and you know it on a Saturday afternoon rather than at 00:15 while watching a temperature probe climb.