On August 4, 2026, Elastic Security Labs identified a self-propagating worm called CHAINDROP moving through the npm registry after attackers compromised the maintainer account behind keyv, a key-value library that received over 600 million downloads in the prior month. Within days the campaign had backdoored more than 400 packages across the maintainer's publish surface. The payload was a credential harvester: npm tokens, GitHub personal access tokens, cloud provider keys for AWS, GCP and Azure, Kubernetes service account tokens, HashiCorp Vault tokens and SSH private keys. For most engineering organisations, the first person who had to do something about it was not a security analyst on a dedicated rotation. It was whoever happened to be holding the pager.

That is the shift worth naming clearly. Security incident on-call used to be a separate discipline with its own roster, its own tooling and its own escalation path, largely invisible to the SRE holding production. In 2026 it increasingly arrives through the same channel as a latency alert, gets triaged by the same tired engineer at the same awkward hour, and competes for the same attention as a failing deploy. Most on-call programmes were never designed for that. This post covers what breaks when a compromise pages the reliability rotation, and how to structure severity, escalation, containment and communication so that the first hour is deliberate rather than improvised.

Why security incidents now land on the reliability rotation

There are three structural reasons this keeps happening, and none of them are going away.

Your dependency tree is production

A compromised package with a preinstall hook is not a theoretical risk sitting in a scanner backlog. It executes with the privileges of whatever ran the install, which in practice means a CI runner with deploy credentials or a developer laptop with a cloud session token. The blast radius of that execution is measured in the same units reliability engineers already use: which services, which environments, which regions. When Elastic published its response guidance, the action list was rotate GitHub tokens, rotate npm automation tokens, rotate cloud credentials, rotate Kubernetes configs, rotate Vault tokens and CI secrets. Every one of those actions has a production availability consequence. Rotating a credential that half your services read at boot is an outage waiting for a restart.

So the work splits awkwardly. The detection and the analysis belong to security. The execution belongs to the people who know which service reads which secret and what happens when it changes. Those people are on the on-call rotation.

The clock runs differently

Reliability incidents have a natural end state: the error rate returns to baseline and you go back to bed. Security incidents do not resolve, they get contained, then investigated, then remediated, and the investigation can run for days. An on-call shift is typically 8 to 24 hours. A credential compromise investigation is not. If your rotation has no explicit mechanism for handing off an open, unresolved, multi-day incident, the person who caught the page ends up carrying it far past the end of their shift, which is exactly the pattern that produces burnout. Industry survey work in 2026 consistently puts on-call stress at the centre of SRE attrition, and open-ended incidents are the worst version of it.

Detection is now noisy in both directions

Security tooling generates alerts at volumes that make reliability alerting look restrained. When a supply chain campaign breaks, every scanner in the stack fires at once, most of it duplicative, some of it about packages you do not actually ship. Routing that raw feed to the production pager is the fastest way to teach an engineer to ignore the pager. Routing none of it means the one alert that mattered sits in a queue nobody reads until Monday. Neither default is acceptable, and the middle path requires deliberate design.

What a security page actually looks like on the reliability rotation

It helps to be concrete, because "security incident" covers wildly different operational shapes. Three of them show up on the production pager regularly.

The upstream compromise

Something you depend on has been backdoored. CHAINDROP is the current example, but the pattern is a decade old and recurs several times a year. Nothing in your system is broken. Your dashboards are green. The work is entirely investigative: determine whether the affected versions ever entered your build, whether they executed, and what they could have reached. Your monitoring will not help you, because there is no anomaly to detect. Your lockfiles, your build logs and your artifact registry will.

The credential exposure

A token has leaked, through a compromised laptop, a public repository, a log line or a third-party breach notification. The response is rotation under time pressure, and rotation is where security response and reliability response collide hardest. Rotating fast risks an outage. Rotating slowly leaves an attacker with valid access. Somebody has to make that call, and it is usually made at 3am by a person with incomplete information about which consumers of the secret will survive the change.

The active intrusion

Somebody is in the environment now. This is the rarest shape and the only one where the correct action may be to deliberately degrade or take down a healthy service. It inverts every instinct the on-call engineer has been trained on. An SRE spends their career keeping things up. Containment sometimes means taking things down, quickly, without a change advisory board, and being right about it.

The severity model breaks first

Most severity scales are written entirely in availability terms. SEV1 is a full outage, SEV2 is major degradation, SEV3 is partial or single-tenant impact. Run a credential compromise through that scale and it scores as a SEV4 or does not score at all, because customer-facing error rates are zero. Meanwhile the actual urgency is high and the actual business risk is severe. The scale produces the wrong answer, so people override it informally, and once people are overriding the scale informally you no longer have a severity model, you have vibes.

A severity scale that covers both

The fix is not to bolt a separate security scale alongside the reliability one, because then every incident starts with an argument about which scale applies. The fix is to make impact multi-dimensional and let the highest dimension set the severity. Score every incident on three axes:

Availability impact: what proportion of requests, users or regions are affected right now.
Data and access impact: what an attacker could read, write or reach with what they currently hold, assuming the worst credible case rather than the confirmed case.
Time sensitivity: whether the impact grows while you deliberate. A worm propagating through packages you publish grows. A historical log exposure does not.

Take the maximum across the three axes. A confirmed leak of a production database credential is a SEV1 even with a perfectly flat error rate, because the data and access axis says so. This single change removes most of the argument from the first ten minutes of a security page, and the first ten minutes are the ones you cannot get back.

Dual-track response: contain and keep serving

The most useful structural idea to borrow from mature security operations is that containment and service continuity are separate workstreams with separate owners, running in parallel, coordinated by one incident commander who owns neither.

In a reliability incident these two things are the same activity. You fix the thing and service returns. In a security incident they are frequently in direct opposition. Blocking an attacker's access path may break a legitimate integration. Rotating a signing key may invalidate sessions for every customer. Taking a compromised build runner offline may stop all deploys, including the deploy that ships the fix. If one person is trying to hold both goals, they will oscillate, and oscillation under time pressure produces the worst decisions.

Who decides to break production on purpose

Name this authority in advance and write it down. The question "are we allowed to take this down?" should never be asked for the first time during an incident. In practice a workable rule is that the incident commander can authorise containment actions with a defined availability cost up to a stated threshold, and anything beyond that threshold requires a named executive who is explicitly on an escalation path with a response time commitment. The threshold matters less than the fact that it exists and everyone knows the number.

The corollary matters too. If the on-call engineer is not authorised to make containment decisions, then the escalation path to whoever is must be as fast and as reliable as the page that woke them up. An escalation policy that ends in "email the security team" is not an escalation policy. It is a delay with paperwork.

A practical runbook for the first hour

Here is a structure that survives contact with an actual 3am page. Adapt the timings, keep the sequence.

Minutes 0 to 10: confirm and declare

Establish that the alert is real and that it concerns your environment specifically, not the industry generally. A vendor advisory is not an incident. A vendor advisory plus a matching entry in your lockfile is an incident. Then declare it, in a dedicated channel, with a severity from the three-axis scale and a named commander. Declaring early and downgrading later is dramatically cheaper than the reverse, because the expensive part of an incident is assembling people, and people assemble slowly.

Resist the urge to start investigating before declaring. The single most common failure mode here is an engineer quietly digging for forty minutes, finding something serious, and only then pulling people in, at which point the response starts forty minutes late with a tired lead who has already formed a hypothesis.

Minutes 10 to 30: scope the blast radius

Answer, in this order: did the affected artifact or credential enter our environment, did it execute or get used, and what could it reach. These are three different questions and teams routinely conflate the first two. A compromised package version present in a lockfile that was never installed on a runner is a very different incident from one that ran a preinstall hook on a machine holding deploy keys.

The tooling for this is unglamorous and should be prepared in advance: a query that greps every lockfile across every repository for a package and version range, access to build logs going back far enough to matter, and an inventory of which credentials live on which runners. Every minute you spend building that query during an incident is a minute the attacker keeps their access.

Minutes 30 to 60: contain, rotate, communicate

Containment first, then rotation, then evidence preservation, and communication running continuously alongside all three. Rotate in dependency order, starting with credentials that grant the ability to obtain other credentials. A token that can mint tokens outranks a token that can read one bucket. Rotating the bucket credential first while the minting credential stays valid accomplishes nothing except making you feel busy.

Keep a running timeline in the incident channel as you go. Not a summary written afterwards from memory, an actual timestamped log of what was observed, what was decided and by whom. Security incidents attract retrospective scrutiny from auditors, customers and sometimes regulators, and the timeline you reconstruct three days later from Slack scrollback will be worse than the one you wrote as it happened.

Communication is harder than the fix

Reliability incidents have a well-worn communication pattern: post to the status page, update every 30 minutes, publish a postmortem. Security incidents break every part of that pattern.

You often cannot say what happened, because the investigation is open and premature claims are worse than silence. You may have legal or contractual notification obligations with clocks that started before you knew there was an incident. Your customers may need to take action on their own systems, which means your communication is not informational, it is operational. And internally, the population who needs to know is different from a reliability incident: legal, support leadership and sometimes the executive team need to be looped in early, not at the end.

The practical mitigation is to pre-write the templates. Have a holding statement for "we are investigating a potential security issue" that legal has already approved. Have a customer notification template with the fields blank. Have a decision tree for who gets told at what confidence level. Writing any of this during an incident guarantees it is written badly by someone who should be doing something else.

Five structural changes to make before the next page

None of these are expensive. All of them are much cheaper before an incident than during one.

Add a data and access axis to your severity scale so a zero-error-rate compromise can legitimately be a SEV1, and take the maximum across axes rather than averaging them.
Write down containment authority with an explicit availability cost threshold and a named escalation target above it, reachable by page rather than by email.
Build the blast radius queries now. Lockfile search across all repositories, build log retention that reaches back far enough, and a credential-to-runner inventory. Test them on a package you actually use.
Define handoff for open incidents. A security incident that outlives a shift needs a documented state transfer, not a hopeful message at 8am. Include current hypothesis, actions taken, actions pending and who has been told what.
Separate the security alert feed from the production pager with an explicit triage rule. High-confidence, environment-specific findings page. Everything else goes to a queue with a named owner and a stated response time.

Where a Slack-native on-call tool fits

The reason this class of incident is so often handled badly is not that engineers lack skill. It is that the response spans tools. Detection sits in a security platform, escalation sits in a paging tool, coordination happens in chat, remediation happens in a terminal, and the record of what was decided is scattered across all four. Every context switch is a place where a decision goes unrecorded and a handoff drops.

Running on-call inside Slack, which is where the incident conversation is happening anyway, collapses most of that. The person who is on call is visible in the channel rather than looked up in another tab. Escalation to the security lead or the executive with containment authority happens in the thread where the question was asked, so the answer lands in the timeline automatically. Overrides and handoffs, which matter enormously for incidents that outlive a shift, are a message rather than a login. And the incident record assembles itself as a byproduct of the conversation rather than as a documentation task nobody wants at 6am. Pagerly is built around exactly that: rotations, escalations, overrides and incident timelines that live where the work already happens.

That is not a claim that tooling solves this. The severity model, the containment authority and the pre-written communications are organisational work that no product performs for you. But given that those decisions have been made, the difference between them being followed and being forgotten is usually whether following them requires opening another tab at 3am.

The takeaway

The boundary between security response and reliability response has been eroding for years, and campaigns like CHAINDROP make the erosion impossible to ignore. Over 400 compromised packages, more than a billion monthly downloads in the blast radius, and a remediation list consisting almost entirely of credential rotations that only the production on-call engineer knows how to execute safely.

Treating security incident on-call as somebody else's rotation is no longer a viable position. The page is going to arrive on the reliability pager because that is where the people with the necessary access and context already are. What you control is whether that engineer has a severity scale that gives the right answer, a written containment authority, prepared blast radius queries, a real handoff process and an escalation path that actually reaches a decision-maker. Build those five things on a quiet week. The next supply chain campaign will not schedule itself around your sprint.