Opsgenie shuts down April 2027 - migrate to Pagerly in one click
PagerlyPagerly
← All postsEngineering

Your Error Budget Is Spent by Your Dependencies

One vendor outage can burn a year of error budget. Measure dependency risk, set honest SLOs, and run on call around the gap.

Pagerly cover image: Your Error Budget Is Spent by Your Dependencies

Your error budget is probably being spent by systems you do not own. On August 17, 2026, GitHub was degraded for 7 hours and 47 minutes. Authentication, Actions, the API, pull requests, issues and Copilot were all affected. If you run a 99.9 percent availability target for a service that cannot deploy without GitHub Actions, that single afternoon consumed roughly 88 percent of the downtime you are allowed for the entire year. You did not push anything. You did not misconfigure anything. Your budget is gone anyway.

That is the uncomfortable arithmetic most teams never actually run. We write SLOs as if our own code and our own infrastructure are the only sources of failure, then we act surprised when a shared dependency quietly drains the whole allowance in one sitting. This post is about running that arithmetic properly, and then reshaping your on call practice around the answer.

The error budget math nobody actually runs

An error budget is the inverse of an availability target. If you promise 99.9 percent, you are also promising yourself that you may be unavailable for 0.1 percent of the period. That sounds abstract until you convert it into wall clock time, which is where the discomfort starts.

What each availability target actually buys you

Over a 365 day year, the allowances look like this:

  • 99 percent (two nines): about 3 days and 15 hours of downtime per year, or roughly 7 hours per month.
  • 99.9 percent (three nines): about 8 hours and 46 minutes per year, or roughly 43 minutes per month.
  • 99.95 percent: about 4 hours and 23 minutes per year, or roughly 22 minutes per month.
  • 99.99 percent (four nines): about 52 minutes per year, or roughly 4 minutes per month.

Read that middle line again. Three nines, the target most mid sized engineering organisations casually claim in a status page footer or a customer contract, gives you under nine hours for the whole year. A single long incident of the kind that happened to GitHub in August, or to Cloudflare's R2 and Workers estate earlier the same month, does not dent that budget. It empties it.

One incident, one year of budget

The specific numbers matter here because they make the point better than any abstraction. GitHub's own retrospective describes an incident that began when traffic hit a new peak and a critical infrastructure component in the Central US data centre failed to scale with it. Capacity pressure spread, authentication started failing, and multiple services degraded together. Recovery took staged traffic rerouting and infrastructure isolation. Most services came back the same afternoon, but some Copilot services lagged, and errors in those services triggered a client side retry loop that increased traffic during recovery and had to be mitigated before traffic could safely be restored.

Total customer impact: 7 hours and 47 minutes. That is 467 minutes. Against a three nines annual budget of 526 minutes, one incident took 88 percent. Against a 99.95 percent budget of 263 minutes, it took 177 percent. You were over budget for the year by lunchtime, on a day when your own code was blameless.

Now add the second incident. GitHub's post notes this was its second significant August incident, following an Actions failure on August 6 with impact measured in hours rather than minutes. Two events, both attributed to capacity rather than to any code or configuration change, and any three nines commitment built on top of that dependency is arithmetically dead for the year.

Why borrowed outages break error budget thinking

Error budgets were designed as a negotiation device between engineering velocity and reliability. You burn budget, you slow down feature work and spend the difference on hardening. That loop works beautifully when the burn comes from your own deploys. It works badly when the burn comes from somebody else's data centre, because the usual remediation lever, shipping less, does not buy you anything.

Capacity failures do not announce themselves

The most instructive line in GitHub's retrospective is the admission that neither August incident was caused by a code or configuration change. Both were capacity failures at their core: critical components were not scaled before demand exceeded their capacity. GitHub cites monthly commits growing from 1.4 billion in April to 2.9 billion in August as the pressure behind it.

This is a very different failure class from the one most incident response is drilled for. A change related outage has an obvious first hypothesis and an obvious first mitigation: find the change, revert the change. A capacity failure has neither. There is no bad deploy in the timeline, the graphs simply bend, and the responder's instinct to look for the culprit commit wastes the first twenty minutes. Worse, capacity failures tend to arrive at the exact moment your traffic is highest, which is also the moment the business cares most.

Retry behaviour is part of your budget too

The second half of the GitHub story is the part every engineering team should take personally. A latent retry loop on the client side amplified traffic during recovery and had to be neutralised before service could be safely restored. GitHub's stated remediation is to apply consistent retry limits, retry budgets and variable timeouts across service to service interactions to prevent retry storms and cascading load.

Your services are somebody else's clients. Every unbounded retry loop you have written, every SDK default you never audited, every cron that hammers a vendor API until it gets a 200, is a contribution to the next amplification event. When a dependency slows down, aggressive retries convert a partial degradation into a total one, and they do it on your side of the boundary. Auditing retry policy is one of the very few concrete things you can do to protect a budget you do not control.

Failure is moving up the stack

Industry reporting through 2026 points the same direction. Power remains the single most common root cause of impactful infrastructure outages, but facility level failures are gradually declining as a share while DNS, BGP, submarine cable and control plane events take their place. The practical consequence for on call teams is that more of your outage minutes will originate outside your perimeter, in systems where you have no telemetry, no runbook and no ability to act. Planning for a world where you cause most of your own downtime is planning for a world that no longer exists.

How to build a dependency error budget

The fix is not to abandon error budgets. It is to make them honest by accounting for the dependencies that can stop your product. Here is a sequence that works without a large platform team.

Step one: inventory what can actually stop you

Write down every external service whose failure produces customer visible impact or blocks your ability to ship a fix. Be ruthless about the second category, because it is the one teams forget. If your CI provider is down you cannot deploy the mitigation for an unrelated incident, which means your CI provider is inside your incident response critical path even though no customer touches it.

For each entry, capture four things: what breaks when it fails, whether a degraded mode exists, how you find out, and who on your team can make the call to switch to the degraded mode. Most teams discover that the fourth column is empty for almost every row. That gap, not the vendor's uptime, is what actually determines your recovery time.

Step two: tier your dependencies and budget them separately

Not everything deserves the same treatment. A workable three tier split:

  • Tier one, hard dependencies: failure stops customer traffic. Identity providers, payment processors, primary cloud region, CDN. These need a documented degraded mode and a rehearsed decision to enter it.
  • Tier two, delivery dependencies: failure stops you shipping or observing. CI, artifact registries, observability vendors, feature flag services. These need a documented manual bypass, including the emergency path to deploy without the usual pipeline.
  • Tier three, soft dependencies: failure degrades a feature but not the product. Analytics, non critical enrichment APIs, marketing tooling. These need a circuit breaker and nothing more.

Then set a separate budget per tier. Your composite availability cannot exceed the availability of your tier one dependencies multiplied together, so if you have four hard dependencies each genuinely running at 99.95 percent, your ceiling is about 99.8 percent before you write a single line of your own code. Publishing a three nines target above that stack is not ambition, it is a maths error, and it is the kind of maths error that turns into a contractual problem.

Step three: measure real burn, not vendor claims

Vendor status pages are marketing surfaces with an engineering veneer. They lag, they scope narrowly, and they define impact in ways that flatter the operator. Measure the dependency yourself from where you consume it: synthetic checks on the specific operations you depend on, error rate and latency on your own client side calls, and a durable record of every minute you spent degraded because of somebody else.

Keep that record in a form you can total up at quarter end. When you can say "we lost 512 minutes this quarter and 71 percent of them originated in three external services", you have moved the conversation from anecdote to evidence. That is the input that justifies building the fallback, negotiating the contract, or choosing a second provider. Without it, dependency risk stays a hallway complaint.

On call practice for outages you cannot fix

Budget accounting tells you where the minutes go. Response practice determines how many minutes each event costs. For third party incidents the levers are different from the ones in a standard runbook, because remediation is off the table and everything else matters more.

Detect it before your customers explain it to you

The worst version of a vendor outage is the one you learn about from a customer, because by then you have already spent the first thirty minutes looking in the wrong place. Two cheap mitigations: alert on your own client side error rate and latency per dependency rather than only on aggregate service health, and pull vendor status feeds into the same channel where your alerts land so that correlation is visual and immediate.

The signal you want is not "our checkout is failing". It is "our checkout is failing and the payment provider's error rate went up 90 seconds earlier". The second version routes itself.

Decide ownership in the first five minutes

Vendor incidents create an ownership vacuum. It is not the database team's problem, it is not the platform team's problem, and so it becomes everybody's problem, which means nobody drives it. Name a single incident commander regardless of whether the root cause is yours, and make that explicit in the escalation policy. The commander's job during a borrowed outage is not repair. It is four things: confirm blast radius, decide on degraded mode, own external communication, and keep a timeline.

Make the degraded mode decision a named person's call with a documented threshold, not a committee discussion. "If the provider is failing more than 25 percent of requests for more than 10 minutes, the on call engineer switches to the queue and drain path without further approval" is a sentence that saves an hour. Its absence is why so many vendor incidents have a twenty minute technical impact and a two hour customer impact.

Communicate while you have nothing to report

During a dependency outage your only real deliverable is information. Customers tolerate downtime far better than silence, and internal stakeholders who are not being updated will generate their own load by asking. Post early, post a scope and a next update time even when you have no ETA, and keep posting on that cadence. "We are affected by an upstream provider incident, checkout is degraded, next update in 20 minutes" is a complete and useful message.

Write the postmortem anyway

Teams routinely skip the retrospective when the cause was external, on the grounds that there is nothing to fix. This is exactly backwards. You cannot fix the vendor, but you can fix detection lag, the missing degraded mode, the unbounded retry, the escalation that went to a person on holiday, and the status page that nobody updated for 40 minutes. Those are all yours, and they are usually responsible for most of the impact minutes.

A good structure for a borrowed outage postmortem asks: when did the dependency start failing, when did we know, when did we tell customers, when did we enter degraded mode, and what was the gap between each of those. Every gap is a defect you own.

Where a Slack native on call tool changes the numbers

Most of the minutes lost in a third party incident are coordination minutes, not engineering minutes. Somebody has to be found, context has to be assembled, a decision has to be made, and people have to be told. If that work happens across a paging tool, a separate incident tool, a status page dashboard and then Slack, each handoff adds delay to an event where delay is the only cost you control.

This is the practical case for keeping on call and incident response inside the place where your engineers already are. With Pagerly, the on call rotation, the escalation, the incident channel, the timeline and the status page updates all live in Slack. When a vendor starts failing at 2am, the alert lands in the channel, the current on call engineer is resolvable with a mention rather than a lookup in another tool, and the escalation path runs without anybody having to remember who covers the weekend. The timeline is being written as a side effect of the conversation, which means the postmortem exists before anybody sits down to write it.

None of that makes the vendor come back faster. It compresses the parts you own: time to acknowledge, time to assemble, time to decide, time to communicate. On a 467 minute upstream outage, shaving 30 minutes off your own coordination overhead is a real reduction in customer impact and a real reduction in error budget burn.

A checklist to run this week

None of the following requires budget approval or a quarter of planning. Pick the ones you cannot already answer.

  • Convert your target to minutes. Write your annual error budget in wall clock minutes on the same page as your SLO. Nine hours reads differently from 99.9 percent.
  • Total your last quarter's burn by origin. Split it into your code, your infrastructure and somebody else's. The ratio will probably surprise you.
  • List your tier one and tier two dependencies. For each, name the person who can declare degraded mode and the threshold that triggers it.
  • Audit retry policy on every outbound call. Cap attempts, add jitter and exponential backoff, add a retry budget, and set timeouts that are shorter than your customer's patience. This is the single highest leverage change available to you.
  • Add a client side alert per hard dependency. Alert on the operations you actually call, not on the vendor's status page colour.
  • Test the emergency deploy path. If your CI provider were down right now, could you ship a one line fix to production? Find out on a Tuesday afternoon rather than during an incident.
  • Rehearse one vendor outage. Thirty minutes of game day with a named commander and a simulated provider failure exposes more gaps than a week of documentation review.
  • Recheck your published availability claim. Multiply your hard dependency availabilities together. If the product is below your promise, fix the promise or the architecture.

The takeaway

Reliability targets written as if you are the only possible cause of failure will keep being falsified by afternoons like August 17. The honest version of an error budget accounts for the dependencies that can take you down, tiers them by consequence, measures the burn from where you consume them, and treats coordination speed as the one variable you can still move once somebody else's data centre is the problem.

GitHub's response to its own August incidents is instructive precisely because it is unglamorous: more capacity, better alerting on lower priority resource signals, isolation of critical systems, removal of shared dependencies, and consistent retry limits. Scaled down, that is a decent template for everyone else. Know what you depend on, remove the shared paths you can, bound your retries, alert on the things that actually predict failure, and make sure that when the pager goes off for something you cannot fix, the humans still find each other in under two minutes.

Sources and further reading: GitHub's retrospective on the August 17 outage, the root cause analysis on GitHub Status, the earlier August 6 Actions incident, and the original treatment of error budgets in Embracing Risk from the Google SRE book.