AI-Generated Code Incidents Are Breaking On-Call

Category
Falit Jain
August 7, 2026
5 min read
AI-Generated Code Incidents Are Breaking On-Call
Table of Content

The story of 2026 engineering has a plot twist that on-call teams saw coming. AI coding assistants now write a large share of production code, pull requests per developer are up, and delivery velocity looks great on the dashboard. Then the pager goes off. Fresh industry data shows a clear pattern: AI-generated code incidents are climbing even as review scores and merge counts improve, and the people carrying that weight are the on-call engineers who have to debug code they did not write, at 3am, under pressure. This post breaks down what the 2026 reports actually found, why AI-assisted delivery quietly shifts risk onto incident response, and the concrete practices that keep on-call sane when a machine is generating half your changes.

If you run reliability for a team that has leaned into AI coding tools, the payoff here is simple. You will get a clear read on the numbers, a mental model for where the new failure modes live, and a practical checklist for ownership, runbooks, alerting, and post-incident learning that holds up when AI is in the loop.

The 2026 data: faster code, more production incidents

Several independent 2026 reports landed on the same uncomfortable finding. Speed went up. Stability went down. The gap between the two is now measurable, and it shows up first in production.

New Relic's State of AI Coding 2026 captured the paradox in one line: AI-generated code often grades higher in code review, yet it triggers a rise in production incidents once it ships. Reporting on the same research, VentureBeat noted that 43% of AI-generated code changes need debugging in production, even after passing QA or staging. Read that again. Nearly half of AI-authored changes surface a problem only after they are live, which is exactly where on-call lives.

The delivery-metrics picture is just as pointed. Harness, in its State of Modernization 2026 research, reported that main branch success rates dropped to roughly 70.8%, a five-year low and well under the 90% benchmark most teams target. That means close to 3 in 10 merges are failing somewhere in the pipeline. The Perforce State of DevOps 2026 report added that for every 25 percentage points of AI code adoption, teams saw a meaningful decline in change-failure-rate stability.

The incident-level detail is where it gets real for reliability teams. Across the 2026 surveys, the numbers cluster in the same direction:

  • More rollbacks and hotfixes. For the heaviest AI tool users, around 22% of deployments end in a rollback, hotfix, or customer-impacting incident, compared with roughly 15% for occasional users.
  • More incidents per unit of work. Pull requests per developer rose about 20% with AI assistance, but incidents per pull request rose faster, by roughly 23.5%.
  • Longer recovery. Mean time to recovery for deployment-related incidents ran longer for very frequent AI users (about 7.6 hours) than for occasional users (about 6.3 hours).
  • More senior-engineer firefighting. A large majority of organizations reported an increase in the time senior staff spend fixing code and intervening in emergencies.
  • A trust gap. Nearly two-thirds of engineering leaders admitted their teams often ship AI-generated code without line-by-line verification.

Put together, the trend is not that AI writes bad code. It is that AI changes the shape of risk. More code moves through the pipeline, more of it looks fine in review, and a larger slice of the defects only reveal themselves in production. The cost did not disappear. It moved downstream to the on-call rotation.

Why AI-assisted delivery quietly shifts risk to on-call

To respond well, it helps to understand the mechanism. AI coding tools do not fail randomly. They fail in specific, repeatable ways that happen to be hard to catch before production and hard to debug once there.

The code is unfamiliar to the person paging on it

The oldest rule in incident response is that the fastest fix comes from the person who understands the system. AI breaks that assumption. A model can generate a working-looking change in seconds, but no human built a mental model of it along the way. When it breaks at 2am, the on-call engineer is reading the code for the first time, during the incident, with the clock running. The 2026 research described this as a shift in the ratio of code in production to code the team understands well enough to debug under pressure. That ratio moving the wrong way is a reliability problem, not a productivity one.

Review catches style, production catches context

AI-generated code tends to be clean, idiomatic, and well structured, which is exactly why it grades well in review. Reviewers see tidy functions and reasonable naming and approve. What review does not reliably catch is missing context: an unhandled edge case in a downstream service, a wrong assumption about data shape, a retry that hammers a dependency, a config value that is fine in staging and catastrophic in production. Those are integration and environment failures, and they show up when real traffic hits.

Volume outpaces integration testing

When each developer ships more changes, the pipeline sees more merges competing for the same integration windows. The 2026 data linked longer recovery times to engineers merging more code without adequate integration testing. More concurrent change means more surface area for interactions nobody tested together, and more ambiguity about which of the last twenty deploys actually caused the alert.

The blast radius is wider and less obvious

A human author usually knows which services a change touches. An AI-assisted change spread across several files can quietly alter a shared utility, a serialization format, or an API contract. When it breaks, the symptom can appear far from the cause. That distance between symptom and root cause is precisely what drives up mean time to recovery.

What good on-call looks like in the age of AI code

None of this is an argument against AI coding tools. Velocity is real and valuable. The argument is that reliability practices have to catch up to the new failure modes. Teams that stay calm through this shift tend to do the same handful of things well.

Make ownership explicit, including for AI-authored code

Every service and every critical path needs a clear owner and a clear escalation path, and that has to include code a model generated. "The AI wrote it" is not an owner. Someone accountable should be able to explain the intent of a change and where to look when it misbehaves. Codeowners, service catalogs, and up-to-date on-call schedules are the plumbing that turns a mystery alert into a known escalation. When the page fires, the first question is never "whose is this?" It is already answered.

Write runbooks for symptoms, not just services

Because AI-related failures often surface far from their cause, runbooks organized purely by service can leave the on-call engineer stranded. Complement them with symptom-first runbooks: "checkout latency is spiking, here are the four most common upstreams and the exact queries to run." Symptom-first guidance shortens the gap between the alert and the first useful action, which is where most of the recoverable time in an incident actually sits.

Tune alerts so humans still trust the pager

More deploys and more automated checks can mean more noise. If half your pages are non-actionable, engineers start ignoring the pager, and the one real incident gets lost. Route alerts to the team that owns the failing path, set sane thresholds, and make sure every page maps to something a human can do. Alert quality is a reliability control, not a nicety. A rotation that trusts its pager recovers faster than one drowning in noise.

Keep a human in the loop before code ships, not only after

The trust gap in the data is a leading indicator. Teams shipping AI code without verification are borrowing time from their future on-call rotation. That does not mean line-by-line review of everything. It means proportionate scrutiny: heavier verification and integration testing on high-blast-radius paths, canary or progressive rollouts so a bad AI change hits 1% of traffic instead of 100%, and fast, well-practiced rollback. The cheapest incident is the one a canary caught before it went wide.

Treat every incident as a learning event

Google's SRE guidance on blameless postmortems matters more, not less, in the AI era. When an AI-generated change causes an incident, the goal is not to blame the tool or the engineer who accepted the suggestion. It is to understand what context was missing, what the pipeline let through, and what guardrail would have caught it. Over time those learnings become the review checklist, the canary rule, and the runbook entry that stop the next one.

A practical checklist for reliability teams

If you want to turn the trend into action this quarter, start here:

  • Audit ownership. Confirm every production service and critical path has a named owner and a current escalation path. Close the gaps first.
  • Add progressive delivery. Put canary or staged rollouts and one-click rollback in front of your highest-traffic services so bad changes fail small.
  • Raise integration testing on hot paths. Where AI adoption is highest, add integration and contract tests rather than relying on unit tests and review alone.
  • Write three symptom-first runbooks. Cover your top customer-facing symptoms with the exact dashboards and queries to check first.
  • Prune noisy alerts. Kill or reroute any page that fired in the last month without leading to action.
  • Run blameless postmortems. For every significant incident, capture the missing context and the guardrail that would have caught it, then actually ship that guardrail.
  • Track the right metrics. Watch change failure rate, MTTR, and incidents per deploy, not just deploy frequency and PR volume.

A common failure pattern, start to finish

Abstract numbers are easier to act on when you can see the shape of a real incident. Here is a composite scenario that matches the pattern the 2026 data describes, walked through the way it tends to unfold on a real rotation.

The change that looked fine

A developer asks an AI assistant to add a caching layer to a busy read endpoint. The suggestion is clean: it uses the existing cache client, adds a sensible TTL, and includes a tidy helper. Review takes four minutes. Two reviewers approve because there is nothing to object to on the surface. Unit tests pass. Staging looks healthy because staging traffic is light. The change merges into a main branch that, like the industry average, is already succeeding less often than it used to. Nobody built a mental model of how this cache behaves under production load, because a model wrote it and humans skimmed it.

The page nobody expected

Hours later, a traffic spike hits. The cache key the assistant generated does not account for a per-tenant parameter, so tenants start reading each other's cached responses. The first symptom is not "cache bug." It is a spike in support tickets and a customer-data alert two services away from the change. The on-call engineer, who did not write the code and has never seen this helper, starts from the symptom and has to work backward across service boundaries. This is the widened blast radius and the symptom-to-cause distance that push recovery time up. Without a symptom-first runbook and clear ownership, the first thirty minutes go to figuring out which of the day's many deploys is even responsible.

What would have changed the outcome

Several cheap guardrails would have shortened this incident dramatically: a canary rollout that exposed the cache bug to 1% of tenants, an integration test that exercised multi-tenant reads, a symptom-first runbook that pointed straight from "cross-tenant data" to recent caching changes, and an escalation path that put the owning team on the page in seconds. None of these are exotic. They are the difference between a ten-minute rollback and a multi-hour scramble, and they are exactly the investments the trend data argues for.

Where a Slack-native on-call tool fits

Most engineering teams already live in Slack, and incidents are messy, fast, and collaborative. When the tooling for on-call sits inside the place where people are already talking, the friction that slows recovery drops. This is the practical case for a Slack-native approach to on-call and incident management, and it is exactly the problem Pagerly is built to solve.

In the AI-code era, a few capabilities matter more than usual. Clear on-call schedules and one-tap overrides mean the right owner is reachable when an unfamiliar change breaks, without hunting through a separate app. Escalation policies that route a page to the team that actually owns the failing path cut the "whose is this?" delay that inflates recovery time. Bringing alerts, acknowledgement, and the incident conversation into one Slack channel keeps the timeline in a single place, which is a gift when the postmortem asks what happened and when. And because the response happens where the team collaborates, pulling in a second responder or a service owner is a mention away, not a context switch.

The point is not that tooling fixes AI-generated code. It does not. The point is that when a larger share of incidents comes from code no single human fully modeled, the speed and clarity of the human response becomes the deciding factor. Reducing the coordination tax on that response is one of the highest-leverage reliability investments a team can make right now.

The takeaway for engineering leaders

The 2026 reports are not a reason to abandon AI coding tools, and they are not a moral panic. They are a scheduling notice. AI has moved a real fraction of software risk from the authoring stage to the production stage, and production is on-call's home turf. Teams that acknowledge the shift and invest in ownership, progressive delivery, integration testing, alert quality, and blameless learning will keep their velocity and their sleep. Teams that treat rising incident rates as noise will keep paying for the speed in pages, burnout, and churn.

The core lesson from the data is durable regardless of which vendor's survey you read. Code review measures whether code looks right. Production measures whether it is right. As AI writes more of the code, the distance between those two truths is where AI-generated code incidents are born, and closing it is the reliability work of 2026. Get your ownership, runbooks, rollouts, and incident response in order, put the response where your team already works, and the pager becomes a manageable signal instead of a nightly tax.

View all
Design
Product
Software Engineering
Customer Success

Latest blogs

AI-Related Outages Are Reshaping On-Call in 2026
August 7, 2026

AI-Related Outages Are Reshaping On-Call in 2026

New reliability research shows AI-related outages now exceed one in ten incidents, and here is what on-call teams should do about it.
Incident Response Lessons From the Spotify Outage
August 6, 2026

Incident Response Lessons From the Spotify Outage

A vendor-neutral look at the August 5, 2026 Spotify outage and the incident response lessons every on-call team can apply.
AI Provider Outages: An On Call Playbook
August 6, 2026

AI Provider Outages: An On Call Playbook

A practical on call guide to staying reliable when an AI provider outage becomes your incident.