Cron job monitoring is the part of observability most teams skip until a backup turns out to have stopped running three weeks ago. A web server that falls over generates errors, trips a threshold and pages someone inside a minute. A nightly job that quietly stops running generates nothing at all. There is no error rate to alert on, no latency spike, no failed health check. There is only an absence, and absence is invisible to almost every monitoring setup by default.

That asymmetry is why scheduled jobs produce a disproportionate share of the worst incidents: the backup that was never taken, the invoice run that skipped a billing cycle, the data export a customer has been waiting on since Tuesday. This guide covers why cron jobs fail silently, the dead man's switch pattern that catches them, what to measure beyond "did it run", and how to route the resulting alerts so they reach whoever is actually on call.

Why Cron Jobs Fail Silently

Conventional monitoring is built around observing bad signals. A request returns 500, a queue depth climbs, a disk fills up. Something happens, and that something is measurable. Scheduled jobs invert the problem. The failure mode you care about most is that nothing happened, and nothing is hard to observe if you are not deliberately looking for it.

Absence Of A Signal Is Not A Signal

If your monitoring only reacts to events your job emits, then a job that never starts is perfectly healthy as far as your monitoring is concerned. The dashboard stays green. The error rate stays at zero, because zero runs produce zero errors. Teams frequently discover this the hard way, when the metric that looked reassuring for weeks turns out to have been measuring nothing at all.

The Failure Modes That Actually Bite

In practice, scheduled jobs break in a small number of recurring ways, and it is worth knowing them by name:

The exit zero lie. A shell script runs five commands. The third fails. Without set -e, the script continues and exits zero, reporting success while having done most of its work badly or not at all. Pipelines are worse: without set -o pipefail, the exit status of a pipeline is the status of the last command, so a failing dump piped into a successful gzip looks like a clean run.
Partial completion. The job starts, processes forty percent of the rows, hits a malformed record and dies. It ran, it logged, it even produced output. The output is just incomplete, and nothing downstream knows it.
Silent overrun and overlap. A job scheduled hourly starts taking seventy minutes. Cron does not care. It launches the next instance anyway, and now two copies compete over the same rows or the same file. Data corruption from overlapping cron runs is common and frequently misdiagnosed.
Environment drift. Cron runs with a minimal environment. A different PATH, no shell profile, no virtualenv, no credentials that were exported in someone's interactive session. The job that works perfectly when you run it by hand fails under cron for reasons that have nothing to do with the job's logic.
The host disappeared. The instance was replaced during a rolling deploy, or scaled in, or someone moved the workload and the crontab did not come with it. The job was not failing. It simply no longer exists anywhere.
The MAILTO black hole. Cron's default failure notification is email to the local user. On most modern hosts that mail goes nowhere, or into an inbox nobody has opened since the person who set it up left the company.

The Dead Man's Switch Pattern

The fix for silent failure is to invert the monitoring. Instead of asking your job to tell you when it fails, you require it to tell you when it succeeds, and you alert when that confirmation does not arrive. This is the dead man's switch, sometimes called heartbeat monitoring or inverted monitoring, and it is the single highest value change you can make to cron job monitoring.

Mechanically it is simple. You register a check with an expected interval. At the end of a successful run, your job makes an HTTP request to a unique URL. If the monitoring system does not receive that request within the expected window plus a grace period, it raises an alert.

The important property is that this catches every failure mode in the list above, including the ones no in-process error handler can catch. It catches the job that crashed before its own error handling loaded. It catches the host that vanished. It catches the crontab entry someone commented out during debugging and forgot to restore. It catches a network partition between the job and everything else. If the confirmation does not arrive, you hear about it, and you do not need to have anticipated the specific reason.

Choosing A Grace Period

The grace period is where most implementations go wrong in one of two directions. Set it too tight and you get paged every time a job runs slightly long, which trains everyone to ignore the alert. Set it too loose and a daily backup can be broken for the better part of a day before anyone is told.

A reasonable default is to measure the job's actual runtime for a couple of weeks, take the ninety fifth percentile, and set the grace period to roughly double that, with a floor of a few minutes. A job that normally finishes in ninety seconds and occasionally takes four minutes gets a ten minute grace period, not an hour. Revisit it when the job's workload grows, because runtimes creep.

What To Monitor Beyond "Did It Run"

A heartbeat at the end of the script tells you the script reached the end. That is necessary but not sufficient. Three further signals separate teams who know their jobs are healthy from teams who merely know their jobs are running.

Duration And Overrun

Signal the start of the run as well as the finish, and you get duration for free. Duration is an early warning system. A nightly aggregation that has crept from four minutes to thirty eight minutes over two quarters is going to collide with something eventually, and the graph tells you months before the collision. Alert on duration exceeding a threshold, and separately on any run that is still executing when the next one is scheduled to start.

Exit Code Versus Semantic Success

Report the exit code explicitly rather than only signalling on success. A job that exits non zero should fail its check immediately rather than waiting for the grace period to lapse, because you already know it failed and there is no reason to wait.

Beyond the exit code, consider whether the job did something meaningful. A database dump that exits zero and produces a file of zero bytes is a failure that every exit code check in the world will call a success. So is a sync job that processed no records because an upstream credential silently started returning an empty result set. Assert on the outcome, not just the process:

Backups: file exists, size is within an expected band, and a test restore runs on a schedule. An untested backup is a hypothesis, not a backup.
Data pipelines: row counts land within a plausible range of the previous run, and drop to zero is treated as a failure rather than a quiet Tuesday.
Report generation: the output timestamp is fresh, not a stale file left over from the last successful run.
Cleanup jobs: the thing that was supposed to shrink actually shrank.

The Renewal And Dependency Chain

Many cron jobs exist to keep something else alive. Certificate renewal, token refresh, cache warming, index rebuilds. For these, monitor the downstream state directly in addition to the job itself. Check that the certificate on the live endpoint has a comfortable number of days remaining, not merely that the renewal script exited zero. The script can succeed at doing the wrong thing.

Instrumenting A Cron Job Step By Step

Here is a practical wrapper pattern that covers start, finish, duration and exit code without requiring you to modify the job's own code.

Step 1: Make The Script Fail Loudly

Start every job script with strict mode. This one line eliminates the exit zero lie:

set -euo pipefail

That is: exit on any command failure, treat unset variables as errors, and make a pipeline fail if any stage fails rather than only the last one.

Step 2: Signal Start And Finish Separately

Wrap the job so that it pings a start endpoint, runs, then pings a finish endpoint carrying the exit code. A minimal version looks like this:

#!/usr/bin/env bash
set -uo pipefail
HEARTBEAT="https://your-monitor.example/ping/abc123"
curl -fsS -m 10 --retry 3 "$HEARTBEAT/start" || true
/opt/jobs/nightly-export.sh
CODE=$?
curl -fsS -m 10 --retry 3 "$HEARTBEAT/$CODE" || true
exit $CODE

Note the || true on the curl calls. A monitoring outage should never cause your actual job to fail. Note also the timeout and retries: a heartbeat call that hangs forever is its own kind of incident.

Step 3: Prevent Overlapping Runs

Use flock so a slow run cannot be lapped by the next one:

flock -n /var/lock/nightly-export.lock /opt/jobs/wrapper.sh

With -n, if the lock is held, the new invocation exits immediately rather than queueing. Decide deliberately whether a skipped run should alert. Usually it should, because it means the previous run is overrunning.

Step 4: Capture Output Somewhere Real

Redirect stdout and stderr to a log file or your logging pipeline rather than relying on cron mail. When the alert fires at two in the morning, the responder's first question is what the job printed, and the answer should be one click away rather than a hunt across hosts.

Step 5: Register The Check With Its Schedule

Record the expected schedule in the monitoring system, not only in the crontab. The monitor needs to know a job is expected hourly in order to notice that it has not checked in for ninety minutes. Keep the two in sync, ideally by defining both in the same configuration or infrastructure code.

Deciding What Deserves A Page At Three In The Morning

Instrumenting everything is correct. Paging for everything is not. The value of cron job monitoring collapses the moment responders learn that the alerts are usually noise.

The useful question is not how important the job is in the abstract, but what changes if it waits until morning. Two factors decide this:

Time to the next run. A job that runs every five minutes and fails once will self heal before anyone reaches a laptop. Alert after several consecutive failures, not the first. A job that runs once daily has no second chance today, so a failure has a full day of consequences attached.
Downstream dependency. A nightly aggregation that feeds a report the executive team opens at eight in the morning has a hard deadline. A cleanup job that reclaims disk space has a soft one, right up until the disk is at ninety five percent, at which point it does not.

A workable tiering for most teams: page immediately for jobs where failure means data loss, money not moving or a missed regulatory deadline. Everything else opens a ticket in a channel the team triages during working hours. Review the tiering quarterly, because jobs change importance without anyone updating the alert.

Routing Cron Alerts To Whoever Is Actually On Call

A cron alert is only as useful as its delivery. The most common failure here is not technical at all: the alert fires correctly, lands in a channel or an inbox, and nobody specific owns it. Shared responsibility for an alert reliably becomes nobody's responsibility.

Three properties make the difference. The alert should name the current on call person rather than a group, so there is no diffusion of responsibility. It should arrive where the team already works rather than in a separate console someone has to remember to check. And it should escalate if it is not acknowledged, because the first notification will sometimes be missed no matter how good the routing is.

This is the part Pagerly's uptime monitoring and paging are built for. Monitors run against your endpoints with the timeouts you choose, and alerts arrive in Slack or Discord tagged to the person currently on the rotation rather than to a channel at large, with escalation to email, SMS, phone call or the mobile app if nobody acknowledges. Because the rotation itself lives in Slack through round robin rotations, the routing follows shift swaps automatically rather than pointing at whoever was on call when the integration was first configured.

Common Mistakes Worth Avoiding

Monitoring the scheduler instead of the job. Confirming that the cron daemon is running tells you almost nothing about whether your jobs are doing their work.
One heartbeat for a whole batch. If a single wrapper runs six jobs and pings once at the end, you learn that the sixth job finished. The first five are unmonitored.
Grace periods copied across every check. A five minute grace period on a job that runs monthly is meaningless, and an hour on a job that runs every five minutes hides a lot of failures.
Alerting to email only. Cron's native mail path is the least reliable notification channel in common use.
Never testing the alert. Comment out a job deliberately and confirm someone hears about it. An untested alert path fails exactly when you need it.
No runbook attached. "Nightly export did not check in" is not actionable at three in the morning without a link explaining where the logs are and whether a manual rerun is safe.
Forgetting idempotency. If the standard remediation is to rerun the job, the job has to be safe to rerun. Many are not, and nobody finds out until the first incident.

A Practical Rollout

You do not need to instrument every scheduled job this week. A staged approach gets most of the value quickly:

Inventory first. Collect every crontab, scheduled task, CI cron trigger and managed scheduler rule across your estate. Most teams are surprised by the count, and by how many jobs nobody present can explain.
Rank by blast radius. Sort by what breaks if the job silently stops for a week. Backups, billing, data retention and anything with a compliance deadline go to the top.
Instrument the top ten. Add the wrapper, register the checks, set grace periods from observed runtimes.
Tier the alerts. Decide explicitly which of the ten page overnight and which wait for morning.
Write the runbooks. One short page per paging alert: what it means, where the logs are, whether a rerun is safe, who to escalate to.
Test the path. Break one deliberately in a controlled way and confirm the right person hears about it.
Delete the dead ones. Every inventory turns up jobs that no longer serve a purpose. Removing them is faster than monitoring them.

The Takeaway

Cron job monitoring is not really about cron. It is about a category of failure where the symptom is silence, and silence does not trigger anything unless you build something that expects a signal and complains when it does not arrive. The dead man's switch handles that inversion. Duration tracking and outcome assertions catch the runs that technically succeeded while accomplishing nothing useful. Sensible tiering keeps the resulting alerts credible.

The teams that get burned by scheduled jobs are rarely the ones whose jobs fail. Every team's jobs fail. They are the ones who found out weeks later, from a customer, or from a restore that had nothing to restore from. Start with the inventory, instrument the jobs where silence would be expensive, and make sure the alert reaches a named human rather than a channel.

Want cron and endpoint alerts to reach whoever is on call right now? Pagerly runs your monitors and your rotation in the same place, inside Slack, so alerts follow the schedule instead of a stale integration setting. Get started free.