Visual fallback lifecycle

This guide shows what Epazote does over time: when a failed check advances the failure streak, when a fallback becomes eligible, when a failed command is tried again, how stop limits attempts, and how if_not.group prevents a restart storm.

The most important rule is:

A fallback is an action, not a health check

Running if_not.cmd or if_not.http does not prove that the service recovered. Only the next successful url or test check marks the service healthy and resets the outage state.

The examples use this configuration:

services:
  app:
    every: 30s
    timeout: 5s
    url: http://127.0.0.1:8080/health
    expect:
      status: 200
      if_not:
        threshold: 3
        stop: 2
        timeout: 45s
        group: app-host
        cmd: systemctl restart app
        http: http://alerts.internal/epazote

There are two different timeouts in that example:

Setting	Bounds
Service timeout: 5s	The health check itself
if_not.timeout: 45s	Each fallback action, and separately the queue wait for a grouped command

The whole decision path

Every configured service owns one asynchronous task. Different services can be checked at the same time, but one service never overlaps its own checks.

The failed check is recorded before Epazote waits for or runs the fallback. A slow restart therefore cannot leave epazote_status showing the previous healthy result for the duration of the command.

The counters describe an outage

An outage begins with the first consecutive failed check and ends only when a check succeeds.

State	Meaning	Reset by
Failure streak	Number of consecutive failed checks	A successful check
Used stop budget	Number of fallback attempts spent during this outage	A successful check
Exhaustion report flag	Prevents the same stop-limit error on every later scan	A successful check, or a fully refunded skip
epazote_fallback_executions_total	Process-lifetime Prometheus counter of recorded fallback outcomes	Never; it is a counter

threshold and stop answer different questions:

• threshold — when may fallback start? It counts failed checks.
• stop — how many fallback attempts may be spent? It counts action attempts after the threshold is reached.

They do not form a delay-and-retry loop inside one scan. Epazote performs at most one fallback attempt for one failed check.

threshold defaults to 1, so omitting it makes the first failed check eligible. if_not.timeout defaults to 300s. If stop is omitted, fallback attempts are unlimited for the duration of the outage.

Timeline: threshold: 3 and stop: 2

The following timeline assumes each check and fallback finishes before the next 30-second tick.

The same sequence as state:

Check	Health result	Failure streak	Used attempts	What happens
1	Failed	1	0	Below threshold
2	Failed	2	0	Below threshold
3	Failed	3	1	Run fallback attempt #1/2
4	Failed	4	2	Run fallback attempt #2/2; after it finishes, epazote_fallback_exhausted becomes 1
5	Failed	5	2	Refuse fallback and report exhaustion once at ERROR
6	Failed	6	2	Refuse fallback without repeating the exhaustion error
7	Healthy	0	0	End the outage and restore the complete budget

Notice that the fourth log says the streak is 4/3: the numerator keeps counting the real failed checks after the threshold has been crossed.

The exhausted gauge and the exhaustion log answer slightly different timing questions. The gauge becomes 1 at the end of the scan in which the final allowed attempt was spent. The error is emitted on the next failed check, when Epazote first tries and refuses to go beyond that limit. Below-threshold and execution-progress entries require -v; fallback failure and exhaustion entries are visible at the default ERROR level.

Once threshold is reached, every later failed check starts another fallback attempt until a check succeeds or the stop budget is spent.

When the fallback command fails

A non-zero if_not.cmd exit is a failed fallback attempt, not another health check. It spends one stop attempt, records outcome="failure", and waits for the service's next scheduled check before trying anything again.

This produces several rules that are easy to miss:

1. A failed fallback command does not increment the failure streak again. The streak advances once for the health check that triggered it.
2. A failed fallback does not retry immediately. The next failed health check creates the next attempt.
3. A successful fallback command does not set epazote_status to 1. Exit code 0 says the command succeeded on its own terms; only a healthy check proves the service is healthy.
4. A successful fallback does not reset threshold or stop. Only the next healthy check resets the outage.
5. A command that exits non-zero, cannot be spawned, or exceeds if_not.timeout spends its attempt. Refunding a broken command would let it retry forever despite stop.

test and if_not.cmd are different commands

Both are shell commands, but they have different jobs:

Command	Purpose	What a non-zero or unexpected exit means
Service test	The health check	The check failed; set status down and advance the failure streak
if_not.cmd	The fallback action	Recovery failed; record fallback failure and spend an attempt

For example:

services:
  nginx:
    every: 30s
    test: pgrep -x nginx
    expect:
      status: 0
      if_not:
        threshold: 2
        stop: 3
        cmd: systemctl restart nginx

If pgrep exits 1, that is one failed check. When the streak reaches two, systemctl runs. If systemctl then exits 1, that is one failed fallback, not a second failed check.

Scheduling while a fallback is running

The fallback is part of that service's scan. The same service cannot start its next check until its current fallback finishes, times out, or is skipped.

Epazote does not queue four catch-up checks and does not overlap them. The practical interval between checks is therefore at least every, but a slow check or fallback can make it much longer.

Other services keep checking in their own tasks. The only cross-service waiting is the waiting that you explicitly request by assigning commands to the same if_not.group.

How command serialization works

Without a group, fallback commands for different services run concurrently. With a group, only commands carrying the same group label take the same lock.

Assume orders and billing use group: database, billing also sends an HTTP alert, and frontend has no group. orders happens to reach the lock first:

The boundaries are exact:

Relationship	Can commands overlap?
Same non-empty group	No
Different groups	Yes
One grouped and one ungrouped	Yes
Both ungrouped	Yes
Command and any if_not.http action	Yes

Additional details:

• A group serializes only if_not.cmd. It never delays if_not.http.
• group without cmd is rejected at startup because it would protect nothing.
• Queue order follows lock-request arrival, not YAML order and not alphabetical service name.
• The lock exists inside one Epazote process. The same label in two separate Epazote processes does not coordinate them.
• Giving every fallback command the same group restores process-wide one-at-a-time behavior.
• Group only commands that share a script or contend for the same resource. Grouping unrelated commands adds delay without adding safety.

The two timeout phases of a grouped command

An ungrouped command has one timeout phase: run for at most if_not.timeout.

A grouped command has two independent phases:

The queue wait and command run do not share one deadline. If if_not.timeout is 45s, a grouped command can wait nearly 45s and then run for nearly another 45s.

With the default if_not.timeout of 300s, that is a worst case of nearly ten minutes without another check for that grouped service.

This prevents a dangerous alternative: starting a restart after a long wait with only a few seconds left, then killing it halfway through after it stopped the service but before it started it again.

Timeline: a grouped command times out in the queue

The skip does not reset B's failure streak. It merely ends that scan; the next failed check evaluates the threshold and budget again.

cmd and http run independently

When both actions are configured, Epazote starts both branches together and waits for both. A command failure never suppresses the alert, and a group queue never delays it.

The aggregate outcome uses this precedence:

failure  >  skipped  >  success

Command branch	HTTP branch	Metric outcome	stop attempt
Exit 0	Not configured or 2xx	success	Spent
Non-zero, spawn error, or timeout	Any result	failure	Spent
Skipped in group queue	Not configured	skipped	Refunded
Skipped in group queue	2xx	skipped	Spent
Skipped in group queue	Non-2xx or request error	failure	Spent
Not configured	2xx	success	Spent
Not configured	Non-2xx or request error	failure	Spent

Why does a skipped command plus a successful HTTP action spend stop? Something did execute: the HTTP action was attempted and accepted. Refunding the attempt would allow an alert on every failed scan forever while the command group remained busy.

Why is the metric still skipped in that case? The label explains what happened to the command: it never ran. The stop budget answers a different question: whether any configured action was attempted.

What is visible while fallback runs

The health result and fallback result become known at different times.

Metric	When it changes
epazote_status	As soon as the health result is known, before fallback
epazote_consecutive_failures	With that health result, before fallback
epazote_last_check_timestamp_seconds	With that health result, before fallback
epazote_fallback_configured	Initialized from configuration and remains 1 for a service with if_not, or 0 without it
epazote_fallback_executions_total	After all configured fallback branches finish
epazote_fallback_exhausted	Recomputed at the end of the scan from used attempts >= stop, after the fallback and any skipped-command refund
epazote_failures_total	For HTTP scan errors, when the failed scan returns to the service loop; never for a fallback failure

epazote_failures_total is deliberately separate from the fallback outcome:

• an HTTP request that cannot be made, or a response body that cannot be read, is a scan error; it still sets the service down, advances the streak, and follows the normal fallback path;
• a response that arrives but fails expect sets epazote_status to 0 without incrementing the scan-error counter;
• a service test that cannot be spawned or exceeds the service timeout is a failed check and follows the fallback path, but is not added to epazote_failures_total;
• a failed if_not.cmd or if_not.http increments the fallback outcome="failure" counter, not the scan-error counter.

Shutdown and timeout cleanup

Fallback commands run in their own operating-system process groups.

Killing the process group matters for commands that launch children. Stopping only the shell could leave sleep, systemctl, or a helper script running after Epazote exits.

SIGINT and SIGTERM are handled as an orderly supervisor shutdown, but active service tasks are cancelled immediately and an active fallback command group is sent SIGKILL. Epazote does not wait for a restart script to finish, so avoid restarting Epazote while a recovery action is running. SIGKILL sent to Epazote itself cannot be intercepted; when Epazote runs under systemd, its control group provides the additional process-lifetime boundary.

Common questions

Does a successful fallback mean the service is healthy?

No. It means the configured action reported success. The next health check must still pass.

Does a failed fallback cause an immediate retry?

No. There is one fallback attempt per failed health check. The next scheduled check must fail before another attempt is considered.

Does threshold: 3 run fallback every third failure?

No. It waits through failures 1 and 2. Failure 3 and every later consecutive failure are eligible until the service becomes healthy or stop is exhausted.

Does stop: 2 mean two failed checks?

No. It means at most two spent fallback attempts during one outage. Failed checks below threshold do not spend it, and a grouped command skipped with no HTTP action is refunded.

Does a command exit code 0 reset stop?

No. Only a successful health check resets the stop budget.

Does a group slow down every service?

No. It affects only commands with the same group label. Other groups and ungrouped commands remain concurrent, and HTTP actions never queue.

Is group serialization global across hosts?

No. It is local to one running Epazote process. Use an external distributed lock if separate Epazote instances must coordinate the same resource.

What should I alert on?

Use the metrics together:

• epazote_status == 0 — the last health result failed;
• epazote_consecutive_failures — the current outage length in checks;
• epazote_fallback_configured == 0 — the service has no automatic fallback action at all;
• epazote_fallback_executions_total{outcome="failure"} — configured actions are failing;
• epazote_fallback_executions_total{outcome="skipped"} — a command group is over-subscribed;
• epazote_fallback_exhausted == 1 — no stop-budget attempt remains;
• time() - epazote_last_check_timestamp_seconds — the service task has not completed a health check recently.

The timestamp series does not exist until the first check result is known. To also catch a service that never completed its first check, alert on its absence, for example:

absent(epazote_last_check_timestamp_seconds{service_name="app"})

See Reading the fallback metrics for Prometheus details and if_not for the complete configuration reference.