epazote.yml

Epazote uses a YAML configuration file. Below are the available options:

Strict validation

Unknown or misplaced keys make Epazote exit at startup with an error pointing at the offending field, instead of being silently ignored. For example max_size (instead of max_bytes), or max_bytes/method nested under expect: when they belong at the service level, are reported as unknown field. Invalid body/body_not regex patterns are also rejected at startup, so a bad config can never run. A service url — and an if_not.http — must parse as an http:// or https:// address with a host, which is checked at the same point. test and if_not.cmd must contain a non-whitespace character, and every if_not block must configure at least one of cmd or http.

The basic shape

Every service has the same high-level structure:

services:
  service-name:
    every: 30s
    url: http://127.0.0.1:8080/health
    expect:
      status: 200

Think of it like this:

• every: how often to check
• url or test: what to check
• expect: what counts as healthy
• if_not: what to do when the check is not healthy

services:
  example-service:
    every: 5m
    url: https://example.com
    method: GET
    follow_redirects: true
    timeout: 10s
    expect:
      status: 200

Durations

Every duration in the file — every, timeout and if_not.timeout — is a whole number followed by a unit. The unit is required; there are four:

Unit	Meaning	Example
s	seconds	30s
m	minutes	5m
h	hours	12h
d	days	2d

Fractions are rejected rather than rounded. 1.5m is a startup error, not 90 seconds: reading it as one minute or two would be a schedule you never wrote, and epazote would rather refuse the file than pick one for you.

For anything finer, use seconds

Seconds are the escape hatch from the coarser units. Write 90s instead of 1.5m, and 36h instead of 1.5d.

Seconds are also the floor. There is no millisecond unit, so 500ms is an error and 1s is the shortest duration epazote accepts — comfortably below anything a health probe over a network can resolve, and far below a useful every, since a sub-second scan interval mostly succeeds at hammering the service you are trying to watch.

Zero is rejected as well. 0s reads like "as fast as possible", but an interval of nothing cannot be scheduled and a timeout of nothing expires before the request it is meant to bound, so it is refused with the rest of the configuration rather than accepted and left to fail once the process is already starting.

Each mistake is reported for what it is, so the message tells you which rule you hit rather than blaming the part of the value that was correct:

services:
  app:
    url: https://example.com
    every: 1.5m
    expect:
      status: 200

$ epazote -c epazote.yml
Error: Failed to parse config file

Caused by:
    services.app: Duration must be a whole number: 1.5m. Use a smaller unit
    instead of a fraction, for example '90s' rather than '1.5m' at line 3 column 5

How checks are scheduled

Each service runs in its own task and checks one at a time. A check never starts while the previous one for that service is still running, so every is a floor, not a guarantee: the real interval is whichever is longer, every or how long the check actually takes.

With every: 1s and a check that takes 5s, you get a check roughly every 5s — not five overlapping ones. Ticks missed while a check was running are dropped rather than queued, so there is no burst of catch-up checks afterwards.

The same applies to if_not actions, which run as part of the check:

services:
  app:
    test: /usr/local/bin/health.sh
    every: 1s
    expect:
      status: 0
      if_not:
        cmd: systemctl restart app     # takes 200s

While that restart runs, this service performs no checks at all and its epazote_status, failure streak, and last-check timestamp already reflect the failed check that triggered it. They stay there until another check runs, so a service that recovered early still reads as down until Epazote verifies that recovery. Other services are unaffected; they have their own tasks.

Keeping the gap bounded

Two knobs control how long a service can go unchecked:

• if_not.timeout (default 300s) caps a single recovery. Lower it when a five-minute blind spot is too long, accepting that a slower restart is killed. A grouped command has a separate queue-wait phase with the same budget, so its worst case is nearly ten minutes: up to five minutes waiting, followed by up to five minutes running.
• threshold and stop stop a broken service from restarting back-to-back forever, each attempt blinding monitoring for its full duration.

Note that recovery actions for different services run concurrently by default, so two services whose fallbacks restart the same thing can still collide. Give them a shared if_not.group and those restarts serialise — they take turns instead of firing at once.

Choose the right matcher

Use the simplest matcher that fits your service:

• only the HTTP status matters: use expect.status
• the response is text or HTML: use expect.body
• the response must not contain text or a regex match: use expect.body_not
• the response is JSON: use expect.json
• you need shell logic or external tools: use test

Examples:

services:
  http_status_only:
    url: http://127.0.0.1:8080/health
    every: 30s
    expect:
      status: 200

  plain_text_check:
    url: http://127.0.0.1:8080/health
    every: 30s
    expect:
      status: 200
      body: ok

  forbidden_text_check:
    url: http://127.0.0.1:12345/metrics
    every: 30s
    expect:
      body_not: r"error|failure|Fatal"

  json_check:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success

  shell_check:
    test: pgrep -x nginx
    every: 30s
    expect:
      status: 0

test and if_not.cmd are executed with the current shell from the SHELL environment variable, falling back to sh if it is not set.

Changed in 4.2.0

A test or if_not.cmd that is empty or only whitespace is refused at start-up. The shell accepts such a command and exits 0, so a blank test previously reported the service UP without checking anything, and a blank if_not.cmd recorded outcome="success" on every attempt while repairing nothing. This most often came from a template variable that rendered empty.

Since Epazote automatically wraps commands with sh -c, you don't need to manually add it. For example:

# Both work the same way:
services:
  example:
    url: http://example.com
    every: 1m
    expect:
      status: 200
      if_not:
        cmd: echo "$(date) service failed" >> /tmp/epazote.log

# No need for sh -c wrapper:
# if_not:
#   cmd: sh -c "echo \"$(date) service failed\" >> /tmp/epazote.log"

For more complex scripts, the most reliable option is to make it executable and give it a shebang such as #!/usr/bin/env bash or #!/bin/sh.

every: Specifies how often the service is checked. Supports s (seconds), m (minutes), h (hours), and d (days).

url: The URL to check. (test can be used instead of url to check the exit status of a command.)

Changed in 4.2.0

The URL is validated when the configuration is read, and must be an http:// or https:// address with a host. A malformed value — or one written without a scheme, such as localhost:8080 — previously loaded without complaint and then failed on every check, before the failure was counted and before any if_not recovery ran. Such a service reported down permanently while epazote_consecutive_failures stayed at 0. It is now a start-up error naming the service and the key. The same applies to if_not.http.

method: The HTTP method to use when checking the URL. (default: GET)

INFO

You can use any of the following methods:

CONNECT
DELETE
GET
HEAD
OPTIONS
PATCH
POST
PUT
TRACE

follow_redirects: Follow HTTP redirects. (default: false)

max_bytes: Memory limit in plain bytes (no KB/MB suffixes) used while reading the response body. What it means depends on the matcher:

• body / body_not: the size of the sliding window used to scan the response. The entire body is always scanned no matter how large it is — the window only caps how much is held in memory at once. (default: 65536 bytes - 64KB)
• json: a hard limit. A JSON document must be parsed whole, so a response larger than this is truncated and the check fails; the log reports the truncation and the content-length. (default: 524288 bytes - 512KB)

You rarely need to set max_bytes

expect.body and expect.body_not find the expected text wherever it appears — a 5MB /metrics page works with the defaults. Reading stops as soon as every configured matcher has found its text, and is always bounded by the service timeout.

Set it explicitly only when:

• the response is JSON larger than 512KB: raise it to fit the document, e.g. max_bytes: 2097152 for responses up to 2MB
• a single body/body_not match could be longer than half the window (32KB with the default): matches spanning a window boundary longer than that may be missed

max_bytes: 0 disables body reading entirely, so body checks can never match.

timeout: The maximum time a single check may take. (default: 5s)

For url services this bounds the HTTP request and the body read. For test services it also bounds the command: a check command that runs longer is killed and counted as a failed check, so the normal if_not threshold/stop path runs.

WARNING

timeout bounds the check, not the recovery action. A systemctl restart run from if_not.cmd gets its own, far more generous budget — see if_not.timeout.

HTTPS certificate expiry

For any service whose url starts with https://, Epazote also reads the server certificate and exports the time left before it expires:

# HELP epazote_ssl_cert_expiry_seconds Number of seconds until SSL certificate expiration
# TYPE epazote_ssl_cert_expiry_seconds gauge
epazote_ssl_cert_expiry_seconds{service_name="example"} 5184000

Nothing needs to be configured. The certificate is fetched on the first check and re-read every 12 hours, so a service checked every: 10s does not open a TLS connection on every scan — the exported value is counted down between refreshes.

The connection and TLS handshake are bounded by the service timeout, so a host that accepts the connection and never completes the handshake cannot hold up the check. A failed check is remembered for a minute — timed from when the check finished, so a slow failure is not immediately retried — and a service that is down is therefore not re-probed ahead of every single HTTP check.

INFO

The certificate check never decides whether a service is healthy. If it fails — an unreachable host, a handshake error, an untrusted chain — Epazote logs a warning and continues with the normal HTTP check, so expect and if_not behave exactly as they would for an http:// service. Alert on epazote_ssl_cert_expiry_seconds to catch certificates approaching expiry.

Command-line options

Runtime behavior (which config to load, where to serve metrics, and how to log) is controlled with command-line flags. Every flag also has an environment variable, so the same settings can be provided through a systemd unit, container, or .env file. When both are set, the command-line flag wins; an empty environment variable falls back to the default.

Flag	Environment variable	Default	Description
-c, --config <FILE>	EPAZOTE_CONFIG	epazote.yml	Path to the configuration file.
-p, --port <PORT>	EPAZOTE_PORT	9080	Port the /metrics server listens on.
-b, --bind <ADDRESS>	EPAZOTE_BIND	[::]	Address the metrics server binds to. The default [::] listens on all interfaces (falling back to 0.0.0.0 when IPv6 is disabled); set 127.0.0.1 or ::1 to keep /metrics local-only. An explicit address is used as-is and never falls back.
-v, --verbose	EPAZOTE_VERBOSE	0	Increase verbosity. Repeat for more detail: -v info, -vv debug, -vvv trace.
--json-logs	EPAZOTE_JSON_LOGS	false	Emit structured JSON logs instead of the pretty human-readable format.

# equivalent invocations
$ epazote -c /etc/epazote/epazote.yml --bind 127.0.0.1 --port 9090 -vv

$ EPAZOTE_CONFIG=/etc/epazote/epazote.yml \
  EPAZOTE_BIND=127.0.0.1 \
  EPAZOTE_PORT=9090 \
  EPAZOTE_VERBOSE=2 \
  epazote

OpenTelemetry

OTLP tracing is opt-in and configured through the standard OTEL_* environment variables (for example OTEL_EXPORTER_OTLP_ENDPOINT). Leave them unset for the lightest runtime footprint.

Per-service context passed to fallback scripts uses a separate set of EPAZOTE_* variables documented under Environment variables for if_not.cmd.

Logging

By default, Epazote prints human-readable logs. If you prefer structured output, run it with --json-logs.

For HTTP checks in pretty mode:

• healthy checks are logged as compact INFO entries
• failed expectation checks are logged as WARN entries
• response headers are shown only for failed HTTP checks

Fallback action logging

The default level is ERROR. Epazote is normally run from the packaged systemd unit, which passes no verbosity flag, so only ERROR entries reach the journal unless you add -v. The fallback messages are split along that line deliberately: the ones that mean recovery did not work are ERROR, so they arrive without being asked for, and the routine progress of a fallback is WARN/INFO, behind -v.

Visible by default, at ERROR:

• The recovery command ran and failed — Fallback command for app ran but exited with code 7. The script was executed and reported failure on its own terms.
• The alert was refused — Fallback HTTP request for app was answered with status code 500. The request arrived and the endpoint rejected it, so nobody was notified.
• The fallback never completed — Fallback for service 'app' did not complete: fallback command skipped: waited the fallback 'timeout' of 2s for another fallback command in group 'db' to finish: …. The action could not be carried out at all: a grouped command that never got its turn (see if_not.group), a command that could not be spawned, or an alert endpoint that could not be reached.
• The service has been given up on — Service 'app' reached stop limit (2), skipping fallback. The stop budget is spent and Epazote will not run the fallback again for the rest of this outage. This is emitted once when the spent budget first refuses an attempt, rather than on every later failed check. It matters because epazote_status reads 0 whether a service is still being retried or has no automatic attempts left, so this line and epazote_fallback_exhausted are the only places that distinction exists.

Visible with -v, at WARN and INFO:

• Below threshold — Service 'app' failure count 1/3 below threshold, skipping fallback. Ordinary progress toward threshold; it fires on every failed check that has not reached it yet, which is why it is not raised to ERROR.
• Threshold reached — Service 'app' threshold reached (3/3), executing fallback (execution #1/2).
• The fallback succeeded — Executed fallback command for app with exit code 0, Executed fallback HTTP request for app with status code 200.
• Per-action failures — the individual Fallback command for app failed: … and Fallback HTTP request for app failed: … entries behind the single ERROR summary above.

"Error scanning service" is about the check, not the recovery

Error scanning service 'app': … means the check itself could not be completed — the request could not be made, or the response body could not be read. It is the log counterpart of epazote_failures_total.

A fallback that was skipped or that failed is reported under its own line above and never here, so the two can be read independently: this line tells you Epazote could not reach or parse the service, the fallback lines tell you what recovery did about it. A service that answers and merely fails its expectations does not produce this entry at all — it moves epazote_status to 0 and, if if_not is configured, produces the fallback lines only.

expect

expect defines expected responses from the service.

expect:
  status: 200
  body: "success"
  if_not:
    cmd: "sudo systemctl restart example-service"

• status: Expected HTTP status code. For HTTP checks, this is optional if another matcher such as body, body_not, or json is configured. When using test instead of url, status is required because Epazote checks the command exit status.

• body: Expected response body using a plain substring match by default, or a raw regex when prefixed with r"...".

• body_not: Forbidden response body using the same plain substring or r"..." raw regex matching as body. If the pattern is found, the check fails.

Regex anchors

In r"..." patterns, ^/$ (and \A/\z) refer to the start and end of the whole response body — they are matched against the true body start/end, not against the internal scan windows. Edge cases: in a mixed alternation like r"^foo|bar" the anchored branch may rarely match near a window boundary, and (?m) line anchors are approximated near window boundaries. Plain substring patterns are always exact.

• json: Expected response body parsed as JSON and matched structurally.

• if_not: Actions to take if expectations fail.

INFO

expect.header is present in the config schema but response-header matching is not enforced yet. For now, use status, body, body_not, or json to validate responses.

if_not

if_not defines actions to take if the check fails

Start with the visual lifecycle

The visual fallback lifecycle follows a check from its first failure through threshold, command execution, retries, stop exhaustion, group queues, metrics, and the healthy check that resets the outage. Use this reference section for individual keys and the visual guide for how they interact over time.

services:
  example:
    url: http://example.com
    every: 1m
    expect:
      status: 200
      if_not:
        threshold: 3
        stop: 2
        timeout: 15m
        cmd: "systemctl restart example-service"
        http: "http://alert-service/restart"

• threshold: Number of consecutive failed checks required before the fallback action is executed. (default: 1)
• stop: Number of times to run the cmd or http during one outage. A healthy check resets this counter for the next outage.
• timeout: How long each fallback action may take — cmd before it is killed, and http before the request is abandoned. (default: 300s)
• group: An optional label that serialises the fallbacks sharing it, so their cmds run one at a time instead of concurrently. Without a group a service's fallback runs immediately, taking no lock. See if_not.group.
• cmd: Command to run if the check fails.
• http: HTTP endpoint to call if the check fails.

An if_not must have an action

At least one of cmd or http is required. A block carrying only threshold, stop, timeout or group is refused at start-up: it declares a budget for an action that does not exist, and before 4.2.0 it recorded a successful fallback execution on every failed check while doing nothing.

cmd and http are independent. When both are configured they both run, and a failing cmd does not skip the http call — the alert still goes out even when the restart did not work. Failures from either are logged.

They run concurrently, so the alert fires as the recovery command starts rather than after it finishes. This matters because a cmd can be made to wait: services that share an if_not.group run their fallback commands one at a time, so a grouped command may sit in a queue before it starts. Because http never queues, the alert still goes out the moment the check fails — even while the command is waiting behind others in its group, and even if that command is ultimately skipped. A command with no group takes no lock and runs immediately.

For a grouped cmd, timeout applies to each phase separately: the command waits up to timeout for its turn in the group's queue, and once it starts it gets the whole of timeout to run in. An ungrouped command never waits, so only the run phase applies to it. A command still queued when its wait runs out is skipped and logged, and the next failed check retries it. The two phases are budgeted separately on purpose: sharing one deadline would let a restart start with only a sliver of time left and be killed part-way through, stopping a service without starting it again. The trade-off is that a grouped fallback can occupy up to twice timeout in the worst case.

Being skipped is not the same as failing. stop limits how many times the fallback actions actually run, so a command that never started is normally handed its attempt back rather than spending it — that is the case when cmd is the only action configured. When an http alert is configured too, the refund is withheld: the alert takes no lock and was still sent, which counts as an execution, and refunding it would uncap alerting for as long as the contention lasted. If that alert succeeds, the attempt is still recorded as skipped because the command did not run; the label follows the command, while the refund follows what executed. If the alert fails, failure takes precedence over the simultaneous skip. Either way the failed check itself still counts toward threshold. This refund only arises inside a group; an ungrouped command always runs, so it always spends its stop attempt.

threshold counts consecutive failures. A successful check resets the failure counter to 0.

stop is not a failure threshold. It only limits how many times Epazote will execute the fallback action after the threshold has been reached during the current outage. When the service becomes healthy again, the stop counter is reset.

This is the easiest way to think about the two together:

• threshold: when fallback starts
• stop: when fallback stops

Example:

services:
  example:
    url: http://example.com
    every: 30s
    expect:
      status: 200
      if_not:
        threshold: 3
        stop: 2
        cmd: systemctl restart example-service

With every: 30s, that means:

1. first failed check: WARN - failure count 1/3 below threshold
2. second failed check: WARN - failure count 2/3 below threshold
3. third failed check: INFO - threshold reached (3/3), executing fallback (execution #1/2)
4. fourth failed check: INFO - threshold reached (4/3), executing fallback (execution #2/2)
5. fifth failed check: ERROR - reached stop limit (2), skipping fallback
6. later failed checks: no repeated stop-limit error; the budget remains exhausted

If the next check is healthy, both counters reset. A later outage starts again at failure count 1/3 and can use the same stop: 2 retry budget.

The execution counter format #1/2 shows the current execution number and the stop limit for the current outage. If no stop is configured, it displays as #1/unlimited.

Example with expect.json and if_not:

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success
        data:
          activeTargets:
            - labels:
                job: DBMI-lab-nico
              health: up
      if_not:
        threshold: 3
        stop: 3
        cmd: systemctl restart vmagent

if_not.group

By default each service recovers on its own: when its check fails, its if_not.cmd runs straight away, with no regard for what any other service is doing. That is fine when services are unrelated, but it turns against you the moment their recovery actions share something.

Picture a database host that several services check against. The host bounces, every one of those checks fails on the same tick, and every cmd fires its systemctl restart at the same instant — a restart storm that pounds the very host you are trying to bring back up. The same failure mode shows up in your logs when several services share one restart script: their output interleaves in a single file and you can no longer tell which line belonged to which service.

if_not.group solves both by running the fallbacks that share a group one at a time:

• Same group → serialised. While one member's cmd runs, the others in that group wait their turn rather than piling on together.
• No group → concurrent. A service without a group takes no lock and runs its fallback immediately — exactly how every fallback behaved before groups existed.

A realistic config mixes the two. Here two APIs that lean on the same database host share a group, while an unrelated marketing site is left ungrouped:

services:
  orders-api:
    url: http://127.0.0.1:8080/health
    every: 30s
    expect:
      status: 200
      if_not:
        group: db-host-3                 # shares a database host with billing-api…
        cmd: systemctl restart orders-api

  billing-api:
    url: http://127.0.0.1:8081/health
    every: 30s
    expect:
      status: 200
      if_not:
        group: db-host-3                 # …so the two restarts take turns
        cmd: systemctl restart billing-api

  marketing-site:
    url: https://example.com/health
    every: 1m
    expect:
      status: 200
      if_not:
        # no group: nothing it restarts is shared, so it never waits
        cmd: systemctl restart nginx

If the database host bounces and both API checks fail on the same tick, orders-api and billing-api restart one after the other instead of at once. marketing-site has no group, so its restart fires immediately no matter what the grouped services are doing.

The group name is an arbitrary label you choose. Epazote never interprets it — it only compares one service's group against another's — so db-host-3 above could equally be mysql, slow-restarts, or any string you like. What matters is that the services which must not restart together carry the same label. A few rules:

• Any non-empty string is valid.
• Matching is case-sensitive: mysql and MySQL are different groups.
• Surrounding whitespace is ignored, so "mysql " and mysql are the same group.
• An empty or whitespace-only string is rejected at start-up — group: "" and group: " " refuse to start, since serialising against a blank name is never what was meant.
• A key written with no value at all is rejected too. group:, group: null and group: ~ all halt start-up rather than being read as no group: writing the key is an intent to serialise, and silently doing the opposite is the kind of quiet surprise groups exist to remove. To run a command ungrouped, leave the key out entirely. The same rule covers the rest of the if_not block — cmd:, http:, stop:, threshold: and timeout: — where an empty key would otherwise read as configured recovery that never runs.
• A non-empty group requires cmd. Groups serialise commands only; if_not.http never queues, so putting a group beside an HTTP-only action would advertise protection that does not exist.
• YAML coerces scalars to strings, so group: 123 becomes the label "123".

Use a group when two or more services would otherwise collide — when they run the same fallback script, or when their restarts contend for the same resource (the same database host, the same disk, the same cluster). Group exactly those services and no others.

Leave genuinely independent services ungrouped. Grouping restarts that never touch the same thing only makes them queue behind one another for no reason, and that needless queuing is the very problem groups were added to remove: a service left waiting long enough has its turn expire and is skipped altogether. If two restarts never contend, let them run concurrently.

Epazote warns when it spots a likely shared restart

At start-up — at normal verbosity, so no -v is needed — Epazote warns when two services either run a byte-identical cmd or invoke the same script with different arguments, unless all of them share one group. It names the services and suggests giving them a group.

A group only covers the services actually in it, so grouping one side of a shared script and forgetting the other is reported too, rather than passing as handled. The script test looks past a leading wrapper — sudo, env, a shell or an interpreter — because a recovery command routinely needs privileges, and a rule that stopped at the first token would miss sudo /opt/restart.sh entirely.

It deliberately does not treat a system utility as a shared script, whether written systemctl or /usr/bin/systemctl: those services have nothing in common but the tool, and a warning that cries wolf is one operators quickly learn to ignore. Where a command is too ambiguous to read — a wrapper carrying its own flags, or a shell construct such as cd /srv && ./restart.sh — it stays silent rather than guess.

For the same reason it is not a guarantee: a shared resource cannot be detected at all, since nothing in systemctl restart mariadb reveals what else lives on that host. Treat the warning as a useful nudge, not proof that your grouping is complete.

To restore the old process-wide behaviour, give every service the same group. Version 4.1.0 serialised every fallback across the whole process; 4.2.0 narrowed that to the group so unrelated services no longer wait on each other. If you genuinely want one restart at a time everywhere, put every service in one shared group:

services:
  orders-api:
    url: http://127.0.0.1:8080/health
    every: 30s
    expect:
      status: 200
      if_not:
        group: everything          # one shared group…
        cmd: systemctl restart orders-api

  billing-api:
    url: http://127.0.0.1:8081/health
    every: 30s
    expect:
      status: 200
      if_not:
        group: everything          # …so only one restart runs at a time, anywhere
        cmd: systemctl restart billing-api

if_not.timeout

Recovery is not a health probe. A service timeout of a few seconds is right for deciding whether an endpoint answers, but far too short for the work done to fix it, so the fallback actions get their own budget: 300s by default. It applies to both cmd and http.

services:
  example:
    url: http://example.com
    every: 1m
    timeout: 5s          # the check must answer quickly
    expect:
      status: 200
      if_not:
        timeout: 15m     # the restart may take a while
        cmd: "systemctl restart example-service"

Raise it when recovery is genuinely slow (a database restart, a rebuild); lower it when a recovery command must never linger. It follows the same rules as every other duration: a whole number and a required unit, so timeout: 30 is a config error — write timeout: 30s.

The budget still exists for a reason: a recovery command that hangs forever would otherwise stall every future check for that service, so a fallback that exceeds its timeout is killed and the failure is logged.

Commands run in their own process group, so a timeout kills anything the command started as well, not just the shell.

Diagnosing a failing command

Anything a test or if_not.cmd command writes to stderr is logged: as a warning when the command exits non-zero, and at debug level otherwise, since plenty of healthy commands write to stderr. stdout is discarded. Run with -vv to see the debug-level output.

Environment variables for if_not.cmd

When Epazote runs if_not.cmd, it passes service context through EPAZOTE_* environment variables. This makes alert scripts easier to write without parsing log output.

Available variables:

• EPAZOTE_SERVICE_NAME
• EPAZOTE_SERVICE_TYPE (http or command)
• EPAZOTE_URL for HTTP checks
• EPAZOTE_TEST for command checks
• EPAZOTE_EXPECTED_STATUS
• EPAZOTE_ACTUAL_STATUS when available. A command that could not run at all — a spawn failure, or one killed for exceeding its timeout — never produced an exit status, so this is not set for it.
• EPAZOTE_ERROR. For command checks this is command_failed when the command ran and exited with an unexpected status, and command_error when it could not run or was killed at the timeout. A command that could not run is always unhealthy, even when expect.status happens to match the shell's own failure code.
• EPAZOTE_FAILURE_COUNT
• EPAZOTE_THRESHOLD

Example:

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success
      if_not:
        threshold: 3
        stop: 1
        cmd: /usr/local/bin/send-alert.sh

Example script:

#!/usr/bin/env bash
set -euo pipefail

printf 'service=%s\n' "${EPAZOTE_SERVICE_NAME:-}"
printf 'type=%s\n' "${EPAZOTE_SERVICE_TYPE:-}"
printf 'error=%s\n' "${EPAZOTE_ERROR:-}"
printf 'expected=%s actual=%s\n' "${EPAZOTE_EXPECTED_STATUS:-}" "${EPAZOTE_ACTUAL_STATUS:-}"
printf 'failure_count=%s threshold=%s\n' "${EPAZOTE_FAILURE_COUNT:-}" "${EPAZOTE_THRESHOLD:-}"

Body options (json,form,text)

If you want to submit data using for example the POST method, you have three options:

• json - Sends the data as JSON
• form - Sends the data as a form
• text - Sends the data as text

The headers are set automatically based on the body type, but can be changed if needed using the option headers.

Example submitting data as JSON:

services:
  example-service:
    every: 5m
    url: http://example.com
    method: POST
    body:
      json:
        key: value

Example submitting data as a form:

services:
  example-service:
    every: 5m
    url: http://example.com
    method: POST
    body:
      form:
        key: value

Example submitting data as text:

services:
  example-service:
    every: 5m
    url: http://example.com
    method: POST
    body: "Hello World!"
    headers:
      content-type: text/plain

TIP

You can override the default headers by adding a headers key.

For example in the case of sending a text body, you can set the content-type to text/plain, together with other custom headers:

    headers:
      content-type: text/plain
      X-Custom-Header: TestValue

Body regular expressions

You can match the body of the response in two ways.

Without the r"..." prefix, body is treated as plain text and matched as a substring. For example, to match the word "success" in the body:

services:
  example-service:
    every: 5m
    url: http://example.com
    expect:
      status: 200
      body: success

For more complex regular expressions, prefix the body with r"<your regex>":

services:
  example-service:
    every: 5m
    url: http://example.com
    expect:
      status: 200
      body: r"success|ok"

That means:

• body: success checks whether the response contains the text success
• body: r"success|ok" uses a raw regular expression

If the response is JSON, prefer expect.json over regex. It is easier to read and less fragile.

Reject body matches

Use body_not when the response is healthy only if a plain string or regex is absent:

services:
  alloy_metrics:
    every: 30s
    url: http://127.0.0.1:12345/metrics
    expect:
      body_not: r"error|failure|Fatal"
      if_not:
        cmd: /script/when/failure.sh

body_not supports the same matching rules as body:

• body_not: Failure fails when the response contains the text Failure
• body_not: r"error|failure|Fatal" fails when the response matches the raw regex

When body_not fails, if_not.cmd receives EPAZOTE_ERROR=body_not_match.

JSON response matching

Use expect.json when the response is JSON and you want structural matching instead of text matching:

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success

Nested objects are matched recursively, so you can check only the fields you care about:

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success
        data:
          activeTargets:
            - labels:
                job: DBMI-lab-nico
              health: up

Notes:

• expect.body and expect.json are mutually exclusive
• HTTP checks may omit expect.status when expect.body, expect.body_not, or expect.json is configured
• command checks using test require expect.status
• objects are matched as subsets, so extra fields in the response are allowed
• array expectations match when each expected element matches at least one element in the actual response array
• if_not works with expect.json and expect.body_not the same way it works with expect.body
• if_not.threshold defaults to 1, which preserves the previous behavior

Test command

Instead of using a URL, you can use the test key to check the exit status of a command:

services:
  example-service:
    every: 5m
    test: "pgrep -x httpd"
    expect:
      status: 0

test: is a shell command that will be executed status: is the expected exit status of the command

Epazote runs test with the current shell from SHELL, falling back to sh. For more complex logic, prefer calling an executable script:

services:
  example-service:
    every: 5m
    test: /usr/local/bin/check-httpd.sh
    expect:
      status: 0

It can be used also with if_not and perform actions if the command fails:

services:
  example-service:
    every: 5m
    test: pgrep -x httpd
    expect:
      status: 0
      if_not:
        cmd: sudo systemctl restart httpd

A test command is bounded by the service timeout (default 5s). A check that runs longer is killed and counted as a failed check, which is what keeps one stuck command from silently stopping every later check for that service. Give slow probes more room explicitly:

services:
  example-service:
    every: 5m
    test: /usr/local/bin/deep-health-check.sh
    timeout: 30s
    expect:
      status: 0
      if_not:
        timeout: 10m
        cmd: sudo systemctl restart httpd

Supervisor Behavior

Epazote acts as a supervisor for your services. To ensure high availability and proper incident response:

• Task Supervision: If any individual service monitoring task crashes or completes unexpectedly, the main Epazote process will gracefully terminate.
• Auto-Restart: It is highly recommended to run Epazote under a process manager like systemd or supervisord. This allows the process to be automatically restarted if it exits due to a task failure, ensuring continuous monitoring.

See the Install page for a sample systemd service configuration.