Create the configuration file

When running epazote it defaults to the epazote.yml configuration file, you can specify a different file using the -c flag.

$ epazote -c /path/to/epazote.yml

Basic configuration

The configuration file is a YAML file that contains the services you want to monitor, here is an example:

---
services:
  app:
    url: http://0.0.0.0:8080
    every: 1m
    expect:
      status: 200
      if_not:
        cmd: systemctl restart app

In this example we are monitoring a service called app that runs on http://0.0.0.0:8080, we are checking the status code every minute and if the status code is not 200 we restart the service using systemctl restart app.

Mental model

For each service, Epazote does the same loop:

1. Wait every
2. Run either an HTTP check with url or a shell check with test
3. Compare the result with expect
4. If the check fails, run if_not if it is configured

In practice, most services fit one of these three patterns:

1. Check only the HTTP status code

services:
  app:
    url: http://127.0.0.1:8080/health
    every: 30s
    expect:
      status: 200

2. Check a JSON API

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success

3. Reject failure text in an HTTP response

services:
  alloy_metrics:
    url: http://127.0.0.1:12345/metrics
    every: 30s
    expect:
      body_not: r"error|failure|Fatal"
      if_not:
        cmd: /script/when/failure.sh

Use body_not when a service can return a body with a failure marker. This example does not check the HTTP status; it only fails if the body matches error|failure|Fatal.

4. Check a command exit code

services:
  nginx_process:
    test: pgrep -x nginx
    every: 30s
    expect:
      status: 0

If the endpoint returns JSON and you want to match fields instead of raw text, use expect.json:

---
services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success

If you want a fallback action, add if_not:

services:
  vmagent_targets:
    url: http://127.0.0.1:8429/api/v1/targets
    every: 30s
    expect:
      status: 200
      json:
        status: success
      if_not:
        threshold: 3
        stop: 2
        cmd: systemctl restart vmagent

That means:

• wait for 3 consecutive failures before running the command
• after that, run the command at most 2 times during that outage
• reset both counters after a healthy check, so a later outage gets the same retry budget

See the visual fallback lifecycle for complete timelines showing exactly when the counter advances, when a failed command is retried, how stop is spent, and how grouped commands take turns.

If you use a script in if_not.cmd, Epazote also exports EPAZOTE_* environment variables such as EPAZOTE_SERVICE_NAME, EPAZOTE_ERROR, EPAZOTE_FAILURE_COUNT, and EPAZOTE_THRESHOLD. That is the easiest way to build alert scripts without parsing logs.

run epazote

Within the same directory as the epazote.yml file you can run epazote:

$ epazote -v

-v flag is for verbose output

By default, Epazote prints human-readable logs. Use --json-logs if you want structured JSON logs instead.

For HTTP checks in pretty mode:

• healthy checks are logged as compact INFO entries
• failed expectation checks are logged as WARN entries
• response headers are shown only for failed HTTP checks

When if_not is configured, fallback execution is visible in the log. The default level is ERROR, so the entries that mean recovery did not work arrive without asking, and the routine progress of a fallback needs -v:

• At ERROR, by default: a recovery command that ran and exited non-zero, an if_not.http action answered with a non-2xx status, a fallback that never completed at all (Fallback for service 'app' did not complete: …), and a service whose stop budget is spent. Exhaustion is reported once for the outage; later failed checks remain exhausted without repeating the same error
• With -v: failures below threshold with the current count, the threshold being reached with its execution counter (e.g. execution #1/2), and the per-action success and failure entries

Error scanning service '…' is separate: it means the check could not be completed, not that recovery failed. See Fallback action logging for the full list.

Metrics

After running epazote you can access the metrics at http://0.0.0.0:9080/metrics

you can change the port using the -p flag (or the EPAZOTE_PORT environment variable)

$ curl 0:9080/metrics

By default the metrics server binds to all interfaces ([::], with an automatic fallback to 0.0.0.0 on systems where IPv6 is disabled). Use the -b / --bind flag (or the EPAZOTE_BIND environment variable) to restrict it — for example, to keep /metrics reachable only from localhost:

$ epazote --bind 127.0.0.1

when an explicit address is given it is used as-is; only the default [::] falls back to 0.0.0.0.

Output example:

# HELP epazote_build_info Build information for the running epazote binary (always 1)
# TYPE epazote_build_info gauge
epazote_build_info{revision="fbd1b3c90f0fca0458b66a421c91c658fd3bcd62",version="4.2.0"} 1
# HELP epazote_consecutive_failures Consecutive failed checks for the service, reset to 0 after a successful check
# TYPE epazote_consecutive_failures gauge
epazote_consecutive_failures{service_name="app"} 0
# HELP epazote_failures_total Total number of scan errors: checks that could not be completed at all. A check that ran and failed its expectations sets epazote_status to 0 instead.
# TYPE epazote_failures_total counter
epazote_failures_total{service_name="app"} 0
# HELP epazote_fallback_configured 1 when the service declares an if_not action that can actually run
# TYPE epazote_fallback_configured gauge
epazote_fallback_configured{service_name="app"} 1
# HELP epazote_fallback_executions_total Total number of if_not fallback attempts, by outcome
# TYPE epazote_fallback_executions_total counter
epazote_fallback_executions_total{outcome="failure",service_name="app"} 0
epazote_fallback_executions_total{outcome="skipped",service_name="app"} 0
epazote_fallback_executions_total{outcome="success",service_name="app"} 0
# HELP epazote_fallback_exhausted 1 when the service used up its 'stop' budget and will no longer run its configured fallback actions
# TYPE epazote_fallback_exhausted gauge
epazote_fallback_exhausted{service_name="app"} 0
# HELP epazote_last_check_timestamp_seconds Unix timestamp of the last completed check for the service
# TYPE epazote_last_check_timestamp_seconds gauge
epazote_last_check_timestamp_seconds{service_name="app"} 1788115363
# HELP epazote_response_time_seconds Service response time in seconds
# TYPE epazote_response_time_seconds histogram
epazote_response_time_seconds_bucket{service_name="app",le="0.005"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.01"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.025"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.05"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.1"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.25"} 3
epazote_response_time_seconds_bucket{service_name="app",le="0.5"} 3
epazote_response_time_seconds_bucket{service_name="app",le="1"} 3
epazote_response_time_seconds_bucket{service_name="app",le="2.5"} 3
epazote_response_time_seconds_bucket{service_name="app",le="5"} 3
epazote_response_time_seconds_bucket{service_name="app",le="10"} 3
epazote_response_time_seconds_bucket{service_name="app",le="+Inf"} 3
epazote_response_time_seconds_sum{service_name="app"} 0.005090209
epazote_response_time_seconds_count{service_name="app"} 3
# HELP epazote_status Service status (1 = OK, 0 = FAIL)
# TYPE epazote_status gauge
epazote_status{service_name="app"} 1

That is a healthy service that has never failed. Most series are present at 0 rather than missing, so that "healthy, never failed" can be told apart from "epazote is not running, or this service was renamed".

What is exported

Every metric is labelled with service_name, except epazote_build_info, which carries the build in its own labels.

Metric	Type	Meaning
epazote_status	gauge	1 when the check matched every expectation, 0 when it did not.
epazote_response_time_seconds	histogram	How long the check took.
epazote_failures_total	counter	Scan errors — a check that could not be completed at all. A check that ran and simply failed its expectations moves epazote_status to 0 and does not increment this.
epazote_ssl_cert_expiry_seconds	gauge	Seconds until the certificate expires, for https:// services.
epazote_consecutive_failures	gauge	The current run of consecutive failed checks, reset to 0 by the first success. Reported for every service, whether or not it has an if_not block.
epazote_fallback_executions_total	counter	Fallback attempts, additionally labelled outcome: success, failure or skipped. success means every action that ran reported success and no grouped command was held back — it is not proof the service was repaired.
epazote_fallback_exhausted	gauge	1 once the stop budget is spent, meaning no further fallback action will run during this outage.
epazote_fallback_configured	gauge	1 when the service declares an if_not.cmd and/or if_not.http action, 0 when it has no if_not at all.
epazote_last_check_timestamp_seconds	gauge	Unix time of the last completed check.
epazote_build_info	gauge	Always 1; the running version and revision are the labels.

url and test services both report epazote_status and epazote_response_time_seconds, so a failing command check is visible as epazote_status == 0 just like an HTTP one.

Changed in 4.0.0

Before 4.0.0, test services exported no metrics at all — a failing command service was invisible to Prometheus. If your dashboards or alert rules were built against an older version, they will now show command services that were previously absent.

Reading the fallback metrics

epazote_status reports 0 whether a failing service still has a fallback action available, has spent its stop budget, or has no if_not at all. Two gauges tell the three apart:

epazote_fallback_configured	epazote_fallback_exhausted	Meaning
1	0	A fallback action is armed: a failed check runs it once threshold is reached.
1	1	The stop budget is spent. No further fallback action runs during this outage.
0	0	The service declares no if_not. The failure is reported and nothing else happens.

The services where nothing further will happen automatically are the last two rows:

epazote_status == 0 and epazote_fallback_exhausted == 1

epazote_status == 0 and epazote_fallback_configured == 0

Reading epazote_fallback_exhausted == 0 on its own is the trap. It is seeded to 0 for every service, so it reports "has not given up" even where there is nothing to give up on — which is why the second query exists rather than being implied by the first.

An armed action is not a promise of repair

epazote_fallback_configured reports that an action is configured, not what it does. An if_not.http action often just posts an alert or triggers an external workflow, so a service can be ARMED and still need a human. Read these gauges as action availability; use epazote_fallback_executions_total and the action's own logs to judge whether anything was actually fixed.

epazote_consecutive_failures makes a service alertable before it crosses its threshold and runs its fallback, rather than only afterwards. It is reported for every service, including those with no if_not block at all.

epazote_fallback_executions_total splits fallback attempts by outcome. success means every action that ran reported success and no grouped command was held back. failure means at least one action reported otherwise or could not be started. skipped means a grouped command never got its lock and nothing that did run failed; an if_not.http action may still have succeeded alongside it. The label follows the command, while the stop refund is a separate decision: the attempt is refunded only when no HTTP action ran. A failed action takes precedence over a simultaneous skip. A rising failure count means a configured action is not working; a rising skipped count means a group is over-subscribed.

Finally,

time() - epazote_last_check_timestamp_seconds

ages out a check that has stopped running altogether. epazote_status cannot show that on its own, because a task that stops ticking keeps publishing its last value indefinitely. This series is deliberately not seeded at start-up — a zero would claim a check happened at the Unix epoch and trip the alert immediately — so it appears only after a service's first check.

A ready-made Grafana dashboard covering all of these ships in the repository as epazote-grafana-dashboard.json.