Skip to content

Add /livez and /readyz health endpoints - #2706

Open
tmilnthorp wants to merge 6 commits into
AppDaemon:devfrom
tmilnthorp:health-endpoints
Open

tmilnthorp wants to merge 6 commits into
AppDaemon:devfrom
tmilnthorp:health-endpoints

Conversation

@tmilnthorp

@tmilnthorp tmilnthorp commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Adds Kubernetes-style liveness and readiness probes to the HTTP component, as proposed in #739.

Endpoints

Answers 503 when
GET /livez Alive and not wedged? The utility loop has stopped, or nothing has made progress in max(300s, 10 × utility_delay). Never depends on a plugin.
GET /readyz Doing its job? Startup is incomplete, AppDaemon is stopping, or an enabled plugin isn't loaded or connected.
  • Status code is the verdict: 200 for pass/warn, 503 for fail.
  • Failed apps (initialize_error/compile_error) make /readyz report warn with the names, but stay 200, so one broken app doesn't get the instance pulled.
  • Plugins: each enabled plugin is a plugin:<name> check. With no plugins, only startup and apps are checked.
  • Body: IETF health check format (application/health+json, Cache-Control: no-store).
    • Without credentials it's just {"status": ...}.
    • With the API's credentials, or no password configured, it adds the checks map.
  • Routing: registered whenever http is configured, ahead of the dashboard /{name} catch-all.
  • Docker: no HEALTHCHECK in the Dockerfile. CONFIGURE.rst has a compose example against /livez using the image's curl.

What counts as wedged

App reloads run inside the utility loop, and initialize() has no timeout. So progress is either a loop iteration finishing or any app's initialize() returning (or raising).

  • A long reload stays live. A hung initialize() fails after the threshold and names the app. Its hang stalls reloads, plugin refresh and thread checks.
  • Trade-off: a deterministic hang restart-loops under an orchestrator, with its backoff.
  • Also counts as progress: app initialization outside the loop (API restart, plugin reconnect).
  • The client sets patience: Kubernetes failureThreshold, Docker retries. The server only decides what counts as stuck, and JSON clients can apply their own threshold to observedValue.

Example

/livez with an app stuck in initialize() (HTTP 503):

{"status": "fail", "checks": {"utility:lastProgress": [{"status": "fail", "observedValue": 312.0, "observedUnit": "s", "time": "2026-11-06T10:07:23+00:00", "output": "no progress in 312s; app 'slow_app' has been initializing since 2026-11-06T10:02:11+00:00"}]}}

docs/CONFIGURE.rst also has /readyz pass, warn (failed app) and fail (plugin disconnected), plus the unauthenticated body.

Other changes

  • One credential check: securedata, secure and route_secure now share HTTP.is_authorized(). The dashboard decorators still accept only the cookie, and secure's logon page and route_secure's token branch are unchanged.
  • Bad cookies:
    • A malformed adcreds cookie (not a bcrypt hash) used to raise a 500.
    • A wrong cookie on securedata/route_secure used to return no response.
    • Both now fail authentication with a 401.
  • Hardening:
    • Header and query passwords are compared with hmac.compare_digest.
    • A cookie whose bcrypt cost isn't the configured work_factor is refused before checkpw. A $2b$31$… cookie would otherwise pin an executor thread for hours.
  • Plugin readiness requires active and is_ready. MQTT never clears its ready_event, so is_ready alone stayed green through a broker outage.
  • App state strings are constants in app_management.py, read through AppManagement.failed_apps() / initializing_apps().
  • New Utility state: last_progress/record_progress() and startup_complete.
  • Behaviour changes in HISTORY:
    • a dashboard named livez/readyz is now shadowed;
    • users log in again after changing work_factor.

Tests

tests/unit/test_health.py drives the endpoints through aiohttp's TestClient. It uses real HTTP, Utility, AppManagement and PluginManagement objects and autospec'd plugins.

  • Readiness: pass, warn and fail; a plugin stopped via notify_plugin_stopped while is_ready is still true (the MQTT case); apps disabled.
  • Liveness:
    • the 300s boundary and utility_delay scaling;
    • a dead loop task fails, except while stopping;
    • a long reload stays live, and a hung initialize() names the oldest initializing app;
    • the real _init_loop's startup ordering.
  • Auth:
    • missing, wrong, empty, wrong-case, malformed and high-cost credentials get only {"status": ...};
    • valid header, query or cookie credentials, or no password, show the detail;
    • no decorator 500s on a bad cookie.
  • Headers: Cache-Control and observedUnit.

The load-bearing branches were mutation-checked. pytest tests/unit passes (277), and the docs build passes with sphinx -W --keep-going.

Serve Kubernetes-style liveness and readiness probes from the HTTP
component, reporting in the IETF health check format. Liveness is based
on the utility loop's last iteration and never depends on plugins.
Readiness requires completed startup and all enabled plugins ready, and
reports failed apps as a warning only. Check detail is withheld from
unauthenticated requests when a password is configured.

Refs AppDaemon#739
A long reload of many apps blocks the utility loop for longer than the
liveness threshold even though apps keep finishing initialize(). Count
an app's initialize() returning as progress, raise the minimum threshold
to 300s, and name the app that has been initializing the longest when
there has been no progress.
…ormed login cookies

A login cookie that isn't a bcrypt hash made bcrypt.checkpw raise, which
the health endpoints and the API's securedata surfaced as a 500. It is
now treated as invalid credentials.
Report a plugin as ready only while PluginManagement marks it active, since
the MQTT plugin never clears its ready event. Find stuck and failed apps in
the admin namespace through AppManagement instead of tracking them twice.
Make secure, route_secure and securedata share HTTP.is_authorized, which now
compares secrets in constant time and refuses login cookies whose bcrypt
cost is not the configured work factor. Fill gaps in the tests, compact the
documented examples and fix the docs build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant