Repository navigation
Add /livez and /readyz health endpoints - #2706
Open
tmilnthorp wants to merge 6 commits into
Open
tmilnthorp wants to merge 6 commits into
tmilnthorp wants to merge 6 commits into
Conversation
Serve Kubernetes-style liveness and readiness probes from the HTTP component, reporting in the IETF health check format. Liveness is based on the utility loop's last iteration and never depends on plugins. Readiness requires completed startup and all enabled plugins ready, and reports failed apps as a warning only. Check detail is withheld from unauthenticated requests when a password is configured. Refs AppDaemon#739
A long reload of many apps blocks the utility loop for longer than the liveness threshold even though apps keep finishing initialize(). Count an app's initialize() returning as progress, raise the minimum threshold to 300s, and name the app that has been initializing the longest when there has been no progress.
…ormed login cookies A login cookie that isn't a bcrypt hash made bcrypt.checkpw raise, which the health endpoints and the API's securedata surfaced as a 500. It is now treated as invalid credentials.
Report a plugin as ready only while PluginManagement marks it active, since the MQTT plugin never clears its ready event. Find stuck and failed apps in the admin namespace through AppManagement instead of tracking them twice. Make secure, route_secure and securedata share HTTP.is_authorized, which now compares secrets in constant time and refuses login cookies whose bcrypt cost is not the configured work factor. Fill gaps in the tests, compact the documented examples and fix the docs build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Kubernetes-style liveness and readiness probes to the HTTP component, as proposed in #739.
Endpoints
GET /livezmax(300s, 10 × utility_delay). Never depends on a plugin.GET /readyzinitialize_error/compile_error) make/readyzreportwarnwith the names, but stay 200, so one broken app doesn't get the instance pulled.plugin:<name>check. With no plugins, only startup and apps are checked.application/health+json,Cache-Control: no-store).{"status": ...}.checksmap.httpis configured, ahead of the dashboard/{name}catch-all.HEALTHCHECKin the Dockerfile.CONFIGURE.rsthas a compose example against/livezusing the image'scurl.What counts as wedged
App reloads run inside the utility loop, and
initialize()has no timeout. So progress is either a loop iteration finishing or any app'sinitialize()returning (or raising).initialize()fails after the threshold and names the app. Its hang stalls reloads, plugin refresh and thread checks.failureThreshold, Dockerretries. The server only decides what counts as stuck, and JSON clients can apply their own threshold toobservedValue.Example
/livezwith an app stuck ininitialize()(HTTP 503):{"status": "fail", "checks": {"utility:lastProgress": [{"status": "fail", "observedValue": 312.0, "observedUnit": "s", "time": "2026-11-06T10:07:23+00:00", "output": "no progress in 312s; app 'slow_app' has been initializing since 2026-11-06T10:02:11+00:00"}]}}docs/CONFIGURE.rstalso has/readyzpass, warn (failed app) and fail (plugin disconnected), plus the unauthenticated body.Other changes
securedata,secureandroute_securenow shareHTTP.is_authorized(). The dashboard decorators still accept only the cookie, andsecure's logon page androute_secure's token branch are unchanged.adcredscookie (not a bcrypt hash) used to raise a 500.securedata/route_secureused to return no response.hmac.compare_digest.work_factoris refused beforecheckpw. A$2b$31$…cookie would otherwise pin an executor thread for hours.activeandis_ready. MQTT never clears itsready_event, sois_readyalone stayed green through a broker outage.app_management.py, read throughAppManagement.failed_apps()/initializing_apps().Utilitystate:last_progress/record_progress()andstartup_complete.livez/readyzis now shadowed;work_factor.Tests
tests/unit/test_health.pydrives the endpoints through aiohttp'sTestClient. It uses realHTTP,Utility,AppManagementandPluginManagementobjects and autospec'd plugins.notify_plugin_stoppedwhileis_readyis still true (the MQTT case); apps disabled.utility_delayscaling;initialize()names the oldest initializing app;_init_loop's startup ordering.{"status": ...};Cache-ControlandobservedUnit.The load-bearing branches were mutation-checked.
pytest tests/unitpasses (277), and the docs build passes withsphinx -W --keep-going.