Skip to content

fix(workloadapi): harden watch error handling - #491

Open
maxlambrecht wants to merge 6 commits into
spiffe:mainfrom
maxlambrecht:fix/workloadapi-watch-errors
Open

maxlambrecht wants to merge 6 commits into
spiffe:mainfrom
maxlambrecht:fix/workloadapi-watch-errors

Conversation

@maxlambrecht

@maxlambrecht maxlambrecht commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What

Harden error handling in DefaultWorkloadApiClient for X.509 context, X.509 bundle, and JWT bundle watches:

  • Retry PERMISSION_DENIED with backoff. Before the first successful update, suppress the error while retries can be scheduled.
  • After material has been delivered, report the first denial immediately through Watcher.onError, then retry if the callback returns normally and the retry policy permits it. A successful update allows a subsequent denial to be reported again.
  • Suppress repeated denial notifications, while still reporting a later failure with another status when retries are exhausted or scheduling fails.
  • Wrap invalid trust-domain keys in the exception type appropriate to the watch, preserving the cause and allowing subsequent updates if onError returns normally.
  • Consistently report X509BundleException for X.509 bundle watch failures.

Why

The SPIFFE Workload Endpoint specification permits retrying PERMISSION_DENIED because a workload may start before its registration is available. Previously, this status stopped the watch immediately.

After material has been delivered, the Workload API specification says clients should cease using material withdrawn by a subsequent denial. Reporting that denial immediately lets watchers react while reconnection remains possible.

This change reports withdrawal to watchers. Invalidating cached material in the built-in X.509 and JWT sources remains a separate follow-up.

Unchecked trust-domain validation failures could also escape onNext, cancel the stream, and bypass the watcher’s error callback. X.509 bundle watches additionally reported inconsistent exception types.

Validation

Added in-process tests covering all three watches:

  • Initial permission denial followed by a successful update, without a premature error callback.
  • Successful update, permission denial, and successful recovery, with one typed error between updates.
  • Invalid trust-domain key followed by a valid update, confirming typed error reporting and continued delivery.

Observer tests cover repeated denials, notification reset after recovery, non-retryable statuses, retry exhaustion, scheduling rejection, and throwing error callbacks. Regression cases verify that a reported denial does not hide a later UNAVAILABLE failure when retries are exhausted or scheduling is rejected.

All 241 Workload API tests pass.

@maxlambrecht
maxlambrecht force-pushed the fix/workloadapi-watch-errors branch 3 times, most recently from 109c885 to 8eb59e9 Compare October 30, 2026 20:08
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
@maxlambrecht
maxlambrecht force-pushed the fix/workloadapi-watch-errors branch from 8eb59e9 to 2fe1e76 Compare October 30, 2026 20:22
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
@maxlambrecht
maxlambrecht force-pushed the fix/workloadapi-watch-errors branch from 2fe1e76 to e3ef1f5 Compare November 6, 2026 15:30
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
@maxlambrecht

Copy link
Copy Markdown
Member Author

Hey @NopeCharon @cvas22! Would either of you be willing to take a look at this PR? Since you've both been contributing to the library recently, your feedback would be really valuable. Thanks!

@cvas22

cvas22 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@maxlambrecht One moment, will review.

@cvas22 cvas22 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through this. The observer changes look right to me, and moving PERMISSION_DENIED into the retry path lines up with go-spiffe, whose handleWatchError only stops on Canceled and InvalidArgument. A couple of things I noticed though.

The biggest user facing change isn't in the CHANGELOG. An unregistered workload used to make DefaultX509Source.newSource() / DefaultJwtSource.newSource() fail right away, but with the default initTimeout of 0 and unlimited retries it now blocks forever and only logs SEVERE on every retry. I think that's the right behavior per the spec, but it probably deserves a "Changed" entry and a line in the newSource javadoc pointing people at initTimeout.

Also, once a denial has been reported, a later PERMISSION_DENIED that exhausts retries or fails to schedule is swallowed, so a watcher can't tell "denied but still retrying" apart from "denied and the watch is gone". Since the sources keep serving the old material after onError, reporting the terminal "Cancelling" error in that case too (or a distinct exception) would let them notice the watch ended.

Small nit: the same javadoc paragraph is copied onto three methods in WorkloadApiClient, maybe it could live once on DefaultWorkloadApiClient.

@cvas22 cvas22 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved my notes from the earlier comment onto the lines they're about.

Comment thread CHANGELOG.md Resolved
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
@maxlambrecht

Copy link
Copy Markdown
Member Author

Thanks for the review, @cvas22 ! I addressed all three points:

  • Added the changelog entry and initialization-timeout guidance, including CachedJwtSource.
  • All three watchers now report terminal Cancelling errors even after a reported permission denial. Tests cover retry exhaustion, scheduling rejection, and callback ordering.
  • Consolidated the repeated Javadoc into DefaultWorkloadApiClient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants