Repository navigation
fix(workloadapi): harden watch error handling - #491
maxlambrecht wants to merge 6 commits into
Conversation
109c885 to
8eb59e9
Compare
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
8eb59e9 to
2fe1e76
Compare
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
2fe1e76 to
e3ef1f5
Compare
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
|
Hey @NopeCharon @cvas22! Would either of you be willing to take a look at this PR? Since you've both been contributing to the library recently, your feedback would be really valuable. Thanks! |
|
@maxlambrecht One moment, will review. |
cvas22
left a comment
There was a problem hiding this comment.
Went through this. The observer changes look right to me, and moving PERMISSION_DENIED into the retry path lines up with go-spiffe, whose handleWatchError only stops on Canceled and InvalidArgument. A couple of things I noticed though.
The biggest user facing change isn't in the CHANGELOG. An unregistered workload used to make DefaultX509Source.newSource() / DefaultJwtSource.newSource() fail right away, but with the default initTimeout of 0 and unlimited retries it now blocks forever and only logs SEVERE on every retry. I think that's the right behavior per the spec, but it probably deserves a "Changed" entry and a line in the newSource javadoc pointing people at initTimeout.
Also, once a denial has been reported, a later PERMISSION_DENIED that exhausts retries or fails to schedule is swallowed, so a watcher can't tell "denied but still retrying" apart from "denied and the watch is gone". Since the sources keep serving the old material after onError, reporting the terminal "Cancelling" error in that case too (or a distinct exception) would let them notice the watch ended.
Small nit: the same javadoc paragraph is copied onto three methods in WorkloadApiClient, maybe it could live once on DefaultWorkloadApiClient.
cvas22
left a comment
There was a problem hiding this comment.
Moved my notes from the earlier comment onto the lines they're about.
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
Signed-off-by: Max Lambrecht <maxlambrecht@gmail.com>
|
Thanks for the review, @cvas22 ! I addressed all three points:
|
What
Harden error handling in
DefaultWorkloadApiClientfor X.509 context, X.509 bundle, and JWT bundle watches:PERMISSION_DENIEDwith backoff. Before the first successful update, suppress the error while retries can be scheduled.Watcher.onError, then retry if the callback returns normally and the retry policy permits it. A successful update allows a subsequent denial to be reported again.onErrorreturns normally.X509BundleExceptionfor X.509 bundle watch failures.Why
The SPIFFE Workload Endpoint specification permits retrying
PERMISSION_DENIEDbecause a workload may start before its registration is available. Previously, this status stopped the watch immediately.After material has been delivered, the Workload API specification says clients should cease using material withdrawn by a subsequent denial. Reporting that denial immediately lets watchers react while reconnection remains possible.
This change reports withdrawal to watchers. Invalidating cached material in the built-in X.509 and JWT sources remains a separate follow-up.
Unchecked trust-domain validation failures could also escape
onNext, cancel the stream, and bypass the watcher’s error callback. X.509 bundle watches additionally reported inconsistent exception types.Validation
Added in-process tests covering all three watches:
Observer tests cover repeated denials, notification reset after recovery, non-retryable statuses, retry exhaustion, scheduling rejection, and throwing error callbacks. Regression cases verify that a reported denial does not hide a later
UNAVAILABLEfailure when retries are exhausted or scheduling is rejected.All 241 Workload API tests pass.