Skip to content

[fix] Prevent blocked follower elections from delaying AuditorElector shutdown - #4914

Closed
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix-auditor-elector-shutdown
Closed

void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix-auditor-elector-shutdown

Conversation

@void-ptr974

Copy link
Copy Markdown
Contributor

Motivation

AuditorElector.shutdown() queues manager cleanup behind the blocking election task on the same single-thread executor. A follower waiting for leadership therefore prevents cleanup until the hardcoded ten-second timeout interrupts the worker.

That interruption does not establish task completion. Election can return and construct/start an Auditor while resources are being closed, and concurrent shutdown callers can return before the first caller finishes cleanup.

Changes

  • Gate election, Auditor construction and startup against a shared stopping state. Let admitted construction/startup finish without interruption; close a successfully constructed Auditor even if shutdown prevents it from starting.
  • Run cleanup independently of the election executor. Close the manager first, then interrupt any remaining election wait and await actual executor termination before shutting down the Auditor and owned BookKeeper client.
  • Make the native ZooKeeper manager release its follower latch on close and coordinate vote creation with close, preventing a vote from being created after cleanup. Closing before fallback interruption also avoids interrupting an in-flight vote creation before its path is recorded.
  • Share cleanup completion across concurrent and repeated shutdown calls. Interrupted waiters finish waiting for cleanup before receiving interruption, while shutdown requests from the election/cleanup thread avoid waiting for themselves.
  • Preserve externally owned clients (ownBkc=false) and the existing log-only handling of close failures, while still attempting subsequent supported close operations.

Validation

29 tests passed: 16 new elector lifecycle tests, 5 new ZooKeeper manager tests, and the existing AuditorBookieTest (5) and AutoRecoveryMainTest (3). Checkstyle, Apache Rat, Spotless and SpotBugs passed.

JAVA_HOME=/path/to/jdk-17 mvn -ntp -P'!mac,!cargo-zigbuild' \
  -pl bookkeeper-server -am checkstyle:check test \
  -Dtest=AuditorElectorTest,ZkLedgerAuditorManagerTest,AuditorBookieTest,AutoRecoveryMainTest \
  -Dsurefire.failIfNoSpecifiedTests=false -Dspotbugs.skip \
  -DtestRetryCount=0 -DredirectTestOutputToFile=true

Static checks used the same module/profile selection:

mvn -ntp -P'!mac,!cargo-zigbuild' -pl bookkeeper-server -am \
  apache-rat:check spotless:check spotbugs:check -DskipTests

The native build profiles were disabled for these local Java lifecycle checks on macOS. The full project test suite was not run locally.

Scope

Manager implementations must follow the documented cancellation contract: close() may wait for admitted metadata operations, but must return independently of the election invocation; remaining waits must respond to close or interruption. This does not terminate arbitrary uninterruptible waits in external implementations; the Pulsar manager requires its own fix.

Completion covers election exit and the existing supported close operations. This change does not add failure-unwinding inside Auditor constructors or strengthen Auditor.shutdown()'s internal handling of close failures.

@void-ptr974 void-ptr974 changed the title [fix] Make AuditorElector shutdown cancel election waits and await cleanup [fix] Avoid 10-second follower shutdown delays and Auditor initialization/cleanup races Oct 30, 2026
@void-ptr974 void-ptr974 changed the title [fix] Avoid 10-second follower shutdown delays and Auditor initialization/cleanup races [fix] Prevent blocked follower elections from delaying AuditorElector shutdown Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant