Skip to content

[improve][test] Reduce AuditorBookieTest wall time from 180s to 69s - #27094

Merged
merlimat merged 2 commits into
apache:masterfrom
void-ptr974:improve/auditor-parallel-teardown
Oct 30, 2026
Merged

merlimat merged 2 commits into
apache:masterfrom
void-ptr974:improve/auditor-parallel-teardown

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This change reduces the five-test AuditorBookieTest JVM wall time from 180 seconds to 69 seconds (61.7%) in a local paired run by closing independent electors concurrently during test teardown. It applies the same bounded cleanup to all three Auditor test classes while preserving their functional assertions and wait windows.

The comparison used upstream master commit 1733139a1d1f, JDK 21, and BookKeeper 4.18.2:

Measurement Serial teardown Parallel teardown
Whole test-class JVM wall time 179.71 s 68.78 s
Sum of teardown time 117.87 s 6.52 s
Sum of test-body time 49.37 s 49.34 s

Teardown time fell by 94.5%, while test-body time stayed essentially unchanged. Previously, teardown closed independent electors sequentially; BookKeeper election/shutdown waits could therefore accumulate after the functional checks had completed. Concurrent shutdown overlaps these waits.

Both runs used the same runtime dependencies, resource-ownership fixes, CI listeners, leak settings, 1.3 GB heap, retries disabled, and four shared CPUs. The serial control changed only the teardown shutdown call back to the existing serial method. This is one pair of fresh-JVM local measurements; the complete CI critical-path reduction has not been measured.

Related upstream production fix: apache/bookkeeper#4914 addresses blocked follower election waits and shutdown completion in BookKeeper's AuditorElector. This test teardown optimization is independent of that change; uninterruptible election waits in the Pulsar manager still require a separate Pulsar fix.

Modifications

  • Close independent electors concurrently during teardown of the three Auditor test classes, using at most six workers and deduplicating elector instances.
  • Wait for every submitted shutdown and actual worker-thread exit, including Netty thread-local cleanup, before releasing the remaining fixture resources. Aggregate failures and restore the caller's interrupt flag after cleanup. Crossing the 90-second deadline records a failure; cleanup continues until the workers finish, so this is not a hard execution-time limit.
  • Clear closed elector maps, register the restarted bookie with its fixture, and close method-owned auditor and underreplication-manager resources.
  • Add nine deterministic helper regression tests covering bounded concurrency, duplicates, failure aggregation, interruption, timeout/drain behavior, and thread-local release/errors. Preserve existing functional assertions, test-body shutdown sequences, and wait windows, including the current upstream restart/write-failure fixes.

Only test sources change. BookKeeper dependencies, Pulsar manager code, and production lifecycle behavior are unchanged. Waiting for helper workers does not imply that every BookKeeper internal resource closes synchronously.

Verifying this change

Personal CI is still running on the fork PR: Pulsar CI and Pulsar CI Flaky. CI results are pending.

  • Make sure that the change passes the CI checks.

The following passed on the PR sources with JDK 21 and retries disabled: repository quickCheck, all 33 tests in the three existing Auditor classes, and all nine new helper tests (42 tests, zero failures/skips).

THREAD_LEAK_DETECTOR_WAIT_MILLIS=10000 ./gradlew quickCheck :pulsar-metadata:test \
  --tests org.apache.bookkeeper.replication.AuditorBookieTest \
  --tests org.apache.bookkeeper.replication.AuditorPeriodicCheckTest \
  --tests org.apache.bookkeeper.replication.AuditorLedgerCheckerTest \
  --tests org.apache.bookkeeper.replication.ParallelAuditorShutdownTest \
  -PtestMaxParallelForks=1 -PtestRetryCount=0 -PtestFailFast=false \
  --no-daemon --max-workers=4

The separate serial/parallel timing runs each passed all five functional tests and had no unexpected surviving threads at the final strict resource check. The Gradle log also contained a ChildWatcher null-path callback error. These results do not establish that the suite is free of flakes; the watcher is outside this change's scope.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model — test teardown only; bounded shutdown workers.
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Close independent electors concurrently during Auditor test teardown, wait for
shutdown tasks and worker-thread exit, and preserve cleanup failures and
interruptions. Release fixture-owned resources and add deterministic cleanup
regression tests.

Assisted-by: Codex

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 19a607304e9325846ed025f07c507e5309542b74. No actionable findings identified within the reviewed scope.

Validation caveat: all nine shutdown-helper tests passed. One integration test failed in the combined run but passed in isolation; full-suite and paired performance validation remain incomplete.

@merlimat
merlimat merged commit 43f1ce1 into apache:master Oct 30, 2026
42 checks passed
lhotari pushed a commit that referenced this pull request Nov 2, 2026
…27094)

Co-authored-by: 道君- Tao Jiuming <daojun@apache.org>
(cherry picked from commit 43f1ce1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants