Repository navigation
[improve][test] Speed up the BookKeeper replication tests in pulsar-metadata - #27304
Merged
Merged
Conversation
….tryToBecomeAuditor() on close A follower in tryToBecomeAuditor() waits on the manager until it becomes the leader or its session is lost. close() withdrew the candidacy but left that wait in place, so the calling thread only returned once interrupted. close() now wakes it up and the call fails like a lost session does.
…etadata - Stop a bookie's auto recovery before the bookie, as a bookie server does, and close an auditor elector's auditor manager before shutting it down: AuditorElector.shutdown() otherwise waits 10 s for a follower's election. - Re-enable the ledger replication that a test disabled, and cancel the delayed audit a test left pending, before the auditors stop: an auditor waits 30 s for a task blocked on either. - Run the auto recovery of all the BookieAutoRecoveryTest bookies on the Pulsar metadata drivers, not only of those started by the tests. - Shorten the replication worker retry backoff and the lock hold time of a failed ledger where no test checks them, and disable the speculative reads that delay closing the TestReplicationWorker client. - Wait for the replication or the worker state instead of fixed sleeps, and check that a getLedgerToRereplicate() call is waiting after a scan instead of waiting 1 s for it not to return.
…tTest The test checked that nothing got audited during the restart with pollLedgerToRereplicate(), which skips the ledgers another client has locked. With the replication enabled, the auto recovery worker locks the ledger that the auditor marks, fails to replicate it with a bookie down and holds the lock, so the check passed even when the auditor ran.
…t stops an elector AuditorBookieTest stops the elector of a follower bookie in testBookieClusterRestart and testEnsureOnlySingleAuditor, and each of those shutdowns waited 10 s for the follower's election.
lhotari
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Six classes take ~1,100 s of every run of the Pulsar Metadata unit job (2 Nov 12:18 UTC scheduled run:
AuditorPeriodicCheckTest296 s,LedgerUnderreplicationManagerTest241 s,AuditorLedgerCheckerTest188 s,BookieAutoRecoveryTest144 s,TestReplicationWorker134 s,AuditorRollingRestartTest106 s), about the same inthe last eight scheduled runs.
AuditorBookieTest(64 s) has the same problem. Most of that time goes to waits that nocheck depends on:
AuditorElector.shutdown()(BookKeeper 4.18.2,same on BookKeeper master) closes the auditor manager on the elector thread and waits up to 10 s for it. On a
follower that thread is blocked in
LedgerAuditorManager.tryToBecomeAuditor(), which only returns once the bookiebecomes the auditor, so the shutdown always times out, interrupts the election and closes the manager itself
("Failed to close auditor manager in 10 seconds").
PulsarLedgerAuditorManager.close()doesn't even wake thatwait. Tests stop followers all the time: they kill "a non-auditor bookie" on purpose to avoid an election.
Counted in the baseline logs: 15 in the bodies of
AuditorLedgerCheckerTest, 9 inAuditorPeriodicCheckTest,7 in
BookieAutoRecoveryTest(itskillBookiestops the bookie's auto recovery), 5 inAuditorBookieTest, one inthe
AuditorRollingRestartTestteardown. (Stopping all electors in parallel, as [improve][test] Reduce AuditorBookieTest wall time from 180s to 69s #27094's teardown does, doesn'twait: each follower becomes the auditor in turn as the previous one leaves.)
Auditor.shutdown()waits 30 s for itsexecutor to terminate before interrupting it ("Executor not shutting down, interrupting"). Its audits wait for the
ledger replication to be enabled, and a
ScheduledThreadPoolExecutorstill runs delayed tasks aftershutdown().AuditorRollingRestartTestdisables the replication and never enables it again: 30 s when it kills the auditorbookie, and 30 s for each of the two auditors elected during the teardown (36 s in the test, 70 s of cleanup that
the Gradle class time leaves out).
testPeriodicCheckWhenDisabledleaves it disabled too (30 s), and the threetestDelayBookieAuditOf*tests close an auditor whose lost-bookie audit is delayed byInteger.MAX_VALUEs (30 seach).
ReplicationWorkerwaitsrwRereplicateBackoffMs(5 s) and keeps the ledger locked for 1/32 of
lockReleaseOfFailedLedgerGracePeriod(9.4 s, doubling on each failureup to 37.5 s).
testDeferLedgerLockReleaseForReplicationWorkerwaits for ten failures of each worker (47 s);BookieAutoRecoveryTestwaits 9-19 s per test after starting the replacement bookie for a worker that failedbefore it to release the ledger.
PendingReadOpschedules a speculative read for each entry of a read but keeps (andcancels) only the last one, so closing the
TestReplicationWorkerclient afterverifyRecoveredLedgerswaits up tothe 2 s
speculativeReadTimeoutfor the others (8 teardowns, ~1.6 s each).gettimeouts as "no ledger to replicate" checks in sevenLedgerUnderreplicationManagerTesttests on five metadata stores (41 s), a 3 s sleep for an ensemble change intestFailedWriteRecovery,getAuditor()sleeping 2 s per call, a 3 s sleep before a bookie restart intestBookiesNotAvailableScenarioForReplicationWorker, 1 s polling.Also:
BookieAutoRecoveryTestswitchesbaseConfto themetadata-store:drivers only aftersetUp()started thethree bookies, so their auto recovery ran on BookKeeper's ZooKeeper drivers and the bookies started by the tests on
Pulsar's: two separate elections, two auditors at once.
AuditorRollingRestartTestalready converts the URI instartBKCluster().For example,
AuditorRollingRestartTeston master:The killed auditor sees its own bookie leave first (
ServerTester.shutdown()stops the bookie before its autorecovery), and that audit waits for the replication to be enabled.
Modifications
No timer whose length a check depends on is shortened, and no check is removed or relaxed.
PulsarLedgerAuditorManager(main code).close()sets aclosedflag and wakes a caller waiting intryToBecomeAuditor(), which then fails like it does on a lost session. Before, a closed follower manager kept thecaller waiting until it was interrupted. New test:
PulsarLedgerAuditorManagerTest.testCloseReleasesWaitingCandidate(ZooKeeper and Oxia).
BookKeeperClusterTestCaseshutdownAuditorElector(elector): closes the elector's auditor manager first (the package-privateAuditorElector.shutdownTask, whatshutdown()itself runs after its 10 s timeout), then callsshutdown(). Withthe change above, the follower's election ends right away and
shutdown()doesn't wait.order of a bookie server (its
LifecycleComponentStackcloses components in the reverse order of their start).stopReplicationService()uses the helper too.getAuditor()sleptdeathWatchInterval + 1 s(2 s) on every candidate to see whether theAutoRecoveryDeathWatcherwould kill it. It checks what the death watcher checks instead: the daemon is runningand so are its elector and its replication worker.
AuditorLedgerCheckerTest,AuditorBookieTest:shutdownBookie()andstopAuditorElectors()use the helper.AuditorPeriodicCheckTesttestDelayBookieAuditOf*(3 tests): after the checks, cancel the delayed audit before closing the auditor.testPeriodicCheckWhenDisabled: enable the replication again after the checks.testFailedWriteRecovery: waits until the ledger metadata no longer lists the bookie that rejected writes,instead of sleeping 3 s and then checking it.
AuditorRollingRestartTestpollLedgerToRereplicate()skips theledgers another client locked, and with the replication enabled the auto-recovery worker locks the ledger the
auditor marks, fails to replicate it with a bookie down and holds the lock for 9.4 s. It now also checks that the
ledger has no under-replicated record (
getLedgerUnreplicationInfo), which no lock hides.BookieAutoRecoveryTeststartBKCluster()converts the URI tometadata-store:likeAuditorRollingRestartTest, so all bookies run theirauto recovery on the Pulsar drivers (one election, one auditor), and the helper applies to them.
lockReleaseOfFailedLedgerGracePeriod"64" (64 ms, the valueTestReplicationWorkeruses): the workers fail whilethe test hasn't started the replacement bookie yet; no test checks how long they keep the ledger locked.
TestReplicationWorkerrwRereplicateBackoffMs100 ms andlockReleaseOfFailedLedgerGracePeriod"64" for the whole class, the valuestestRepairedNotAdheringPlacementPolicyLedgerFragmentsalready used ([fix][test] Fix flaky TestReplicationWorker.testRepairedNotAdheringPlacementPolicyLedgerFragmentsOnRack #26308).testDeferLedgerLockReleaseForReplicationWorkerchecks the deferral periods computed from its own 64 ms grace period, not their timing; the windows in which
replication must keep failing (
test2RWs...,testRWShouldRetryUntil...) now see a retry every ~100 ms insteadof one.
testBookiesNotAvailableScenarioForReplicationWorker: restarts the bookie once each worker failed three times(bounded at 30 s) instead of after a 3 s sleep in a background thread; the checks in the loop are unchanged.
speculativeReadTimeout0 on the test client: the reads only verify the replicated entries.testRWZKConnectionLost,testRWZKConnectionLostOnNonRecoverableZkError: Awaitility on the same condition withthe same 10 s bound instead of polling every 1 s.
LedgerUnderreplicationManagerTest: "Shouldn't be able to find a ledger to replicate" no longer waits 1 s forgetLedgerToRereplicate()not to return. It waits until the call is in theObject.wait()that follows a scan thatfound no ledger it could lock (
testBasicInteraction,testLocking,testMarkingAsReplicated,testRelease,test2reportSame), or in theReplicationEnableCb.await()of a disabled replication (testDisableLedgerReplication,testEnableLedgerReplication), then checks the call hasn't returned. The poll thread's stack identifies the wait.Verifying this change
./gradlew :pulsar-metadata:test --tests <class> --rerun -PtestFailFast=false, one class per run, locally (macOS),master (the merge base 5b28b72, checked out over this branch's sources) and this branch back to back for each class,
after an untimed compile, while other sessions ran tests. CI column: the 2 Nov 12:18 UTC scheduled master run.
AuditorPeriodicCheckTestLedgerUnderreplicationManagerTestAuditorLedgerCheckerTestBookieAutoRecoveryTestTestReplicationWorkerAuditorRollingRestartTestAuditorBookieTestAuditorRollingRestartTest's class time leaves out the 70 s of cleanup (Gradle's class time stops before the@AfterMethod/@AfterClasscleanup that the job waits for), hence the wall column. Master'sBookieAutoRecoveryTestvaries with which bookies are followers when they get killed: 148.7 s (Gradle wall 153 s) in a first master run 75 min
earlier, 109.5 s in this one; the branch took 37.1 s and 35.0 s. The other classes were within a few seconds of the
first runs.
Methods that changed by 1 s or more (sum over invocations; master -> branch, same runs as above):
AuditorPeriodicCheckTesttestDelayBookieAuditOfCheckAllLedgersAuditorPeriodicCheckTesttestDelayBookieAuditOfPlacementPolicyAuditorPeriodicCheckTesttestDelayBookieAuditOfReplicasCheckAuditorPeriodicCheckTesttestFailedWriteRecoveryAuditorPeriodicCheckTesttestInitialDelayOfCheckAllLedgersAuditorPeriodicCheckTesttestInitialDelayOfPlacementPolicyCheckAuditorPeriodicCheckTesttestPeriodicCheckWhenLedgerDeletedLedgerUnderreplicationManagerTesttest2reportSameLedgerUnderreplicationManagerTesttestBasicInteractionLedgerUnderreplicationManagerTesttestDisableLedgerReplicationLedgerUnderreplicationManagerTesttestEnableLedgerReplicationLedgerUnderreplicationManagerTesttestLockCleanupAfterMigrationChangesDuringReadLedgerUnderreplicationManagerTesttestLockingLedgerUnderreplicationManagerTesttestMarkingAsReplicatedLedgerUnderreplicationManagerTesttestReleaseAuditorLedgerCheckerTesttestDelayedAuditOfLostBookiesAuditorLedgerCheckerTesttestDelayedAuditWithMultipleBookieFailuresAuditorLedgerCheckerTesttestDelayedAuditWithPeriodicBookieCheckAuditorLedgerCheckerTesttestDelayedAuditWithRollingUpgradeAuditorLedgerCheckerTesttestMultipleBookieFailuresAuditorLedgerCheckerTesttestReadOnlyBookieShutdownAuditorLedgerCheckerTesttestRescheduleOfDelayedAuditOfLostBookiesToStartImmediatelyAuditorLedgerCheckerTesttestRescheduleOfDelayedAuditOfLostBookiesToStartLaterAuditorLedgerCheckerTesttestRestartBookieAuditorLedgerCheckerTesttestSimpleLedgerAuditorLedgerCheckerTesttestToggleLedgerReplicationBookieAutoRecoveryTesttestClosedLedgersBookieAutoRecoveryTesttestEmptyLedgerLosesQuorumEventuallyBookieAutoRecoveryTesttestLedgerMetadataContainsIpAddressAsBookieIDBookieAutoRecoveryTesttestNoSuchLedgerExistsBookieAutoRecoveryTesttestOpenLedgersBookieAutoRecoveryTesttestStopWhileReplicationInProgressTestReplicationWorkertest2RWsShouldCompeteForReplicationOf2FragmentsAndCompleteReplicationTestReplicationWorkertestBookiesNotAvailableScenarioForReplicationWorkerTestReplicationWorkertestDeferLedgerLockReleaseForReplicationWorkerTestReplicationWorkertestRWShouldReplicateTheLedgersAfterTimeoutIfLastFragmentIsURTestReplicationWorkertestRWShouldRetryUntilThereAreEnoughBksAvailableForReplicationTestReplicationWorkertestRWZKConnectionLostTestReplicationWorkertestRWZKConnectionLostOnNonRecoverableZkErrorTestReplicationWorkertestReplicateEmptyOpenStateLedgerAuditorRollingRestartTesttestAuditingDuringRollingRestartAuditorBookieTesttestBookieClusterRestartAuditorBookieTesttestEnsureOnlySingleAuditorAll the waits removed are fixed lengths (10 s and 30 s shutdown timeouts, 5 s and 9.4-37.5 s retry delays, 2 s and
1 s sleeps), and the CI times match the local ones, so CI should save about the same. These seven classes took
1,173 s in the 2 Nov 12:18 run (per-class wall time, so
AuditorRollingRestartTestincludes its cleanup); their Gradlewall times on the branch add up to 489 s here, startup included. That is ~680 s off the Pulsar Metadata job's ~1,820 s
of test time, and
LedgerUnderreplicationManagerTest(~212 s) becomes its longest class.Mutation checks. Each breaks what a replaced wait or check guards; only the affected methods,
-PtestRetryCount=0, sources restored after each run:PulsarLedgerAuditorManager.close()fix revertedtestCloseReleasesWaitingCandidate(ZooKeeper, Oxia)AuditorRollingRestartTesttestFailedWriteRecoverytestDelayedAuditWithMultipleBookieFailurestestBookiesNotAvailableScenarioForReplicationWorkertestRWShouldRetryUntilThereAreEnoughBksAvailableForReplicationtestDeferLedgerLockReleaseForReplicationWorkerm1releases its lock beforem2pollstestLockingx5 storestestBasicInteractionx5testDisableLedgerReplicationx5,testEnableLedgerReplicationx5Not falsifiable by a mutation: the cleanups after the checks (enabling the replication, cancelling the delayed audit),
the elector shutdown order and
getAuditor()'s check (it returns the same auditor; the race the old sleep guardedagainst doesn't happen in these tests).
BookieAutoRecoveryTestwaits on unbounded latches for the same events asbefore; only the worker's lock hold time changed.
Regression: the other 13 classes of
org.apache.bookkeeper.replicationandPulsarLedgerAuditorManagerTestpass(47 tests). The seven classes and
PulsarLedgerAuditorManagerTesttogether, in the default 4 parallel forks with-PtestRetryCount=0, twice: 260/260 tests pass both times (278 s and 275 s of Gradle wall; class times within ~3 s ofthe single-class runs).
./gradlew quickCheckpasses.Not changed here:
LedgerUnderreplicationManagerTestis the longest class left (~210 s):testLockCleanupAfterAmbiguousDeletewaitsout the 30 s
BLOCKING_CALL_TIMEOUTin two cases (its check is that the cleanup times out), andtestLockCleanupAfterSameSessionRecoverythe loss of a 12 s ZooKeeper session in two (the outage must stay shorterthan the session). Overlapping each pair would save ~40 s but means merging data-provider cases.
AuditorLedgerCheckerTest(3-10 s), its 5 s and 1 s "nothing getsmarked" windows,
testPeriodicCheckWhenDisabled's 2 s windows against a 1 s check interval,BookieAutoRecoveryTest's 3 sopenLedgerRereplicationGracePeriod,AuditorRollingRestartTest's two 2 s windows.testRWZKConnectionLostteardown (~4 s): clients reconnecting after the test restarted ZooKeeper.AuditorElector.shutdown()should close the manager or interrupt beforewaiting;
Auditor.shutdown()should drop delayed tasks (setExecuteExistingDelayedTasksAfterShutdownPolicy(false))or interrupt a task waiting for the replication;
PendingReadOpleaks all but the last speculative read task;SingleThreadExecutor.awaitTermination()returns the inverse ("The mainWorkerPool did not shutdown cleanly" on everyclient close).
PulsarLedgerAuditorManagerignores the listenertryToBecomeAuditor()gets, so an auditor that loses itsleadership or its session isn't told (BookKeeper's ZooKeeper manager sends
SessionLost/VoteWasDeleted). Notverified beyond reading the code.
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes