Skip to content

[improve][test] Speed up the BookKeeper replication tests in pulsar-metadata - #27304

Merged
merlimat merged 4 commits into
apache:masterfrom
merlimat:mmerli/faster-replication-tests
Nov 2, 2026
Merged

merlimat merged 4 commits into
apache:masterfrom
merlimat:mmerli/faster-replication-tests

Conversation

@merlimat

@merlimat merlimat commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Motivation

Six classes take ~1,100 s of every run of the Pulsar Metadata unit job (2 Nov 12:18 UTC scheduled run:
AuditorPeriodicCheckTest 296 s, LedgerUnderreplicationManagerTest 241 s, AuditorLedgerCheckerTest 188 s,
BookieAutoRecoveryTest 144 s, TestReplicationWorker 134 s, AuditorRollingRestartTest 106 s), about the same in
the last eight scheduled runs. AuditorBookieTest (64 s) has the same problem. Most of that time goes to waits that no
check depends on:

  • 10 s for each auditor elector of a follower that a test stops. AuditorElector.shutdown() (BookKeeper 4.18.2,
    same on BookKeeper master) closes the auditor manager on the elector thread and waits up to 10 s for it. On a
    follower that thread is blocked in LedgerAuditorManager.tryToBecomeAuditor(), which only returns once the bookie
    becomes the auditor, so the shutdown always times out, interrupts the election and closes the manager itself
    ("Failed to close auditor manager in 10 seconds"). PulsarLedgerAuditorManager.close() doesn't even wake that
    wait. Tests stop followers all the time: they kill "a non-auditor bookie" on purpose to avoid an election.
    Counted in the baseline logs: 15 in the bodies of AuditorLedgerCheckerTest, 9 in AuditorPeriodicCheckTest,
    7 in BookieAutoRecoveryTest (its killBookie stops the bookie's auto recovery), 5 in AuditorBookieTest, one in
    the AuditorRollingRestartTest teardown. (Stopping all electors in parallel, as [improve][test] Reduce AuditorBookieTest wall time from 180s to 69s #27094's teardown does, doesn't
    wait: each follower becomes the auditor in turn as the previous one leaves.)
  • 30 s for each auditor that stops with a blocked or a delayed task. Auditor.shutdown() waits 30 s for its
    executor to terminate before interrupting it ("Executor not shutting down, interrupting"). Its audits wait for the
    ledger replication to be enabled, and a ScheduledThreadPoolExecutor still runs delayed tasks after shutdown().
    AuditorRollingRestartTest disables the replication and never enables it again: 30 s when it kills the auditor
    bookie, and 30 s for each of the two auditors elected during the teardown (36 s in the test, 70 s of cleanup that
    the Gradle class time leaves out). testPeriodicCheckWhenDisabled leaves it disabled too (30 s), and the three
    testDelayBookieAuditOf* tests close an auditor whose lost-bookie audit is delayed by Integer.MAX_VALUE s (30 s
    each).
  • BookKeeper's replication worker defaults. After a failure a ReplicationWorker waits rwRereplicateBackoffMs
    (5 s) and keeps the ledger locked for 1/32 of lockReleaseOfFailedLedgerGracePeriod (9.4 s, doubling on each failure
    up to 37.5 s). testDeferLedgerLockReleaseForReplicationWorker waits for ten failures of each worker (47 s);
    BookieAutoRecoveryTest waits 9-19 s per test after starting the replacement bookie for a worker that failed
    before it to release the ledger.
  • Leaked speculative reads. PendingReadOp schedules a speculative read for each entry of a read but keeps (and
    cancels) only the last one, so closing the TestReplicationWorker client after verifyRecoveredLedgers waits up to
    the 2 s speculativeReadTimeout for the others (8 teardowns, ~1.6 s each).
  • Fixed sleeps and slow polls for conditions: 1 s get timeouts as "no ledger to replicate" checks in seven
    LedgerUnderreplicationManagerTest tests on five metadata stores (41 s), a 3 s sleep for an ensemble change in
    testFailedWriteRecovery, getAuditor() sleeping 2 s per call, a 3 s sleep before a bookie restart in
    testBookiesNotAvailableScenarioForReplicationWorker, 1 s polling.

Also: BookieAutoRecoveryTest switches baseConf to the metadata-store: drivers only after setUp() started the
three bookies, so their auto recovery ran on BookKeeper's ZooKeeper drivers and the bookies started by the tests on
Pulsar's: two separate elections, two auditors at once. AuditorRollingRestartTest already converts the URI in
startBKCluster().

For example, AuditorRollingRestartTest on master:

underReplicationManager.disableLedgerReplication();
...
BookieId auditor = lam.getCurrentAuditor();
ServerConfiguration conf = killBookie(auditor);   // 30 s
Thread.sleep(2000);
startAndAddBookie(conf);
Thread.sleep(2000); // give it time to run
assertEquals("shouldn't be anything under replicated", -1,
        underReplicationManager.pollLedgerToRereplicate());
// teardown: 10 s + 30 s + 30 s
09:39:16,447 [AuditorBookie-127.0.0.1:56798] Auditor - LedgerReplication is disabled externally ..., so waiting until it is enabled
09:39:16,562 [TestNG-method=...] Auditor - Shutting down auditor
09:39:46,565 [TestNG-method=...] Auditor - Executor not shutting down, interrupting
...
09:39:51,322 [Test worker] AutoRecoveryMain - Shutting down AutoRecovery
09:40:01,326 [Test worker] AuditorElector - Failed to close auditor manager in 10 seconds
09:40:01,584 [Test worker] Auditor - Shutting down auditor
09:40:31,590 [Test worker] Auditor - Executor not shutting down, interrupting
09:40:31,954 [AutoRecoveryDeathWatcher-56798] AutoRecoveryMain - Shutting down auto recovery
09:41:01,838 [Test worker] Auditor - Executor not shutting down, interrupting

The killed auditor sees its own bookie leave first (ServerTester.shutdown() stops the bookie before its auto
recovery), and that audit waits for the replication to be enabled.

Modifications

No timer whose length a check depends on is shortened, and no check is removed or relaxed.

PulsarLedgerAuditorManager (main code). close() sets a closed flag and wakes a caller waiting in
tryToBecomeAuditor(), which then fails like it does on a lost session. Before, a closed follower manager kept the
caller waiting until it was interrupted. New test: PulsarLedgerAuditorManagerTest.testCloseReleasesWaitingCandidate
(ZooKeeper and Oxia).

BookKeeperClusterTestCase

  • shutdownAuditorElector(elector): closes the elector's auditor manager first (the package-private
    AuditorElector.shutdownTask, what shutdown() itself runs after its 10 s timeout), then calls shutdown(). With
    the change above, the follower's election ends right away and shutdown() doesn't wait.
  • Killing, stopping or restarting a bookie stops its auto recovery first (through the helper), then the bookie, the
    order of a bookie server (its LifecycleComponentStack closes components in the reverse order of their start).
    stopReplicationService() uses the helper too.
  • getAuditor() slept deathWatchInterval + 1 s (2 s) on every candidate to see whether the
    AutoRecoveryDeathWatcher would kill it. It checks what the death watcher checks instead: the daemon is running
    and so are its elector and its replication worker.

AuditorLedgerCheckerTest, AuditorBookieTest: shutdownBookie() and stopAuditorElectors() use the helper.

AuditorPeriodicCheckTest

  • The tests that stop all electors before creating their own auditor use the helper.
  • testDelayBookieAuditOf* (3 tests): after the checks, cancel the delayed audit before closing the auditor.
  • testPeriodicCheckWhenDisabled: enable the replication again after the checks.
  • testFailedWriteRecovery: waits until the ledger metadata no longer lists the bookie that rejected writes,
    instead of sleeping 3 s and then checking it.

AuditorRollingRestartTest

  • Enables the replication again after the checks.
  • The kill is fast now: the auditor stops before its bookie, so it isn't waiting for the replication.
  • Its check couldn't fail (found by the mutation check below; same on master): pollLedgerToRereplicate() skips the
    ledgers another client locked, and with the replication enabled the auto-recovery worker locks the ledger the
    auditor marks, fails to replicate it with a bookie down and holds the lock for 9.4 s. It now also checks that the
    ledger has no under-replicated record (getLedgerUnreplicationInfo), which no lock hides.

BookieAutoRecoveryTest

  • startBKCluster() converts the URI to metadata-store: like AuditorRollingRestartTest, so all bookies run their
    auto recovery on the Pulsar drivers (one election, one auditor), and the helper applies to them.
  • lockReleaseOfFailedLedgerGracePeriod "64" (64 ms, the value TestReplicationWorker uses): the workers fail while
    the test hasn't started the replacement bookie yet; no test checks how long they keep the ledger locked.

TestReplicationWorker

  • rwRereplicateBackoffMs 100 ms and lockReleaseOfFailedLedgerGracePeriod "64" for the whole class, the values
    testRepairedNotAdheringPlacementPolicyLedgerFragments already used ([fix][test] Fix flaky TestReplicationWorker.testRepairedNotAdheringPlacementPolicyLedgerFragmentsOnRack #26308). testDeferLedgerLockReleaseForReplicationWorker
    checks the deferral periods computed from its own 64 ms grace period, not their timing; the windows in which
    replication must keep failing (test2RWs..., testRWShouldRetryUntil...) now see a retry every ~100 ms instead
    of one.
  • testBookiesNotAvailableScenarioForReplicationWorker: restarts the bookie once each worker failed three times
    (bounded at 30 s) instead of after a 3 s sleep in a background thread; the checks in the loop are unchanged.
  • speculativeReadTimeout 0 on the test client: the reads only verify the replicated entries.
  • testRWZKConnectionLost, testRWZKConnectionLostOnNonRecoverableZkError: Awaitility on the same condition with
    the same 10 s bound instead of polling every 1 s.

LedgerUnderreplicationManagerTest: "Shouldn't be able to find a ledger to replicate" no longer waits 1 s for
getLedgerToRereplicate() not to return. It waits until the call is in the Object.wait() that follows a scan that
found no ledger it could lock (testBasicInteraction, testLocking, testMarkingAsReplicated, testRelease,
test2reportSame), or in the ReplicationEnableCb.await() of a disabled replication (testDisableLedgerReplication,
testEnableLedgerReplication), then checks the call hasn't returned. The poll thread's stack identifies the wait.

Verifying this change

  • Make sure that the change passes the CI checks.

./gradlew :pulsar-metadata:test --tests <class> --rerun -PtestFailFast=false, one class per run, locally (macOS),
master (the merge base 5b28b72, checked out over this branch's sources) and this branch back to back for each class,
after an untimed compile, while other sessions ran tests. CI column: the 2 Nov 12:18 UTC scheduled master run.

Class CI (s, 2 Nov 12:18) master: class (s) branch: class (s) master: Gradle wall (s) branch: Gradle wall (s) tests
AuditorPeriodicCheckTest 296 255.7 52.8 259 58 12/12 -> 12/12 pass
LedgerUnderreplicationManagerTest 241 245.0 211.6 247 214 198/198 -> 198/198 pass
AuditorLedgerCheckerTest 188 188.0 84.4 191 88 16/16 -> 16/16 pass
BookieAutoRecoveryTest 144 109.5 35.0 114 39 7/7 -> 7/7 pass
TestReplicationWorker 134 140.5 52.9 146 56 14/14 -> 14/14 pass
AuditorRollingRestartTest 106 36.0 6.0 110 9 1/1 -> 1/1 pass
AuditorBookieTest 64 62.2 21.8 65 25 5/5 -> 5/5 pass
total 1037 464 1132 489

AuditorRollingRestartTest's class time leaves out the 70 s of cleanup (Gradle's class time stops before the
@AfterMethod/@AfterClass cleanup that the job waits for), hence the wall column. Master's BookieAutoRecoveryTest
varies with which bookies are followers when they get killed: 148.7 s (Gradle wall 153 s) in a first master run 75 min
earlier, 109.5 s in this one; the branch took 37.1 s and 35.0 s. The other classes were within a few seconds of the
first runs.

Methods that changed by 1 s or more (sum over invocations; master -> branch, same runs as above):

Class Method invocations master (s) branch (s)
AuditorPeriodicCheckTest testDelayBookieAuditOfCheckAllLedgers 1 41.8 1.8
AuditorPeriodicCheckTest testDelayBookieAuditOfPlacementPolicy 1 41.7 1.7
AuditorPeriodicCheckTest testDelayBookieAuditOfReplicasCheck 1 41.7 1.7
AuditorPeriodicCheckTest testFailedWriteRecovery 1 4.5 1.7
AuditorPeriodicCheckTest testInitialDelayOfCheckAllLedgers 1 22.5 2.6
AuditorPeriodicCheckTest testInitialDelayOfPlacementPolicyCheck 1 22.1 2.3
AuditorPeriodicCheckTest testPeriodicCheckWhenLedgerDeleted 1 12.2 2.2
LedgerUnderreplicationManagerTest test2reportSame 5 7.7 3.0
LedgerUnderreplicationManagerTest testBasicInteraction 5 5.7 1.2
LedgerUnderreplicationManagerTest testDisableLedgerReplication 5 5.2 0.7
LedgerUnderreplicationManagerTest testEnableLedgerReplication 5 5.4 0.9
LedgerUnderreplicationManagerTest testLockCleanupAfterMigrationChangesDuringRead 2 4.3 2.8
LedgerUnderreplicationManagerTest testLocking 5 5.4 0.9
LedgerUnderreplicationManagerTest testMarkingAsReplicated 5 6.2 1.8
LedgerUnderreplicationManagerTest testRelease 5 5.7 1.2
AuditorLedgerCheckerTest testDelayedAuditOfLostBookies 1 11.4 6.3
AuditorLedgerCheckerTest testDelayedAuditWithMultipleBookieFailures 1 14.4 6.1
AuditorLedgerCheckerTest testDelayedAuditWithPeriodicBookieCheck 1 11.9 6.8
AuditorLedgerCheckerTest testDelayedAuditWithRollingUpgrade 1 13.5 9.3
AuditorLedgerCheckerTest testMultipleBookieFailures 1 20.7 0.7
AuditorLedgerCheckerTest testReadOnlyBookieShutdown 1 11.4 1.4
AuditorLedgerCheckerTest testRescheduleOfDelayedAuditOfLostBookiesToStartImmediately 1 11.4 5.2
AuditorLedgerCheckerTest testRescheduleOfDelayedAuditOfLostBookiesToStartLater 1 11.4 7.2
AuditorLedgerCheckerTest testRestartBookie 1 10.5 0.5
AuditorLedgerCheckerTest testSimpleLedger 1 10.3 0.3
AuditorLedgerCheckerTest testToggleLedgerReplication 1 21.7 1.7
BookieAutoRecoveryTest testClosedLedgers 1 9.7 0.7
BookieAutoRecoveryTest testEmptyLedgerLosesQuorumEventually 1 32.2 3.9
BookieAutoRecoveryTest testLedgerMetadataContainsIpAddressAsBookieID 1 14.1 5.1
BookieAutoRecoveryTest testNoSuchLedgerExists 1 3.3 0.7
BookieAutoRecoveryTest testOpenLedgers 1 12.3 3.5
BookieAutoRecoveryTest testStopWhileReplicationInProgress 1 10.1 1.2
TestReplicationWorker test2RWsShouldCompeteForReplicationOf2FragmentsAndCompleteReplication 1 10.8 2.4
TestReplicationWorker testBookiesNotAvailableScenarioForReplicationWorker 1 8.9 4.2
TestReplicationWorker testDeferLedgerLockReleaseForReplicationWorker 1 47.3 3.7
TestReplicationWorker testRWShouldReplicateTheLedgersAfterTimeoutIfLastFragmentIsUR 1 6.2 4.3
TestReplicationWorker testRWShouldRetryUntilThereAreEnoughBksAvailableForReplication 1 10.1 4.0
TestReplicationWorker testRWZKConnectionLost 1 3.4 2.2
TestReplicationWorker testRWZKConnectionLostOnNonRecoverableZkError 1 4.1 2.0
TestReplicationWorker testReplicateEmptyOpenStateLedger 1 5.6 0.8
AuditorRollingRestartTest testAuditingDuringRollingRestart 1 34.8 4.8
AuditorBookieTest testBookieClusterRestart 1 33.6 3.6
AuditorBookieTest testEnsureOnlySingleAuditor 1 14.7 4.7

All the waits removed are fixed lengths (10 s and 30 s shutdown timeouts, 5 s and 9.4-37.5 s retry delays, 2 s and
1 s sleeps), and the CI times match the local ones, so CI should save about the same. These seven classes took
1,173 s in the 2 Nov 12:18 run (per-class wall time, so AuditorRollingRestartTest includes its cleanup); their Gradle
wall times on the branch add up to 489 s here, startup included. That is ~680 s off the Pulsar Metadata job's ~1,820 s
of test time, and LedgerUnderreplicationManagerTest (~212 s) becomes its longest class.

Mutation checks. Each breaks what a replaced wait or check guards; only the affected methods,
-PtestRetryCount=0, sources restored after each run:

Mutation Test Result
PulsarLedgerAuditorManager.close() fix reverted testCloseReleasesWaitingCandidate (ZooKeeper, Oxia) fails: the candidate is still waiting after 10 s
Replication left enabled during the rolling restart AuditorRollingRestartTest master: passes; branch: fails ("shouldn't be anything under replicated")
No replication worker started testFailedWriteRecovery fails (ensemble wait times out)
Second bookie not killed, so the 10 s delayed audit stays delayed testDelayedAuditWithMultipleBookieFailures fails ("Ledger is not marked as underreplicated")
Bookie restarted before the workers start testBookiesNotAvailableScenarioForReplicationWorker fails ("should be underreplicated")
Bookie restarted before the window in which replication must fail testRWShouldRetryUntilThereAreEnoughBksAvailableForReplication fails ("Expecting that replication should not complete")
Workers' grace period doubled (128 ms) against the 64 ms expectation testDeferLedgerLockReleaseForReplicationWorker fails ("RW1 delayperiod expected:<2> but was:<4>")
m1 releases its lock before m2 polls testLocking x5 stores 5 of 5 fail
A 5th ledger marked before the last poll testBasicInteraction x5 5 of 5 fail
Replication not disabled testDisableLedgerReplication x5, testEnableLedgerReplication x5 10 of 10 fail

Not falsifiable by a mutation: the cleanups after the checks (enabling the replication, cancelling the delayed audit),
the elector shutdown order and getAuditor()'s check (it returns the same auditor; the race the old sleep guarded
against doesn't happen in these tests). BookieAutoRecoveryTest waits on unbounded latches for the same events as
before; only the worker's lock hold time changed.

Regression: the other 13 classes of org.apache.bookkeeper.replication and PulsarLedgerAuditorManagerTest pass
(47 tests). The seven classes and PulsarLedgerAuditorManagerTest together, in the default 4 parallel forks with
-PtestRetryCount=0, twice: 260/260 tests pass both times (278 s and 275 s of Gradle wall; class times within ~3 s of
the single-class runs). ./gradlew quickCheck passes.

Not changed here:

  • LedgerUnderreplicationManagerTest is the longest class left (~210 s): testLockCleanupAfterAmbiguousDelete waits
    out the 30 s BLOCKING_CALL_TIMEOUT in two cases (its check is that the cleanup times out), and
    testLockCleanupAfterSameSessionRecovery the loss of a 12 s ZooKeeper session in two (the outage must stay shorter
    than the session). Overlapping each pair would save ~40 s but means merging data-provider cases.
  • Timer-bound checks kept: the audit delays of AuditorLedgerCheckerTest (3-10 s), its 5 s and 1 s "nothing gets
    marked" windows, testPeriodicCheckWhenDisabled's 2 s windows against a 1 s check interval,
    BookieAutoRecoveryTest's 3 s openLedgerRereplicationGracePeriod, AuditorRollingRestartTest's two 2 s windows.
  • testRWZKConnectionLost teardown (~4 s): clients reconnecting after the test restarted ZooKeeper.
  • Upstream BookKeeper (all still on master): AuditorElector.shutdown() should close the manager or interrupt before
    waiting; Auditor.shutdown() should drop delayed tasks (setExecuteExistingDelayedTasksAfterShutdownPolicy(false))
    or interrupt a task waiting for the replication; PendingReadOp leaks all but the last speculative read task;
    SingleThreadExecutor.awaitTermination() returns the inverse ("The mainWorkerPool did not shutdown cleanly" on every
    client close).
  • PulsarLedgerAuditorManager ignores the listener tryToBecomeAuditor() gets, so an auditor that loses its
    leadership or its session isn't told (BookKeeper's ZooKeeper manager sends SessionLost/VoteWasDeleted). Not
    verified beyond reading the code.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

….tryToBecomeAuditor() on close

A follower in tryToBecomeAuditor() waits on the manager until it becomes
the leader or its session is lost. close() withdrew the candidacy but left
that wait in place, so the calling thread only returned once interrupted.
close() now wakes it up and the call fails like a lost session does.
…etadata

- Stop a bookie's auto recovery before the bookie, as a bookie server does,
  and close an auditor elector's auditor manager before shutting it down:
  AuditorElector.shutdown() otherwise waits 10 s for a follower's election.
- Re-enable the ledger replication that a test disabled, and cancel the
  delayed audit a test left pending, before the auditors stop: an auditor
  waits 30 s for a task blocked on either.
- Run the auto recovery of all the BookieAutoRecoveryTest bookies on the
  Pulsar metadata drivers, not only of those started by the tests.
- Shorten the replication worker retry backoff and the lock hold time of a
  failed ledger where no test checks them, and disable the speculative
  reads that delay closing the TestReplicationWorker client.
- Wait for the replication or the worker state instead of fixed sleeps,
  and check that a getLedgerToRereplicate() call is waiting after a scan
  instead of waiting 1 s for it not to return.
…tTest

The test checked that nothing got audited during the restart with
pollLedgerToRereplicate(), which skips the ledgers another client has
locked. With the replication enabled, the auto recovery worker locks the
ledger that the auditor marks, fails to replicate it with a bookie down and
holds the lock, so the check passed even when the auditor ran.
…t stops an elector

AuditorBookieTest stops the elector of a follower bookie in
testBookieClusterRestart and testEnsureOnlySingleAuditor, and each of those
shutdowns waited 10 s for the follower's election.
@merlimat merlimat added this to the 5.1.0 milestone Nov 2, 2026
@merlimat
merlimat merged commit d26baab into apache:master Nov 2, 2026
82 of 84 checks passed
@merlimat
merlimat deleted the mmerli/faster-replication-tests branch November 3, 2026 00:50
lhotari pushed a commit that referenced this pull request Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants