Repository navigation
perf(beacon): back the participation lists and pending deposits with persistent trees - #762
Draft
MegaRedHand wants to merge 7 commits into
Draft
MegaRedHand wants to merge 7 commits into
MegaRedHand wants to merge 7 commits into
Conversation
…persistent trees On mainnet (about 2.4M validators) the post-block state_root check costs about 27 ms per block, and perf attributes most of BeaconState::hash_tree_root to plain SszList fields re-merkleized from scratch on every call: the two participation lists (2.4 MB each) and pending_deposits. Lighthouse keeps every state list in milhouse trees with cached node hashes. EpochParticipation (altair through fulu), PendingDeposits (electra, fulu) and their gloas progressive counterparts now use ethlambda_ssz_tree lists. Only the current epoch's participation list is written by a block, so the previous epoch's list and an untouched deposit queue reuse their cached roots. The SSZ encoding and roots are unchanged (tests compare against the flat lists), so DB_VERSION is not bumped. Participation writes are buffered densely, not in a BTreeMap: at one slot's worth of scattered attesters the BTreeMap inserts cost several times the hashing they save. A quick ignored benchmark at 2.4M entries and 75k writes measures about 13 ms for the flat lists against about 5 ms for the trees. Reads go through ParticipationRef and ParticipationMut, since a tree has no slice view. The participation scan walks the list in lockstep with the registry instead of a tree descent per validator. Both lists are also rebased onto a resident relative of a state decoded from storage.
This was referenced Nov 2, 2026
Collaborator
Author
|
Combined result for the four Lighthouse ports (#759 pubkey index, #760 column KZG, #761 progressive target balances, #762 tree-backed participation), squashed onto Offline import replay on ethlambda-5, same three mainnet corpora, both legs built at the commit and run with
Per STF step, ordinary blocks (n=162): Live on the ethlambda-2 mainnet follower (nightly 1d06bf9): steady |
…peat Three primitives the participation-list fixes that follow need, kept in their own commit so the fixes read as pure call-site changes. - `List::set` / `ProgressiveList::set` buffer a value without first descending the tree to copy the old one into the write buffer, which `get_mut` does even when the caller already knows the old value. - `sorted_reader()` reads non-decreasing indices keeping the current leaf's slice, descending only when an index leaves it. Pending writes are consulted first, exactly as `get` does, so the answers are identical. - `List::repeat` / `ProgressiveList::repeat` build a list of identical elements by sharing one full leaf and one full inner node per level, each caching its hash once, so building and hashing cost the tree height plus the partial last leaf instead of the whole length. Nodes are immutable and every write copies the path it touches, so a shared leaf can never be mutated; tests write into a repeated list and check its clone and the untouched leaves, and compare roots against the Vec-built list at lengths around every leaf and node boundary.
…rotation Two micro-benchmarks next to `participation_root_after_a_block_of_writes_bench`, at 2.4M validators: the participation part of one `process_attestation` (about 75k ascending reads, then the writes) in each combination of per-index `get`/sorted reader and `get_mut`/`set`, and the epoch rotation's new list built from a zeroed `Vec` against `List::repeat`, with the root that follows. Run with `cargo test --profile release-fast -p ethlambda-types --lib -- --ignored --nocapture attestation_pass_bench participation_rotation_bench`.
The write phase did `get_mut(index)` and `*flags |= new_flags` for every attester. On a tree-backed list `get_mut` descends the tree to copy the old value into the write buffer, although the read phase had already read it as `current_flags`. Micro-benchmark at 2.4M validators and about 74k ascending attesters (best of five, release-fast): the write phase takes 2.16 ms with `get_mut` and 0.94 ms with the blind `set`. The read phase now records `current_flags | new_flags` and the write phase stores it with the new `ParticipationMut::set`. That is exactly what `|=` would leave: the attesting indices are distinct (members of disjoint committees, and the signature set built before the read phase refuses a repeated index), so each index is written once, and nothing writes the list between the two phases, so the stored value is still `current_flags`. Applies to the altair, deneb, electra (electra and fulu) and gloas versions of `process_attestation`.
… reader Once the participation lists became trees, the read phase of `process_attestation` paid a full tree descent per attester (`get` was a slice index before). An attestation carries tens of thousands of attesters. Its indices are sorted ascending and consecutive ones almost always share a leaf, which holds thousands of flags. `sorted_reader()` keeps the current leaf's slice and descends only when an index leaves it, consulting pending writes first exactly as `get` does, so the answers are identical. Micro-benchmark at 2.4M validators and about 74k ascending attesters (best of five, release-fast): the read phase takes 0.70 ms per-index and 0.10 ms with the reader; with the blind write from the previous commit the whole participation pass goes from 2.86 ms to 1.03 ms. Every `process_attestation` (altair, deneb, electra and fulu, gloas) now uses it; the reader debug-asserts that indices do not decrease, which `get_attesting_indices` guarantees by sorting.
…o leaf `process_participation_flag_updates` installed the new current list as `vec![0; validator_count]` converted into a tree: about 2.4M bytes allocated and filled into hundreds of separate leaves, every one of which the next state root then hashed. At epoch boundaries `process_epoch` went from 96.3 ms to 105.5 ms in the offline import replay once the lists became trees. `List::repeat` / `ProgressiveList::repeat` (added with the other ssz-tree primitives) build the list from one full leaf and one full inner node per level, each shared through its `Arc` with its cached hash, so building and hashing cost the tree height plus the partial last leaf. Micro-benchmark at 2.4M validators (release-fast, steady state over three runs): the old build plus the root takes 3.0 to 6.0 ms, the repeated list 0.10 to 0.13 ms. Nodes are immutable and every write copies its path, so sharing is invisible to later writes; the roots and SSZ bytes are tested equal to the flat list's at lengths around every leaf and node boundary, and a write to a repeated list is tested to leave its clone and the other leaves alone. Used by the altair-through-fulu rotation, gloas's own copy of it, and the altair upgrade's two fresh lists. PendingDeposits is rebuilt from a `Vec` once per epoch by `process_pending_deposits`, which only drops a prefix and appends postponed deposits. The tree cannot keep the suffix's cached hashes: leaves are aligned runs, so dropping a prefix that is not a multiple of a leaf shifts every later element into a different leaf, and the API has no operation that re-chunks while carrying cached element roots. An ignored benchmark in `ethlambda-types` puts the cost at about 7 ms for the copy and rebuild plus about 24 ms to hash, for a queue of 200k deposits (about 1 ms plus 3 ms at 20k), once per epoch. The hash was paid on every block before the trees, so this is left as it is.
Collaborator
Author
|
Per-PR attribution on
#760 does not run in the replay (no custody columns), so the second step is #761's. Regressions found by the per-step split, and what the follow-up commits (now pushed to #761 and #762) did about them:
ethlambda-2 (mainnet follower) runs nightly 2b51a33 since 2026-11-03 14:42 UTC. |
MegaRedHand
added
performance
2 of 3 tasks
3 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description / Motivation
The post-block state root check cost ~27 ms per block on the mainnet follower. Perf put 87% of
BeaconState::hash_tree_rootin plainSszListfields re-merkleized from scratch on every call: the two participation lists (2.4 MB each) andPendingDeposits; only 13% was in the tree-backedethlambda_ssz_treelists. Lighthouse keeps every state list inmilhousepersistent trees with cached node hashes.What Changed
EpochParticipation(altair through fulu) isethlambda_ssz_tree::List<u8, LIMIT>with the denseVecMapwrite buffer;PendingDeposits(electra, fulu) isList<PendingDeposit, LIMIT, BTreeMap>; gloas's areethlambda_ssz_tree::ProgressiveList. All intree_fields!, soapply_pending_mutationsflushes them, andBeaconState::rebase_onshares them with a resident relative.altair_validator_lists()returnsParticipationRefs,epoch_participation_mut()aParticipationMut.helpers/participation.rs::scanwalks both participation iterators in lockstep withiter_validators()instead of an indexed read per validator.process_attestationvariants, epoch rotation, fork upgrades andprocess_pending_deposits;docs/beacon_stf.mdandCLAUDE.mdupdated.Correctness / Behavior Guarantees
DB_VERSIONbump: new tests compare the old and new representations at lengths 0, 1, 31, 32, 33, 1000, 65541 and 300001 (bounded and progressive participation, pending deposits), plus writes and a push after a warm hash.BTreeMapbuffer was a clear loss for participation (75k scattered writes: 29-32 ms vs 13 ms flat); the denseVecMapbuffer gives ~5 ms for the two lists.Benchmarks
Offline
ethlambda benchmark import replayon ethlambda-5 (Ryzen 9700X), three mainnet corpora, 2 rounds, steady state n=164, paired per block against main012dc4f49:Per round -15.3% / -13.6%. Epoch-boundary blocks (n=2) -4.9%.
Follow-up commits: regressions this PR introduced
Measured on nightly with all four Lighthouse ports, this PR's first commit cost
attestations+1.5 ms per block (a tree descent per attester to read its flags, and another perget_mutwrite) andprocess_epoch+9.1 ms at epoch boundaries. The last five commits address that:ethlambda_ssz_tree:List::set/ProgressiveList::set(a blind buffered write),sorted_reader()(reads non-decreasing indices, descending only when an index leaves the leaf last read), andrepeat(value, len)(one full leaf and one full inner node per level, shared, each hashing once).process_attestation: the read phase uses the sorted reader (attesting indices are sorted) and recordscurrent | new; the write phase stores it withset. That is exact because an attestation's indices are distinct and nothing writes the list between the two phases.repeat.Micro-benchmarks (2.4M validators, ~74k ascending attesters): read 0.56-0.70 -> 0.10 ms, write 1.6-2.2 -> ~1.0 ms; rotation plus the next hash 3-6 ms -> 0.1-0.2 ms.
On nightly, together with #761's follow-up, offline replay on ethlambda-5, paired, steady state n=164:
attestations(ordinary blocks)process_epoch(epoch boundaries)Still above nightly before the ports (2.9 ms and 97.2 ms):
attestations+1.4 ms (shared with #761) andprocess_epoch+7.6 ms per epoch boundary; being profiled.PendingDepositsis left as it is: dropping a processed prefix that is not a multiple of a leaf's element count shifts every later element into another leaf, so the cached hashes cannot be kept without a re-chunking operation. Its per-epoch cost is about 1 ms plus 3 ms of hashing for 20k deposits, against a hash that used to be paid on every block.Tests Added / Run
participation_root_after_a_block_of_writes_bench(cargo test --profile release-fast -p ethlambda-types --lib -- --ignored --nocapture participation_root).set, the sorted reader andrepeatchecked againstgetand against Vec-built lists (roots and SSZ bytes) at lengths around every leaf and inner-node boundary up to 2.4M; writes into a repeated list throughset,get_mut,iter_cow,rebase_onandpushleave the clone and untouched leaves unchanged. Ignored benchesattestation_pass_bench,participation_rotation_bench,pending_deposits_epoch_rebuild_bench.cargo test --profile release-fast -p ethlambda-ssz-tree --lib(80, also in the dev profile),-p ethlambda-types --lib(237),-p ethlambda-state-transition --lib(630),-p ethlambda-storage --lib(210);cargo checkof blockchain, rpc, p2pssz_static/1761,sanity/595,epoch_processing/804,/attestation/356,fork/138,transition/172,/random/208,fork_choice/206,/deposit_request/43,rewards/372,finality/41Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test— not run; the crate lib tests and spec filters above were run instead