perf(beacon): cache fork-choice vote weights in a node array by MegaRedHand · Pull Request #775 · lambdaclass/ethlambda

MegaRedHand · 2026-11-04T15:32:06Z

🗒️ Description / Motivation

get_head costs ~17-18 ms per imported block on the mainnet follower (12.4 ms with #767's faster hasher), and a live import pays it twice for the first timely block of a slot: on_block's proposer-boost gate runs walk_head, then the chain actor runs get_head. Most of it is the vote tally in compute_node_weights: every call walks every validator's latest message (~1M on mainnet) and hashes each into a map.

This keeps the vote weights cached in a node array instead, updated when a batch of votes arrives, so a head computation no longer touches the validator set.

What Changed

crates/storage/src/proto_array.rs (new): the cache. It uses Lighthouse unstable's proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL weights (own votes and cached subtree totals). Each validator's counted vote is remembered by node index.
- A vote batch moves own weights and marks the array dirty. The next read refreshes every total in one reverse pass over the blocks.
- No best_child/best_descendant pointers. Under gloas the choice between children moves without any vote: the previous slot's EMPTY/FULL weigh 0, the PTC tiebreaker, a FULL branch appearing when its envelope verifies, and boost gating. That is why Lighthouse stopped maintaining them too, and the walk from the justified node is cheap.
Store (crates/storage/src/store.rs):
- The array lives in BeaconScratch, under the lock that guards latest_messages, so a vote and its weight change are one step.
- New record_latest_messages: one attestation is one batch, under one lock. It replaces the per-attester loop that took the lock three times per attester.
- New insert_fork_choice_node, which on_block calls just before insert_signed_block.
- New proto_array_weights: the read.
- Hooks on set_latest_message, insert_equivocating_index, prune_live_chain and delete_live_chain_entries.
fork_choice.rs:
- walk_head reads cached_node_weights instead of compute_node_weights.
- The proposer boost moves into add_proposer_boost, which both paths share. It depends on the clock, as do gloas's previous-slot rule and the walk bound, so all three are applied at read time and never cached.
- update_latest_messages makes one record_latest_messages call.
Metrics: lean_beacon_proto_array_rebuilds_total{reason} and lean_beacon_proto_array_rebuild_seconds (documented in docs/metrics.md).
foldhash is now a dependency of ethlambda-storage, for the root lookups. It was already in the lockfile, so no new version is pulled in.

Correctness / Behavior Guarantees

Same weights as before. For every node at or above the walk bound, cached_node_weights equals compute_node_weights on the same store. compute_node_weights stays, public, as the reference, like compute_head and gloas_get_head.
Staleness. Whatever the cache cannot follow incrementally marks it stale. The next read then rebuilds it from the latest messages, at the old tally's cost. That covers:
- finality pruning and live-window removals (payload invalidation, restart head repair);
- a restart;
- a new justified balances snapshot;
- the gloas rule switch;
- a block reaching LiveChain without its node.
On a live node that is about one rebuild per epoch, on the first head computation after the justified checkpoint or finality moves, which can land inside a block import.
Race-safe rebuilds. A rebuild scans LiveChain itself under the lock rather than trusting the caller's index. An API read that scanned before a removal therefore cannot rebuild the removed blocks back in.
Rule set. The cache records the rule set it was built under. Pre-gloas rules read every block as a single full node, as the tally does.
Underflow. A weight that would go below zero (only a bug reaches that) saturates, marks the cache stale and logs a warning; the next read repairs it.
No behavior change in what fork choice selects.

Benchmarks

Local micro-benchmark, cached_weights_timing (ignored test): 1M validators, 73 blocks, on a laptop under load, so the tally varied 27 to 60 ms between runs.

	time
tally (`compute_node_weights`, the old path)	27 ms
cached read, no new votes	7 µs
cached read after a 15,625-vote batch	49 µs
the batch itself (`record_latest_messages`, at vote arrival)	1.3 ms
a rebuild (about once per epoch)	17 ms

The eth-5 import replay has not been run yet; numbers will follow in a comment.

Tests Added / Run

Differential testing against compute_node_weights:

Proptest (proto_array::tests::cached_weights_match_a_fresh_tally). After every operation of a random sequence the cached weights equal a naive tally, and the incrementally kept array equals a fresh rebuild node by node. Operations: block insertions, vote batches, equivocations, balance swaps, rule switches, stale marks.
Random trees (the_bottom_up_walk_matches_the_spec_literal_references_on_random_trees): cached weights equal the tally at every step of every seed.
New the_cached_weights_follow_votes_blocks_and_equivocations. Feeds votes, blocks and equivocations through update_latest_messages, on_block's insertion order and insert_equivocating_index, under both rule sets. After each step it checks the incremental cache against a rebuild and against the tally, and the head against the spec-literal reference.
Fixtures. check_head_against_reference now also compares cached weights with the tally at every head check of every fork-choice fixture.
Mutation checks, all reverted:
- a wrong parent bucket fails the proptest;
- dropping a vote from each batch fails the new test;
- the same bug fails 33 of 206 mainnet fork-choice fixture cases, so the fixture check reads the incremental cache, not a fresh rebuild.

Run:

cargo test --workspace --profile release-fast --lib --bins
make test-beacon (v1.7.0-beta.2 fixtures, both presets):
- mainnet: lib 647, spec 7,493, blockchain with beacon-spec-tests 335;
- minimal: lib 648, spec 49,577;
- all passing.

Related Issues / PRs

Supersedes perf(beacon): hash fork-choice votes with a seeded fast hasher #767: its faster hasher sped up the tally, which is no longer on the head computation's path.
Related to perf(beacon): resolve pubkeys through a cached index and reuse the total active balance in churn #759, perf(beacon): stop re-verifying stored custody columns at block import #760, perf(beacon): keep unslashed target balances incrementally for the pulled-up tip #761, perf(beacon): back the participation lists and pending deposits with persistent trees #762 (the other block-import optimizations from the same profile).

✅ Verification Checklist

Ran make fmt — clean
Ran make lint (clippy with -D warnings) — clean
Ran make test equivalent: cargo test --workspace --profile release-fast --lib --bins plus make test-beacon (the leanSpec spec suites were not run)

get_head tallied every validator's latest message on every call, about 1M on mainnet, and a live import pays it twice for the first timely block of a slot (on_block's boost-gate walk, then the actor's get_head). That tally was most of get_head's ~17 ms in the eth-5 replay. The store now keeps the vote weights in a node array (Lighthouse unstable's proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL weights). One attestation is one vote batch: Store::record_latest_messages records the messages and moves the attesters' weight under one lock, and the next read refreshes the subtree totals in one reverse pass over the blocks. The head walk reads them through cached_node_weights. The proposer boost, gloas's previous-slot rule and the walk bound depend on the clock, so they stay read-time. There are no best-child pointers: under gloas the choice moves without any vote, which is why Lighthouse stopped maintaining them. What the cache cannot follow incrementally (finality pruning, a live-window removal, a restart, a new justified balances snapshot, the gloas rule switch, a block stored without its node) marks it stale, and the next read rebuilds it from the latest messages at the old tally's cost, about once per epoch. A rebuild rescans LiveChain under the lock, so an API read that scanned before a removal cannot rebuild the removed blocks back in. compute_node_weights stays, as the reference. Equivalence is checked by a proptest against a naive tally (with the incremental state also equal to a fresh rebuild after every operation), on the random trees at every step of every seed, by a new test that feeds votes, blocks and equivocations through update_latest_messages, on_block's insertion order and insert_equivocating_index, and at every head check of every fork-choice fixture. Local benchmark (1M validators, 73 blocks, laptop): the tally 27 ms; a warm cached read 7 us; a read after a 15.6k-vote batch 49 us; the batch itself 1.3 ms; a rebuild 17 ms.

…r" (#767) Reverts nightly's a20c52e. #767 is closed in favor of #775, which takes the per-validator vote tally off the head computation's path: the head walk reads cached vote weights, and compute_node_weights, the tally #767 sped up, stays only as the reference the tests compare against.

get_head tallied every validator's latest message on every call, about 1M on mainnet, and a live import pays it twice for the first timely block of a slot (on_block's boost-gate walk, then the actor's get_head). That tally was most of get_head's ~17 ms in the eth-5 replay. The store now keeps the vote weights in a node array (Lighthouse unstable's proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL weights). One attestation is one vote batch: Store::record_latest_messages records the messages and moves the attesters' weight under one lock, and the next read refreshes the subtree totals in one reverse pass over the blocks. The head walk reads them through cached_node_weights. The proposer boost, gloas's previous-slot rule and the walk bound depend on the clock, so they stay read-time. There are no best-child pointers: under gloas the choice moves without any vote, which is why Lighthouse stopped maintaining them. What the cache cannot follow incrementally (finality pruning, a live-window removal, a restart, a new justified balances snapshot, the gloas rule switch, a block stored without its node) marks it stale, and the next read rebuilds it from the latest messages at the old tally's cost, about once per epoch. A rebuild rescans LiveChain under the lock, so an API read that scanned before a removal cannot rebuild the removed blocks back in. compute_node_weights stays, as the reference. Equivalence is checked by a proptest against a naive tally (with the incremental state also equal to a fresh rebuild after every operation), on the random trees at every step of every seed, by a new test that feeds votes, blocks and equivocations through update_latest_messages, on_block's insertion order and insert_equivocating_index, and at every head check of every fork-choice fixture. Local benchmark (1M validators, 73 blocks, laptop): the tally 27 ms; a warm cached read 7 us; a read after a 15.6k-vote batch 49 us; the batch itself 1.3 ms; a rebuild 17 ms. Cherry-picked from perf/beacon-proto-array at b4adbaf. Adapted onto nightly: both vote writers (set_latest_message and the new record_latest_messages) grow the dense vote table through one grow_latest_messages helper, which keeps #740's exact growth plus headroom; the storage and fork-choice imports are merged with #759's and #761's caches. Replaces #767, reverted in the previous commit.

…ches On mainnet a block (c2302e2b, slot 15587649) took 826 ms to import, 766 ms of it queued in the chain actor's mailbox behind the slot tick, whose FCR run took 1-2 s. The rule runs on the actor, so it has to cost about what the head computation does. It now reads what fork choice already keeps: - get_current_target_score skips latest messages from other epochs, which can never vote for the current target. Resolving their checkpoints walked roots below the live chain index, decoding a block per parent hop; that walk was the >1 s tail. - Balance sources come from the store's per-checkpoint JustifiedBalances, which now also carry the active-but-slashed validators and the total active balance, instead of a registry pass per source per run. The pulled-up head reads the current target's entry unless the head state shows a slashing processed in the current epoch, and then builds from the pulled-up state as before: activity and effective balances are fixed within an epoch, and slashings[epoch % EPOCHS_PER_SLASHINGS_VECTOR] starts every epoch at zero. - Attestation scores are the proto-array's PENDING subtree totals (#775) when the source is the justified checkpoint's, which it is on every run but an epoch's first; the remaining fold hashes roots with foldhash. - Committee weights are memoized per (source, slot), and a range across an epoch boundary reuses the later part's members, shared by every range ending at the same slot; equivocators are a bitmap. At 1M validators (the ignored fast_confirmation_timing bench), per run: mid-epoch ~27 ms -> ~0.5 ms (get_head: ~80 us); an epoch's first slot 113 ms -> 14.7 ms (get_head: 12.9 ms); its second and third ~40 ms -> 2.3-2.8 ms. New tests compare the cached scores with the fold, pre-gloas and under gloas; the filtered target score with the unfiltered sum; and the reused pulled-up balances with the pulled-up state's, with and without a slashing in the epoch. Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

…ches On mainnet a block (c2302e2b, slot 15587649) took 826 ms to import, 766 ms of it queued in the chain actor's mailbox behind the slot tick, whose FCR run took 1-2 s. The rule runs on the actor, so it has to cost about what the head computation does. It now reads what fork choice already keeps: - get_current_target_score skips latest messages from other epochs, which can never vote for the current target. Resolving their checkpoints walked roots below the live chain index, decoding a block per parent hop; that walk was the >1 s tail. - Balance sources come from the store's per-checkpoint JustifiedBalances, which now also carry the active-but-slashed validators and the total active balance, instead of a registry pass per source per run. The pulled-up head reads the current target's entry unless the head state shows a slashing processed in the current epoch, and then builds from the pulled-up state as before: activity and effective balances are fixed within an epoch, and slashings[epoch % EPOCHS_PER_SLASHINGS_VECTOR] starts every epoch at zero. - Attestation scores are the proto-array's PENDING subtree totals (#775) when the source is the justified checkpoint's, which it is on every run but an epoch's first; the remaining fold hashes roots with foldhash. - Committee weights are memoized per (source, slot), and a range across an epoch boundary reuses the later part's members, shared by every range ending at the same slot; equivocators are a bitmap. At 1M validators (the ignored fast_confirmation_timing bench), per run: mid-epoch ~27 ms -> ~0.5 ms (get_head: ~80 us); an epoch's first slot 113 ms -> 14.7 ms (get_head: 12.9 ms); its second and third ~40 ms -> 2.3-2.8 ms. New tests compare the cached scores with the fold, pre-gloas and under gloas; the filtered target score with the unfiltered sum; and the reused pulled-up balances with the pulled-up state's, with and without a slashing in the epoch. Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

…ash (#749): run off the fork choice's caches On mainnet a block (c2302e2b, slot 15587649) took 826 ms to import, 766 ms of it queued in the chain actor's mailbox behind the slot tick, whose FCR run took 1-2 s. The rule runs on the actor, so it has to cost about what the head computation does. It now reads what fork choice already keeps: - get_current_target_score skips latest messages from other epochs, which can never vote for the current target. Resolving their checkpoints walked roots below the live chain index, decoding a block per parent hop; that walk was the >1 s tail. - Balance sources come from the store's per-checkpoint JustifiedBalances, which now also carry the active-but-slashed validators and the total active balance, instead of a registry pass per source per run. The pulled-up head reads the current target's entry unless the head state shows a slashing processed in the current epoch, and then builds from the pulled-up state as before: activity and effective balances are fixed within an epoch, and slashings[epoch % EPOCHS_PER_SLASHINGS_VECTOR] starts every epoch at zero. - Attestation scores are the proto-array's PENDING subtree totals (#775) when the source is the justified checkpoint's, which it is on every run but an epoch's first; the remaining fold hashes roots with foldhash. - Committee weights are memoized per (source, slot), and a range across an epoch boundary reuses the later part's members, shared by every range ending at the same slot; equivocators are a bitmap. At 1M validators (the ignored fast_confirmation_timing bench), per run: mid-epoch ~27 ms -> ~0.5 ms (get_head: ~80 us); an epoch's first slot 113 ms -> 14.7 ms (get_head: 12.9 ms); its second and third ~40 ms -> 2.3-2.8 ms. New tests compare the cached scores with the fold, pre-gloas and under gloas; the filtered target score with the unfiltered sum; and the reused pulled-up balances with the pulled-up state's, with and without a slashing in the epoch. Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

…740): review fixes - fix(storage): reserve the vote table once per attestation, not per index. An aggregate lists its attesters in ascending order, so growing the table one index at a time copied it whenever an index passed the headroom: hundreds of copies of a registry-sized table for one aggregate spanning it. Store::reserve_latest_messages makes room up to the highest index first. - refactor(storage): check the vote table's capacity arithmetic (checked_add), and say why the resize after it cannot reallocate. - test(storage): assert the growth contract (index plus headroom) rather than a ratio, name the test after it, and check that a reserved batch of ascending votes never moves the table. On nightly the table has a second writer, #775's Store::record_latest_messages, which records a whole attestation under one lock and is the only one on the production path. It reserves for the batch's highest index itself, under that lock, so #740's call from fork_choice::update_latest_messages is not needed here; both writers share the slot and reservation helpers. Updates nightly's #740 from fix/storage-latest-messages-exact-growth at 83aabfc to be37485 (the branch head, bfb4697, adds only a merge of main). Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

MegaRedHand added performance Performance improvements or possible performance improvements beacon Ethereum Beacon Chain client labels Nov 4, 2026

MegaRedHand mentioned this pull request Nov 4, 2026

perf(beacon): hash fork-choice votes with a seeded fast hasher #767

Closed

2 of 3 tasks

MegaRedHand mentioned this pull request Nov 5, 2026

feat(beacon): fast confirmation rule, its event and the engine safe hash #749

Draft

5 of 6 tasks

style(storage): rustfmt the equivocating check in set_latest_message

b951d47

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

Navigation Menu

Repository navigation

perf(beacon): cache fork-choice vote weights in a node array - #775

perf(beacon): cache fork-choice vote weights in a node array#775
MegaRedHand wants to merge 2 commits into
mainfrom
perf/beacon-proto-array

MegaRedHand commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MegaRedHand commented Nov 4, 2026

🗒️ Description / Motivation

What Changed

Correctness / Behavior Guarantees

Benchmarks

Tests Added / Run

Related Issues / PRs

✅ Verification Checklist

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant