Skip to content

fix(storage): grow the beacon vote table exactly instead of doubling - #740

Merged
pablodeymo merged 9 commits into
mainfrom
fix/storage-latest-messages-exact-growth
Nov 18, 2026
Merged

pablodeymo merged 9 commits into
mainfrom
fix/storage-latest-messages-exact-growth

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

Store::latest_messages (the beacon fork choice's latest vote per validator) is a dense table indexed by validator index. set_latest_message grew it with resize, which at least doubles the capacity whenever an index lands past it. A registry that keeps gaining validators crosses that edge soon after the table is first sized. So on our mainnet follower the table had room for about twice the registry, and roughly half of its 256 MB (heap profile) was never used.

Exact growth has its own trap: an indexed attestation's attesters are sorted ascending. Growing the table one index at a time, each index past the headroom reallocated and copied the whole table, so the first aggregate spanning the registry after a start copied a registry-sized table hundreds of times while holding the beacon lock.

What Changed

  • crates/storage/src/store.rs:
    • When a new index is past the capacity, the table grows to that index plus LATEST_MESSAGES_HEADROOM (4096) spare entries with reserve_exact, then resizes as before. The capacity arithmetic is checked, and the growth lives in one helper, reserve_latest_message_slot.
    • New Store::reserve_latest_messages(highest): makes room for every index up to highest in one reallocation at most.
  • crates/blockchain/state_transition/src/beacon/fork_choice.rs: update_latest_messages reserves for the batch's highest index before recording it. The highest is taken by max, not by position, so an unsorted caller is still covered.

Correctness / Behavior Guarantees

  • Same contents and indexing; only the allocation size changes.
  • One attestation reallocates the table once at most, whatever order its indices come in.
  • A registry growing by a few validators an epoch still reallocates rarely, once per headroom's worth of new indices.
  • Expected saving on the mainnet follower: about 120 MB.

Tests Added / Run

  • the_latest_message_table_grows_with_headroom_rather_than_doubling:
    • the first index sizes the table to at most that index plus the headroom;
    • the highest index inside the headroom leaves the capacity unchanged;
    • an index past the headroom grows the table to at most the new index plus the headroom.
  • a_reserved_batch_of_ascending_votes_never_moves_the_table: indices spaced wider than the headroom, recorded in ascending order after one reservation, never move the table's buffer or change its capacity.
  • cargo test --profile release-fast -p ethlambda-storage --lib: 212 passed, 1 ignored.
  • cargo test --profile release-fast -p ethlambda-state-transition --lib fork_choice: 93 passed.

Related Issues / PRs

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran make lint (clippy with -D warnings) — clean. Ran cargo clippy --locked -p ethlambda-storage -p ethlambda-state-transition --all-targets -- -D warnings (clean), not the full workspace target
  • Ran make test (test-consensus plus test-node, at release-fast) — all passing. Ran only the tests above

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 740: grow the latest-message table exactly rather than by doubling

The change looks correct and I found nothing blocking.

Correctness

  • In set_latest_message, reserve_exact(slot + 1 + HEADROOM - len) guarantees capacity >= slot + 1 + HEADROOM. The resize(slot + 1, None) after it therefore never reallocates.
  • The capacity() <= slot guard is the right condition. Indices that land inside leftover capacity skip the reserve call, so no reallocation happens.
  • The subtraction slot + 1 + HEADROOM - messages.len() can't underflow, because the outer check already gives len <= slot.
  • MAX_PLAUSIBLE_VALIDATOR_INDEX (2^26) bounds slot above this code, so the addition can't overflow.
  • The lock is still held across the whole operation, and taking messages as &mut beacon.latest_messages avoids repeated field access.

Performance

  • The tradeoff is sound: for a roughly 1M-entry registry it saves about half the table's memory, and the cost is more frequent reallocation.
  • The cost shows up when indices first arrive in ascending order across a cold start (checkpoint sync or restart). Each time an index passes the headroom, the whole table is copied. With a 4096-entry headroom that is about N/4096 copies, roughly 250 for 1M validators.
  • That is acceptable. In practice a high index usually arrives early and sizes the table in one step.
  • Nit: if this ever shows up in profiles, make the headroom proportional, for example a few percent of slot, instead of a fixed 1 << 12. I wouldn't do it now.

Tests

  • the_latest_message_table_grows_exactly_rather_than_doubling covers three cases: first sizing, a higher index inside the headroom (no reallocation), and growth past the headroom (grown < 2 * sized).
  • The assertions on capacity() are slightly allocator-dependent: reserve_exact guarantees only >=. The upper-bound checks do pass in practice, since Vec honors reserve_exact exactly. This is fine.

Style

  • The comment explains why the code departs from resize's doubling, which fits the surrounding code.
  • The constant's doc comment links Store::set_latest_message correctly.

Approve.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

I found one correctness concern.

  • crates/storage/src/store.rs:4290 — reserve_exact does not guarantee the final capacity will be slot + 1 + LATEST_MESSAGES_HEADROOM; allocators are free to over-allocate. The new test at crates/storage/src/store.rs:9148 and crates/storage/src/store.rs:9158 assumes an upper bound on capacity, so it can be allocator/platform dependent and flaky even if the production code is fine.
  • crates/storage/src/store.rs:9133 — because the test asserts capacity() <= target + headroom, it is checking a property Rust’s Vec API does not promise. A safer test would assert the behavioral contract only: no reallocation within the headroom window, and growth occurs once crossing it.
  • crates/storage/src/store.rs:4291 — arithmetic is currently safe under today’s MAX_PLAUSIBLE_VALIDATOR_INDEX, but the expression is brittle if that bound changes. Using named locals plus saturating_sub/checked math would make this consensus-adjacent storage path easier to audit.

No fork-choice / attestation / justification / SSZ logic changes are in this PR; the production change is a localized memory-usage optimization and otherwise looks reasonable.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which optimizes memory growth for the latest_messages vector in the Beacon Chain store.

Overall Assessment

This is a well-scoped optimization PR that addresses a real memory inefficiency. The logic is mostly sound, but I have several concerns ranging from edge cases to code clarity.


Detailed Findings

1. Potential Panic on Integer Overflow (Lines 4288-4290)

if messages.capacity() <= slot {
    messages.reserve_exact(slot + 1 + LATEST_MESSAGES_HEADROOM - messages.len());
}

Bug: slot + 1 + LATEST_MESSAGES_HEADROOM can overflow usize on 32-bit systems. slot is derived from a u64 validator index (capped at 1 << 26 ≈ 67M), and LATEST_MESSAGES_HEADROOM is 1 << 12 = 4096. While this won't overflow on 64-bit, Rust's usize is 32-bit on some targets. The expect at line 4287 already assumes usize >= u64::MAX behavior isn't hit, but this addition is unchecked.

Fix: Use checked_add or saturating_add:

let new_cap = slot
    .checked_add(1 + LATEST_MESSAGES_HEADROOM)
    .expect("validator index + headroom fits in usize");
messages.reserve_exact(new_cap.saturating_sub(messages.len()));

2. reserve_exact Followed by resize is Redundant (Lines 4289-4291)

messages.reserve_exact(slot + 1 + LATEST_MESSAGES_HEADROOM - messages.len());
messages.resize(slot + 1, None);

Issue: reserve_exact allocates capacity, then resize both grows capacity and initializes elements. But resize may reallocate if the capacity isn't sufficient. The intent is to separate allocation from initialization, but resize will do its own capacity check.

Actually, re-reading: reserve_exact ensures capacity() >= len() + additional, so resize won't reallocate. However, this is subtle. A comment explaining why reserve_exact + resize is used instead of just resize would help.

Suggestion: Add comment: // resize won't reallocate due to reserve_exact above.


3. Test Name Mismatch with Behavior (Line 9132)

fn the_latest_message_table_grows_exactly_rather_than_doubling()

Issue: The function name claims "grows exactly," but the implementation uses reserve_exact which allocates at least the requested capacity—the allocator may round up. The test assertion allows for this:

assert!(sized <= usize::try_from(first).unwrap() + 1 + LATEST_MESSAGES_HEADROOM);

The name oversells the guarantee. Consider grows_with_headroom_rather_than_doubling.


4. Test Fragility: 2 * sized Assertion (Line 9159)

assert!(grown < 2 * sized);

Issue: This relies on sized being large enough that 2 * sized exceeds the headroom-based allocation. If first were small (e.g., 10), and headroom is 4096, then sized ≈ 4107, and grown for higher = 10 + 4096 + 1000 = 5106 would be ~5107, which is not < 2 * 4107 = 8214. The test passes with first = 100_000 by coincidence of scale.

Actually, let me recalculate: sized ≈ 100_000 + 1 + 4096 = 104_097. higher = 100_000 + 4096 + 1000 = 105_096. grown ≈ 105_096 + 1 + 4096 = 109_193. 2 * sized = 208_194. Yes, 109_193 < 208_194. The test works but is structurally fragile.

Suggestion: Make the anti-doubling assertion more robust by computing the expected "doubled" value explicitly:

let doubled = sized.checked_mul(2).unwrap_or(usize::MAX);
assert!(grown < doubled, "grown={grown} should be less than doubled={doubled}");

Or better, assert the exact expected growth formula:

let expected_grown = usize::try_from(higher).unwrap() + 1 + LATEST_MESSAGES_HEADROOM;
assert_eq!(grown, expected_grown, "allocator may round up, so use <= check");

5. Missing Edge Case: Validator Index Near MAX_PLAUSIBLE_VALIDATOR_INDEX (Line 683)

The constant MAX_PLAUSIBLE_VALIDATOR_INDEX is 1 << 26, but set_latest_message doesn't enforce this. The old code and new code both allow slot up to usize::MAX (via the expect). If someone calls with index 1 << 26 - 1, the allocation is ~67M + 4096 entries. At Option<LatestMessage> size (likely 40+ bytes), that's ~2.7GB.

Question: Is this a DoS vector? Gossip validation should filter these, but set_latest_message is called from multiple paths. Not a new issue in this PR, but the headroom makes it slightly worse (4096 extra entries ≈ 160KB, negligible).


6. Comment Grammar (Line 4283)

// Exact growth plus headroom rather than `resize`'s amortized
// doubling: attestations name validators in no particular order,

"amortized doubling" is misleading—Vec::resize doesn't double, Vec growth strategy on push does. resize to a larger size will reallocate to exactly the requested size if current capacity is insufficient.

Wait, let me re-check: Vec::resize calls extend_with, which calls self.buf.reserve(len - self.len), and RawVec::reserve uses amortized_double_size. So yes, resize can trigger amortized doubling!

But actually, the comment says "resize's amortized doubling" which is correct. However, the old code was resize(slot + 1, None) which would indeed potentially double. The new code avoids this.

Minor: "attestations name validators" → "attestations reference validators" or "name" is jargon. Acceptable.


7. Lock Contention Pattern Unchanged (Line 4282)

let mut beacon = self.beacon.lock().unwrap();

Not new in this PR, but holding a mutex across allocation-heavy operations is a performance concern. The latest_messages resize could be large. Consider if this can be optimized separately.


Correctness Verifications (Good)

Aspect Status
latest_message read path unchanged ✓ No diff
None initialization preserves "no vote" semantics ✓
slot bounds check (messages.len() <= slot) preserved ✓
MAX_PLAUSIBLE_VALIDATOR_INDEX still enforced elsewhere ✓ (unchanged)

Recommendations Summary

Priority Item Location
Medium Fix potential usize overflow in capacity calculation store.rs:4289
Low Clarify reserve_exact + resize interaction with comment store.rs:4289-4291
Low Rename test for accuracy store.rs:9132
Low Strengthen test assertions against fragility store.rs:9154-9159
Trivial Fix comment grammar ("name" → "reference") store.rs:4285

The core optimization is sound and the test coverage is appreciated. Address the overflow check (Item 1) before merge.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

MegaRedHand added a commit that referenced this pull request Oct 30, 2026
…740)

Store::latest_messages is a dense table indexed by validator index, and
set_latest_message grew it with resize, which at least doubles the
capacity when an index lands past it. A registry that keeps gaining
validators crosses that edge soon after the table is first sized, so on
the mainnet follower the table had room for about twice the registry
and roughly half of its 256 MB was never used.

Growth now reserves the new index plus a fixed headroom, so a registry
that gains a few validators an epoch still reallocates rarely, but the
table stays within the headroom of the registry's size. That frees about
120 MB on the mainnet follower.

Cherry-picked from fix/storage-latest-messages-exact-growth at 83aabfc.
@pablodeymo

Copy link
Copy Markdown
Collaborator

@MegaRedHand could you address the Kimi review before this goes in? The items
on the current diff:

  1. store.rs:4289 — make the capacity arithmetic explicit and checked, e.g.
    a named target = slot.checked_add(1 + LATEST_MESSAGES_HEADROOM) with an
    expect, and reserve_exact(target - messages.len()). The debug assert
    bounds index, but this path is consensus-adjacent and should read as
    safe on its own.
  2. store.rs:4289-4291 — add the one-line comment that resize cannot
    reallocate after the reserve_exact.
  3. store.rs:9132 — rename the test; reserve_exact guarantees at least,
    not exactly, so something like
    the_latest_message_table_grows_with_headroom_rather_than_doubling.
  4. store.rs:9154-9159 — assert the behavioural contract rather than
    grown < 2 * sized: no reallocation inside the headroom, and growth past
    it lands at the new index plus headroom (use <= for the upper bound).
    Codex flagged the same allocator-dependence.
  5. store.rs:4285 — "attestations name validators" → "reference validators".

Items 5 and 7 in Kimi's list (index bound, mutex held across the allocation)
are pre-existing and can stay out of this PR.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

@pablodeymo all five are in, one commit each:

  1. 451b9ca04: a named target = slot.checked_add(1 + LATEST_MESSAGES_HEADROOM) with an expect, then reserve_exact(target - messages.len()).
  2. 3a49d1073: the comment that resize cannot reallocate after the reserve.
  3. 275ef362d: renamed to the_latest_message_table_grows_with_headroom_rather_than_doubling.
  4. 276cf37be: the test asserts the contract now. The highest index inside the headroom leaves the capacity unchanged, and growth past it lands at most at the new index plus the headroom.
  5. 56faefb74: "reference validators".

Kimi's items 5 and 7 stay out, as you said.

One more from a second review pass, be3748538. An indexed attestation's indices are sorted ascending, so with exact growth each index past the headroom reallocated and copied the whole table. The first aggregate spanning the registry after a start copied it hundreds of times under the beacon lock. The bot review's "a high index usually arrives early" doesn't hold inside one aggregate. update_latest_messages now calls a new Store::reserve_latest_messages for the batch's highest index first, so one attestation reallocates once at most. a_reserved_batch_of_ascending_votes_never_moves_the_table covers it.

MegaRedHand added a commit that referenced this pull request Nov 4, 2026
get_head tallied every validator's latest message on every call, about 1M
on mainnet, and a live import pays it twice for the first timely block of a
slot (on_block's boost-gate walk, then the actor's get_head). That tally was
most of get_head's ~17 ms in the eth-5 replay.

The store now keeps the vote weights in a node array (Lighthouse unstable's
proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL
weights). One attestation is one vote batch: Store::record_latest_messages
records the messages and moves the attesters' weight under one lock, and the
next read refreshes the subtree totals in one reverse pass over the blocks.
The head walk reads them through cached_node_weights. The proposer boost,
gloas's previous-slot rule and the walk bound depend on the clock, so they
stay read-time. There are no best-child pointers: under gloas the choice
moves without any vote, which is why Lighthouse stopped maintaining them.

What the cache cannot follow incrementally (finality pruning, a live-window
removal, a restart, a new justified balances snapshot, the gloas rule switch,
a block stored without its node) marks it stale, and the next read rebuilds it
from the latest messages at the old tally's cost, about once per epoch. A
rebuild rescans LiveChain under the lock, so an API read that scanned before a
removal cannot rebuild the removed blocks back in.

compute_node_weights stays, as the reference. Equivalence is checked by a
proptest against a naive tally (with the incremental state also equal to a
fresh rebuild after every operation), on the random trees at every step of
every seed, by a new test that feeds votes, blocks and equivocations through
update_latest_messages, on_block's insertion order and
insert_equivocating_index, and at every head check of every fork-choice
fixture.

Local benchmark (1M validators, 73 blocks, laptop): the tally 27 ms; a warm
cached read 7 us; a read after a 15.6k-vote batch 49 us; the batch itself
1.3 ms; a rebuild 17 ms.

Cherry-picked from perf/beacon-proto-array at b4adbaf. Adapted onto
nightly: both vote writers (set_latest_message and the new
record_latest_messages) grow the dense vote table through one
grow_latest_messages helper, which keeps #740's exact growth plus headroom;
the storage and fork-choice imports are merged with #759's and #761's caches.
Replaces #767, reverted in the previous commit.
MegaRedHand added a commit that referenced this pull request Nov 5, 2026
…740): review fixes

- fix(storage): reserve the vote table once per attestation, not per index.
  An aggregate lists its attesters in ascending order, so growing the table
  one index at a time copied it whenever an index passed the headroom:
  hundreds of copies of a registry-sized table for one aggregate spanning
  it. Store::reserve_latest_messages makes room up to the highest index
  first.
- refactor(storage): check the vote table's capacity arithmetic
  (checked_add), and say why the resize after it cannot reallocate.
- test(storage): assert the growth contract (index plus headroom) rather
  than a ratio, name the test after it, and check that a reserved batch of
  ascending votes never moves the table.

On nightly the table has a second writer, #775's
Store::record_latest_messages, which records a whole attestation under one
lock and is the only one on the production path. It reserves for the
batch's highest index itself, under that lock, so #740's call from
fork_choice::update_latest_messages is not needed here; both writers share
the slot and reservation helpers.

Updates nightly's #740 from fix/storage-latest-messages-exact-growth at
83aabfc to be37485 (the branch head, bfb4697, adds only a merge of
main).

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
@MegaRedHand
MegaRedHand changed the base branch from main to fix/p2p-disconnect-slow-gossipsub-peers November 17, 2026 22:07
@MegaRedHand
MegaRedHand added this pull request to stack #798 November 17, 2026 22:12
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 740 (dense vote table growth strategy)

Overall: The change is correct and well-reasoned. The exact-growth-plus-headroom strategy avoids leaving up to half of a registry-sized table unused. That is roughly 56 B × N entries, which is tens of MB at mainnet scale. The arithmetic in reserve_latest_message_slot is sound. len <= capacity <= slot < target holds because the branch is only entered when capacity <= slot. The follow-up resize therefore cannot reallocate. A few notes:

1. The fork-choice comment misattributes the cost (fork_choice.rs ~4708)

growing the table one ascending index at a time would copy it many times over

This is only true under the new exact-growth policy. The old resize(slot + 1, None) used amortized doubling, which gives O(log n) reallocations. The reservation is required because of this PR's change, not because of Vec defaults. Consider rewording, for example: "set_latest_message grows the table exactly (plus headroom), so reserve once for the whole batch." That way a future reader doesn't remove the reserve thinking it's a micro-optimization.

2. Prefer a batch API over a doc convention (store.rs set_latest_message / reserve_latest_messages)

Correctness of the "reallocate at most once" property depends on every batch caller remembering to call reserve_latest_messages first. Any batch caller that forgets regresses to one full-table copy per 4096-index gap. Examples might include block-attestation replay or anchor/checkpoint loading paths, if they iterate indices.

A single entry point would enforce this by construction and also take the mutex once instead of N+1 times:

pub fn set_latest_messages(&mut self, votes: impl IntoIterator<Item = (u64, LatestMessage)>)

Alternatively, audit the other set_latest_message call sites. If update_latest_messages is the only batch caller, this is fine as-is.

3. Reservation precedes the equivocation filter (fork_choice.rs ~4711)

max() is taken over all attesting_indices, including equivocators that are skipped below. Normally this is harmless because the index is in the registry anyway. However, it means the table can grow for an index that will never be written. A cheap fix is to filter equivocators before computing max. This is optional.

4. Release-mode guard is still only a debug_assert (latest_message_slot)

This is pre-existing, but the PR widens the surface slightly, since reservation now runs before any per-index logic. In release builds, an index that slipped past validation would trigger reserve_exact of up to ~2^64 elements. That ends in a capacity-overflow panic or an OOM abort, which means a node crash rather than a rejected message.

Since MAX_PLAUSIBLE_VALIDATOR_INDEX already exists, consider making this a hard check:

  • either return early or ignore the vote with a warn!,
  • or bound it by the actual registry length.

An unconditional bound is cheap insurance against a validation bug turning into a DoS.

5. Minor nits

  • attesting_indices.iter().max(): the comment states the indices are sorted, so .last() would be O(1). Keeping max() is more robust if that invariant ever breaks, so this is fine either way.
  • The test assertion sized <= slot(first) + 1 + LATEST_MESSAGES_HEADROOM relies on Vec reporting exactly the requested capacity after reserve_exact. The docs only guarantee >=. In practice std returns the exact value, so this is acceptable, but the assertion is technically implementation-dependent.
  • The table never shrinks. This is pre-existing and acceptable, but worth a one-line note in the doc comment given the memory-focused rationale.

Tests

Both tests cover the right properties: headroom-bounded growth, and pointer/capacity stability across a reserved ascending batch. Asserting on as_ptr() is a good way to prove no reallocation occurred.

LGTM once Item 1 is reworded. Item 2 and Item 4 are worth considering.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. Batch reservation preserves vote-update semantics, and the allocation arithmetic correctly accounts for Vec length and capacity. Production callers supply validated validator indices.

Tests could not run because rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the surrounding fork-choice and store code so the review is grounded in how the vote table is locked and grown.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look good. Pre-reserving capacity for batches in update_latest_messages and using reserve_exact with fixed headroom (LATEST_MESSAGES_HEADROOM) in reserve_latest_message_slot cleanly avoids both repeated reallocations across sorted attestation indices and large unused capacity from amortized doubling. The unit tests thoroughly cover both growth behavior and pointer stability across batched insertions.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The optimization is reasonable: reserving once for a batch avoids repeated Vec growth, and the capacity/headroom logic in reserve_latest_message_slot is correct (capacity > slot guarantees resize(slot + 1) will not reallocate).

A few issues worth addressing:

  • Flaky test upper bounds – In crates/storage/src/store.rs, test the_latest_message_table_grows_with_headroom_rather_than_doubling, the assertions:

    • assert!(sized <= slot(first) + 1 + LATEST_MESSAGES_HEADROOM);
    • assert!(grown <= slot(higher) + 1 + LATEST_MESSAGES_HEADROOM);

    rely on reserve_exact allocating exactly the requested capacity. Vec::reserve_exact only guarantees at least the requested capacity; the allocator may round up and return a larger block. These upper-bound checks are therefore not portable and can fail spuriously. Suggest asserting only lower bounds, or dropping the upper-bound assertions entirely.

  • Redundant O(n) max – In crates/blockchain/state_transition/src/beacon/fork_choice.rs, the comment states indices are sorted, but the code uses attesting_indices.iter().max(). If the sorted guarantee holds, attesting_indices.last().copied() gives the highest index in O(1) and avoids an extra scan before the loop. This is a minor performance/clarity nit, not a correctness issue.

  • Reservation before equivocation filtering – The batch reservation happens before the loop that skips equivocating indices. If an aggregate contains many equivocating validators with high indices, capacity may be reserved unnecessarily. This is a minor inefficiency, not a bug; if desired, the reservation could be moved after a first pass that filters indices, but that may not be worth the extra code.

Production correctness of the reserve_latest_message_slot / set_latest_message changes looks sound to me.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found. The allocation logic is sound:

  • reserve_latest_message_slot cannot underflow because len <= capacity <= slot < target, and the subsequent resize is guaranteed to fit in the reserved capacity.
  • Using the maximum attesting index in fork_choice.rs:4708 correctly covers the entire batch, even if the input is unexpectedly unsorted.
  • Message-update semantics and locking behavior are unchanged.
  • The tests cover both headroom-based growth and pointer/capacity stability across a reserved batch.

Non-blocking security note: latest_message_slot (store.rs, around line 691) enforces MAX_PLAUSIBLE_VALIDATOR_INDEX only with debug_assert!. Both allocation entry points therefore rely on the documented validated-attestation invariant in release builds. If reserve_latest_messages could ever be called with untrusted input, a runtime bound check would be needed to prevent an excessive allocation. Under the current caller contract, this is not a regression.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

⚠️ Review failed: Kimi API request failed with status 429: The engine is currently overloaded, please try again later


Automated review by Kimi (Moonshot AI)

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look good.

The rationale for combining reserve_exact + fixed headroom with an upfront batch reservation in update_latest_messages is well-documented, the bounds/overflow checks in reserve_latest_message_slot are sound, and the new unit tests directly cover both growth behavior and pointer stability across ascending batches.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 740: dense vote table growth

Overall the change is small, correct, and well documented. The arithmetic in reserve_latest_message_slot is sound: reserve_exact(target - len) guarantees capacity >= target, and the subtraction cannot underflow given len <= capacity <= slot. My main concern is a trade-off in the growth policy, not a bug.

1. Dropping amortized growth makes non-batched callers potentially quadratic (store.rs, set_latest_message)

What changed. Previously, resize(slot + 1, None) went through Vec::reserve, which grows to max(2 * cap, required). Inserting ascending indices one at a time was therefore amortized O(1). With reserve_exact plus a fixed headroom, any caller that does not call reserve_latest_messages first now pays one full-table copy per 4096 new indices.

Cost at mainnet scale. Assume about 1M validators and an Option<LatestMessage> of roughly 56 B. A cold, ascending fill through bare set_latest_message would mean about 250 reallocations of a table up to about 56 MB, which is roughly 7 GB of memcpy in total.

What to check. Please audit every other call site of set_latest_message, for example:

  • block-body attestation processing, if it does not route through update_latest_messages;
  • restoring votes from persisted state or a checkpoint;
  • test fixtures and spec-test runners.

The doc comment's "A caller recording a batch should call reserve_latest_messages first" is an easy contract to miss.

Suggestions:

  • A more robust option is to size the table from the registry, e.g. call reserve_latest_messages(validators.len() - 1) when the head or justified state is updated. The registry is the true upper bound, and the table would be allocated once with near-zero waste.
  • Alternatively, fall back to geometric growth with a smaller factor, such as max(slot + 1 + HEADROOM, cap + cap / 4). This keeps the worst case amortized while still bounding waste to about 25%.

2. Reservation includes indices that are later skipped (fork_choice.rs:4712)

max() is computed before the is_equivocating filter. As a result, the table can be sized for an index that is never written. The impact is negligible: it costs at most one headroom's worth of capacity, and slashed indices are still within the registry. It is fine as-is; I mention it only so the behavior is intentional.

3. Minor nits

  • fork_choice.rs:4712. The comment says the indices are sorted. If that is guaranteed by validation, attesting_indices.last() states the intent and is O(1). max() is the safer choice if sortedness is not enforced at this point, so keep it if in doubt.
  • Release-build plausibility guard. latest_message_slot keeps the plausibility check as debug_assert! only. This is unchanged behavior. However, reserve_latest_messages is now a second entry point that will allocate a table of highest + 4096 entries in release builds from a single index. That is fine as long as attesting_indices always come from a validated committee, and the docs already say so. A cheap release-mode check against MAX_PLAUSIBLE_VALIDATOR_INDEX (returning or logging) would harden both paths.
  • Test brittleness. The upper-bound assertions in the_latest_message_table_grows_with_headroom_rather_than_doubling rely on reserve_exact being exact with the global allocator. This is acknowledged in the comment and is true for std today, so it is acceptable.
  • Pointer-identity test. a_reserved_batch_of_ascending_votes_never_moves_the_table is a good, precise test of the batch guarantee.

Summary

The change is correct. Before merging, please either confirm that no hot path calls set_latest_message in a loop without reserving, or switch to sizing the table from the registry length (preferred) so that the exact-growth policy cannot degrade into quadratic copying.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The reservation arithmetic is correct, attesting indices come from validated paths, and vote selection semantics remain unchanged.

Tests could not run because rustup attempted to write outside the permitted workspace.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall the allocation math is correct: reserve_latest_message_slot only runs when len <= capacity <= slot, then reserves to exactly slot + 1 + LATEST_MESSAGES_HEADROOM, so the following resize(slot + 1) cannot reallocate. The tests look good and exercise the key invariants.

A few notes:

  1. Release bounds check remains debug-only
    crates/storage/src/store.rs:685-688 — latest_message_slot uses debug_assert!, so in release builds a malformed/hostile attestation index could still drive a huge allocation in reserve_latest_messages/set_latest_message. This is pre-existing behavior, but the new batched reservation makes the allocation happen earlier and in one shot. If the “validated attestation” invariant is ever bypassable at this call site, consider an assert! or explicit error for index >= MAX_PLAUSIBLE_VALIDATOR_INDEX.

  2. Minor: extra O(n) pass over attesters
    crates/blockchain/state_transition/src/beacon/fork_choice.rs:4708-4711 — the comment says indexed attestation indices are sorted, but the code uses iter().max(). That is safe and robust, but if sortedness is guaranteed, attesting_indices.last().copied() would avoid a second pass. Committee sizes are small enough that this is not significant.

  3. Public API exposure
    crates/storage/src/store.rs:4329 — reserve_latest_messages is public only because it is called from another crate. That is fine, but it’s worth documenting the caller precondition that highest comes from a validated attestation, since this method can immediately allocate a large dense table.

  4. Lock overhead
    reserve_latest_messages always locks self.beacon even when capacity is already sufficient. In the hot update_latest_messages path this adds one extra mutex acquisition per batch. Likely negligible compared with the rest of attestation processing.

No correctness issues found in the reserve/resize logic or the fork-choice ordering.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The allocation change is correct and the batch reserve is the right fix. reserve_exact(target - len) yields capacity of at least slot + 1 + LATEST_MESSAGES_HEADROOM, so the following resize does not reallocate, and vote contents are unchanged.

  1. reserve_latest_message_slot (store.rs around the new helper) and reserve_latest_messages still only debug_assert against MAX_PLAUSIBLE_VALIDATOR_INDEX. In release, one untrusted index allocates a single index + 4097 slot table and then resize zeroes it. That matches the old resize DoS, but the new path does it eagerly from max() before the equivocation loop. If update_latest_messages is only reached after committee bounds checks, say so on reserve_latest_messages; otherwise reject index >= MAX_PLAUSIBLE_VALIDATOR_INDEX before taking the beacon lock.

  2. The headroom test’s upper bound (sized <= slot + 1 + LATEST_MESSAGES_HEADROOM) is not guaranteed. reserve_exact is allowed to return more than requested; if Vec records allocator slack, this fails without a doubling regression. Assert capacity >= slot + 1 and that a second jump of HEADROOM does not reallocate, or allow slack well below a doubling.

  3. fork_choice.rs (~4712) is the only new batch caller. Any other ascending set_latest_message loop will still copy the table every time an index clears the 4096 headroom. iter().max() is fine even if the sorted-indices assumption slips; last() is only an option if that invariant is actually enforced.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

Base automatically changed from fix/p2p-disconnect-slow-gossipsub-peers to main November 18, 2026 20:43
Store::latest_messages is a dense table indexed by validator index, and
set_latest_message grew it with resize, which at least doubles the
capacity when an index lands past it. A registry that keeps gaining
validators crosses that edge soon after the table is first sized, so on
the mainnet follower the table had room for about twice the registry
and roughly half of its 256 MB was never used.

Growth now reserves the new index plus a fixed headroom, so a registry
that gains a few validators an epoch still reallocates rarely, but the
table stays within the headroom of the registry's size. That frees about
120 MB on the mainnet follower.
The sum was safe only because a debug assertion bounds the index, so the
line could not be read as correct on its own. Name the target and check
the addition.
The reserve above it is what makes the growth exact; without the note,
the resize reads as if it could still double the table.
…ises

reserve_exact guarantees at least the requested capacity, not exactly
it, so the old name claimed more than the code does.
`grown < 2 * sized` held only because of the indices the test picked.
Check what the code promises instead: the highest index inside the
headroom fits without a reallocation, and growth past it lands at the new
index plus the headroom, as an upper bound.
An indexed attestation's attesters are sorted ascending. With exact
growth, each index past the headroom reallocated and copied the whole
table, so the first aggregate spanning the registry after a start copied
a registry-sized table hundreds of times while holding the beacon lock.

update_latest_messages now reserves room for the batch's highest index
first, so the batch reallocates once at most. Indices are taken by max,
not by position, so an unsorted caller is still covered.
From the second automated review round:

- update_latest_messages's comment credited the repeated copies to growing one index at a time; they come from this PR's exact growth, which replaced resize's doubling. Said so, so the reservation does not read as a micro-optimization someone could drop.
- The headroom constant says the table never shrinks, and why that is fine.
- The growth test says why its upper bounds stay although reserve_exact promises only at least: they are what tell exact growth from doubling, and std's Vec takes exactly what it asks for.

update_latest_messages is the only production caller of set_latest_message, so the batch reservation needs no API of its own yet.
@pablodeymo
pablodeymo force-pushed the fix/storage-latest-messages-exact-growth branch from 81de754 to 2ba572f Compare November 18, 2026 20:44
@pablodeymo
pablodeymo enabled stacked auto-merge November 18, 2026 20:51
…s equivocators

From the third automated review round:

- reserve_latest_messages states set_latest_message's precondition (an index from a validated attestation, since the table is allocated up to it at once) and that update_latest_messages is its one caller and the only production path recording votes, the table being rebuilt through it after a restart. Four reviewers asked for this or for a release-mode bound; the bound stays out of this PR, as agreed in review.
- update_latest_messages says why equivocators count towards the reserved index, and why max rather than last.
@pablodeymo
pablodeymo added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 7cbdaf0 Nov 18, 2026
11 checks passed
@pablodeymo
pablodeymo deleted the fix/storage-latest-messages-exact-growth branch November 18, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants