Skip to content

fix(validator): re-send proposer preparations every slot, not once an epoch - #804

Merged
MegaRedHand merged 12 commits into
mainfrom
fix/validator-prepare-proposers-every-slot
Nov 19, 2026
Merged

MegaRedHand merged 12 commits into
mainfrom
fix/validator-prepare-proposers-every-slot

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

A beacon node keeps prepare_beacon_proposer's fee recipients in memory, and a restart forgets them. Until the next call, it builds its own payloads for the zero address. The validator client sent them only from the epoch refresh, so a beacon node restarted mid-epoch paid the rest of that epoch's locally built blocks to nobody.

Found on a kurtosis devnet (ethlambda beacon node + ethlambda validator client, image sha-f4efc82):

  • Fulu: the beacon node was SIGKILLed 14 slots into epoch 105 and restarted. Its next 6 local blocks, all before epoch 106, logged "No fee recipient prepared for the proposer; using the zero address". The validator client logged ERROR "Block pays its execution-layer rewards to an address this client did not ask for; signing it anyway" for each. From the first block of epoch 106 the fee recipient was right again.
  • Gloas: the same fault produced 3 such blocks. Blocks built on a builder's bid were not affected.

What Changed

Where Change
crates/validator/src/lib.rs Preparations: after every slot's duties, the duty loop re-sends the preparations of the validators the last refresh resolved, for the keys the store still holds, one send at a time (InFlight). Not on the slot whose refresh's send reached a node: Refreshed.prepared carries that outcome
crates/validator/src/beacon_node/ prepare_beacon_proposer_quietly: the same request through try_all_quiet, so a node that is down costs a debug! line a slot, not two warnings
docs/cli.md, docs/rpc.md, docs/spec_deviations.md, comments and the mismatch error The cadence, and what a keymanager DELETE or import does to it

Correctness / Behavior Guarantees

  • Matches Lighthouse. Its validator client's start_proposer_prepare_service calls prepare_proposers_and_publish every slot.
  • Next slot is covered. Sent once the slot's duties are done (a third of the way in on a slot with none, after the aggregation or payload vote otherwise), so it reaches the node ahead of the next slot's proposal rather than racing this one's. A restart costs at most a proposal made before the first re-send the node is back for.
  • Not on the duty path. It runs on a spawned task, like the registration sends: a slow or down node never delays a duty.
  • One at a time. A re-send that finds the last one still waiting on a node skips, so an older fee recipient cannot land after a newer one. The refresh's own send is not held back by it (it precedes the epoch's first proposal); a re-send still in flight from the slot before can land after it, and the next slot corrects that.
  • Failure handling. The refresh's send still warns, and a failed one is re-sent on the same slot. The per-slot one logs at debug, per node too.
  • Keymanager changes apply sooner. The settings are read on every send, so a fee-recipient change reaches the node after the duties of the slot it was made in, not at the next epoch. Caveats: a deleted key is no longer sent (the node keeps its last address until it expires it or restarts); a DELETE leaving a validator no address sends nothing for it; a key imported mid-epoch is first sent at the next refresh, which is also when its proposer duties are first fetched.
  • Small cost. One small POST per slot to each configured beacon node.

Tests Added / Run

  • the_preparations_go_out_again_after_every_slot_but_the_refreshs: the duty loop's gate, driven by the refresh's real outcome.
  • a_failed_registration_does_not_fail_the_refresh (extended): a failed refresh send is re-sent on its own slot.
  • a_key_deleted_since_the_refresh_is_not_prepared_again
  • a_re_send_still_waiting_on_a_node_makes_the_next_slot_skip
  • the_quiet_fee_recipient_send_reaches_every_node_too (fallback)
  • Ran cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, cargo test --profile release-fast -p ethlambda-validator (622 passed), cargo test --profile release-fast -p ethlambda-rpc.

Related Issues / PRs

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran clippy with -D warnings on the workspace — clean
  • Ran make test (test-consensus plus test-node, at release-fast) — ethlambda-validator and ethlambda-rpc only; CI runs the rest

… epoch

A beacon node keeps prepare_beacon_proposer's fee recipients in memory, so a
restart forgets them, and until the next call it builds its own payloads for
the zero address. The validator client sent them only from the epoch
refresh, so a beacon node restarted mid-epoch spent the rest of that epoch
paying its local blocks' execution-layer rewards to nobody.

The duty loop now re-sends them at the end of every slot (skipping the slot
whose refresh just sent them), off the slot loop like the registration
sends, as Lighthouse's validator client does. The settings are read on each
send, so a keymanager fee-recipient change reaches the node within a slot.

Found on a kurtosis devnet: an ethlambda beacon node SIGKILLed 14 slots into
an epoch built 6 local blocks for the zero address before the next epoch's
refresh (fulu), and 3 more after the same fault on gloas; the validator
client logged "Block pays its execution-layer rewards to an address this
client did not ask for" for each.
@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look good. Extracting prepare_proposers and re-sending preparations in the background on non-refresh slots cleanly handles mid-epoch beacon node restarts and keymanager fee-recipient updates without adding per-slot allocations (via Arc<Vec<ValidatorEntry>>) or blocking the duty loop.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: re-send proposer preparations every slot (PR 804)

The PR fixes a real failure. A beacon node restarted mid-epoch builds for the zero address until the next refresh. Factoring out prepare_proposers is clean, the docs are updated consistently, and the tests cover the main paths. Nothing touches consensus logic. Comments below, roughly by importance.

1. Tasks can pile up if the beacon node hangs (lib.rs, ~L540–548)

A new task is spawned every slot with no bound. Suppose the node is half-up: TCP accepts, but the request never answers. If the HTTP client has no request timeout, or one longer than a slot, these tasks accumulate.

  • register_changed has the same pattern, but it runs once per epoch. This one runs 32× more often.
  • Please confirm HttpBeaconNode has a per-request timeout shorter than SECONDS_PER_SLOT.
  • Alternatively, keep a JoinHandle and skip spawning while the previous send is unfinished: if handle.as_ref().is_some_and(|h| !h.is_finished()).

2. Fallback semantics (FallbackBeaconNode::prepare_beacon_proposer)

The scenario this PR targets is a restarted node. With a fallback set, the restarted node is often the non-primary one, after which it may become primary.

  • If FallbackBeaconNode sends only to the first healthy node, a restarted secondary never hears the preparation until failover. It then builds at least one block for the zero address.
  • Lighthouse broadcasts preparations to all configured beacon nodes.
  • Worth checking. If it is first-success, consider fanning out for this call.

3. The comment's timing claim ("At the end of the slot", ~L531–534)

The spawn sits at the end of the loop body, which runs right after the slot-boundary wake. It is not literally at the end of the slot.

  • If proposal and attestation work is spawned rather than awaited earlier in the loop, this send fires at roughly the start of the slot, alongside them.
  • That is harmless. Preparations are idempotent, and the payload for slot N+1 is built off FCU around the end of slot N.
  • Please reword so it doesn't promise ordering the code doesn't enforce. For example: "after this slot's duties are dispatched".

4. settings.clone() per slot (~L541)

The test calls settings.set_fee_recipient(...) through &ProposerSettings, which implies interior mutability, so this is presumably an Arc clone. If ProposerSettings is instead a struct holding maps behind its own lock, Clone could deep-copy every slot. Worse, a deep copy would snapshot the map and break the "keymanager change reaches the node within a slot" guarantee. Please verify it is Arc-backed.

5. Stale resolved between refreshes (minor)

A validator deleted through the keymanager mid-epoch stays in resolved until the next refresh. If settings.fee_recipient falls back to the default for unknown pubkeys, the client keeps preparing a validator it no longer manages, for up to one epoch.

This is low impact. If fee_recipient returns None for deleted keys, it is a non-issue. Otherwise, consider filtering resolved against the current store pubkeys in prepare_proposers.

6. Nits

  • ~L545: let _ = sent.await.inspect_err(|err| ...) reads awkwardly. if let Err(err) = sent.await { debug!(...) } is more idiomatic.
  • Test the_per_slot_send_carries_a_fee_recipient_changed_since_the_refresh: no refresh happens, and the setting changes before the only send. As written it only shows that settings are read at call time. To match the name, do a send, change the setting, send again, and assert the second payload differs.
  • prepared is set to true even when the refresh's own prepare_proposers failed (it only warn!s). The next slot retries, so this is fine. A one-line comment saying so would avoid a reader thinking a failure is skipped.

Verdict

Correct and worthwhile. Item 1 and Item 2 are worth addressing or confirming before merge. The rest are minor.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll read the surrounding duty-loop and preparation code so the review is grounded in how this slot path actually runs.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable correctness or security findings in the diff. The per-slot resend reuses the existing preparation logic and sends updated fee recipients to all configured beacon nodes. No consensus processing code changes.

Tests could not run because rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

Findings

  1. The re-send does not actually happen “at the end of the slot” — crates/validator/src/lib.rs:531-548
    This code runs at the end of the loop body, immediately after waking at the slot boundary. Consequently, it can still race the current slot’s proposal, especially because proposal duties were spawned earlier in the iteration. If the intention is to prepare for the next slot without racing the current one, schedule the send at an explicit late-slot offset. Alternatively, send it before proposal work if it must protect the current proposal. Otherwise, update the timing comments and documentation to describe the actual best-effort behavior.

  2. Detached per-slot sends can overlap and complete out of order — crates/validator/src/lib.rs:539-547
    A slow or unresponsive beacon node can accumulate one spawned request per slot. Besides task growth when requests lack a bounded timeout, an older request carrying the previous fee recipient could complete after a newer request and temporarily overwrite the keymanager update on the beacon node. Consider a single preparation worker, or retaining the JoinHandle and skipping/coalescing sends while one is still outstanding. This avoids blocking the slot loop while guaranteeing ordered sends.

The helper extraction itself looks correct: it preserves the previous empty-preparation behavior, keeps refresh failures non-fatal, and uses Arc to avoid copying the validator list each slot. The added tests cover the helper’s payload and settings behavior well, though they do not exercise the duty-loop timing or overlapping sends.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a clean refactor: extracting prepare_proposers is good, and the tests cover the main behaviors, including empty/updated fee recipients.

A few concerns:

  1. Possible stale, out-of-order fee-recipient sends — crates/validator/src/lib.rs:527-548 spawns a new prepare_proposers task every slot without any ordering or sequencing. If a keymanager changes a fee recipient, an older slotted send that is delayed in-flight can arrive after a newer send and overwrite the newer value on the beacon node. prepare_beacon_proposer is effectively last-writer-wins. Consider serializing these sends through a single worker/actor or at least documenting/accepting the race window.

  2. No backpressure or cancellation on the per-slot send — crates/validator/src/lib.rs:527-548 also means every slot creates a detached tokio::spawn task. If the beacon node is slow or down and prepare_beacon_proposer takes longer than one slot, tasks can pile up. These tasks are not tied to the CancellationToken, so they can also outlive shutdown. A timeout shorter than the slot duration, or a single coalescing sender, would bound this.

  3. Comment/readability nit — crates/validator/src/lib.rs around the per-slot spawn says “At the end of the slot”, but the code only runs after the synchronous slot duties; it does not actually wait until the slot boundary. That may be fine functionally, but the comment implies a timing guarantee the code does not provide. Also, passing &resolved to prepare_proposers works via deref coercion, but resolved.as_slice() would be clearer.

The consensus-facing parts of the PR appear unaffected; this is purely validator fee-recipient preparation behavior.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@pablodeymo

Copy link
Copy Markdown
Collaborator

The approach looks right and the main flow reads correctly. Two things I'd like fixed before this merges; the rest is non-blocking.

Before merge

1. A failed per-slot send warns, it does not log at debug.

The comment above the spawn says a failure is "only logged at debug", but prepare_proposers calls FallbackBeaconNode::prepare_beacon_proposer, which goes through the loud try_all (fallback.rs:641). That logs "Beacon node did not accept" at warn for each refusing node, and "No configured beacon node accepted" with duty likely missed when none accepts.

With one of two nodes down that is a warn every slot instead of once an epoch. With a single node down it is two per slot, the second claiming a missed duty on slots that have none.

try_all_quiet exists for a call the caller retries every slot and reports itself, which is how submit_proposer_preferences uses it. prepare_beacon_proposer is a trait method, so the per-slot path needs its own entry point (or a flag) to reach it.

2. A key deleted through the keymanager keeps being prepared.

The send replays resolved from the last successful refresh and never looks at the validator store. After DELETE /eth/v1/keystores (or /remotekeys), settings.forget drops the key's override, so settings.fee_recipient answers --suggested-fee-recipient for it, and that address is sent for the key every slot until the next successful refresh.

If the key was moved to another client on the same beacon node, the two clients overwrite each other's preparation and the key's block can pay the old client's default address. Before this change nothing was sent for a key after its delete.

Filtering resolved by the store's current pubkeys at send time would close it.

Non-blocking

  • prepared is set even when the refresh's own send failed. refresh_epoch only warns on that failure and still returns Ok, so the re-send is skipped on the one slot where it is known to be needed. Carrying the send's outcome in Refreshed would fix it. The warn text "blocks proposed this epoch may pay somewhere else" is also out of date now that the next slot retries.
  • No single-flight guard on the spawned send. Against a slow node, one slot's send can outlive the next slot's and land last, reverting a fee recipient changed in between. InFlight already does this for PreferencesService and RegistrationService.
  • The new tests do not cover the loop's gate. All three call prepare_proposers directly, so removing the spawn block or inverting prepared leaves them green. Two of them also repeat a_validators_own_address_is_registered_for_it and no_configured_address_sends_no_registration.
  • Text that still describes the once-an-epoch cadence. Operator-facing: the mismatch error at proposal.rs:326 ("reaches the node at the next epoch boundary") and docs/cli.md:633. Comments: proposer_settings.rs:27, preferences.rs:61, crates/net/rpc/src/beacon/validator.rs:336 and the test doc at lib.rs:1262.
  • "A keymanager change reaches the node within a slot" needs a caveat. A DELETE that leaves a validator with no address sends nothing, so the node keeps the old one. A key imported mid-epoch is not in resolved until the next refresh.
  • "At the end of the slot" is not quite when it runs. In a slot with no duties the spawn fires right after the first duty offset, about a third of the way in. That still meets the stated bound; only the comment is off.
  • docs/spec_deviations.md. "Where a preference is a signature per proposal" contradicts the same bullet's "as signed": re-sending a kept preference costs a request, not a signature.
  • lib.rs:1045. if let Err(err) = ... { warn!(..) } only logs, which CLAUDE.md asks to write with inspect_err.

…send

The per-slot re-send went through FallbackBeaconNode::prepare_beacon_proposer, the loud try_all: with a node down it warned for each refusing node every slot and, with none accepting, claimed a missed duty on slots with no duty. prepare_beacon_proposer_quietly is the same request through try_all_quiet, as submit_proposer_preferences does; its default implementation is the plain call, for a single node.
…ld keys only

The per-slot re-send moves into Preparations, whose after_slot is the duty loop's gate:

- it sends only the resolved validators the store still holds, so a key deleted through the keymanager is no longer prepared with --suggested-fee-recipient;
- it is single-flight (InFlight), so a send that read an older fee recipient cannot land after a newer one;
- it goes through prepare_beacon_proposer_quietly and logs its failure at debug.

Refreshed carries whether the refresh's own send reached a node, so a failed one is re-sent on its own slot, and its warning no longer says the whole epoch may pay elsewhere.

Tests drive the gate with the refresh's real outcome, and cover a deleted key and a re-send still waiting on a slow node. The two that repeated a_validators_own_address_is_registered_for_it and no_configured_address_sends_no_registration are gone.
…ties

The keymanager mismatch error, docs/cli.md, and the comments in proposer_settings.rs, preferences.rs and the beacon node's FeeRecipients still described the once-an-epoch cadence. Also says what a DELETE leaves the node with, and when an imported key is first sent. spec_deviations.md gave a signature per proposal as the reason preferences are re-sent less often, but a kept one is re-sent as signed: the difference is one request for every validator against one per preference.
The keymanager's log after a fee recipient is set still said it reaches the beacon node at the next epoch. The mismatch error no longer mentions a validator left with no address, which never reaches that error, and the FeeRecipients comment no longer makes a claim about other validator clients' cadence.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Thanks. All of it is in 76d43fc, ec76cf7, 133c4ce and 60650ca, in your order:

Before merge

  1. Per-slot send is quiet now. I added BeaconNodeApi::prepare_beacon_proposer_quietly, which sends the same request. Its default implementation is the plain call, and FallbackBeaconNode overrides it with try_all_quiet, so a refusing node logs at debug and nothing claims a missed duty. The per-slot send uses it and logs its own failure at debug. The refresh's once-an-epoch send still warns. the_quiet_fee_recipient_send_reaches_every_node_too covers delivery. The log level holds by construction, since the crate has no log capture.
  2. Deleted keys. The re-send now keeps only the resolved validators the store still holds. It builds the list under the store's read guard. Both delete routes remove the key and forget its overrides under the write guard, so a send sees both changes or neither. a_key_deleted_since_the_refresh_is_not_prepared_again sets a default, deletes and forgets one key, and checks that only the other goes out. Without the filter it fails, because the deleted key is sent with the default.

Non-blocking

  1. prepared follows the send. Refreshed carries prepared, taken from prepare_proposers(..).inspect_err(..).is_ok() (which also covers the lib.rs:1045 nit). A failed refresh send is re-sent on its own slot. The warning now says a block proposed before the re-send after this slot's duties may pay elsewhere. a_failed_registration_does_not_fail_the_refresh asserts both.
  2. Single flight. The re-send moved into a small Preparations with an InFlight, entered before the store and settings are read. A send that finds the last one running skips. a_re_send_still_waiting_on_a_node_makes_the_next_slot_skip holds the first send at a gate on the mock.
    • I left the refresh's own send out of the guard. It runs on the duty loop just before the epoch's first proposal, so skipping there costs that proposal its preparation, and waiting holds the loop on a hung node.
    • The cost is that a re-send still in flight from the epoch's last slot can land after it, and the next slot's re-send corrects that. It's written down on Preparations.
  3. Gate test. The gate is Preparations::after_slot, which the loop calls and which returns the task. the_preparations_go_out_again_after_every_slot_but_the_refreshs feeds it the real Refreshed from refresh_epoch: no re-send on the refresh's slot, one on the next, carrying a fee recipient changed in between.
    • Each of these turns a test red: removing the spawn, inverting the gate, inverting the outcome, dropping the store filter, dropping the guard.
    • The two duplicates are gone.
  4. Stale text is fixed in all six places, plus the prepare_proposers doc (now on Preparations). A wider sweep also found the keymanager's own log after setting a fee recipient ("registered with the beacon node at the next epoch"), now fixed. The mismatch error doesn't mention the no-address case, since that error never fires without an address.
  5. Caveats.
    • A DELETE that leaves no address is now described in the Preparations doc, proposer_settings.rs and docs/cli.md.
    • I documented the mid-epoch import instead of fixing it, because nothing is lost. Proposer duties are narrowed to the indices the refresh resolves (duties.rs refresh_proposers, only called from refresh_around), so this client proposes nothing for that key before the refresh that also prepares it. A per-slot lookup would cost a request every slot for every key not on chain yet.
  6. Timing comment now says once the slot's duties are done: a third of the way in on a slot with none, later on one with some.
  7. spec_deviations.md now gives the real difference: one request for every validator against one request per kept preference. preferences.rs says the same.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look good. Re-sending prepare_beacon_proposer after each slot's duties (with InFlight deduplication, quiet fallback logging, and filtering against keys still present in ValidatorStore) is clean, avoids lock contention across .await points, and is well-covered by the new tests.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the duty-loop integration and the fallback/in-flight helpers this PR depends on before judging correctness.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: per-slot prepare_beacon_proposer re-send

The design looks sound. It follows Lighthouse's cadence, filters deleted keys by the store, serializes re-sends through InFlight, and takes the settings snapshot outside the store lock. The tests cover the main cases well. A few points to address:

Correctness / robustness

  1. One hung node can stop re-sends to every node (lib.rs, Preparations::resend; fallback.rs, prepare_beacon_proposer_quietly).

    • try_all_quiet sends to every node and waits for all of them. The _running guard is held across that await.
    • If one node accepts the TCP connection and never answers, and the HTTP client has no request timeout below a slot, in_flight never clears. Every later slot then skips with a debug! line.
    • That stops re-sends to the healthy nodes as well, including the restarted node this PR exists to fix.
    • Please confirm that HttpBeaconNode bounds request time to well under a slot. Otherwise, wrap the send in tokio::time::timeout(slot_duration, …). Either way, a test with a gated node that never releases would pin this behaviour down.
  2. A stale re-send can land after the refresh's send, with no re-send that slot to correct it (lib.rs, after_slot and the prepared flag).

    • The doc on Preparations admits a slot-(N−1) re-send still in flight can land after the epoch refresh's send.
    • Because prepared == true makes the refresh slot skip its own re-send, the earliest correction is after slot N+1's duties.
    • So slot N+1's proposal can be built with the stale address. Example: a fee recipient changed during slot N−1 while one node was slow.
    • The window is narrow. But the prepared optimisation saves only one request per epoch, so dropping it would be simpler and close the gap: always re-send after the slot's duties. That would also remove the prepared field from Refreshed, plus its plumbing and comments.

Tests

  1. a_failed_registration_does_not_fail_the_refresh does not check the behaviour it claims (end of lib.rs tests).

    • The new half only checks that the spawned task finishes. It would pass even if resend returned early without calling the node.
    • Since the mock fails prepare_beacon_proposer before recording anything, consider counting calls, or using preparation_entered, to assert the re-send was actually attempted.
  2. Leftover Notify permit (mock.rs, preparation_entered.notify_one()).

    • This runs on every call, so tests that never await it leave a stored permit behind.
    • It is harmless today. If a future test awaits notified() after an earlier, unrelated call, it would return immediately. A short note on the field would help.

Style / nits

  1. Non-idiomatic error discard (lib.rs, resend).

    • let _ = sent.inspect_err(|err| debug!(…)); reads better as:
      if let Err(err) = sent { debug!(%slot, %err, "…"); }
  2. Awkward reflow (preferences.rs, around L61–65). The paragraph wraps mid-sentence: "The round runs at\n//! the poll of an epoch's second-to-last\n//! slot". Please re-wrap it.

  3. Load at scale. Each slot now sends a POST covering every resolved validator to every node. That is fine and matches Lighthouse. For large key sets, a line in docs/cli.md noting the per-slot request would help operators who watch beacon-node request logs.

  4. Docs and log messages. The text in proposer_settings.rs, proposal.rs, docs/cli.md and docs/rpc.md is consistent with the new behaviour. The refresh_epoch warn text ("before the re-send after this slot's duties") is accurate given the prepared = false path.

Nothing here is consensus-critical. Item 1 is the one I would want resolved, or confirmed already covered by client timeouts, before merge.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Serialize epoch preparations with slot resends — lib.rs:1046. The epoch refresh bypasses Preparations::in_flight. A delayed resend from the previous slot can therefore write the old fee recipient after the refresh successfully writes a newly configured address. Because prepared = true suppresses that slot’s resend, the stale address can persist into the next proposal, directing execution rewards to the wrong recipient. Coordinate both paths through the same ordering mechanism, and add a gated test covering an old resend completing after an epoch refresh.

Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

  1. Medium — an older re-send can overwrite the epoch refresh’s newer fee recipients.
    In crates/validator/src/lib.rs, refresh_epoch calls prepare_proposers directly (around line 1047), outside Preparations::in_flight. Consequently:

    • A previous slot’s re-send reads fee recipient A and stalls.
    • The epoch refresh successfully sends fee recipient B.
    • The stalled re-send completes afterward and restores A.
    • The epoch’s first proposal can therefore pay A until the next slot’s re-send.

    The “One send at a time” comment acknowledges this ordering, but it remains a correctness issue, especially because proposal.rs signs fee-recipient-mismatched blocks anyway. Consider serializing the refresh’s send with in-flight re-sends, or otherwise preventing an older request from landing after a newer one. A regression test with a gated re-send overlapping refresh_epoch would cover this.

  2. Low — the documented keymanager latency is not always accurate on a refresh slot.
    Preparations::after_slot returns immediately when prepared is true. If a fee recipient changes after the refresh’s send but before that slot’s duties finish, the end-of-slot re-send is skipped and the change waits until the following slot. This contradicts:

    • crates/validator/src/http_api/proposer.rs:316: “sent … once this slot’s duties are done”
    • The proposer_settings.rs module documentation
    • The docs/cli.md feerecipient table

    Either track whether settings changed after the successful refresh send, or qualify the wording with “usually”/“by the next slot.”

Otherwise, the change is well structured: deleted-key filtering avoids stale preparations, InFlight correctly prevents overlapping per-slot re-sends, and the fallback/tests cover partial failure, total failure, and slow-node backpressure. No consensus, SSZ, or signature-verification logic is affected.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a well-reasoned change and the test coverage for the re-send/skip behavior is good. No consensus-critical or memory-safety issues found in this diff.

A few issues worth addressing before merge:

  1. crates/validator/src/beacon_node/mod.rs — the default prepare_beacon_proposer_quietly implementation is not actually quiet for non-FallbackBeaconNode implementations; it just calls the normal method. The method name and docs imply quieter logging, but that behavior only exists in the fallback override. If this is always called through FallbackBeaconNode in practice, document that assumption explicitly; otherwise implement the quiet behavior in the default or rename the method.

  2. crates/validator/src/lib.rs — in Preparations::resend, the InFlight guard is held across the entire prepare_beacon_proposer_quietly call. Since FallbackBeaconNode::try_all_quiet attempts every node, one slow or hung node can cause every subsequent slot’s re-send to be skipped, delaying fee-recipient updates to healthy nodes. Confirm that the HTTP/fallback layer has a bounded timeout, and consider a metric or rate-limited warning when sends are skipped for many consecutive slots.

  3. crates/validator/src/lib.rs — the JoinHandle returned by Preparations::after_slot is ignored at the call site in the duty loop. The task only logs at debug!; if it panics, that failure is silently swallowed. Given this is a background maintenance task, consider a metric for failed/skipped sends or at least a comment noting that detached panics are intentionally not observed.

  4. crates/validator/src/http_api/proposer.rs — the updated log says "sent to the beacon node once this slot's duties are done", but this is not always true: on the refresh slot prepared can be true, causing after_slot to return None, and if a previous re-send is still in flight the next slot skips. The surrounding docs correctly describe a one-slot-or-two delay; align this log wording with that.

  5. Performance note: the change increases prepare_beacon_proposer traffic from once per epoch to once per slot, with the full validator list each time. This is intentional and matches Lighthouse, but for large validator sets the beacon node sees a 32x increase in this endpoint’s payload volume. Worth keeping an eye on via metrics if not already tracked.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

The refresh's own slot skipped the per-slot re-send when the refresh's send reached a node. Two things could then stand past the next slot's proposal: a re-send from the slot before that landed after the refresh's send, carrying the address a keymanager change had replaced, and a change made after the refresh's send. Re-sending after every slot's duties puts the current address back before the next proposal, for one extra request an epoch. Refreshed loses its prepared field.

The Preparations doc now says the HTTP client's 8 s request timeout is what keeps a hung node from stopping the re-sends to the healthy ones.
…uest load

A change reaches the node a slot later while a slow node holds the last send. docs/cli.md says the addresses go out as one request per slot to each beacon node. The default prepare_beacon_proposer_quietly says why it is the plain call, the mock's preparation_entered notes the permit every call stores, and a paragraph in preferences.rs is re-wrapped.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round three (the automated review of 60650ca). Fixed in e2b58d4 and 2faeed9.

Verification:

  • make fmt: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --profile release-fast -p ethlambda-validator: 622 passed.
  • Mutation check on the gate. Each of these turns a test red: skipping the refresh's slot again, removing the spawn, dropping the store filter, dropping the guard.

Fixed

  • Claude 2, Codex P2, Kimi 1 and 2, DeepSeek 4: the refresh's slot skipped its re-send.
    • Two things could stand past the next proposal: a re-send from the slot before that landed after the refresh's send, carrying the address a keymanager change had replaced, and a change made after the refresh's send.
    • The refresh's slot now re-sends after its duties like every other slot, for one extra request an epoch, and Refreshed.prepared is gone.
    • the_preparations_go_out_again_after_every_slot_the_refreshs_included appends the late preparation after the refresh and checks the refresh slot's re-send puts the current address back.
    • I still don't serialize the refresh's own send with the re-sends. Skipping it would cost the epoch's first proposal its preparation, and waiting would hold the duty loop on a hung node for up to the 8 s request timeout. The late send can therefore still cost the epoch's first proposal, and the Preparations doc says so.
  • Claude 3: the failed-refresh test's re-send half. It is moot now that the refresh's slot always re-sends. The test is back to main's form, and the gate test asserts what that re-send carries.
  • DeepSeek 4: keymanager log and docs. The log, proposer_settings.rs and the docs/cli.md row now say a change goes out after the duties of the slot it is made in, or the next slot's while a slow node holds the last send.
  • Claude 4: preparation_entered's stored permit. The field doc says every call stores one.
  • Claude 6: preferences.rs. The paragraph is re-wrapped.
  • Claude 7, DeepSeek 5: request load. --suggested-fee-recipient in docs/cli.md says the addresses go to each beacon node as one request per slot.
  • DeepSeek 1: the default prepare_beacon_proposer_quietly. Its doc says why it is the plain call: the logging is the fallback's alone, and HttpBeaconNode logs nothing about its answer.

Not changed

  • Claude 1, DeepSeek 2: hung node.
    • HttpBeaconNode bounds every request at 8 s (REQUEST_TIMEOUT, http.rs:63), so on a 12 s slot a hung node costs the healthy ones at most every other re-send.
    • That is now in the Preparations doc. No metric: the skips are bounded, and the slot's duties already warn about a node that is down.
  • Claude 5: if let Err. CLAUDE.md asks for inspect_err when an error is only logged.
  • DeepSeek 3: detached task. The other sends in the loop are detached the same way. A panic still reaches the panic hook, and the in-flight guard clears on unwind.

Gemini found nothing. Grok posted only its opening line.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look solid and well-tested.

Key highlights:

  • Preparations::resend acquires the InFlight guard before reading ValidatorStore and ProposerSettings, ensuring that concurrent or slow sends cannot reorder and overwrite a newer fee recipient with a stale one.
  • Filtering resolved against store.contains(&entry.pubkey) under a short-lived read lock (released before the network call) properly handles mid-epoch key deletions without blocking keymanager writes or reverting a deleted key to --suggested-fee-recipient.
  • Using prepare_beacon_proposer_quietly on the per-slot path avoids spamming warnings every slot when a fallback node is down while preserving the epoch-boundary warning in refresh_epoch.

LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the surrounding validator and beacon-node code so the review is grounded in how preparations, fallback logging, and in-flight gating actually work.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: per-slot prepare_beacon_proposer re-send

The change is sound. It matches Lighthouse's cadence and closes a real gap: after a mid-epoch beacon-node restart, local blocks paid the zero address until the next epoch. The ordering hazard is handled carefully:

  • InFlight makes concurrent re-sends skip rather than overlap.
  • The re-send is entered before the settings are read.
  • The refresh's own slot re-sends too, to repair a late overtaking send.

Deleted keys are filtered via store.contains, so a migrated key is not overwritten with the default address. Tests cover the three interesting paths: the refresh slot, a deleted key, and the skip while a send is in flight. A few points:

Correctness / robustness

  1. The timeout reasoning assumes 12 s slots (lib.rs, Preparations doc, "One send at a time"). It says a hung node "can cost the healthy ones every other re-send, never all of them". That relies on the 8 s HTTP timeout being shorter than a slot minus the duty time.

    • On a devnet or preset with shorter slots (e.g. 6 s), a hung node holds in_flight past the next slot's after_slot. Healthy nodes could then miss most re-sends.
    • Either word the doc in terms of SECONDS_PER_SLOT vs the request timeout, or consider a per-request timeout for the quiet path below the slot duration.
  2. Re-imported keys get the default address (resend, store.contains filter). A key deleted and then re-imported within the same epoch passes the filter again. Its overrides were forgotten on delete, so it goes out with --suggested-fee-recipient until the user sets it again.

    • This is probably acceptable, since it is the same as a fresh import, but it is not among the three documented cases.
    • Worth a line in the "What is sent" list.
  3. Early continues skip the re-send (the preparations.after_slot(...) call at the end of the loop body). Any early continue in the slot loop, such as a skipped or overrun slot, skips it for that slot. That is fine, but if any such path is common (e.g. a refresh failure path), confirm it is intended. A short comment at the call site would help.

Mock / tests

  1. Check the mock constructors (mock.rs). The new preparation_gate and preparation_entered fields are not visible in MockBeaconNode::new() / failing(). If those constructors use a struct literal rather than ..Default::default(), the patch will not compile as shown. Please confirm.

  2. Fragile notify permit (a_re_send_still_waiting_on_a_node_makes_the_next_slot_skip). It relies on notify_one storing a permit before notified().await. That is correct today. However, any future test that calls prepare_beacon_proposer on the same node beforehand (e.g. via refresh_epoch) would satisfy the wait early, as the field doc warns. Consider asserting node.preparations().is_empty() before the gate release, to make the "slow send is still parked" precondition explicit.

Idioms / readability

  1. Unidiomatic error logging (lib.rs, refresh_epoch and resend). The let _ = sent.inspect_err(|err| { ... }); pattern is less clear than:

    if let Err(err) = sent {
        warn!(%epoch, %err, "...");
    }

    The let _ = also reads like it is silencing a #[must_use], which is misleading.

  2. Doc phrasing (after_slot). "The task, for a test to await; the loop leaves it detached" is a bit cryptic. Consider: "Returns the spawned task's handle so tests can await it; the duty loop drops it."

  3. Over-specific docs (rpc.md). "the node builds its own payloads for the zero address until the next call" is ethlambda-BN-specific behaviour, which is correct here since this is the BN's doc. The FeeRecipients comment in validator.rs now names specific VCs' cadence ("ethlambda's and Lighthouse's every slot"). Other VCs (Teku, Prysm) re-send per epoch, so "relearns the map from its next one" can still mean up to an epoch. Consider saying so.

Performance

  1. Request volume. One JSON request per slot per node carrying every resolved validator is negligible at typical operator scale and matches Lighthouse. On the refresh slot the preparations are sent twice; that is documented and intentional.

Logging

  1. Quiet-path visibility. The quiet path drops both the per-node warnings and the "all failed" warning to debug!. Persistent refusal, such as a 400 from a malformed address, still surfaces once per epoch through the refresh's warn!, which seems like the right balance.

None of these block merging. Item 1 and Item 4 are the ones I would resolve first.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The re-send guard, deleted-key filtering, and fallback broadcasting look sound. No consensus, signature-verification, or SSZ logic changes.

Tests could not run: rustup attempted to write to the read-only /home/runner/.rustup/tmp directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Reviewed the diff. No consensus-layer state transition, fork choice, attestation, or SSZ logic is changed; this is a validator-client fee-recipient resend change. The core design is sound, and the in-flight guard plus tests cover the important races.

  1. Misleading API log for unresolved validators — in crates/validator/src/http_api/proposer.rs, the new message says a fee recipient is sent “once this slot's duties are done,” but that is only true for validators already in resolved. A newly imported key is not resent until the next successful refresh_epoch, as docs/cli.md itself documents. The log should account for that case, e.g. “once the next epoch refresh resolves it” when applicable.

  2. Potential lock-order concern in Preparations::resend — crates/validator/src/lib.rs, Preparations::resend, holds the store.read() guard while calling settings.fee_recipient() for every held validator. If any keymanager/write path can hold a ProposerSettings write lock and then wait on store.write(), this can deadlock. Even if current code avoids that ordering, this is fragile; consider snapshotting the store keys first and then reading settings without the store read guard, or explicitly documenting the lock ordering.

  3. Per-slot preparation traffic is now O(validators) per node — this is a deliberate design choice and probably acceptable, but for large validator sets it is worth keeping an eye on. The comment already documents Lighthouse parity; a metric for preparation batch size or send latency would be useful.

  4. FallbackBeaconNode::prepare_beacon_proposer_quietly may have an unnecessary map — in crates/validator/src/beacon_node/fallback.rs, .map(|_| ()) is only needed if try_all_quiet returns something other than Result<()>. If it already returns Result<()>, this is redundant and may trigger clippy’s result_map_unit_fn.

  5. Detached preparation tasks are unobserved — after_slot spawns a task and the production loop ignores the JoinHandle. This matches nearby existing patterns, so not blocking, but a panic in a re-send task would be silent. Consider logging on task failure or joining at shutdown if lifecycle tracking is added later.

Otherwise, the implementation is well-documented and the new tests cover the important skipping and overwrite-recovery behavior.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness, security, or performance issues found.

The implementation is careful in the important areas:

  • Preparations::resend acquires the InFlight guard before reading settings, preventing per-slot sends from landing out of order with stale fee recipients.
  • Deleted keys are filtered out, and the store read lock is released before network I/O.
  • The epoch refresh still sends preparations eagerly, avoiding added latency before the epoch’s first proposal.
  • The fallback’s quiet variant preserves the existing “try every node, fail only when all fail” semantics.
  • Tests cover deleted keys, slow-send skipping and recovery, refresh-slot re-sends, and partial/total fallback failure.

Residual risk, already documented in crates/validator/src/lib.rs: the epoch refresh send is outside InFlight, so an older re-send can still temporarily overwrite it around an epoch boundary. The refresh-slot re-send and pre-signing fee-recipient mismatch check mitigate this. If stronger ordering is desired later, a test using a gated re-send that remains in flight across refresh_epoch would be valuable.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

The keymanager's log after a fee recipient is set, the feerecipient row in docs/cli.md and the proposer_settings.rs module doc said the address goes out after the slot's duties. That holds only for a validator the last epoch refresh resolved; one imported since, or with no index yet, is first sent once a refresh resolves it.

The Preparations doc no longer bounds a hung node's cost only for 12 s slots, says a key imported again after a delete goes out with the defaults, and words after_slot's return plainly. The single-flight test asserts the slow re-send is still parked before the gate opens.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round four (the automated review of 2faeed9). Fixed in db66094.

Verification:

  • make fmt: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --profile release-fast -p ethlambda-validator: 622 passed.

Fixed

  • DeepSeek 1: the keymanager log after setting a fee recipient.
    • It said the address goes out after this slot's duties. That holds only for a validator the last epoch refresh resolved; one imported since, or with no index yet, is first sent once a refresh resolves it.
    • The log now says it goes out "with the next proposer preparation that includes this validator".
    • The feerecipient row in docs/cli.md and the proposer_settings.rs module doc spell out the imported-key case.
  • Claude 1: the hung-node bound. The Preparations doc said "on a 12 s slot" only. It now says a hung node costs the healthy nodes the re-sends that come due while a send waits on it: at most every other one on 12 s slots, more on shorter ones, never all of them, since the 8 s request timeout ends each send.
  • Claude 2: a key deleted and imported again in the same epoch. The deleted-key bullet now says it goes out again, with the defaults it starts on.
  • Claude 5: the single-flight test now asserts that nothing reached the node while the slow re-send is parked, before the gate opens.
  • Claude 7: after_slot's doc now says it returns the spawned task's handle so a test can await it, and that the loop drops it.

Not changed

  • Claude 3: early continues. The loop has none. A retryable refresh failure falls through to after_slot, and the only exit stops the client.
  • Claude 4: mock constructors. MockBeaconNode derives Default, and both constructors use it.
  • Claude 6, DeepSeek 3 and 5: inspect_err, request volume and detached tasks were settled in round three.
  • Claude 8: other clients' cadence. "From its next one" is true for any cadence, and round three removed an unchecked claim about other clients.
  • DeepSeek 2: lock order. Both a key delete and the re-send take the store before the settings, and the settings lock is never held across anything else.
  • DeepSeek 4: .map(|_| ()). try_all_quiet returns Result<Delivery>, so the map is needed.

Codex, Gemini and Kimi found nothing to change. Grok posted only its opening line.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks good. The per-slot prepare_beacon_proposer re-send logic, InFlight serialization against slow beacon nodes, key deletion handling via ValidatorStore, quiet fallback logging, and test coverage (using deterministic gate/notify primitives) are all clean and well-structured.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: per-slot prepare_beacon_proposer re-send

The design looks sound: it matches Lighthouse's cadence, serializes re-sends through InFlight, filters deleted keys, and the tests are deterministic (gate plus Notify instead of sleeps). Nothing blocking. A few points to check, in rough order of importance.

Correctness

  1. Ordering between key deletion and settings.forget (lib.rs, Preparations::resend). resend reads settings.fee_recipient while holding the store read guard. That only rules out "key still in store, overrides already forgotten → sent with the --suggested-fee-recipient default" if the keymanager DELETE handler does one of these:

    • removes the key from the store before forgetting its settings; or
    • forgets the settings while holding the store write guard.

    The test does store.write().await.remove(..) and then settings.forget(..) as separate statements, which is the safe order. Please confirm the real DELETE /eth/v1/keystores and /remotekeys handlers do the same. If they forget first, a re-send landing between the two steps sends exactly the overwrite the doc comment says is avoided. A short comment at the handler pinning this order would help.

  2. Early continues in the duty loop. after_slot sits at the very end of the loop body. Any branch that continues earlier skips the re-send for that slot, which the restart-recovery argument depends on. Examples could be the served overrun check or a not-synced or failed-duties path. That is probably acceptable, but the doc's guarantee ("within a slot of its return") only holds if those branches are rare. Consider calling it from a point every served slot reaches, or note the exception.

  3. The "8 s timeout" claim (Preparations doc, "One send at a time"). The bound "at most every other one on 12 s slots" assumes try_all_quiet queries nodes concurrently. If it awaits them sequentially, two hung nodes take about 16 s and skip more re-sends than stated. Also, hard-coding "8 s" in prose will drift from the HTTP client constant. Reference the constant by name instead.

Rust / style

  1. let _ = sent.inspect_err(|err| { ... }); appears in refresh_epoch and resend. This is a roundabout way to log an error. The idiomatic form is clearer:

    if let Err(err) = sent {
        debug!(%slot, %err, "Failed to re-send this client's fee recipients");
    }
  2. Name shadowing. The free function preparations(...) shares its name with two locals:

    • let preparations = Arc::new(Preparations::new(..)) in run;
    • let preparations = preparations(entries, settings) in prepare_proposers and resend.

    It compiles, because the right-hand side is evaluated before the binding, but it reads poorly. Renaming the function to build_preparations or preparations_for would fix it.

  3. Type nit. Arc<Vec<ValidatorEntry>> could be Arc<[ValidatorEntry]> (resolved = entries.into()), which drops one indirection. Minor.

  4. Mock constructors (mock.rs). The new preparation_gate and preparation_entered fields rely on the mock's constructors filling them through Default. If MockBeaconNode::new or failing list fields explicitly, this will not compile. Presumably they use ..Default::default(), so this is fine as long as it builds.

Observability

  1. Persistent failures are only visible once per epoch. Per-slot failures now log at debug!. A node that accepts the refresh's send but rejects every re-send, for example because it restarted into a bad state, is invisible at info. The refresh's warn! covers the common case. A counter metric for failed re-sends, alongside inc_fee_recipient_mismatches, would make this observable without log noise.

Docs

  1. The cli.md, rpc.md, proposer_settings.rs and preferences.rs updates are consistent with the new behaviour. The feerecipient row in cli.md is getting very long, so consider moving the delivery details into a paragraph below the table.

Tests cover the important cases: the re-send on the refresh's own slot correcting a late overwrite, deleted keys being excluded, and a skipped re-send not waiting on a slow node.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The per-slot resend, deleted-key filtering, and fallback broadcast behavior look sound. This PR does not change consensus processing, signature verification, or SSZ encoding.

Tests could not run: rustup attempted to update the toolchain in a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found.

The implementation in crates/validator/src/lib.rs handles the important concurrency cases well:

  • InFlight is acquired before reading settings, preventing concurrent re-sends from landing out of order.
  • The validator-store read lock is released before the network request, avoiding keymanager stalls behind a slow beacon node.
  • Deleted keys are filtered before preparing the resend.
  • The epoch-refresh send remains independent, with the refresh-slot resend limiting the stale-write race.
  • prepare_beacon_proposer_quietly preserves fallback partial-success semantics while reducing repetitive warnings.

The tests cover the significant cases: all-node failure, deleted keys, refresh-slot resends, fee-recipient changes, and slow in-flight requests.

The only residual consideration is operational: request size and beacon-node work now scale as O(resolved validators) per slot, per fallback node. That matches the intended Lighthouse-style behavior, but large validator deployments may benefit from metrics tracking preparation request duration and skipped resends.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The per-slot re-send and the deleted-key filter are the right fix for a restarted beacon node building to the zero address. A few correctness gaps remain.

  1. Stale in-flight send can outlive more than the epoch’s first proposal. Preparations::resend skips when in_flight is held, and the epoch refresh’s prepare_beacon_proposer does not take that lock (lib.rs, resend / the “One send at a time” comment). Sequence: slot N−1 re-send is still waiting with the old fee recipient; refresh writes the new one; the late request then overwrites it; the refresh slot’s corrective after_slot also skips because that same guard is still held. The next send is only after the following slot’s duties, so it lands after that slot’s proposal as well. The comment’s “at most the epoch’s first proposal” is wrong on the skip path — the next proposal can pay the old address too. When a send is skipped, coalesce a follow-up that runs as soon as the in-flight guard drops (re-read settings, send once), instead of waiting for the next after_slot.

  2. MockBeaconNode::preparation_entered is a required Arc<Notify> (mock.rs ~127) and Notify is not Default, but no constructor/Default update is in the diff. new() will not compile until it sets Arc::new(Notify::new()).

  3. The proposal warning (proposal.rs ~323) says a keymanager fee-recipient change reaches the node after that slot’s duties. That is not true while a slow node holds the previous send (Item 1). The settings docs already qualify this; the log line should too.

The quiet fallback path is sound: try_all_quiet still fans out to every node, fails only when none accept, and the tests cover both. Filtering re-sends with store.contains correctly avoids clobbering another client’s preparation with --suggested-fee-recipient after a key delete. Off-loop spawn plus InFlight is the right backpressure for an 8s HTTP timeout.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a well-structured change with thorough tests. I have a few substantive comments and some minor clarity issues.

  1. crates/validator/src/beacon_node/mock.rs — The diff adds preparation_gate and preparation_entered to MockBeaconNode, but I don’t see the corresponding initialization in MockBeaconNode::new/Default or a use tokio::sync::Notify; import. If those aren’t present elsewhere in the PR, this won’t compile. Please confirm they’re included.

  2. crates/validator/src/lib.rs, Preparations::resend / after_slot — The ordering guarantee is weaker than the comment suggests. InFlight::enter() is synchronous, and tokio::spawn does not guarantee that the slot N task enters InFlight before the slot N+1 task, even though after_slot(N) is called first. If the slot N task is scheduled late, slot N+1 can send newer fee recipients first, and then slot N can enter afterward and send stale settings, overwriting the newer ones. The slot parameter is currently only used for logging. To make the ordering explicit, consider:

    • tracking the latest slot/generation in Preparations or InFlight and ignoring stale sends, or
    • making InFlight::enter() an async queue where send order is preserved.

    This is a low-probability race in normal scheduling, but for fee-recipient correctness it’s worth hardening.

  3. crates/validator/src/lib.rs, Preparations::resend — The guard is bound as _running; this works, but it’s easy to misread as an unused variable rather than the thing keeping InFlight occupied. Prefer _guard or _in_flight_guard, especially because the early return on empty preparations relies on dropping it.

  4. crates/validator/src/lib.rs, refresh_epoch — resolved is only returned in Refreshed after duties.refresh_around(epoch).await?. If that fallible call fails after duties.set_indices and prepare_proposers have already run, the duty loop does not update resolved, so the new per-slot re-sends keep using the previous resolution. Verify whether duties.refresh_around can fail after set_indices has already taken effect; if so, a restarted node could miss newly activated validators until the next successful refresh.

  5. Load / large validator sets — preparations() builds a full Vec<ProposerPreparationDto> for every resolved validator every slot and sends it to every fallback node. The PR explicitly documents this as intentional and Lighthouse-like, but it’s worth making sure HTTP body limits and fallback timeouts remain acceptable for operators with thousands of validators.

The test coverage is good, especially the cases for deleted keys, same-slot re-send after a late prior send, and the in-flight skip behavior.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit e69132b Nov 19, 2026
14 checks passed
@MegaRedHand
MegaRedHand deleted the fix/validator-prepare-proposers-every-slot branch November 19, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants