Skip to content

feat(validator): publish proposer preferences (gloas) - #707

Merged
MegaRedHand merged 55 commits into
feat/beacon-builder-vcfrom
feat/beacon-builder-preferences
Nov 17, 2026
Merged

MegaRedHand merged 55 commits into
feat/beacon-builder-vcfrom
feat/beacon-builder-preferences

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

First gloas step of the builder work: the validator client publishes signed proposer preferences (fee recipient, target gas limit) for its gloas proposals. Stacked on #705.

Why: from gloas, beacon-APIs makes SignedProposerPreferences the source of a proposer's fee recipient and gas limit, superseding prepare_beacon_proposer. A proposer without preferences also gets no trustless bids: p2p bids for its slot are ignored.

What it adds

Piece Behavior
Publishing Every slot, for the current and next epoch that are gloas epochs (so warm-up starts one epoch before the fork). Fetches v2 proposer duties, whose dependent_root is the shuffling dependent root the preferences must carry; the duty loop keeps v1. Keeps only duties inside the target epoch, one per slot, so a node's answer cannot make it sign more than an epoch's worth. Signs each own duty whose slot has not started and which not every node holds yet for that (slot, dependent_root). Sends one preference per request to every node, all of a poll's at once (both target epochs' fetches and sends too), so one refusal does not hold back the others and a hung node costs a poll one timeout. A preference only some nodes accepted is re-sent, as first signed, each slot until every node holds it. One every node holds is re-sent once an epoch, at the epoch's second-to-last slot (its last, if a slow node made the client skip that poll), for a node that restarted. A reorg that moves the dependent root re-publishes. With the slot clock, a slot that started while the poll waited is not sent
Unconditional Not gated on --builder-proposals, as in Lighthouse: from gloas the preferences carry the fee recipient for every proposal. Validators with no fee recipient are skipped, with one warning per epoch
Signing domain DOMAIN_PROPOSER_PREFERENCES at the fork version of the proposal epoch, as consensus-specs v1.7.0-beta.4 signs and verifies it, and as Lighthouse, Prysm, Teku, Lodestar and Nimbus do. ethlambda's beacon node also still accepts beta.2's domain (the fork of the epoch before the proposal's), since our fixtures are pinned to beta.2
Eth-Consensus-Version The target epoch's fork (gloas)
Gloas boost factor and bid floor With --builder-proposals, produceBlockV4's body carries the configured builder_boost_factor, so pooled p2p bids can win on value, and --builder-min-bid (gwei, default 0) is sent as min_bid with or without it. Without it the factor stays 0: the local payload unless it cannot be built, so a node whose execution client fails at the slot may still build on a p2p bid, and the client signs that block, as Lighthouse, Prysm and Teku do (#796 revisits the default)
--gas-limit The default target gas limit (60000000), for registrations before gloas and preferences from it, which produceBlockV4 builds this client's own payload toward. Not tied to --builder-proposals
Fee recipient check A gloas block may pay the current address or that of a preference a node accepted for the slot, which a node pays in preference to the current one
Metrics ethlambda_validator_proposer_preferences_published_total, ..._failures_total (sends some node refused or did not answer, re-sends included), ..._duty_fetch_failures_total (failed v2 duty fetches). A refusal by every node is warned once per preference, then logged at debug; preferences newly held by only some nodes are warned once per poll

Known limitations

  • A preference cannot be replaced for the same slot and dependent root, so a fee recipient changed after publishing applies from the next dependent root on.
  • A beacon node restarted mid-epoch lacks the rest of that epoch's preferences; the once-an-epoch re-send covers the next epoch from its first slot.
  • GAS_LIMIT_SCHEDULE is not read, so a default configuration on a chain whose gas limit is not 60M still pulls toward 60M until --gas-limit is set (validator: take the default target gas limit from GAS_LIMIT_SCHEDULE #795). Sepolia and Platåberget schedule 200M, and the docs say to pass --gas-limit 200000000 there.

Tests

  • Unit: the v2 path; per-entry submission; refusal isolation; reorg re-publish; the pre-gloas gate; started slots, by the poll's slot and by the clock; missing fee recipients; no keys loaded; single-flight polling; the fork-version choice on both sides of the boundary; the boost factor and min bid in the gloas body; the once-an-epoch re-send (identical bytes, counted once, the next epoch's first slot covered, also when the round's first poll is skipped), and a re-send one of two nodes or every node refuses; five sends to a slow node costing one wait; --gas-limit and --builder-min-bid parsing; the fee recipient check's accepted addresses, the proposer's own only; a zero gas limit refused by the keymanager.
  • End to end through ethlambda's own beacon node: preferences signed by the validator client from its own v2 duties pass the node's gossip rules and are gossiped.

make lint passes, as do the validator lib, rpc lib and binary tests. The second-round changes (once-an-epoch re-send, concurrent sends, --gas-limit, --builder-min-bid, the fee recipient check) are not devnet-tested yet.

Devnet results

Kurtosis (ethereum-package) with this branch's head (18a907cf), Lighthouse and Geth glamsterdam-devnet-8, and buildoor as the builder. Fulu from genesis, gloas at epoch 6. Three nodes: Lighthouse alone; ethlambda's validator client on a Lighthouse beacon node (--builder-proposals --prefer-builder-proposals, 40% of stake); ethlambda for both (--builder-proposals). All three stayed on one head through the fork.

Works

  • v2 duties: 0 fetch failures, against both ethlambda's and Lighthouse's beacon node.
  • Publishing: from the warm-up epoch on, once per (slot, dependent_root), 0 failures through ethlambda's beacon node, which validated, pooled and gossiped every preference.
  • Lighthouse accepts our preferences from the second gloas epoch on: buildoor, which follows a Lighthouse node, cached all 32 slots of epochs 7 and 8, ours included.
  • Fee recipient timing is exactly as documented: changed through the keymanager in epoch 9, epoch 10's preferences and blocks kept the old address, and epoch 11's carried and paid the new one. The pre-sign check logs an ERROR for every block in between (20 here), and its hint ("reaches the node only at the next one") is wrong at gloas, where it can take two epochs.
  • Metrics move as documented.
  • The builder stack underneath (feat(validator): propose builder blocks (fulu) #705 and below) works in fulu: registrations, filtering, blinded blocks through both ethlambda's and Lighthouse's beacon node, the bid gas-limit check, boost factor 0, and the circuit breaker.

Fails at the fork boundary

  1. Signing domain (fixed since: the client now signs at the proposal epoch's fork version, as beta.4 and every client do). Lighthouse refused all 16 epoch-6 preferences from the validator client with BadSignature, then accepted the same client's epoch-7 and epoch-8 ones. The client retried each refused preference every slot until its slot started (749 failures).
  2. Warm-up preferences are gossiped to nobody (follow-up: Warm-up proposer preferences are gossiped before peers subscribe to the gloas topics #794). ethlambda's beacon node joins the gloas topics an epoch before the fork and gossips the warm-up preferences at once, but Lighthouse subscribes only 2 slots before the fork. No node gossips them again (the client's once-an-epoch re-send gets a 200 for an identical entry, which the node does not republish), so no Lighthouse node, and no builder, saw them. The node still logs Published proposer preferences to gossipsub at info, because the publish error is logged only at debug (swarm_adapter.rs). Possible fixes: re-send the pooled preferences at the fork, or publish the warm-up epoch's preferences closer to it.

Together, on a network with Lighthouse, none of our validators had a preference for the first gloas epoch.

Not tested

  • Gloas bid weighing (builder_boost_factor in produceBlockV4): buildoor's builder was never onboarded (the builder registry was empty on both clients), so there were no bids and every gloas block was self-built. Both beacon nodes accepted the request body.
  • Preferences without --builder-proposals, the reorg re-publish, and validators with no fee recipient.
  • Whether ethlambda's beacon node accepts the other signing version: Lighthouse's epoch-6 preferences never reached it.

Found along the way, not from this PR (same on main)

  • Finality stopped at the fork. At gloas the validator client omits committee_index from attestation_data and sends payload_attestation_data?slot=, as beacon-APIs master says. Lighthouse glamsterdam-devnet-8 requires committee_index and serves /payload_attestation_data/{slot}, answering 400 to both. The client on Lighthouse missed every attestation from the fork on: target participation fell from 97% to 58%, and justification stopped.
  • The validator client had no --gas-limit: it registered (and from gloas preferred) 60M unless set per validator through the keymanager. On a 200M chain every builder bid failed the gas-limit check. Added in this PR, since from gloas the preference also steers the node's own payload.
  • ethlambda's local payloads are empty (getPayload right after forkchoiceUpdated), so any bid beats a local build, and boost factors above 0 cannot be told apart. --builder-min-bid now sets a floor.
  • Lighthouse glamsterdam-devnet-8's gloas blocks pay the zero address, which our client flags as a fee recipient mismatch.

https://claude.ai/code/session_011vghNjfTQmZNRCjPkRs1wP

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 707: validator client publishes signed proposer preferences

I read the diff and the checked-out preferences.rs, signing.rs and the fork-choice helper. I did not build the PR or run its tests. The change looks correct and well tested, with only minor points.

Correctness

  • The DependentRoot::V2 refactor in rpc/src/beacon/validator.rs is behavior-preserving. compute_shuffling_dependent_slot is start_slot(epoch.saturating_sub(MIN_SEED_LOOKAHEAD)).saturating_sub(1), which matches the old inline code when MIN_SEED_LOOKAHEAD is 1. Sharing it with the state transition also removes a drift risk.
  • sign_proposer_preferences signs at the fork version of proposal_epoch - MIN_SEED_LOOKAHEAD, matching the gossip rule. Both sides of the fork boundary are tested. The PR documents the one case where this differs from the validator guide.
  • publish_epoch skips started slots, sends one preference per request, and remembers each success by (slot, dependent_root). Pruning is keyed correctly. Poisoned mutexes are recovered, and InFlightGuard clears the flag on panic.
  • The mock now bumps proposer_duties_v2_calls only after guard(), so failed calls are not counted. The mock's doc comments say the same.

Minor points

  1. preferences.rs:269-310: each preference is sent in turn, and each send goes to every node. A client with many keys and a slow node could take a while per poll. The in_flight guard keeps polls from stacking, but a poll that overruns skips the next slot's poll. That could cost preferences for the slots after it, which gossip IGNOREs once they have started. A handful per epoch is fine, so this only matters at high key counts. Sending with bounded concurrency (join_all or buffer_unordered) would remove the risk cheaply.
  2. preferences.rs:183-198: a failed duties fetch is counted and logged at debug!, and also warns once per epoch. That is intentional, but fetch_warned is pruned with retain(>= epoch), so a node that stays down warns once per target epoch. That is acceptable.
  3. published is keyed on (slot, dependent_root) only. A fee recipient or gas limit changed later is not re-sent for the same root, which the PR documents under "No replacement". Fine.
  4. http.rs: builder_config formats the JSON by hand. That is safe because the factor is a u64, but serde_json::json! would be more idiomatic. This is a nit.
  5. lib.rs: the spawn of preferences.poll(...) is a detached task, so panics are swallowed apart from the guard reset. The guard makes this safe.

Tests
Coverage is thorough: v2 path, per-entry submission, refusal isolation, reorg re-publish, the pre-gloas gate, started slots, missing fee recipients, single-flight, fork-version choice, the boost factor on the wire, and an end-to-end run against ethlambda's own beacon node. The Lighthouse fork-boundary interop is flagged as unverified, which is honest.

No blocking issues.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

I reviewed the PR diff and the touched Rust paths in context. Overall this looks careful and the added tests materially improve confidence around proposer-preferences signing and v2 duties. I did not find a clear consensus-breaking bug in the new logic.

  • No blocking correctness/security issue stands out in the added proposer-preferences flow.
  • The main behavior change — signing DOMAIN_PROPOSER_PREFERENCES using the epoch before the proposal and sourcing dependent_root from v2 proposer duties — is well-motivated and covered by tests in crates/validator/src/signing.rs:335, crates/validator/src/preferences.rs:183, and crates/net/rpc/src/beacon/builder_market_tests.rs:1065.

Findings

  • Minor robustness concern: PreferencesService keeps published, refusal_warned, no_fee_warned, and duty_fetch_warned behind std::sync::Mutex in async code (crates/validator/src/preferences.rs:99). If any of those locks are ever held across future .await refactors, this becomes a latent footgun. It looks safe today, but I’d consider tokio::sync primitives or extra care/comments around lock scope for maintainability.
  • Potential operator-confusion/API concern: the CLI help now says --builder-proposals “boost factor weighs p2p bids in produceBlockV4” even though the option still sounds boolean and gloas publishing is now unconditional (bin/ethlambda/src/validator.rs:80). The implementation seems intentional, but the UX semantics have shifted enough that this may surprise operators. Not a correctness bug, but worth ensuring docs/help text make it crystal clear that on gloas this flag only influences boost factor, not whether proposer preferences are published.
  • Test gap, low severity: the end-to-end test explicitly notes it does not pin the fork-boundary version choice (crates/net/rpc/src/beacon/builder_market_tests.rs:1072). Since fork-boundary domain mismatches are exactly the kind of subtle consensus-adjacent bug that can reappear later, I’d keep an eye on whether the unit coverage in crates/validator/src/signing.rs:335 is enough, or add an integration-style boundary case if practical.

What looks good

  • Correct use of v2 proposer duties for dependent_root, which matches the gossip validation contract and avoids a likely source of silent rejection (crates/validator/src/beacon_node/http.rs:642, crates/validator/src/preferences.rs:183).
  • Good defensive behavior around blinded responses when builder proposals are off; refusing a blinded block in that mode avoids signing something this client cannot publish (crates/validator/src/beacon_node/http.rs:668).
  • Sensible deduplication semantics keyed by (proposal_slot, dependent_root), which handles reorgs without suppressing valid resubmission (crates/validator/src/preferences.rs:99).
  • Strong test coverage for HTTP encoding/decoding, gossip acceptance, duplicate handling, and signature-domain behavior.

Looks good to me aside from the minor maintainability/UX notes above.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which implements proposer preferences for the Gloas fork. Let me analyze the changes systematically.

Overall Assessment

This is a well-structured PR adding Gloas fork proposer preferences support. The core logic is sound, but I've identified several issues ranging from minor to potentially serious.


Critical Issues

1. Potential Division by Zero in builder_boost_factor Documentation vs. Implementation

File: crates/validator/src/beacon_node/http.rs, lines 88-96

The documentation states 0 prefers local payload, but the JSON sends "0" as a string. The spec's produceBlockV4 expects this to be a number, not a string. However, looking more carefully, the format string produces "0" (quoted string), not 0 (number).

fn builder_config(boost_factor: Option<u64>) -> String {
    let factor = boost_factor.unwrap_or(0);
    format!(r#"{{"min_bid":"0","builder_boost_factor":"{factor}","builders":[]}}"#)
}

Issue: builder_boost_factor is serialized as a string ("0", "250"), but the Beacon API spec defines it as a number (uint64). This will cause spec-compliant beacon nodes to reject the request or parse it incorrectly.

Fix: Remove quotes around {factor}:

format!(r#"{{"min_bid":"0","builder_boost_factor":{factor},"builders":[]}}"#)

Similarly, verify whether min_bid should be a string or number per spec.


Security Issues

2. Missing Input Validation on builder_boost_factor

File: crates/validator/src/beacon_node/http.rs, line 93

boost_factor is passed directly from CLI/config into JSON without bounds checking. While u64 is inherently bounded, consider whether extremely large values (e.g., u64::MAX) could cause issues in downstream parsing.

Mitigation: The current code is probably safe, but consider capping or documenting maximum sensible values.


Correctness Issues

3. Fork Version for Signing Domain at Fork Boundary

File: crates/validator/src/signing.rs, lines 335-342

let proposal_epoch = compute_epoch_at_slot(preferences.proposal_slot);
let domain = self.domain(
    DOMAIN_PROPOSER_PREFERENCES,
    proposal_epoch.saturating_sub(preset::MIN_SEED_LOOKAHEAD),
);

Issue: MIN_SEED_LOOKAHEAD is 4 epochs (mainnet), not 1. This subtracts 4 epochs, not 1.

Wait—let me re-check. The comment says "the epoch before the proposal's" but the code subtracts MIN_SEED_LOOKAHEAD (4).

Looking at the test at line 689:

let preferences = preferences_for(gloas_epoch * preset::SLOTS_PER_EPOCH + 2);

And the test verifies it signs with fulu_fork_version when proposal_epoch == GLOAS_EPOCH. With MIN_SEED_LOOKAHEAD = 4, GLOAS_EPOCH - 4 is still Fulu (assuming Fulu ends at GLOAS_EPOCH - 1).

But what if proposal_epoch == GLOAS_EPOCH + 3? Then proposal_epoch - 4 = GLOAS_EPOCH - 1, still Fulu. Only at GLOAS_EPOCH + 4 does it become Gloas.

This contradicts the comment which says "the epoch before the proposal's." The spec's validate_proposer_preferences_gossip uses the fork at the dependent block's epoch, which for v2 duties is compute_shuffling_dependent_slot(epoch) — that's start_slot(epoch) - 1, i.e., the last slot of the previous epoch, which falls in epoch - 1.

So the correct fork epoch should be proposal_epoch - 1, not proposal_epoch - MIN_SEED_LOOKAHEAD.

File: crates/validator/src/signing.rs, line 339

proposal_epoch.saturating_sub(preset::MIN_SEED_LOOKAHEAD),

Fix: This should be proposal_epoch.saturating_sub(1) to match the actual gossip rule domain.

However, looking at the test a_preference_for_the_first_gloas_epoch_is_signed_under_the_fulu_version — it passes with GLOAS_EPOCH because GLOAS_EPOCH - 4 is still in Fulu territory. But the test a_preference_for_the_second_gloas_epoch_is_signed_under_the_gloas_version uses GLOAS_EPOCH + 1, and GLOAS_EPOCH + 1 - 4 = GLOAS_EPOCH - 3, which is still Fulu!

Wait, let me re-read the test... The test at line 691 only checks that it verifies under gloas_fork_version, but doesn't check that it fails under fulu_fork_version. Let me check...

Actually, looking more carefully at line 720-724:

assert!(verify(
    &pubkey,
    &signature,
    under(context.config.gloas_fork_version)
));

This only asserts it verifies under Gloas, not that it fails under Fulu. So the test doesn't catch the MIN_SEED_LOOKAHEAD issue.

This is a bug: The signing domain uses MIN_SEED_LOOKAHEAD (4) instead of 1, meaning for epochs GLOAS_EPOCH through GLOAS_EPOCH+2, the signature would verify under Fulu in ethlambda but potentially fail against other clients' gossip validation that uses the correct epoch-1 semantics.

Wait—I need to re-read the spec deviation document more carefully. It says:

validate_proposer_preferences_gossip verifies against the dependent block's state advanced to the lookahead start

The "lookahead start" might indeed be MIN_SEED_LOOKAHEAD epochs before. Let me re-check the consensus spec...

Actually, looking at the CLAUDE.md change:

the gossip rule's signing domain

And the spec deviation says:

the gossip rule's state (at the start of the previous epoch)

Hmm, "previous epoch" suggests 1 epoch, not 4. But "lookahead start" in the spec might mean MIN_SEED_LOOKAHEAD.

Given the explicit test and the detailed comment, this might be intentional. However, the comment says "the epoch before the proposal's" which is 1, not 4. This inconsistency between comment and code is confusing and potentially wrong.

Recommendation: Verify against the actual consensus spec for validate_proposer_preferences_gossip. If the spec uses epoch - 1, fix the code. If it uses epoch - MIN_SEED_LOOKAHEAD, fix the comment.

4. compute_shuffling_dependent_slot vs. V1 Definition Consistency

File: crates/net/rpc/src/beacon/validator.rs, lines 540-544

Self::V2 => compute_shuffling_dependent_slot(epoch),

Good: This delegates to the state transition function, preventing drift. But verify that compute_shuffling_dependent_slot is actually exported and correct.


Performance Issues

5. Lock Contention in PreferencesService::poll

File: crates/validator/src/preferences.rs, lines 181-186

for target in [epoch, epoch + 1] {
    if self.context.config.fork_at_epoch(target) < ForkName::Gloas {
        continue;
    }
    self.publish_epoch(current_slot, target, &pubkeys, &store).await;
}

The loop processes epochs sequentially. Each publish_epoch awaits v2 duty fetches and submissions. Since the two epochs are independent, they could be parallelized with try_join! or join!.

Mitigation: Low impact (only 2 epochs), but worth considering for high-latency nodes.

6. Per-Preference Sequential Submissions

File: crates/validator/src/preferences.rs, lines 252-280

Preferences are sent one-by-one in a loop with await each iteration. For validators with many proposal slots, this serializes network round-trips.

Mitigation: Consider limited concurrency (e.g., futures::stream::iter(...).buffer_unordered(4)) to parallelize submissions while maintaining per-preference error handling.


Rust Best Practices

7. Mutex Poison Recovery Hides Bugs

File: crates/validator/src/preferences.rs, lines 121-125

fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
    mutex
        .lock()
        .unwrap_or_else(std::sync::PoisonError::into_inner)
}

While documented, silently recovering from poisoned mutexes can hide panics in other threads. In a consensus client, this could mask critical failures.

Recommendation: Consider whether parking_lot::Mutex (which doesn't poison) would be cleaner, or at least log when recovering from poison.

8. AtomicBool Ordering Could Be Relaxed

File: crates/validator/src/preferences.rs, line 178

if self.in_flight.swap(true, Ordering::AcqRel) {

AcqRel is stronger than needed for a simple flag. Acquire on load, Release on store would suffice, or even Relaxed since there's no data dependency being synchronized (the actual data goes through Arc clones).

Fix: Use Ordering::Acquire for the load part of swap... actually swap doesn't let you split. Consider compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) for better performance on some architectures.

9. std::sync::Mutex in Async Code

File: crates/validator/src/preferences.rs, multiple locations

Using std::sync::Mutex and immediately locking/unlocking in async code is acceptable for very short critical sections, but parking_lot::Mutex or tokio::sync::Mutex would be more idiomatic. The current pattern of lock(&self.published).insert(...) is fine since it doesn't hold across await points.


Error Handling & Edge Cases

10. Empty preferences List Early Return

File: crates/validator/src/beacon_node/http.rs, lines 837-839

if preferences.is_empty() {
    return Ok(());
}

Good defensive coding. But verify this matches the spec—does the Beacon API allow empty POSTs? The mock test confirms this behavior, but ensure it's consistent with other similar endpoints.

11. Missing Handling of dependent_root Mismatch

File: crates/validator/src/preferences.rs

The service remembers published preferences by (slot, dependent_root), but what if a node accepts a preference with a different dependent_root than what the service expects? The v2 duties fetch gives one root, but the node's head might have moved on.

Current behavior: The preference would be signed with root A, sent to node with head at root B, and the node's gossip rule would reject it. The service would retry next slot.

Potential issue: If there's a consistent mismatch (e.g., node sees different fork choice), the service would warn once and then debug-log retries, potentially missing proposals silently.

Mitigation: Consider adding metrics or more visible logging for persistent mismatches.


Testing Issues

12. Test verifies Function Duplicates Production Logic

File: crates/validator/src/preferences.rs, lines 437-463

The verifies helper reconstructs domain computation manually instead of using SigningContext. This creates a maintenance burden and risk of test-production drift.

Recommendation: Use SigningContext::domain or a test helper that shares code.

13. Missing Test for Fork Boundary Signing

File: crates/validator/src/signing.rs

The tests cover first and second Gloas epochs, but miss:

  • Epoch exactly at GLOAS_EPOCH + 3 (where MIN_SEED_LOOKAHEAD subtraction would still yield Fulu if using 4, but Gloas if using 1)
  • The transition from Fulu to Gloas during the epoch before Gloas (when signing preferences for GLOAS_EPOCH during GLOAS_EPOCH - 1)

Documentation & Comments

14. Inconsistent Terminology: "gloas" vs "Gloas"

Minor: Some comments use lowercase "gloas" for the fork name. Standardize to "Gloas" when referring to the fork.

15. CLAUDE.md Update Is Good but Could Be Clearer

The CLAUDE.md change correctly documents the behavior, but the sentence:

From the epoch before gloas it signs proposer preferences every slot for its proposals in the current and next epoch

Could be misread as "from the epoch before Gloas starts" vs "starting from the epoch before each target epoch." The actual behavior is: starting from GLOAS_EPOCH - 1, sign preferences for slots in GLOAS_EPOCH and GLOAS_EPOCH + 1.


Minor Issues

16. builder_config Function Name

File: crates/validator/src/beacon_node/http.rs, line 88

The name builder_config is slightly misleading—it's the self-build configuration, not general builder configuration. produce_block_v4_body or self_build_config might be clearer.

17. Redundant std::slice::from_ref Import Style

File: crates/validator/src/preferences.rs, line 269

std::slice::from_ref(&entry)

Consider importing std::slice::from_ref at the top for consistency, or use &[entry] directly if the trait accepts it.

Actually, looking at the trait:

preferences: &[gloas::SignedProposerPreferences],

std::slice::from_ref(&entry) produces &[SignedProposerPreferences], which is correct. &[&entry] would be wrong. This is fine.


Summary Table

Priority Issue Location Fix
Critical builder_boost_factor serialized as string not number http.rs:94 Remove quotes: "{factor}" → {factor}
High Signing domain uses MIN_SEED_LOOKAHEAD (4) vs epoch-1; comment/code mismatch signing.rs:339 Verify spec, fix code or comment
Medium Mutex poison recovery hides bugs preferences.rs:121 Use parking_lot or log recovery
Medium Sequential epoch processing preferences.rs:181 Consider join!
Medium Sequential preference submission preferences.rs:252 Consider limited parallelism
Low AtomicBool ordering could be relaxed preferences.rs:178 compare_exchange_weak
Low Test logic duplicates production preferences.rs:437 Share domain computation

The PR is well-tested and the architecture is sound. The string-vs-number JSON serialization is the most critical fix needed before merge.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

@MegaRedHand
MegaRedHand added this pull request to stack #731 October 29, 2026 19:41
@MegaRedHand
MegaRedHand force-pushed the feat/beacon-builder-preferences branch from ae1a3d2 to 5931f19 Compare October 29, 2026 20:11
@MegaRedHand
MegaRedHand removed this pull request from stack #731 October 29, 2026 20:41
@MegaRedHand
MegaRedHand added this pull request to stack #733 October 29, 2026 20:41
@MegaRedHand
MegaRedHand force-pushed the feat/beacon-builder-preferences branch from 5931f19 to 652bfc0 Compare October 29, 2026 20:50
…nces

Signed proposer preferences must carry the shuffling dependent root, which
only v2 of the proposer duties endpoint reports (v1's root is one epoch
later). Add both calls to the beacon node trait, with the fallback sending
preferences to every node since each one gossips what it accepts.
…main

The gossip validator verifies a preference at the fork version of the epoch
before its proposal, while the validator guide signs with the head state.
The two differ only in the first epoch of a fork, where the gossip rule is
what peers accept, so sign for that one.
From gloas a proposer's fee recipient and gas limit travel in signed
proposer preferences, and a node ignores trustless bids for a slot whose
proposer published none. A per-slot service signs one per owned proposal in
the current and next gloas epoch, keyed by (slot, dependent root) so a reorg
re-publishes, and sends it to every node.
…oas bids by the boost factor

Wire the preferences service into the slot loop, unconditionally since from
gloas they carry every proposal's fee recipient. produceBlockV4 now sends the
configured builder_boost_factor in its body, so with --builder-proposals a
pooled p2p bid can win on value; without it the factor stays 0 as before.
…'s gossip rules

The signing domain is the part of the preferences flow most likely to drift
between the signer and the verifier, so drive the validator client's own
signer and HTTP client against the real route and gossip rules.
Document the per-slot preferences, their two counters, the signing domain
departure from the validator guide, and that the boost factor now also
weighs gloas p2p bids.
… their reporting

A node answers 400 for a whole call when any entry fails, and one it holds
for the same slot and root with other content is such a failure, so a
batch kept every entry unpublished and re-sent until that slot passed (a
restart after a fee recipient change triggers it). Send each preference
alone and remember each success.

Also: a failed v2 duties fetch no longer warns "duty likely missed" twice a
slot (the service warns once per epoch and counts it), the fork header is
the target epoch's, the poll is spawned ahead of the proposal, the
signing-domain explanation is corrected (get_domain reads state.fork, so
the guide and the gossip rule differ only for a proposal in a fork's first
epoch signed during it), v2 dependent slot reuses the state transition's
definition, and the end-to-end test now reads its duty and dependent root
through the client.
A preference a node keeps refusing is retried every slot until its slot
passes, which warned up to two epochs of slots per entry. Warn on the first
refusal of each (slot, dependent root), log repeats at debug, and keep
counting every failure. Reword the failures metric to what it now counts.
…split the fetch counter

try_all warned per node on every refusal, so a preference a node kept
refusing still warned every slot despite the per-entry dedup; add a quiet
variant for it. Poll after the proposal rather than before, so the next
epoch duties fetch does not compete with block production. Count failed v2
duty fetches on their own counter, since a syncing node is not a refused
preference.
From gloas the proposer preferences carry each validator's fee recipient
and target gas limit, so the keymanager's `feerecipient` and `gas_limit`
reach the node through them, from the next dependent root on, whether or
not `--builder-proposals` is set. The README's duty list, written on main
by #716, gains the preferences, which are published every slot.
@MegaRedHand
MegaRedHand force-pushed the feat/beacon-builder-preferences branch from 652bfc0 to 34a914f Compare October 30, 2026 16:03
@pablodeymo

Copy link
Copy Markdown
Collaborator

Security read of this PR, with the signing side and the network/node side reviewed separately. No critical or high issues: the message is signed under DOMAIN_PROPOSER_PREFERENCES alone, so nothing here is slashable or replayable under another domain; only keys in the store are ever signed for, and a key removed between the duties fetch and signing is skipped; no secret is logged or cloned; no lock is held across an await; the single-flight guard clears on panic and drop; and the signing domain (fork version at proposal_epoch - 1), the v2 dependent root (compute_shuffling_dependent_slot, the same function the gossip rule uses) and the Eth-Consensus-Version all line up with what the node verifies, pinned by the signing.rs tests and the end-to-end gossip test. Two things I'd fix before merging, plus smaller ones.

1. Bound and check the duties before signing (preferences.rs::publish_epoch, http.rs::proposer_duties_v2)

The v2 duties are taken as they come: no cap on the entry count and no check that duty.slot lies in the target epoch. The fallback's try_each hands the list from the first node that answers, so one buggy or hostile node can return N entries naming our keys at far-future slots with a rotating dependent_root. Each slot the client then signs N messages while holding the store read guard (tokio's RwLock is write-preferring, so a keymanager write queued behind it blocks the slot loop's next read), sends N sequential POSTs to every node at up to 8 s each inside the single-flight poll, and inserts N keys into published or refusal_warned. The prune keeps every slot at or above the current epoch's start, so far-future keys are never dropped. Nothing dangerous is signed (own domain, own fee recipient), so this is amplification and memory growth, not key exposure. Dropping duties whose slot is outside [start(target), start(target + 1)) bounds everything to 32 entries per target and makes the prune effective.

2. A node that refused once never receives the preference again (fallback.rs::try_all_inner, preferences.rs::publish_epoch)

try_all answers Ok as soon as one node accepts, and the caller then marks (slot, dependent_root) published and skips it on every later poll. The retry-every-slot the module doc promises only happens when every node refused. A node that was syncing, restarting, or lacked the dependent block at that moment ends up with no entry in its builder market, and since gossip never queues or replays, nothing fills it in later. If the client fails over to that node for produceBlockV4 at the slot, it ignores every p2p bid and self-builds, and the gas target falls back to the parent bid's. Registrations (#705) re-send the whole set every epoch and so self-heal; preferences have no periodic resend. Keying published by (slot, root, position), or marking it published only when every configured node accepted, would close it.

Smaller

  • validator_index and dependent_root go into the signed message verbatim from the node. The duty loop already resolved the indices, so a cross-check is cheap; a wrong index is rejected by gossip today, so the cost is only a refused send.
  • The zero address passes --suggested-fee-recipient (hex and length only), and from gloas this PR signs and publishes it unconditionally, so a mistyped zero address burns rewards. Same point as on feat(validator): propose builder blocks (fulu) #705, with more reach here.
  • The settings are read twice per message (fee recipient, then gas limit), so a keymanager write in between pairs an old recipient with a new gas limit for that (slot, root), which cannot then be replaced. One snapshot per message would do.
  • Response bodies are still reflected uncapped into BeaconNodeStatus and from there into the once-per-key warn line; the client keeps reqwest's default redirect policy. Both pre-existing (feat(validator): propose builder blocks (fulu) #705).
  • The route comment about a validator client naming fulu during the epoch before gloas describes other clients; this one always sends the target epoch's fork.

Tests I'd add: duties at slots outside the target epoch are dropped, and a list of hundreds of entries is bounded; a node that refused while another accepted receives the preference on a later poll; a key removed between the duties fetch and signing is skipped without losing the others; a preference signed during the first gloas epoch itself passing the node's rules end to end (the gossip test has gloas at epoch 0, so the boundary is unit-pinned only).

Checked and clean: pre-gloas target epochs skipped before any fetch; started slots skipped, matching the node's IGNORE; one request per preference, so one refusal cannot hold back the others; an empty list never reaches the wire; builder_boost_factor is a u64 in a format!, None renders 0, and the node multiplies in u128, so no clamp is needed; poisoned mutexes recovered; a bounced poll exits at once; no_fee_warned and fetch_warned hold at most two live epochs; the node route accepts a fulu or gloas header and caps entries at the lookahead length; nothing new derives Debug on a secret, and logs carry pubkeys, slots and indices only.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks great. The implementation of PreferencesService, the v2 proposer duties client method, the produceBlockV4 builder boost factor handling, and the gossip signing domain handling at the fork boundary are all clean, well-documented, and thoroughly covered by unit and integration tests.

A few highlights:

  • Using compute_shuffling_dependent_slot directly in DependentRoot::slot (crates/net/rpc/src/beacon/validator.rs:543) ensures the RPC route and state transition logic cannot drift.
  • Sending preferences individually and tracking state per (slot, dependent_root) in PreferencesService (crates/validator/src/preferences.rs:270-310) cleanly avoids a single stale/conflicting preference blocking the rest of the epoch's preferences, while handling reorgs and mid-epoch configuration updates naturally.
  • Signing at proposal_epoch.saturating_sub(MIN_SEED_LOOKAHEAD) (crates/validator/src/signing.rs:342-345) accurately matches the fork version in the lookahead state used by validate_proposer_preferences_gossip during the first epoch of Gloas.

No issues found — ready to merge.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Retry preferences on nodes that failed to accept them — preferences.rs:281. FallbackBeaconNode::try_all_inner returns success when any node accepts, so this permanently marks the preference published even if another node was unavailable. When that node recovers, subsequent polls skip delivery; gossip does not guarantee replay of preferences published during its downtime. If block production fails over to it, builder bids may be ignored and the requested gas target unavailable. Track acceptance per node and retry failed deliveries using the same signed preference. Add a regression test covering partial failure followed by recovery.

Tests could not run: Rustup attempted to write to its read-only temporary directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Review: proposer preferences in the validator client (PR 707)

Overall this is well-structured. Single-flight polling, per-entry submits, per-(slot, root) dedup and once-per-epoch warnings are all sensible, and the tests are thorough. The main gap is in the gloas proposal flow (Item 1); the rest are smaller.

Correctness / consensus

1. A non-zero gloas boost factor lets a p2p bid win, but the proposal path is unchanged.

  • Where: http.rs builder_config, BlockRequest docs in mod.rs.
  • Problem: With --builder-proposals, produceBlockV4 can now return a block whose bid has builder_index != BUILDER_INDEX_SELF_BUILD. The trait doc for execution_payload_envelope still describes it as "the envelope the node cached for the self-built payload". The diff does not touch proposal.rs.
  • Consequence: If the proposal service unconditionally fetches, signs and publishes the envelope after a gloas block, a won builder bid will produce a 404. That gets logged as a missed envelope, or retried, on every builder-won proposal. Nothing should ever sign an envelope for a builder's payload.
  • Fix: Confirm the proposal path branches on the bid's builder index. Add a test where the mock returns a non-self-build gloas block.

2. Unconditional publishing combined with factor 0 sends builders a mixed signal.

  • Without --builder-proposals, preferences are published, so builders are invited to bid. The body then carries builder_boost_factor: "0", so no bid can ever win.
  • Publishing is defensible (fee recipient), and the docs say this. Still, consider stating explicitly in docs/cli.md that bids are solicited but never taken without the flag. Operators reading the metrics may otherwise be confused.

3. The signing domain deviates from validator.md, which is an interop risk.

  • Where: signing.rs sign_proposer_preferences.
  • The reasoning about get_domain(state.fork) vs. the schedule is sound. In the common case, first-gloas-epoch preferences are signed during the fulu epoch, so both definitions agree.
  • The divergence only hits re-signs inside the first gloas epoch, e.g. after a reorg moves the root or after a restart.
  • Risk: If Lighthouse, Prysm or others verify against the head state rather than the advanced dependent state, those re-signed preferences will be rejected on their mesh. Only ethlambda's endpoint is known to accept both.
  • Ask: Worth confirming against another client's gossip implementation before relying on it. The deviation doc should mention which side other clients took.

4. Verify that the DependentRoot::V2 refactor matches the old formula at the edges.

  • Where: rpc/beacon/validator.rs, around line 540.
  • The old code was compute_start_slot_at_epoch(epoch.saturating_sub(1)).saturating_sub(1), which gives slot 0 for both epoch 0 and epoch 1.
  • Please confirm compute_shuffling_dependent_slot saturates identically for epochs 0 and 1. If it does not, the served root changes there.
  • A unit test pinning DependentRoot::V2.slot(0), slot(1) and slot(2) would lock this down.

Behaviour

5. Only the first node's duties are signed against.

  • Where: fallback.rs proposer_duties_v2 / submit_proposer_preferences.
  • Duties come from the first node that answers (try_each_quiet), but the submit goes to all nodes. A secondary node on a different head refuses the entry, and that refusal is only logged at debug!.
  • published is then set as soon as one node accepts, so the refusing node is never retried even after it converges.
  • This is acceptable given the cost, but worth a sentence in the module doc ("Why every node gets them") so nobody assumes full fan-out is guaranteed.

6. Pre-fork submissions carry a gloas Eth-Consensus-Version.

  • Where: preferences.rs publish_epoch.
  • During the epoch before gloas, fork_at_epoch(target) yields Gloas while the node's head is still fulu.
  • That is correct per the endpoint semantics ("the fork the preferences belong to"). However, the only test with a fulu header is the mock/HTTP one; the node-side integration test runs with gloas at epoch 0.
  • An ethlambda-node test at a gloas boundary with this header would close the gap.

Minor

  • preferences.rs, fee-recipient check: lock(&self.published) is taken once per duty inside the loop. The lists are tiny so this is fine, but a single guard held across the loop would read cleaner.
  • metrics.rs, failures counter: the doc comment says "no beacon node accepted", while docs/metrics.md says "refused or failed to take". Align the wording.
  • Docs layout: CLAUDE.md and README.md reflow is fine. The README line now exceeds the surrounding wrap width.
  • builder_market_tests.rs, integration test: the spawned axum::serve task is never shut down. That is fine in tests, but holding the JoinHandle and calling abort() at the end is tidier.

Nice work extracting test_support and the quiet fallback variants; the try_*_inner split keeps the logging policy in one place.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

Finding

  • Partial fallback submission is incorrectly remembered as fully published — crates/validator/src/preferences.rs, in publish_epoch’s successful submit_proposer_preferences arm, together with crates/validator/src/beacon_node/fallback.rs’s try_all_inner. try_all_inner returns Ok(()) when at least one node accepts, as the new three-node fallback test demonstrates. The service then inserts (slot, dependent_root) into the global published set and never submits that preference again. If node A accepts while node B is down or refuses, B is never retried once it recovers. This contradicts the module’s “a node that refused a send must be tried again” behavior and can reduce gossip/builder coverage or leave a later block-producing node without the preference’s fee recipient. Track publication per node, or return incomplete coverage from the fallback submission and retry until every configured node has accepted. Add a test where one node accepts initially and a failing node recovers on a later poll.

Otherwise, the change looks well considered: the v2 dependent-root calculation now shares the state-transition helper, the fork-boundary signing domain is explicitly tested, single-flight cleanup is panic-safe, and one-preference-per-request avoids batch-wide rejection poisoning unrelated entries.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: proposer preferences in the validator client (PR 707)

This is a well-reasoned PR. The domain switch to the proposal epoch's fork version, the v2 dependent_root, and the "keep as signed when the outcome is unknown" state machine all look correct, and the test coverage is thorough. Remarks below, most important first.

1. crates/net/rpc/src/beacon/proposer_preferences.rs: dedup now depends on validate's rule order

Before this change, the identical-entry short-circuit was a direct market.preferences(..) == Some(&signed) check ahead of validate. Now it only fires when validate returns Ignore(AlreadySeen). That holds only while AlreadySeen stays the first rule.

  • If a timing rule (e.g. "slot already started") or any other rule is ever moved ahead of it, an identical resubmission becomes a 400.
  • That matters because the validator client re-sends every kept preference once an epoch, and each slot while Partly.

Nothing is lost by keeping the cheap check first and adding the post-record_preferences race check:

if holds(market, &signed) { debug!(..); continue; }
let verdict = validate(market, store, &signed, now_ms);
...
if !market.record_preferences(signed.clone(), wall_slot) {
    if holds(market, &signed) { continue; }
    ...
}

This drops the coupling. The race the new comment describes is still covered by the second holds.

Nit: in holds, the key tuple only exists to be destructured again. Pass the fields directly.

2. crates/validator/src/preferences.rs (submit, Err(err) if late): a late, outcome-unknown send leaves no record

When the slot starts while the send is in flight and the error is outcome-unknown (a timeout or 5xx), nothing is inserted into sent. A node may still hold that preference and build the block with its fee recipient. accepted_fee_recipients won't list it, so check_fee_recipient logs an error! and bumps fee_recipient_mismatches_total, a metric documented as "should read zero forever".

This only happens if settings changed between signing and proposing, so it is rare. Still, recording Sent::Unconfirmed(entry) when err.outcome_unknown() would cost nothing, even when late. The slot is skipped afterwards anyway via started().

3. bin/ethlambda/src/validator.rs, http.rs builder_config: builder-bid blocks are signed without opt-in

Without --builder-proposals, produceBlockV4 now gets builder_boost_factor: 0 and min_bid defaults to 0. A node whose EL fails at the slot can then build on any pooled bid, including a 1-gwei one, and this client signs it. The rationale (parity with Lighthouse/Prysm/Teku, the slot is otherwise lost) is sound and documented. Operators should still be able to see it happen, though:

  • Consider an info!/warn! plus a counter in proposal.rs when a gloas block whose bid is not BUILDER_INDEX_SELF_BUILD is signed while builder_boost_factor.is_none().
  • Today it would only surface through the generic "both addresses" log in the else branch of compares_fee_recipient.

4. Fallback with nodes on different heads

proposer_duties_v2 is try_each_quiet, so the dependent root comes from the first node that answers. If a later node sits on a different head, it refuses the preference indefinitely. The entry stays Partly, is re-sent every slot, and proposer_preferences_failures_total climbs until the root changes.

This is correct behaviour, but it deserves a sentence in the metrics doc next to the "restart with other settings" note, so operators don't chase it as a fault.

5. crates/validator/src/lib.rs: poll spawned after an awaited propose

tokio::spawn(preferences.clone().poll(..)) runs only after propose(..).await. On a proposal slot, the poll for the next slot is therefore delayed by the full proposal round-trip. Normally preferences go out an epoch ahead, so the impact is small. A reorg that changes the root right before the next slot could make that preference late, though.

Spawning before propose would avoid this. The poll's v2 duties fetch could then make the node advance a state while it produces the block, which is the stated reason for the current order. That reason is fine; just be aware of the trade-off.

6. crates/validator/src/error.rs classify: reordering changes blinded-publish semantics

Putting is_connect() ahead of is_timeout() turns a connect timeout from Timeout (outcome unknown) into Connect (definite). The changed assertion in http.rs (Timeout → Connect) suggests the blinded-publish path is affected too.

That is correct, since nothing was sent. But a blinded publish that hit a connect timeout now counts as a refusal rather than BlindedUnconfirmed. Please confirm that the blinded path still never re-proposes after a definite failure. Slashing protection already recorded the slot, so it shouldn't, but a regression test asserting this would pin it down.

7. Minor

  • state_transition/.../proposer_preferences.rs: reordering the domains so the beta.4 domain is tried first is good, since the common case now costs one verification. At a fork boundary, an invalid signature still costs up to two or three pairings, which predates this PR. Worth a TODO to drop the beta.2 domains once fixtures move.
  • fallback.rs submit_proposer_preferences: AtomicBool with Relaxed is fine. Note that join_all polls on one task, so it is only needed for Send.
  • proposer_settings.rs with_default_gas_limit: it does not validate > 0. The CLI and keymanager both do, but a direct library caller of ValidatorConfig could pass 0. Consider a debug_assert! or a NonZeroU64.
  • preferences.rs within_epoch: capping hostile duty lists at one per slot is a nice touch. A hostile duties node can still make the client sign arbitrary dependent_roots and validator_indexes for its own slots. These are harmless because they are unslashable and verified against the index's pubkey, but a one-line note in the module doc's "Why v2 proposer duties" section would make the trust boundary explicit.

The CLI, metrics and docs updates are consistent with the code.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR is well-designed and thoroughly tested. The handling of the v1.7.0-beta.4 signing domain, v2 shuffling dependent roots, per-node delivery tracking across fallbacks/restarts, and the TOCTOU check in post_proposer_preferences are all solid.

A couple of minor edge-case suggestions in crates/validator/src/preferences.rs:

  1. Recording Unconfirmed when a submission fails with outcome_unknown after its slot started (crates/validator/src/preferences.rs:489-497)
    In PreferencesService::submit, if submit_proposer_preferences returns Err(err) after key.0 has started (late == true), the Err(err) if late arm returns early without updating self.sent:

    Err(err) if late => {
        debug!(
            epoch = target,
            slot = key.0,
            validator,
            %err,
            "A proposer preference's slot started before a beacon node took it"
        );
        Tally::default()
    }

    If before was Before::Unsent or Before::Refused and err.outcome_unknown() is true (e.g., the request was sent before the slot started, the beacon node accepted it, and the response timed out after the slot boundary), self.sent won't record Sent::Unconfirmed(entry). Recording Sent::Unconfirmed(entry) when err.outcome_unknown() here won't cause any extra retries (since started is already true for future polls), and ensures accepted_fee_recipients still includes the preference's fee recipient if a proposal for that slot is in flight.

  2. Defensive saturating arithmetic on slot/epoch bounds (crates/validator/src/preferences.rs:146, 562, crates/validator/src/lib.rs:306)
    In round_of(current_slot) (current_slot.saturating_add(2)), publish (epoch.saturating_add(1)), and within_epoch (target.saturating_add(1)), using saturating_add avoids potential overflow if a mock or malformed response ever supplies a near-u64::MAX slot/epoch.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found.

The implementation handles the important edge cases well:

  • crates/validator/src/preferences.rs preserves the originally signed preference after uncertain delivery, preventing different nodes from holding conflicting preferences for the same (slot, dependent_root). Untrusted duties are epoch-bounded and deduplicated, and store locks are not held across network awaits.
  • crates/validator/src/beacon_node/fallback.rs correctly distinguishes definite refusal from unknown outcomes and reports partial delivery so missing nodes are retried.
  • crates/net/rpc/src/beacon/proposer_preferences.rs handles the gossip/submission race without republishing identical entries.
  • The beta.4 signing-domain change and v2 dependent-root calculation have strong fork-boundary and end-to-end test coverage.

One non-blocking operational tradeoff: the once-per-epoch re-send schedule in PreferencesService can leave a beacon node restarted just after a round without next-epoch preferences until late in that epoch. This is accurately documented, but it means builder bids may be ignored and stale fee-recipient/gas-limit settings may persist for much of an epoch. An additional epoch-start re-send would shorten that gap if the extra requests are acceptable.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The preferences work is careful and well tested: beta.4 signing, v2 dependent roots, one-preference-per-request, and the unconfirmed-message stickiness all line up with the gossip rules. A few correctness gaps are worth fixing before this ships.

  1. A late, unconfirmed send is dropped, which breaks the “may be held” invariant. In PreferencesService::submit (crates/validator/src/preferences.rs), Err(err) if late returns without writing sent. That is wrong when err.outcome_unknown() is true: the node may already have cached and gossiped that exact message. The slot will not be retried (started stays true), so the signed fee recipient never lands in accepted_fee_recipients. If the keymanager address changed while the POST was in flight, check_fee_recipient then false-alarms on a payment this client actually authorized. Same hole if before was Refused and this attempt signed a new message: state stays Refused even though the new message may be the one the node holds. On outcome_unknown, store Sent::Unconfirmed(entry) even when late. A definite 4xx/503 can stay untracked.

  2. Produced blocks are not checked against the gas limit this PR exists to set. check_fee_recipient / pays_this_validator (crates/validator/src/proposal.rs, around the new helper) allow the current address or any tracked preference address, then sign anyway on mismatch. Nothing compares the bid or payload gas limit to target_gas_limit / --gas-limit. A beacon node that ignores preferences (or still builds toward 60M) fails silently, and this client will sign the block that pulls the network limit the wrong way. Sepolia and Platåberget are exactly that case. Log and count a mismatch the same way as fee recipient; do not need to refuse the block.

  3. BeaconNodeFailure::classify now treats connect timeout as Connect (definite, nothing sent). That is the right call for preferences, and the reqwest ordering (is_connect before is_timeout) matches the comment in crates/validator/src/error.rs. Confirm blinded-block publish still treats only post-connect Timeout / Request as “may have been handed to the relay”. A connect failure mis-labeled the other way is an equivocation; this direction is safe only if is_connect() cannot fire after bytes were written.

  4. Next fork will reject preferences. post_proposer_preferences allows only Fulu and Gloas (crates/net/rpc/src/beacon/proposer_preferences.rs, version check). The validator sends fork_at_epoch(target), so the first fork after gloas fails the header check on this node until that list is updated. Fine while gloas is newest; worth a comment or “header fork ≥ Fulu” so it does not get lost.

Nits: builder_config in crates/validator/src/beacon_node/http.rs interpolates JSON by hand. u64s are safe today; serde_json::json! will not rot when a string field is added. The AlreadySeen label when record_preferences loses to a different message is slightly misleading; the validator treats every 4xx the same, so behavior is fine.

The domain reorder, compute_shuffling_dependent_slot for v2 duties, the within_epoch cap (hostile duties cannot blow the sign/send loop), and the end-to-end fork-boundary tests are in good shape. Accepting beta.2 domains as a fallback does not widen the set, and signing under the proposal epoch is what other clients verify.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Reviewed the proposer preferences service, builder min-bid, gas-limit, and domain changes. The state machine is clear and the test coverage is thorough. Two issues worth addressing:

  • In crates/validator/src/preferences.rs, PreferencesService::submit, the Err(err) if late branch returns Tally::default() without updating the sent map. Compare the Ok(Delivery::Partly) late case, which still records Sent::Partly. If the error was outcome_unknown (for example a timeout after the node may have received and accepted the preference), the node can hold/gossip that preference before the slot starts, but accepted_fee_recipients will not return it. Later ProposalService::check_fee_recipient can then falsely report a fee-recipient mismatch for a block that actually paid the preference’s recipient. I would record Sent::Unconfirmed(entry) when err.outcome_unknown() is true, even though the slot has already started, because that map is still consulted during block signing for the current slot.

  • In crates/validator/src/proposer_settings.rs, with_default_gas_limit accepts any u64, including zero. The CLI rejects zero via clap’s range(1..) and POST gas_limit rejects zero, but ValidatorConfig is public, so a programmatic caller can set gas_limit: 0. That would cause proposer preferences to be signed with target_gas_limit: 0. Add an assertion, clamp to a positive value, or otherwise validate the input here as well.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

Claude, Gemini, Grok and DeepSeek (seventh review): a send whose slot
started before its answer left no record, even when the failure was one a
node may have taken. Its address was then missing from the block's fee
check, and a message signed afresh after a refusal stayed recorded as
refused though a node may hold it. Such a send of a message no node was
known to hold is now kept as Unconfirmed; one already kept stays as it is.
The mock waits before failing too, so a test can have a slot start while
the send is in flight.

The node now checks for an identical held entry before validate rather than
relying on AlreadySeen being validate's first rule, so no rule can refuse a
resubmission (Claude, in every round). docs/metrics.md also names nodes on
different heads as a steady, expected source of preference failures.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable correctness or security findings in the reviewed diff. The signing domain, preference retries, and Gloas builder configuration look consistent with the intended behavior.

Tests could not run: rustup attempted a toolchain update but could not write to its read-only temporary directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 707 — proposer preferences in the validator client

This is a substantial, carefully reasoned PR. The domain fix (signing at the proposal epoch's fork version) and the V2 dependent-root unification look correct. The Sent/Before state machine is coherent, and the "keep the message as first signed" invariant is well defended in tests. The comments below are mostly about edge cases and robustness.

Correctness / interop

  1. Duplicate-resend answers from other clients (preferences.rs, submit, the Before::Everywhere and Before::Partly arms). The design relies on a node answering 200 to an identical, already-held preference.

    • For ethlambda that is guaranteed: holds() in rpc/.../proposer_preferences.rs returns success.
    • For Lighthouse, Prysm and Teku the module doc says "treats it as already seen". That is a gossip outcome, not necessarily an HTTP 200.
    • If any of them answers 400 on an IGNORE/duplicate, the consequences are:
      • every epoch's re-send round flips Everywhere → Partly;
      • from then on each such preference is re-sent every slot until its slot starts;
      • proposer_preferences_failures_total climbs, and the "Not every beacon node took…" warning fires each round.
    • Please confirm the HTTP behaviour against each client. If you can't, consider treating a 400 whose body names "already seen/known" as acceptance.
  2. BeaconNodeFailure::classify reorder (error.rs). Moving is_connect() ahead of is_timeout() is semantically right, and it changes the expectation in http.rs from Timeout to Connect. However, it also changes behaviour for every existing caller, not just preferences:

    • Blinded-publish handling used to treat a connect timeout as "unconfirmed". It now treats it as a definite failure.
    • That is correct if reqwest only sets is_connect before any bytes are written, but it deserves a line in the PR description and a check of any other matches!(…, Timeout) sites.
    • Adding Build also means any exhaustive match on the enum (metric labels, Display) needs reviewing.
  3. Hostile or flapping node and the sent map (publish_epoch). A node that answers v2 duties with a different dependent_root on every poll makes the client do all of the following each slot:

    • sign a fresh preference per owned slot;
    • send it to every node;
    • add a map entry.

    Pruning bounds this to about two epochs of entries, and the preferences are not slashable, so the risk is low. Still, a per-slot cap on distinct roots, or a debug log when a slot's root changes more than once per epoch, would make it observable.

  4. check_fee_recipient leniency. accepted_fee_recipients includes Unconfirmed entries that no node may ever have taken, and it ignores the dependent root. Because every address in it was signed by this same validator, this is safe. Noting it only so it is a deliberate choice.

  5. Signing builder-built gloas blocks without --builder-proposals (proposal.rs module doc, CLI). This is defensible and matches Lighthouse, Prysm and Teku. Two things operators should know:

    • --builder-min-bid defaults to 0, so an operator who never opted into builders can end up proposing a 1-gwei bid block whenever the EL hiccups.
    • Consider logging at info when a non-self-build gloas block is signed with builder proposals off, so the fallback is visible.

Tests

  1. a_late_send_a_node_may_hold_is_kept is timing-flaky (preferences.rs).
    • genesis = now + 1 - slot*12 uses whole seconds, so the target slot starts somewhere in (0, 1] s from the test's start.
    • If the sub-second fraction is near 1, the slot has already started when publish_epoch evaluates self.started(...). The duty is then skipped, nothing is inserted, and stages(&service)[&(slot, root)] panics on a missing key.
    • The 1100 ms delay also leaves only about 100 ms of margin on the other side.
    • Prefer an injectable clock, or start_paused with a SlotClock built off tokio::time. a_slot_the_clock_says_started_is_not_sent is fine, since it has a margin of a second or more.

Rust / maintainability

  1. try_all_inner (fallback.rs) discards each node's Ok(T). That is fine today because T is always a single node's answer. If a FallbackBeaconNode is ever nested, an inner Delivery::Partly would be reported upward as full acceptance. Mapping Ok(Delivery::Partly) from a node to "not accepted everywhere" would make this robust for little cost.

  2. MockBeaconNode::on_proposer_duties_v2: Option<Box<dyn Fn()>>. Fine as test-only code, but it blocks a future #[derive(Debug)] on the mock.

  3. Spawn every slot (lib.rs, tokio::spawn(preferences.clone().poll(...))). Pre-gloas this returns immediately, which is cheap. Still, gating the spawn on fork_at_epoch(epoch + 1) >= Gloas would avoid a task per slot for months on mainnet.

  4. Documentation volume. The module doc for preferences.rs is about 110 lines, and much of it is repeated in docs/cli.md, spec_deviations.md and several function docs. That much repetition will drift. Consider keeping the full rationale in one place and linking to it.

    • Minor: docs/cli.md has broken reflow ("Duties outside their epoch, and all but the\nfirst for a slot").

Small nits

  • http.rs, builder_config: building JSON with format! is safe here because both values are u64. A serde_json::json! would still be more future-proof if a builders list is added.
  • set_gas_limit rejects 0, consistent with the CLI. There is no upper sanity bound, which is acceptable.
  • proposer_preferences.rs (rpc): the post-record_preferences holds() re-check is a good fix for the gossip race.

Overall this is close to approve. I'd like Items 1 and 6 addressed (or confirmed) before merge.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The implementation, fork-boundary domain handling, and end-to-end test coverage look solid. One minor race window in the RPC handler worth tightening:

  1. crates/net/rpc/src/beacon/proposer_preferences.rs (lines 129–141):
    holds(market, &signed) is checked before validate(...) and again after market.record_preferences(...) to handle the case where gossip caches the identical preference concurrently. However, the first rule inside validate is AlreadySeen (market.has_preferences(...)). If gossip records the identical preference in the brief window after the initial holds(market, &signed) check on line 129 and before validate's AlreadySeen check on line 133, validate will return Outcome::Ignore(IgnoreReason::AlreadySeen) and the handler will record a 400 failure instead of treating it as cached meanwhile.

    Checking holds(market, &signed) when !matches!(outcome, Outcome::Accept) (or specifically on Outcome::Ignore(IgnoreReason::AlreadySeen)) closes that window:

    let outcome = validate(store, market, &signed, wall_slot);
    if !matches!(outcome, Outcome::Accept) {
        if holds(market, &signed) {
            debug!(%slot, validator, "Proposer preferences cached meanwhile; not republishing");
            continue;
        }
        let reason = describe(&outcome);
        warn!(%slot, validator, %reason, "Rejected submitted proposer preferences");
        failures.push(Failure {
            index,
            message: reason,
        });
        continue;
    }

Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The proposal-epoch signing domain and v2 dependent-root changes are well covered at the Fulu→Gloas boundary. The fallback’s definite-refusal versus unknown-outcome handling is appropriately conservative, and PreferencesService avoids replacement conflicts by retransmitting the originally signed message. Duty bounding, single-flight polling, pruning, and lock scopes also look sound.

One non-blocking test concern:

  • crates/validator/src/preferences.rs, test a_late_send_a_node_may_hold_is_kept: this depends on real wall-clock timing and a 1.1-second sleep. Scheduling delays could make the slot start before submission, causing the test to skip the send and fail; it also slows the suite. Consider an injectable/controllable clock or paused-time approach similar to a_slow_node_costs_a_poll_one_wait.

The temporary acceptance of legacy beta.2 signature domains is documented and bounded to the fork-boundary compatibility case.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The preferences state machine, beta.4 signing domain, and v2 dependent-root alignment look correct and are well tested. Fork choice, justification, finalization, and the STF are untouched. A few issues are worth fixing before this ships.

  1. Untrusted validator_index is signed and then used as the fee-recipient allowlist key. In publish_epoch the pubkey is checked against the local store, but the index written into ProposerPreferences is taken from the duty the beacon node returned. Gossip will reject a mismatch, yet a node that returns 200 is recorded as held, and accepted_fee_recipients filters on that index (proposal.rs passes duty.validator_index the same way). A hostile or buggy node can suppress the mismatch warning across this client's validators. Resolve the index from a trusted mapping, or refuse to sign when the same pubkey is paired with more than one index.

  2. --builder-min-bid is not enforced on the client. produceBlockV4 is sent the floor, but ProposalService still signs any non-blinded gloas block, including a builder bid when --builder-proposals is off. If the node ignores min_bid (or applies it after the boost factor), a 1-gwei bid is signed anyway. That is the case --prefer-builder-proposals is most exposed to. Confirm this node's produceBlockV4 rejects bids below the floor, and drop or warn on a bid that does not clear builder_min_bid before signing.

  3. Late-preference tests are wall-clock races. a_slot_the_clock_says_started_is_not_sent and a_late_send_a_node_may_hold_is_kept derive genesis from SystemTime::now() and are not start_paused. A slow CI machine can cross the slot before started runs, so the second test never inserts Unconfirmed. Drive SlotClock from a paused clock or an injected time.

  4. Duty fetch latency can skip the re-send window. round_of (current_slot + 2) is right for a 32-slot epoch, but proposer_duties_v2 fails over one node at a time and each dead node adds a full timeout before any send. Enough dead nodes listed first push the round past the next epoch's first slots, which then never get the restart re-send. Worth a metric or a cap on failover delay for this path; the module doc already describes the failure mode.

Smaller notes:

  • outcome_unknown treating connect timeouts as definite (BeaconNodeFailure::Connect before Timeout) is the right fix, and keeping the original signed bytes on Unconfirmed / Partly is what makes a non-deterministic BLS nonce safe. Good.
  • DependentRoot::V2 delegating to compute_shuffling_dependent_slot, with the beacon-API slot test, is the right way to keep the duties route and gossip from drifting.
  • Trying the proposal-epoch domain first in proposer_preferences_domains is correct for v1.7.0-beta.4. Keeping the beta.2 domains is fine as a temporary gossip leniency; it is not a forgery risk, but it should stay tied to the fixture pin so it does not become permanent.
  • Up to two epochs of per-preference POSTs are fired with join_all and no concurrency cap (about 64 per epoch, times every beacon node). Correct for per-entry status, but a large validator set will burst the node at each epoch boundary. A small semaphore would be enough.

Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall, this is a thorough, heavily tested PR. The preference signing-domain reordering and gossip dedup changes look correct.

A few points worth addressing:

  • crates/validator/src/beacon_node/http.rs, builder_config: --builder-min-bid is documented and parsed as gwei, but the value is emitted directly into produceBlockV4’s min_bid without conversion. If the node’s min_bid follows Beacon API/execution-layer conventions, it is in wei, so any non-zero --builder-min-bid would be off by 10^9. Either convert here (min_bid * 1_000_000_000, using a wider type) or explicitly document that ethlambda’s min_bid is gwei if the node intentionally deviates.

  • crates/validator/src/preferences.rs, PreferencesService::sign: the validator-store read guard is held across all BLS signings for an epoch. Since publish runs both target epochs concurrently, this can hold the read lock for up to 2 * SLOTS_PER_EPOCH signatures and block keymanager writes for the whole batch. Acquiring the read guard per signature, or otherwise minimizing the locked section, would reduce write starvation.

  • crates/validator/src/preferences.rs, publish_epoch + within_epoch: duties are deduplicated by slot before checking whether the pubkey belongs to this client. A buggy or malicious beacon node can return an unowned duty for a slot first and cause this client to skip signing a preference for its own validator at that slot. The node is already trusted for duties, so this is low severity, but filtering to owned pubkeys before dedup would be safer and better match the “untrusted input” comment.

  • Minor API validation: ValidatorConfig::gas_limit and ProposerSettings::with_default_gas_limit accept 0 from library callers. The CLI and keymanager API reject zero, so consider validating at the library boundary too.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

Gemini (eighth review): checking for the identical held entry before
validate (6760670) opened a window. Gossip can record the same message
between that check and validate's AlreadySeen rule, which then refused the
submission with a 400. A refused verdict now asks again whether the node
holds the identical entry before refusing.

Also:
- a_late_send_a_node_may_hold_is_kept now has a slot starting two to three
  seconds in and a node answering after three and a half, so only a
  two-second stall can break it (Claude, Kimi and Grok: the one-second
  margins could flake).
- A docs/cli.md paragraph is reflowed (Claude).
# Conflicts:
#	crates/validator/src/beacon_node/fallback.rs
#	crates/validator/src/beacon_node/http.rs
#	crates/validator/src/beacon_node/mock.rs
#	crates/validator/src/http_api/proposer.rs
#	crates/validator/src/signing.rs
@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue because a pull request earlier in the stack was removed Nov 17, 2026
# Conflicts:
#	crates/validator/src/lib.rs
@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit f4efc82 Nov 17, 2026
13 checks passed
@MegaRedHand
MegaRedHand deleted the feat/beacon-builder-preferences branch November 17, 2026 22:19
MegaRedHand added a commit that referenced this pull request Nov 18, 2026
Brings the validator client work that landed on main (durable EIP-3076 slashing protection #734, its import and export #736, the review follow-ups #789, voluntary exits #779, builder blocks #705 and proposer preferences #707) along with the builder PRs they stack on (#694, #695, #696, #704), the generated node key (#726) and #787, #793.

Conflicts resolved by keeping both sides: the engine client's getBlobsV2 timeout next to main's one-attempt getPayload timeout, and their wire tests; nightly's batched block_signature_set next to main's verify_proposer_signature (the blinded block's check); gloas_proposal.rs importing both EventBus and the sync status.
MegaRedHand added a commit that referenced this pull request Nov 18, 2026
…s limit in validator_definitions.yml (#780)

The keymanager API's feerecipient, graffiti and gas_limit routes overrode
one validator's settings in memory only, so a restart put every validator
back on the command line's defaults. They are now written onto the
validator's entry in validator_definitions.yml, in Lighthouse's layout
(suggested_fee_recipient, graffiti, gas_limit), before memory changes, and
seeded from the file at startup. A re-import keeps an entry's values, an
import that enables a disabled entry loads and validates the values it
carries, and keystore and password files are flushed before the entry that
names them is saved.

Resolved against main, which this PR predates (slashing protection #734,
#736, #789 and the builder flow #705, #707):

- The definitions file is saved through main's write_private_durably, which
  already flushes the file and its directory, so the PR's own rename plus
  sync_dir is dropped. The settings routes save through save_definitions,
  keeping definitions_unflushed accurate, and treat Error::Unflushed as
  written (memory follows the file, with a warning), as the PR's logged
  directory flush did. The import seeds the settings in both the Ok and the
  Unflushed arm and returns main's Overwritten.
- write_private and write_private_no_symlink keep the PR's flush, the latter
  staying public for the node key.
- Overrides keeps main's registered() next to the PR's apply(), so seeding
  and every route still bump the generation the builder registrations
  cache on.
- A gas_limit of zero in the file is refused, as --gas-limit and POST
  gas_limit refuse it since #707 signs it into proposer preferences.
- Main's newer tests build ValidatorDefinition with the three fields as None.
MegaRedHand added a commit that referenced this pull request Nov 18, 2026
…nd proposer preferences (#694-#707) and the slashing-protection follow-ups (#789), and sign the new messages through the async signing path.

- Builder registrations and proposer preferences sign through `sign_under` like every other message, so a remote key's are Web3Signer requests: `VALIDATOR_REGISTRATION` with no `fork_info` (the signer builds the builder domain from its own genesis version), and `PROPOSER_PREFERENCES`, versioned gloas and told the fork at the proposal epoch. `SigningRequest::fork_info` is optional for that.
- `RegistrationService::current` decides under the cache's locks, signs the chunk through `sign_all` with every lock released, and writes the cache last. Each chunk still runs on the blocking pool (driven there with `Handle::block_on`), so local signatures stay off the runtime and a chunk stays the unit of failure. `PreferencesService::sign` goes through `sign_all` with no read guard held.
- `refresh_epoch` returns `Refreshed { moved, registrable }`. The epoch's full registration send is spawned after the slot's block, beside the subnet subscriptions, so its signer requests never queue ahead of the block's.
- `ProducedBlock::header` covers the blinded block, whose header is the full block's.
- The validator crate depends on `futures-util` alone: main added it for the same `join_all` this branch took `futures` for.
pablodeymo added a commit that referenced this pull request Nov 18, 2026
…s-limit and the keymanager's POST gas_limit refuse it, and correct the --enable-keymanager help text.

- `ValidatorDefinition::overrides()` parsed `suggested_fee_recipient` and `graffiti` but passed `gas_limit` through unchecked, so a hand-written `gas_limit: 0` was seeded as that validator's gas limit, at startup and on an import that enables the entry. The command line and the keymanager route both refuse zero: the value goes into builder registrations, and since #707 it is signed into the gloas proposer preferences, which cannot be replaced for their dependent root. It is refused in `overrides()` now, with an error naming the entry and the field, as a malformed fee recipient or graffiti is. Raised in the review of #780, with this check and its test.
- `--enable-keymanager`'s help text still said the three settings are held in memory only. It now reads as the flag's row in `docs/cli.md` does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants