Skip to content

feat(validator): propose builder blocks (fulu) - #705

Merged
MegaRedHand merged 114 commits into
mainfrom
feat/beacon-builder-vc
Nov 17, 2026
Merged

MegaRedHand merged 114 commits into
mainfrom
feat/beacon-builder-vc

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Fifth step of MEV-Boost builder support: the validator client side. With --builder-proposals, the client registers its validators with the beacon node's builder and accepts a blinded block when the builder's bid wins. It signs the blinded block and publishes it back to the node that produced it. Stacked on #704.

Scope:

  • Covered: fulu and earlier (electra blinded blocks are accepted for third-party beacon nodes).
  • Gloas is unchanged: produceBlockV4 keeps its self-build configuration, and registrations stop at the gloas fork. The gloas builder API comes in phase B.

Flags

Flag Behavior
--builder-proposals Off by default. Registers validators and sends builder_boost_factor with every pre-gloas block request
--builder-boost-factor <u64> Default 100. Requires --builder-proposals
--prefer-builder-proposals Sends u64::MAX, so the builder wins whenever there is a valid bid, unless the node's execution client sets shouldOverrideBuilder. Requires --builder-proposals; conflicts with --builder-boost-factor

Without --builder-proposals, nothing changes: the client sends builder_boost_factor=0 and refuses a blinded answer, as before.

What it adds

Piece Behavior
Registrations One ValidatorRegistrationV1 per validator the node resolved, signed under the builder domain (genesis fork version, zero validators root). The signature and timestamp are cached until the validator's fee recipient or gas limit changes, and a re-signed one is always newer than the last (max(now, last + 1)). The full set is sent after each epoch refresh and changed keys every slot, on spawned tasks with signing off the async runtime, in batches of 500 to every configured node, each node working through its own batches beside the others; a node that cannot be reached is sent no more batches that round. One send runs at a time, and each reads the cache only once the previous one was answered, so an older copy does not arrive after a newer one: a per-slot send that finds another running skips and a later slot sends the change, a full send waits for a per-slot one, and a full send that finds the last full one still running skips its epoch. A per-slot send no node accepts is sent again, as signed, every slot until a node takes it or the next full send ends the retry. Validators with no fee recipient are skipped with one warning, and a zero --suggested-fee-recipient is reported at error at startup
Blinded answer ProducedBlock gains a blinded variant. Its root is the full block's, so the signing path is unchanged, and its slot goes into the durable EIP-3076 slashing-protection record (#734) before it is signed, like a full block's. Slot and proposer are checked as for a full block. The fee-recipient check is skipped, since the header names the builder's address
Blinded publish POST /eth/v2/beacon/blinded_blocks, SSZ, only on the node that produced the block, with no failover: only that node's builder holds the payload, and only that node claimed the slot. The produced block carries its producer (Producer, filled in by the fallback), and the proposal path sends the signed block back to that node. A blinded answer is refused before signing when builder proposals are off. The request runs on a spawned task, so the proposal budget cannot cut it off
Unconfirmed publish A 5xx other than 503, or no answer once connected, is reported as "unconfirmed", not as a missed proposal: the relay may still publish the block. Nothing retries, since the slot is signed and a second block would be an equivocation. A 4xx, a 503 or a connection never made (refused, or past the client's 3 s connect timeout) is a definite failure, counted as a failed proposal. The budget is enforced inside ProposalService::propose_by, which returns a Proposed outcome: past the budget a signed blinded block comes back as StillPublishing, and whichever of the duty loop and the publish task ends up holding the outcome (decided by a oneshot channel) reports it
Metrics ethlambda_validator_builder_registrations_sent_total, ..._builder_registration_failures_total, ..._blinded_blocks_proposed_total, ..._blinded_blocks_unconfirmed_total

Known limitations

  • Per-validator builder toggles (Lighthouse's PATCH route) are out of scope; the flags apply to the whole process.
  • A full registration send that fails is not retried until the next epoch's. Only a per-slot send, which carries a keymanager change, is retried every slot until then.

Tests

  • Unit: the blinded decode and its root, the HTTP query and status mapping, producer-only publish in the fallback, the registration signature domain, the cache and batching, the gloas stop, the proposal path's outcomes, and the flags.
  • End to end through ethlambda's own beacon node and the mock relay:
    • registration → blinded produce → sign → publish → relay;
    • a full block when builder proposals are off;
    • a full block when the relay is down.

make lint passes, as do the validator lib, rpc lib and binary tests.

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR for the builder proposals feature in ethlambda. This is a substantial change adding MEV-Boost style builder block proposals to the validator client.

Security & Correctness

Critical: Blinded block publish isolation is correct but fragile

crates/validator/src/beacon_node/fallback.rs:294-317 — The publish_blinded_block implementation correctly pins blinded publishes to the producing node. However, the blinded_producer lock uses unwrap_or_else(std::sync::PoisonError::into_inner) which recovers from poisoned locks by clearing the error. This is dangerous: if a thread panicked while holding the lock, the Option<(Slot, usize)> could be in an inconsistent state. Given this is consensus-critical (prevents equivocation), consider whether panicking is safer.

Critical: u64::MAX as "prefer builder" sentinel

bin/ethlambda/src/validator.rs:95-100 — Using u64::MAX as the boost factor for --prefer-builder-proposals is correct per the spec, but the arithmetic local >= factor * bid / 100 could overflow if implemented naively on the node side. The PR doesn't show the node-side implementation, but verify that multiplication is checked or uses uint256.

Medium: Race between register_changed and register_all

crates/validator/src/registration.rs:335-341 — register_all stores signed_generation after current_blocking completes, but register_changed reads it before checking. If register_changed fires during register_all's current_blocking, it will see stale generation, re-sign everything, and send duplicates. The register_changed spawn in lib.rs:304 happens after register_all completes, so this is unlikely in practice, but the ordering is subtle.

Performance

Good: Chunked signing off async runtime

crates/validator/src/registration.rs:280-302 — SIGN_CHUNK = 64 with spawn_blocking properly avoids blocking the async runtime. The per-chunk store read lock is correct.

Concern: register_changed every slot even with no changes

crates/validator/src/lib.rs:368-370 — The spawn happens unconditionally off the slot loop. The generation check is fast (atomic read), but this still allocates a task per slot. Consider whether this is necessary when builder is None or when no validators are registered.

Rust Idioms & Code Quality

Good pattern: then_some for conditional construction

bin/ethlambda/src/validator.rs:93 — self.builder_proposals.then_some(...) is idiomatic.

Issue: std::sync::Mutex in async code

crates/validator/src/beacon_node/fallback.rs:46 — blinded_producer: std::sync::Mutex<Option<(Slot, usize)>> is held across await points in produce_block (line 137) and publish_blinded_block (line 311). This is technically fine since the critical sections are brief, but tokio::sync::Mutex would be more idiomatic for async contexts. The std::sync::Mutex is used elsewhere in this file for consistency with existing code.

Issue: unwrap_or_else(PoisonError::into_inner) pattern repeated

Multiple files (fallback.rs, proposal.rs, registration.rs) use this pattern. Consider a helper or parking_lot which doesn't poison.

Error Handling

Good: BlindedUnconfirmed distinction

crates/validator/src/error.rs:72-80 — Properly distinguishes "relay may have published" from definite failures. The comment correctly notes this is never retried.

Issue: task.await.unwrap_or_else in publish_blinded

crates/validator/src/proposal.rs:589-593 — If the spawned task panics, this returns Ok(BuilderUnconfirmed), which is correct (unknown state), but the panic info is lost. Consider logging the panic payload if available.

Consensus & Spec Compliance

Good: Fork-aware blinded block handling

crates/validator/src/beacon_node/block_contents.rs:348-369 — Explicitly rejects blinded blocks for forks that don't support them (Phase0 through Deneb, Gloas, Lean). The exhaustive match forces future fork additions to make a decision.

Question: Gloas builder market transition

crates/validator/src/registration.rs:257-260 — Registrations stop at Gloas, but the PR description says "from gloas the builder market works through proposer preferences instead." Is there a corresponding proposer_preferences implementation, or is this a TODO? The produceBlockV4 path in http.rs doesn't show preference handling.

Testing

Good: Comprehensive test coverage

The validator_client_tests.rs builder flow module tests end-to-end: registration, blinded production, publish, and failure modes. The fallback.rs tests verify no failover on blinded publish.

Missing test: Clock observation for blinded publish delay

crates/validator/src/proposal.rs:550-558 — The publication delay metric is only observed when slot_start is Some. No test verifies this path with a real SlotClock.

Documentation

Minor: Typo in CLAUDE.md

CLAUDE.md:855 — "pre-gloas slots register validators with the node's builder and may sign a blinded block, published to the producing node only" — "published" should be "publishable" or rephrase for clarity.

Minor: docs/cli.md inconsistency

docs/cli.md:296 — --builder-boost-factor default is 100, but bin/ethlambda/src/validator.rs:84 shows default_value = "100". The description says "0 prefers the local payload unless it cannot be built" but the CLI help says "0 prefers the local payload unless it cannot be built" — verify this matches node behavior.

Suggestions

  1. Line 137 of fallback.rs: Consider tokio::sync::Mutex for blinded_producer to avoid blocking the async runtime, or document why std::sync::Mutex is sufficient (hold time is brief).

  2. Line 589 of proposal.rs: Log the panic payload if the publish task panicked, not just the JoinError.

  3. Line 335-341 of registration.rs: Move signed_generation.store before self.send to reduce the window for the race with register_changed, or document why the current ordering is safe.

  4. General: The PoisonError::into_inner recovery pattern appears 10+ times. Consider whether parking_lot::Mutex (no poisoning) would simplify and be safer.

Summary

The implementation is correct and well-tested for the core security property: blinded blocks don't fail over to wrong nodes. The ProposalGuard integration prevents double-signing. The main concerns are around lock poisoning recovery in consensus-critical paths and a subtle race in registration caching. The code is idiomatic Rust with good async patterns for the signing workload.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

Looks good overall. The validator-side builder flow is careful about proposer-safety: the slot is recorded at signing time, blinded publishes are isolated onto a background task, and the timeout path no longer misclassifies a slow relay as a missed proposal.

  • No correctness or security bug jumped out in the changed paths around crates/validator/src/proposal.rs:515 and crates/validator/src/lib.rs:677.
  • The most important consensus property here—never re-signing a second block for the same slot even if publication overruns—is preserved by the guard plus signed_blinded_at, and the new tests cover that well in crates/validator/src/proposal.rs:978 and crates/validator/src/proposal.rs:889.

Minor nits only:

  • crates/validator/src/proposal.rs:544 logs Published::Imported / Published::BuilderUnconfirmed as unexpected for blinded publish, but still increments success counters for any non-BuilderUnconfirmed outcome at crates/validator/src/proposal.rs:552. If Imported is truly impossible on this path, consider treating it as an error instead of counting it as a successful blinded proposal; otherwise metrics can hide a beacon-node/API contract regression.
  • crates/validator/src/lib.rs:683 / crates/validator/src/lib.rs:692 key off signed_blinded_at(slot) after the proposal future resolves. That is fine with the current one-duty-per-slot model, but it couples loop behavior to shared mutable state. A slightly cleaner pattern would be to return an explicit “signed blinded” marker from ProposalService::propose so the caller does not have to re-infer path selection from side state.

If you want, I can also do a second pass on the beacon-node / RPC-side builder changes from the same PR for spec conformance and edge cases.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 705 (validator client: builder proposals and blinded publish)

I read the diff through the end of the HTTP client changes, including the CLI flags, the fallback node, http.rs, block_contents.rs, the rpc test refactor and the end-to-end tests. I did not read the rest of the diff, which likely holds the validator-service wiring, registration signing and caching, and metrics. I did not build the code or run the tests. The review below covers what I read.

The design is sound. The main safety properties hold:

  • No failover on blinded publish. publish_blinded_block goes only to the producing node, recorded by (slot, position) in FallbackBeaconNode. A second node cannot be handed a signed block it never produced.
  • No double-sign. Nothing retries after the slot is signed.
  • Stale claims are cleared. The record is overwritten on every successful produce_block, so a full block clears an earlier blinded claim.
  • Unconfirmed vs. definite failures. The split in publish_blinded_block (crates/validator/src/beacon_node/http.rs) is sensible. Connect errors and 503 are definite because the node cannot have acted on the block. Other 5xx errors, timeouts and resets are unconfirmed.
  • Blinded answers are only decoded when requested. A blinded answer is refused when builder_boost_factor is None, so a client with builder proposals off never signs a block it cannot publish.
  • Gloas and the other forks are refused by name. from_blinded_ssz has no _ arm, so a new fork forces a decision.
  • CLI rules are tested. The requires and conflicts_with rules are exercised by parsing real argument lists.

Minor points and questions

  1. 503 as "definite" (http.rs, publish_blinded_block). This holds for ethlambda and Lighthouse, where 503 means syncing before the builder is touched. A reverse proxy or load balancer can also return 503 after forwarding the request, and then the relay may still publish. The doc comment says "the node cannot have acted", which is only true for that assumption. Consider saying so in the doc, or treating a 503 as definite only when the body or headers match a sync refusal. The cost of the current choice is only a mislabelled metric, since nothing retries either way.

  2. Fallback record with produce_block failure (fallback.rs, produce_block). If every node fails, the early return via ? leaves the previous record in place. This is harmless because publish checks the slot. Writing None on failure as well would make the invariant "record reflects the latest produce" literal, but I do not think it is needed.

  3. register_validators timeout on the fallback path. REGISTRATION_TIMEOUT is 12 s per node, and try_all is sequential. With N unreachable nodes a registration round can take up to N × 12 s. The PR says this runs on spawned tasks, so it does not block duties. Please confirm that no per-slot "changed keys" send can queue behind a previous round indefinitely.

  4. Unused parameter. publish_blinded_block in http.rs takes _slot, which only the fallback layer uses. That is fine for the trait, but a one-line comment would stop someone from "fixing" it.

  5. Copy of the body. post_ssz_body calls body.to_vec(). A blinded block is small, so this does not matter. For full blocks with blobs it was already the existing behaviour, and the refactor does not make it worse.

  6. Boost factor semantics. The doc on --builder-boost-factor says the local payload wins iff local >= factor * bid / 100. With u64::MAX the node side must use widened arithmetic or special-case the value. That is in the prior PR, but it is worth confirming that feat(rpc): serve and publish builder blocks (fulu) #704 handles it without overflow.

Tests. Coverage is good. It includes status mapping, the connect-failure case, fallback routing and no-failover, a registration JSON round trip, and end-to-end tests against the real router with a mock relay. Moving the builder test world into proposal::test_support is a clean reuse.

Overall: LGTM on what I read. The points above are minor, with Point 1 the most worth a doc tweak.


Automated review by Claude (Anthropic) · sonnet · custom prompt

Registrations and builder bids are signed under builder-specs'
DOMAIN_APPLICATION_BUILDER, an application domain outside the
specification's own. The mask's doc said this crate never uses one;
it now does, so the doc names it.
A blinded block carries the header of a payload this node never sees,
and blinding a locally built block needs the same conversion. The
header also needs Deserialize now, since bids and blinded blocks
arrive as JSON.
Real encodings to decode against, so the builder containers are checked
against what relays and other clients send rather than only against
themselves.
Only the bid is a response; the registrations and the blinded block
are request bodies, and the `value` key is OpenAPI's example wrapper.
Record the full commit, since nothing else in the repo pins it. The
header's JSON round trip moves beside the other round-trip tests.
The builder API's domain and gloas's in-protocol builder domain share
a name and nothing else. Copying an existing get_domain call for the
builder API would sign under the wrong one, and the only symptom is a
relay rejecting every signature.
What a validator client signs and its beacon node forwards to relays
through registerValidator, checked against builder-specs' example in
both encodings.
The signed offer getHeader returns: a payload header, the blob
commitments and execution requests that go into the body with it, and
the value paid to the proposer.
The block a validator signs when a relay supplies the payload. Its
root equals the full block's because a payload header's root is the
payload's, which is what lets one signature serve both; the tests pin
that, and builder-specs' example pins the encoding.
The root-equality test used a mostly empty body, and empty lists with
the same limit share a root, so swapping two of them still passed under
the minimal preset. Build it from the fully populated JSON test block
instead. The example tests now also check the JSON this node would send
a relay, not only the JSON it reads.
Registrations and bids are signed over their own roots, relays reject
an older registration rather than requiring an unchanged one, and
from_parts copies a state root its caller must recompute.
A blinded block needs the same body, framing and process_block run as
a full one, then patches the post-state before taking its root. No
behavior change.
The state transition takes full payloads only. Rather than thread a
payload abstraction through it, run it on a stand-in payload built from
the builder's header and patch the two post-state fields that depend on
the payload itself. Safe because fulu, the last fork with blinded
blocks, is final; the guard test fails on any drift.
The stand-in payload is what makes process_block's parent hash,
prev_randao, timestamp and blob checks apply to a builder's header,
and only if it copies the right fields. One test per check, plus the
withdrawals root and the fork gate, which this path checks itself.
Both block paths pass a constant execution engine, so on the blinded
path no one checks the builder's header beyond the fields the stand-in
copies. That is right for the state root but easy to mistake for bid
validation, so the doc now says the caller owns the rest.
The guard's state had no withdrawal due and its body no execution
request, so the withdrawals root was only ever checked against the
empty list and the "identical" rows of the doc's table were argued
rather than tested. Six payload fields the stand-in leaves empty were
empty in the test payload too.
They were single lines past the width the rest of the file wraps at.
The stand-in-and-patch approach is unusual (every other client threads
a payload abstraction through its STF), so the next person to touch
fulu's process_block should find the reason before the guard test does.
fulu_state's doc named a field it deliberately does not cover and a
step that has since been written. assemble_blinded_block's list of
what the caller must still check was short, and A2 codes against it.
The rest are small accuracy fixes found reviewing the branch whole.
Registrations and bids are signed under DOMAIN_APPLICATION_BUILDER at
the genesis fork version and a zero validators root, the way deposits
are. The known-answer test pins mainnet's value, so a call that used
get_domain's current fork and real validators root instead would fail
here rather than at every relay.
@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

PR 705 review: validator-client builder proposals (blinded blocks and registrations)

This is a careful, well-tested PR. The core safety properties hold:

  • Slashing protection. The slot is recorded before signing. The blinded root equals the full root, and the test a_blinded_block_is_signed_over_the_full_blocks_root checks this.
  • No failover for blinded publish. A blinded block goes back only to the node that produced it.
  • Refused before signing. A blinded block is rejected before signing when builder proposals are off.
  • Correct registration domain. compute_builder_domain uses the genesis fork version and a zero validators root.

Most of what follows is verification items and design nits rather than bugs.

Correctness / things to verify

  1. lib.rs refresh_epoch: is entries the full resolved set every epoch?

    • register_all treats the returned pubkeys as authoritative. It calls signed.retain(...) and lapsed.retain(...), and overwrites known.
    • If refresh_epoch only looks up newly resolved validators after the first epoch, entries would be a delta. Each epoch would then purge the cache and the known set for every already-resolved validator, and stop registering them.
    • The new test only covers a first call. Please confirm, or add a second-epoch test asserting that already-resolved keys stay registered.
    • The same question applies to status freshness, so that an exited validator is actually dropped later.
  2. http.rs, a_blinded_publish_whose_connection_times_out_is_a_definite_failure.

    • The test depends on reqwest being built with a TLS backend.
    • Without one, https:// fails at build or request time with a non-connect error. The BeaconNode { failure: Timeout } assertion would then fail.
    • Worth confirming the crate's reqwest features, or gating the test on them.
  3. registration.rs register_changed: a change made before the first register_all is never marked done.

    • Example: a keymanager write arrives before the first refresh succeeds. Every slot then takes sending, clones an empty known, and returns without calling mark_signed.
    • This is harmless but needless work every slot.
    • Consider either leaving signed_generation untouched and returning before try_lock when known is empty, or doing the cheap generation check before tokio::spawn in lib.rs. The second option also avoids spawning a task every slot for the common no-op case.

Design / API

  1. Published::HandedToBuilder is shared between two calls that can never both produce it.

    • Adding it to the shared Published enum forces an "outcome that belongs to another call" warn arm in two places: ProposalService::publish and blinded_outcome.
    • A separate enum for the blinded publish (HandedToBuilder | BroadcastNotImported) would make both impossible states unrepresentable instead of logged.
  2. Producer is a raw position into the fallback's node list.

    • FallbackBeaconNode::publish_blinded_block forwards Producer::default() to the inner node, so nesting fallbacks would silently route to node 0.
    • That nesting isn't done today. A one-line doc note that Producer is only meaningful one level deep would help.
  3. ethlambda_types::text: a log-sanitizer sitting in the consensus types crate.

    • It's pragmatic, since both rpc and validator need it. A small common/util crate would be a better home if one ever appears.
    • Separately, rewrites_the_line skips other invisible format characters: U+200B–U+200D, U+2060–U+2064, U+FEFF. These don't break lines or restyle text, but they can hide content in a log line.
    • Optional: add them for completeness alongside the bidi set.
  4. fallback.rs try_all now fans out concurrently.

    • This affects prepare_beacon_proposer, subscribe_committees and subscribe_sync_committees too, not just registrations.
    • It's the right change, since a hung node no longer holds back the healthy ones. It is a behavior change outside the builder scope, though, and is worth calling out in the PR description.

Robustness

  1. proposal.rs publish_blinded: no graceful shutdown for the in-flight publish.

    • The spawned publish task is the right design.
    • However, docs/spec_deviations.md admits that SIGINT kills a signed-but-unsent blinded block.
    • A follow-up that tracks these tasks (JoinSet or TaskTracker) and awaits them on shutdown, bounded by the 8 s request timeout, would close that window cheaply.
  2. registration.rs newer_than can push the timestamp far into the future.

    • After a large backward clock step, the timestamp lands well ahead of the clock.
    • Your doc explains the trade-off: flashbots refuses a timestamp more than 10 s in the future and drops the whole batch.
    • A metric or warn when newer_than returns more than now_secs + 10 would make that failure mode visible rather than silent.

Nits

  • proposal.rs compares_fee_recipient: this is a one-line wrapper around !is_blinded(). Inlining it with the existing comment would read just as well.
  • http.rs read_error_body: the loop exits silently on a mid-stream error. This is fine for an error body, but a debug! would help when diagnosing truncated pool-submission 400s.
  • PR size: at roughly 3k lines, this PR mixes three changes: the text sanitizer refactor, the fallback concurrency change, and the builder flow. Splitting them would have eased review.

Overall: approve once item 1 is confirmed. The rest are optional follow-ups.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR is well-designed and thoroughly tested. The implementation of builder proposals and validator registrations in the validator client handles the subtle edge cases cleanly:

  • Blinded block routing & error classification (crates/validator/src/beacon_node/fallback.rs, http.rs, proposal.rs): Pinning publish_blinded_block to the producing node (Producer), distinguishing definite refusals/connect failures from BlindedUnconfirmed, and detaching the blinded publish onto its own task with oneshot::Receiver::close + try_recv handoff at the proposal deadline avoids both false failure counts and cancelled relay submissions.
  • Registration caching & concurrency (crates/validator/src/registration.rs, proposer_settings.rs, in_flight.rs): Serialization under sending, non-blocking per-slot checks via ProposerSettings::generation, monotonic timestamps in newer_than, tracking of unsent and lapsed keys across chunk failures or cleared settings, and concurrent per-node batching in FallbackBeaconNode are all solid.
  • Remote error text sanitization (crates/common/types/src/text.rs): one_line properly escapes control characters, line separators, and Unicode bidirectional controls while truncating on character and escape boundaries.

LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The PR is well-structured and heavily tested; the design around Producer, the InFlight guard, and the registration retry logic is particularly careful. A few issues remain:

  1. Full block publish can be cancelled after the request is sent — In crates/validator/src/proposal.rs, the before helper wraps publish_full with tokio::time::timeout_at, which drops the future on deadline expiry. For an HTTP publish, the request may have already been sent and the node may import the block, but the client returns ProposalDeadline and counts it as a missed proposal. The comment “a completed future is never discarded” is not guaranteed by Tokio’s timeout. Blinded blocks are handled correctly by spawning a detached task, but full blocks still have this race. Consider using a non-cancellable publish (e.g., spawn the request onto a separate task and await with select!/oneshot) or accept that deadline cancellation may incorrectly report failure.

  2. read_error_body does not drain the remaining response body — In crates/validator/src/beacon_node/http.rs, after reaching ERROR_BODY_READ_LIMIT, the function breaks without reading the rest of the response. This may prevent reqwest from reusing the connection, causing resource leaks or degraded performance when error responses are frequent. Continue reading and discarding chunks after the limit is hit.

  3. Registration cache is cleared when refresh_epoch returns an empty set — In crates/validator/src/registration.rs, register_all receives resolved from refresh_epoch. When the beacon node is syncing or optimistic, refresh_epoch returns Ok(Vec::new()) (see crates/validator/src/lib.rs), and register_all then retains only the empty set, clearing known, signed, and lapsed. This forces a full re-sign and re-registration after the node returns to sync, and leaves a gap. It would be safer to keep the previous known set unless the refresh produced a non-empty result.

  4. register_validators_in_batches assumptions about batch count — In crates/validator/src/beacon_node/fallback.rs, the fallback computes batches = registrations.len().div_ceil(REGISTRATION_BATCH) and then zips outcomes with that count. If a node’s implementation uses a different batch size or returns a different number of outcomes, the per-node warning may undercount refusals (missing batches are ignored in the loop), and the final error message for missing outcomes is correct but the logging is misleading. The trait contract should explicitly require exactly one outcome per REGISTRATION_BATCH chunk, and the fallback should perhaps log missing outcomes as refusals for that node.

  5. Minor performance/correctness note in post_ssz_body — In crates/validator/src/beacon_node/http.rs, body.to_vec() copies the entire body even when it already owns the bytes. For large full blocks (especially with blobs), this adds an unnecessary allocation. reqwest::Body::from(Vec<u8>) can take ownership directly, avoiding the copy. The same copy existed before, but now the helper centralizes it, making it a good place to fix.

  6. Potential fragility in one_line — In crates/common/types/src/text.rs, one_line uses ExactSizeIterator::len for the escaped length, which is a character count rather than bytes. It is correct because escape_default only produces ASCII, but this invariant is not documented in the code. A brief comment would prevent a future regression if the escaping logic changes.

No consensus-layer correctness issues were found in this PR; the changes are confined to the validator client and its interaction with builder APIs.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The builder-proposal path is carefully built: blinded blocks stay on the producing node, the slot is recorded before signing, and a blinded answer is refused before that record when proposals are off. A few gaps are still worth fixing.

  1. A future-dated registration can take down every validator in its batch. newer_than in registration.rs never moves a timestamp backward, and a clock step (or two changes in the same second) stores held + 1. The flashbots relay rejects the whole request if any timestamp is more than 10 seconds ahead, and this client does not split a 400. The poisoned signature stays cached, so every later full re-send includes it until wall clock catches up. Isolate a timestamp that is ahead of the clock (own batch, or do not cache it) so one key cannot block the other 499.

  2. Skipping an in-flight full send drops that epoch's validator set. register_all returns at full_send.enter() before updating known or signed. The in-flight send keeps the old pubkey list; keys imported or removed in the skipped epoch are invisible to register_changed until a later refresh actually sends. On mainnet a ~100s send fits in an epoch, but a node that answers 500 after its 10s relay wait, or a short-slot devnet, overlaps epochs. Keep the latest resolved set even when the send itself is skipped.

  3. Blinded-root equality is only tested for an empty block (a_blinded_block_decodes_and_keeps_the_full_blocks_root). Slashing protection and the relay both assume BlindedBeaconBlock's root is the full block root. A mismatch once transactions, withdrawals, blob commitments, or execution requests are present is an invalid signature or a same-slot equivocation. Add a non-empty case, including a fulu body if that container is shared with electra.

  4. A blinded publish is detached from process shutdown. After the signature, the slot is spent and the POST lives only on a spawned task (publish_blinded). SIGINT in the relay's 5s window drops that task; nothing retries. spec_deviations.md already says this. It is still a signed block that never leaves. A shutdown drain, even a short one, would close the hole.

Minor: an ethlambda node applies fee recipients in arrival order, ignoring timestamps. The sending lock orders this client's own requests, but a registration that outlives REGISTRATION_TIMEOUT (12s) can still land after a newer one and stick until the next epoch's full send. Worth a node-side "newest timestamp wins" fix; the client cannot close that race itself.

Log sanitization in text.rs (controls, U+2028/U+2029, bidi) and the connect-timeout split for blinded publishes (definite vs BlindedUnconfirmed) look right, and the fallback tests cover no-failover publish.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking findings. The security-sensitive paths look carefully handled:

  • Blinded blocks use the full block root and retain durable slashing protection before signing.
  • Blinded publishes are pinned to the producing node without unsafe failover or retries.
  • Registration signatures use the correct builder domain with a zero validators root.
  • Registration caching, generation tracking, retry behavior, and send ordering have strong concurrency tests.
  • Remote error bodies are bounded and sanitized before reaching logs.
  • Fallback batching handles missing or extra node outcomes without panicking.

The remaining trade-offs—heuristic 5xx classification, stale registration possibilities after partial fan-out, and documented reporting edges for dropped or panicked publish tasks—are explicit and do not introduce slashing risk. The accompanying documentation and test coverage are notably thorough.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

Claude (seventh review): a registration newer_than signs more than ten
seconds ahead of the clock (after a backward clock step) makes the flashbots
relay refuse its whole batch until the clock catches up, and nothing said
so. It is now warned at signing, naming the key, the timestamp and the
clock.

Also notes in one_line that an escape's char count is its byte count, since
escape_default writes ASCII only (DeepSeek; Grok in the sixth review).
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 705 — validator client builder proposals (blinded blocks and registrations)

This is a carefully built PR. The core consensus-safety properties are correct:

  • One signature per slot. The blinded root equals the full block's root, so slashing protection records the slot before signing, whichever way the node answers.
  • No failover after signing. A blinded publish goes back to its Producer only.
  • No retry of an unconfirmed send, so there is no equivocation path.
  • Blinded blocks are refused when builder proposals are off. This is checked both in HttpBeaconNode::produce_block and again in produce_and_sign.
  • The registration signing domain is correct: compute_builder_domain(genesis_fork_version) with a zero root. Tests verify it does not collide with DOMAIN_BEACON_BUILDER or with a context-derived domain.

The items below are mostly things to verify or minor improvements.

Correctness

  1. lib.rs refresh_epoch, the return value: confirm entries is the full resolved set every epoch. register_all treats the pubkeys it receives as authoritative:

    • It runs signed.retain(...) and lapsed.retain(...).
    • It overwrites known.

    If entries only covered validators resolved this epoch ("Resolve any newly activated validators…"), the second epoch would evict every previously cached registration. It would also shrink known, and per-slot changes for those keys would stop going out. The prepare_beacon_proposer assertion (preparations().len() == 6) suggests it is the full set. A test calling refresh_epoch twice and asserting the second returned list still has every key would lock this in.

  2. proposal.rs propose_within / before: deadline during the slashing-protection write. The comment says a deadline before signing can leave the slot recorded with nothing signed. Since the protection write runs on the blocking pool, that window is real. It is safe (the slot is forgone, never double-signed), and behaviour is unchanged from the old timeout(...) wrapper. It may still be worth a one-line mention in docs/cli.md, because it now applies to blinded proposals too.

  3. proposal.rs publish_blinded: the spawned task is untracked. On shutdown or SIGINT, a signed blinded block mid-publish is lost along with its reporting. This is documented in spec_deviations.md. If a graceful shutdown path is ever added, it should join these handles (e.g., via a JoinSet held by the service).

HTTP / error handling

  1. http.rs publish_blinded_block: the 8 s timeout. REQUEST_TIMEOUT (8 s) bounds the request against ethlambda's 5 s relay wait. Third-party beacon nodes may wait longer on submitBlindedBlock, and those will surface as BlindedUnconfirmed. That is the safe classification, but consider a dedicated, slightly longer timeout, as was done with REGISTRATION_TIMEOUT.

  2. http.rs read_error_body: a truncated UTF-8 sequence becomes U+FFFD. Cutting at exactly ERROR_BODY_READ_LIMIT can split a multi-byte character, which from_utf8_lossy replaces with U+FFFD. This is harmless for logs. For handle_pool_submission, a 1 MiB cut means the JSON fails to parse and falls back to the whole-batch error, which is fine.

  3. http.rs a_blinded_publish_whose_connection_times_out_is_a_definite_failure: depends on reqwest's TLS feature. Without TLS, an https:// URL errors immediately at build or send rather than at the handshake, and the assertion on BeaconNodeFailure::Timeout would fail confusingly. A comment, or a cfg on the TLS feature, would help.

Performance

  1. http.rs post_ssz_body: the body is copied twice. It does body.to_vec() on a slice the caller already owns as a Vec<u8> (SignedBlock.body, and the moved body in the blinded task). Taking Vec<u8> (or bytes::Bytes) through publish_block / publish_blinded_block would avoid copying a blob-laden full block, roughly 768 KiB with six blobs. This is minor, and the code it replaces did the same copy.

  2. lib.rs slot loop: register_changed is spawned every slot. The fast path is one atomic compare, but every slot still pays a spawn. Gating the spawn on settings.generation() != signed_generation would need that state exposed. This is optional; the cost is negligible.

Readability

  1. text.rs one_line. The kept_if_cut bookkeeping is subtle but correct. I traced the cap-below-marker case, an escape that lands exactly at the cap, and a backtrack over a whole escape. The tests cover each. Consider a short comment noting that kept_if_cut <= written.len() <= max_bytes, so max_bytes - kept_if_cut cannot underflow.

  2. fallback.rs register_validators_in_batches. It defends against a node returning the wrong number of outcomes (take(batches), with missing outcomes treated as not accepted). Good. registration.rs::send has the same defence.

Overall this looks good to merge, once Item 1 is confirmed.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings in the reviewed diff. Blinded-block signing preserves slashing protection, publication stays pinned to the producing node, and registration failures remain isolated from validator duties.

Tests could not run: the pinned Rust toolchain was unavailable, and the installed toolchain’s offline run lacked the libssz Git dependency.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks great. The builder proposal and validator registration flows are well-designed and thoroughly tested:

  • Blinded block handling & slashing protection (proposal.rs, block_contents.rs, fallback.rs, http.rs): Blinded blocks are validated (slot, proposer, fork, and builder_boost_factor presence) prior to recording in slashing protection and signing, pinned to the producing beacon node via Producer without failover, and published on a detached task with a race-free oneshot handoff (close + try_recv) so proposals running past the attestation deadline do not drop an in-flight blinded publish.
  • Validator registrations (registration.rs, proposer_settings.rs, signing.rs): Signatures are properly computed under DOMAIN_APPLICATION_BUILDER with a zeroed genesis validators root, filtered to pending/active statuses, cached with monotonic timestamps (newer_than), signed off the async runtime in bounded chunks, and serialized across full and per-slot sends so newer settings cannot be overwritten by an older in-flight batch.
  • Log sanitization (crates/common/types/src/text.rs): one_line cleanly escapes control characters, line/paragraph separators, and bidi overrides while truncating with ... on character and escape boundaries.

No issues found — LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The builder-proposal path is carefully structured. I am checking a few consensus-sensitive spots in the existing code before concluding.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

⚠️ Review failed: DeepSeek API request failed: curl exit code 56


Automated review by DeepSeek

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found in the normal FallbackBeaconNode<HttpBeaconNode> wiring. The slashing-protection flow, builder signing domain, producer-pinned blinded publish, bounded error-body handling, and no-retry behavior for unconfirmed publishes look sound and well tested.

One non-blocking API correctness issue:

  • Producer routing breaks with nested fallbacks — crates/validator/src/beacon_node/fallback.rs, in publish_blinded_block (around lines 315–318), delegates with Producer::default(). An outer fallback overwrites the block’s producer with its own position, and the inner fallback then publishes to its node 0 rather than necessarily the node that originally produced the block. The public generic API permits FallbackBeaconNode<FallbackBeaconNode<_>>, even though production does not currently construct it. Either document/forbid nested fallbacks, or preserve the inner producer when delegating. A nested-fallback routing test would cover this.

Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

Kimi (eighth review), Claude (seventh): the fallback hands its node
Producer::default(), so a fallback of fallbacks would send every blinded
block to its inner list's first node. The doc now says so; the client
builds one level only.
Threads serve_fulu_pinned's sync_pool and clock_ahead (074b459) through
serve_fulu_pinned_under, which main's #779 split out of it.
@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 17, 2026
Base automatically changed from feat/beacon-builder-blocks to main November 17, 2026 20:57
# Conflicts:
#	crates/net/rpc/src/beacon/builder_client/mod.rs
#	crates/net/rpc/src/beacon/builder_client/tests.rs
#	crates/net/rpc/src/beacon/proposal.rs
#	crates/net/rpc/src/beacon/validator_client_tests.rs
#	crates/validator/src/beacon_node/mod.rs
#	crates/validator/src/lib.rs
#	crates/validator/src/proposal.rs
@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit e437120 Nov 17, 2026
14 checks passed
@MegaRedHand
MegaRedHand deleted the feat/beacon-builder-vc branch November 17, 2026 22:19
MegaRedHand added a commit that referenced this pull request Nov 18, 2026
Brings the validator client work that landed on main (durable EIP-3076 slashing protection #734, its import and export #736, the review follow-ups #789, voluntary exits #779, builder blocks #705 and proposer preferences #707) along with the builder PRs they stack on (#694, #695, #696, #704), the generated node key (#726) and #787, #793.

Conflicts resolved by keeping both sides: the engine client's getBlobsV2 timeout next to main's one-attempt getPayload timeout, and their wire tests; nightly's batched block_signature_set next to main's verify_proposer_signature (the blinded block's check); gloas_proposal.rs importing both EventBus and the sync status.
MegaRedHand added a commit that referenced this pull request Nov 18, 2026
…s limit in validator_definitions.yml (#780)

The keymanager API's feerecipient, graffiti and gas_limit routes overrode
one validator's settings in memory only, so a restart put every validator
back on the command line's defaults. They are now written onto the
validator's entry in validator_definitions.yml, in Lighthouse's layout
(suggested_fee_recipient, graffiti, gas_limit), before memory changes, and
seeded from the file at startup. A re-import keeps an entry's values, an
import that enables a disabled entry loads and validates the values it
carries, and keystore and password files are flushed before the entry that
names them is saved.

Resolved against main, which this PR predates (slashing protection #734,
#736, #789 and the builder flow #705, #707):

- The definitions file is saved through main's write_private_durably, which
  already flushes the file and its directory, so the PR's own rename plus
  sync_dir is dropped. The settings routes save through save_definitions,
  keeping definitions_unflushed accurate, and treat Error::Unflushed as
  written (memory follows the file, with a warning), as the PR's logged
  directory flush did. The import seeds the settings in both the Ok and the
  Unflushed arm and returns main's Overwritten.
- write_private and write_private_no_symlink keep the PR's flush, the latter
  staying public for the node key.
- Overrides keeps main's registered() next to the PR's apply(), so seeding
  and every route still bump the generation the builder registrations
  cache on.
- A gas_limit of zero in the file is refused, as --gas-limit and POST
  gas_limit refuse it since #707 signs it into proposer preferences.
- Main's newer tests build ValidatorDefinition with the three fields as None.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants