Skip to content

validator: check the beacon node's sync status before each slot's duties, not once an epoch #828

Open

Description

@MegaRedHand

The validator client asks its beacon node whether it is syncing or optimistic in one place: the top of refresh_epoch. Between refreshes, every duty relies on the beacon node refusing the request with a 503. The trait's own doc calls that 503 "a backstop, not the check: it puts correctness entirely in the node's hands for a rule the client is the one bound by" (crates/validator/src/beacon_node/mod.rs:393-397).

This issue is about the client only. The beacon node's own gate reading a catch-up as synced is #559, with a fix in #791. A client-side check reads that same status through /eth/v1/node/syncing, so it does nothing for #559. It adds two things: protection against a beacon node, ethlambda or not, that answers a duty endpoint while syncing or optimistic, and a check where the Beacon API does not gate an endpoint at all.

Code references are at main @ 5d8628e9.

What happens today

  • When the check runs. refresh_epoch checks first thing (crates/validator/src/lib.rs:1005-1007). It runs at the first slot of each epoch. last_epoch is set only when a refresh succeeds (lib.rs:427), so a failing refresh, and its check, runs again at every slot until one succeeds. After that, nothing in the client asks until the next epoch.
  • What a failed check stops. Only the refresh. BeaconNodeSyncing is retryable (error.rs:231-242). The loop logs "Failed to refresh duties; keeping the previous schedule" (lib.rs:436-439). It then goes on to propose (lib.rs:459-461) and serve_slot_duties (lib.rs:522-537) with whatever it holds:
    • Proposer duties for the new epoch are not held, since they are not prefetched (duties.rs:13-16), so proposals stop.
    • The attester and PTC schedules for the new epoch were fetched as the previous epoch's lookahead (duties.rs:363-373, lib.rs:1258-1269), so attestations and PTC votes go ahead.
    • The period's sync committee duties are held, so sync committee messages go ahead.
  • With several beacon nodes, FallbackBeaconNode::is_optimistic_or_syncing answers "usable" as soon as any node is (beacon_node/fallback.rs:288-323). Every duty request then starts again at the first node in the list (fallback.rs:111-140). The node that passed the check need not be the node that serves the duty.

Between refreshes, per duty:

Duty The client's own check What it relies on instead
Block proposal None (proposal.rs) beacon-APIs lists a 503 for a syncing node on produceBlockV3 and V4 (block.v3.yaml:137, block.v4.yaml:183). It says nothing about an optimistic head
Attestation and aggregate None (attestation.rs:119-123, aggregation.rs) A 503 on attestation_data and aggregate_attestation when syncing, or when the block is optimistic (attestation_data.yaml:15-16, aggregate_attestation.v2.yaml:7-8)
Sync committee message Optimistic only, every slot: execution_optimistic on GET /eth/v1/beacon/blocks/head/root becomes BeaconNodeSyncing (beacon_node/http.rs:1230-1238, sync_committee.rs:123-130) Nothing for a syncing node. beacon-APIs defines no 503 for blocks/{block_id}/root (apis/beacon/blocks/root.yaml). A syncing node that is not optimistic returns its stale head, and the client signs it
Sync contribution None of its own The message half above, and a 503 on sync_committee_contribution for an optimistic block (sync_committee_contribution.yaml:10-11)
PTC vote (gloas) None (payload_attestation.rs:82-187) A 503 on payload_attestation_data while syncing (payload_attestation_data.yaml:13). Nothing for an optimistic head

The beacon-APIs paths are at ethereum/beacon-APIs master @ 6bc8c7bd.

Why it matters

  • Optimistic heads. The optimistic sync spec puts the rule on the validator: an optimistic validator "MUST NOT produce a block", "MUST NOT participate in attestation" and "MUST NOT participate in sync committees" (specs/bellatrix/optimistic-sync.md:316-329, consensus-specs master @ 2ea9d96a). A node whose execution client falls behind mid-epoch goes optimistic. Until the next epoch, only the sync committee message path in this client notices. Every other duty is signed if the node answers.
  • Syncing nodes. A node that answers while catching up hands the client data from a stale head.
    • An attestation on it votes for the wrong head and target. Slashing protection has then recorded the target epoch, so that validator cannot vote again in the epoch.
    • A block built on it is orphaned, and its slot is spent in slashing protection.
    • A sync committee message signs a root that the next block's sync aggregate will not count. This one gets through even from a node that follows the Beacon API to the letter, because the API does not gate the head root.
  • Several nodes. A healthy second node makes the check pass while the first node, syncing, still answers first. Whether that node then refuses each duty depends on its implementation.

What other clients do

Client The client's own sync or optimistic check Read at
Lighthouse Calls /eth/v1/node/syncing on every node each slot, 2 s before the slot starts. It ranks the nodes by sync distance, optimistic status and el_offline, and requests try them in that order (beacon_node_fallback/src/lib.rs:47-49, :140-156). Sync committee messages need a head root with execution_optimistic == false (sync_committee_service.rs:258-266). The early-attestation trigger skips optimistic head events (beacon_head_monitor.rs:194-200). No other duty is gated in the client unstable @ 11813cfb
Teku Runs a readiness check against /eth/v1/node/syncing once a slot, and only when failover nodes are configured. An optimistic node still counts as ready, and readiness only steers routing (BeaconNodeReadinessManager.java:120-130, :190-199). No per-duty gate master @ 62a54b96
Prysm Checks /eth/v1/node/health every slot and stops the duty runner on anything but 200, which includes 206 for a syncing node (health_monitor.go:104-112, beacon-api/beacon_api_node_client.go:112-121, runner.go:96-101). Aggregation selection refuses a syncing or optimistic node (beacon-api/beacon_api_helpers.go:137). Sync committee messages and contributions refuse an optimistic head root (beacon-api/sync_committee.go:47) develop @ 0c1e3d24
Nimbus Calls /eth/v1/node/syncing every 2 s on nodes not marked synced. It classifies each node as synced, optimistically synced or not synced (fallback_service.nim:182, :286, :321), but requests may still go to the latter two (api.nim:66). Sync committee messages skip an optimistic head root (sync_committee_service.nim:389). Its in-node validator skips every duty while the node is not synced or its head is not execution-valid (beacon_validators.nim:1652-1662, under beacon_chain/validators/) unstable @ 9bed158a
Lodestar Calls /eth/v1/node/syncing every slot (syncingStatusTracker.ts:20-27). All sync committee duties are skipped while the node is optimistic (syncCommittee.ts:70-75). No gate on attestations, aggregates, proposals or PTC votes unstable @ 77088eb7
Vouch None of its own: nothing in Vouch's code reads the sync or optimistic status. Choosing among several nodes is left to its go-eth2-client library, not read here master @ 691f3652
Vero Never calls /node/syncing. Instead it rejects every duty response and head root that carries execution_optimistic: true (providers/beacon_node.py:256-262, used at :408, :452, :469, :487, :506), and drops optimistic head events. Attestation data, aggregates, block production and PTC data are not gated master @ 9d110c1d

File names are relative to each client's validator client directory.

No client has a hard per-duty gate for every duty. Three things are common, though:

  • polling the sync status every slot (Lighthouse, Prysm, Lodestar, and Nimbus every 2 s);
  • steering requests away from a node that reports syncing (Lighthouse, Teku, Nimbus) or stopping duties outright (Prysm);
  • refusing sync committee work on an optimistic head (Lighthouse, Prysm, Nimbus, Lodestar, Vero).

ethlambda already does the last.

Proposal

  • Check every slot, before the slot's first duty. One GET /eth/v1/node/syncing per slot per node is a small request.
    • A slot whose check says syncing or optimistic signs nothing: no block, attestation, aggregate, sync committee message, contribution or PTC vote.
    • A failed refresh then stops attestations from the held schedule too, instead of leaving them to the node's 503.
    • To keep the check off the block's path, run it ahead of the slot boundary, as Lighthouse does 2 s before the slot.
  • Serve each duty from a node that passed. Have the fallback try first the nodes whose last check said synced and not optimistic, instead of the configured order alone. The check and the request then agree. The fallback's module doc already names Lighthouse's health-ranked selection as the refinement (fallback.rs:1-5). The ranking itself is validator: rank beacon nodes by health and broadcast signed messages #830; this issue needs only the synced and optimistic part of it.
  • Close the sync committee gap. Add the syncing half to the head root fetch, or take it from the per-slot check.
  • Keep the 503 handling. It stays the backstop the trait doc describes.

To decide

  • Skip, or reorder only. Prysm stops every duty when its node is unhealthy. Lighthouse, Teku and Nimbus only prefer healthier nodes and still fall back to the rest. With a single node the two are the same question: sign nothing in the slot, or ask anyway and let the 503 decide?
  • Per slot or per duty. A slot's aggregate, contribution and PTC vote come seconds after its first duty. A check per duty puts a request in front of each, and on the proposal path that comes out of the block's budget. A check per slot leaves the rest of the slot to the 503s.
  • PTC votes on an optimistic head. The optimistic sync spec does not name DOMAIN_PTC_ATTESTER, and beacon-APIs asks for a 503 only while syncing. ethlambda's own node refuses PTC data for an optimistic block anyway (crates/net/rpc/src/beacon/ptc.rs:247-251). Should the client withhold the vote itself?
  • el_offline. The client logs it and does not act on it (http.rs:648-658), on the reasoning that the node turns optimistic soon enough. Lighthouse ranks nodes on it. Keep it as a log line?
  • No clock-based rule. The client must not skip duties because the head looks old by its own clock. During a network-wide stall every head is old, and clients that stop signing then keep the chain stalled. The signal is the node's is_syncing, and fix(sync): judge the sync gate against peers' Status heads #791 is what makes it right on an ethlambda node.

Tests

  • MockBeaconNode already has syncing and optimistic flags (beacon_node/mock.rs:61-65). Flip optimistic after a successful refresh, and assert that the next slot signs no block, attestation, sync committee message or PTC vote.
  • With two mock nodes, the first syncing but answering every endpoint and the second synced, assert every duty is served from the second.
  • With a syncing node that is not optimistic, assert no sync committee message is signed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions