Repository navigation
Description
The validator client asks its beacon node whether it is syncing or optimistic in one place: the top of refresh_epoch. Between refreshes, every duty relies on the beacon node refusing the request with a 503. The trait's own doc calls that 503 "a backstop, not the check: it puts correctness entirely in the node's hands for a rule the client is the one bound by" (crates/validator/src/beacon_node/mod.rs:393-397).
This issue is about the client only. The beacon node's own gate reading a catch-up as synced is #559, with a fix in #791. A client-side check reads that same status through /eth/v1/node/syncing, so it does nothing for #559. It adds two things: protection against a beacon node, ethlambda or not, that answers a duty endpoint while syncing or optimistic, and a check where the Beacon API does not gate an endpoint at all.
Code references are at main @ 5d8628e9.
What happens today
- When the check runs.
refresh_epochchecks first thing (crates/validator/src/lib.rs:1005-1007). It runs at the first slot of each epoch.last_epochis set only when a refresh succeeds (lib.rs:427), so a failing refresh, and its check, runs again at every slot until one succeeds. After that, nothing in the client asks until the next epoch. - What a failed check stops. Only the refresh.
BeaconNodeSyncingis retryable (error.rs:231-242). The loop logs "Failed to refresh duties; keeping the previous schedule" (lib.rs:436-439). It then goes on topropose(lib.rs:459-461) andserve_slot_duties(lib.rs:522-537) with whatever it holds:- Proposer duties for the new epoch are not held, since they are not prefetched (
duties.rs:13-16), so proposals stop. - The attester and PTC schedules for the new epoch were fetched as the previous epoch's lookahead (
duties.rs:363-373,lib.rs:1258-1269), so attestations and PTC votes go ahead. - The period's sync committee duties are held, so sync committee messages go ahead.
- Proposer duties for the new epoch are not held, since they are not prefetched (
- With several beacon nodes,
FallbackBeaconNode::is_optimistic_or_syncinganswers "usable" as soon as any node is (beacon_node/fallback.rs:288-323). Every duty request then starts again at the first node in the list (fallback.rs:111-140). The node that passed the check need not be the node that serves the duty.
Between refreshes, per duty:
| Duty | The client's own check | What it relies on instead |
|---|---|---|
| Block proposal | None (proposal.rs) |
beacon-APIs lists a 503 for a syncing node on produceBlockV3 and V4 (block.v3.yaml:137, block.v4.yaml:183). It says nothing about an optimistic head |
| Attestation and aggregate | None (attestation.rs:119-123, aggregation.rs) |
A 503 on attestation_data and aggregate_attestation when syncing, or when the block is optimistic (attestation_data.yaml:15-16, aggregate_attestation.v2.yaml:7-8) |
| Sync committee message | Optimistic only, every slot: execution_optimistic on GET /eth/v1/beacon/blocks/head/root becomes BeaconNodeSyncing (beacon_node/http.rs:1230-1238, sync_committee.rs:123-130) |
Nothing for a syncing node. beacon-APIs defines no 503 for blocks/{block_id}/root (apis/beacon/blocks/root.yaml). A syncing node that is not optimistic returns its stale head, and the client signs it |
| Sync contribution | None of its own | The message half above, and a 503 on sync_committee_contribution for an optimistic block (sync_committee_contribution.yaml:10-11) |
| PTC vote (gloas) | None (payload_attestation.rs:82-187) |
A 503 on payload_attestation_data while syncing (payload_attestation_data.yaml:13). Nothing for an optimistic head |
The beacon-APIs paths are at ethereum/beacon-APIs master @ 6bc8c7bd.
Why it matters
- Optimistic heads. The optimistic sync spec puts the rule on the validator: an optimistic validator "MUST NOT produce a block", "MUST NOT participate in attestation" and "MUST NOT participate in sync committees" (
specs/bellatrix/optimistic-sync.md:316-329,consensus-specsmaster@2ea9d96a). A node whose execution client falls behind mid-epoch goes optimistic. Until the next epoch, only the sync committee message path in this client notices. Every other duty is signed if the node answers. - Syncing nodes. A node that answers while catching up hands the client data from a stale head.
- An attestation on it votes for the wrong head and target. Slashing protection has then recorded the target epoch, so that validator cannot vote again in the epoch.
- A block built on it is orphaned, and its slot is spent in slashing protection.
- A sync committee message signs a root that the next block's sync aggregate will not count. This one gets through even from a node that follows the Beacon API to the letter, because the API does not gate the head root.
- Several nodes. A healthy second node makes the check pass while the first node, syncing, still answers first. Whether that node then refuses each duty depends on its implementation.
What other clients do
| Client | The client's own sync or optimistic check | Read at |
|---|---|---|
| Lighthouse | Calls /eth/v1/node/syncing on every node each slot, 2 s before the slot starts. It ranks the nodes by sync distance, optimistic status and el_offline, and requests try them in that order (beacon_node_fallback/src/lib.rs:47-49, :140-156). Sync committee messages need a head root with execution_optimistic == false (sync_committee_service.rs:258-266). The early-attestation trigger skips optimistic head events (beacon_head_monitor.rs:194-200). No other duty is gated in the client |
unstable @ 11813cfb |
| Teku | Runs a readiness check against /eth/v1/node/syncing once a slot, and only when failover nodes are configured. An optimistic node still counts as ready, and readiness only steers routing (BeaconNodeReadinessManager.java:120-130, :190-199). No per-duty gate |
master @ 62a54b96 |
| Prysm | Checks /eth/v1/node/health every slot and stops the duty runner on anything but 200, which includes 206 for a syncing node (health_monitor.go:104-112, beacon-api/beacon_api_node_client.go:112-121, runner.go:96-101). Aggregation selection refuses a syncing or optimistic node (beacon-api/beacon_api_helpers.go:137). Sync committee messages and contributions refuse an optimistic head root (beacon-api/sync_committee.go:47) |
develop @ 0c1e3d24 |
| Nimbus | Calls /eth/v1/node/syncing every 2 s on nodes not marked synced. It classifies each node as synced, optimistically synced or not synced (fallback_service.nim:182, :286, :321), but requests may still go to the latter two (api.nim:66). Sync committee messages skip an optimistic head root (sync_committee_service.nim:389). Its in-node validator skips every duty while the node is not synced or its head is not execution-valid (beacon_validators.nim:1652-1662, under beacon_chain/validators/) |
unstable @ 9bed158a |
| Lodestar | Calls /eth/v1/node/syncing every slot (syncingStatusTracker.ts:20-27). All sync committee duties are skipped while the node is optimistic (syncCommittee.ts:70-75). No gate on attestations, aggregates, proposals or PTC votes |
unstable @ 77088eb7 |
| Vouch | None of its own: nothing in Vouch's code reads the sync or optimistic status. Choosing among several nodes is left to its go-eth2-client library, not read here | master @ 691f3652 |
| Vero | Never calls /node/syncing. Instead it rejects every duty response and head root that carries execution_optimistic: true (providers/beacon_node.py:256-262, used at :408, :452, :469, :487, :506), and drops optimistic head events. Attestation data, aggregates, block production and PTC data are not gated |
master @ 9d110c1d |
File names are relative to each client's validator client directory.
No client has a hard per-duty gate for every duty. Three things are common, though:
- polling the sync status every slot (Lighthouse, Prysm, Lodestar, and Nimbus every 2 s);
- steering requests away from a node that reports syncing (Lighthouse, Teku, Nimbus) or stopping duties outright (Prysm);
- refusing sync committee work on an optimistic head (Lighthouse, Prysm, Nimbus, Lodestar, Vero).
ethlambda already does the last.
Proposal
- Check every slot, before the slot's first duty. One
GET /eth/v1/node/syncingper slot per node is a small request.- A slot whose check says syncing or optimistic signs nothing: no block, attestation, aggregate, sync committee message, contribution or PTC vote.
- A failed refresh then stops attestations from the held schedule too, instead of leaving them to the node's 503.
- To keep the check off the block's path, run it ahead of the slot boundary, as Lighthouse does 2 s before the slot.
- Serve each duty from a node that passed. Have the fallback try first the nodes whose last check said synced and not optimistic, instead of the configured order alone. The check and the request then agree. The fallback's module doc already names Lighthouse's health-ranked selection as the refinement (
fallback.rs:1-5). The ranking itself is validator: rank beacon nodes by health and broadcast signed messages #830; this issue needs only the synced and optimistic part of it. - Close the sync committee gap. Add the syncing half to the head root fetch, or take it from the per-slot check.
- Keep the 503 handling. It stays the backstop the trait doc describes.
To decide
- Skip, or reorder only. Prysm stops every duty when its node is unhealthy. Lighthouse, Teku and Nimbus only prefer healthier nodes and still fall back to the rest. With a single node the two are the same question: sign nothing in the slot, or ask anyway and let the 503 decide?
- Per slot or per duty. A slot's aggregate, contribution and PTC vote come seconds after its first duty. A check per duty puts a request in front of each, and on the proposal path that comes out of the block's budget. A check per slot leaves the rest of the slot to the 503s.
- PTC votes on an optimistic head. The optimistic sync spec does not name
DOMAIN_PTC_ATTESTER, and beacon-APIs asks for a 503 only while syncing. ethlambda's own node refuses PTC data for an optimistic block anyway (crates/net/rpc/src/beacon/ptc.rs:247-251). Should the client withhold the vote itself? el_offline. The client logs it and does not act on it (http.rs:648-658), on the reasoning that the node turns optimistic soon enough. Lighthouse ranks nodes on it. Keep it as a log line?- No clock-based rule. The client must not skip duties because the head looks old by its own clock. During a network-wide stall every head is old, and clients that stop signing then keep the chain stalled. The signal is the node's
is_syncing, and fix(sync): judge the sync gate against peers' Status heads #791 is what makes it right on an ethlambda node.
Tests
MockBeaconNodealready hassyncingandoptimisticflags (beacon_node/mock.rs:61-65). Flipoptimisticafter a successful refresh, and assert that the next slot signs no block, attestation, sync committee message or PTC vote.- With two mock nodes, the first syncing but answering every endpoint and the second synced, assert every duty is served from the second.
- With a syncing node that is not optimistic, assert no sync committee message is signed.
Activity