Repository navigation
Open
Description
The validator client times every duty off the local wall clock and never checks that clock against anything. A clock that is off by a second or more makes duties early or late, and nothing the client logs or exports says why.
Code references are at bf66e86c.
What happens today
SlotClockis seeded once from the node's genesis time and/eth/v1/config/spec, then readsSystemTime::now()for the rest of the process (crates/validator/src/slot_clock.rs:1-6,slot_atat:96-101, built inrunatcrates/validator/src/lib.rs:253). The duty loop wakes on it at every slot boundary (lib.rs:381-382).GET /eth/v1/node/syncingis already called once an epoch (refresh_epoch,lib.rs:979).SyncingDtodeclares onlyis_syncing,is_optimisticandel_offline(crates/validator/src/beacon_node/dto.rs:137-156), so the node'shead_slotandsync_distancearrive and are dropped.HttpBeaconNode::getdecodes the body and discards the response headers,Dateincluded (crates/validator/src/beacon_node/http.rs:192-206).- A wall-clock step is absorbed without a log line. After a forward step
next_slot_to_serveserves the current slot at once, and after a backward step it moves on toserved + 1(slot_clock.rs:328-338). A clock stepped back by ten minutes makes the loop sleep ten minutes. ethlambda_validator_publication_delay_secondsis measured against the client's own clock (lib.rs:756-759), so it reads normal however far that clock is off.
Why it matters
Against an ethlambda beacon node:
- Clock ahead by more than about 500 ms: proposals are refused. The client asks for its block at its own slot boundary, after signing the RANDAO reveal (and, on an epoch's first slot, after the epoch refresh). The node refuses to produce a block for a slot that has not started by its clock, with
MAXIMUM_GOSSIP_CLOCK_DISPARITY(500 ms) of allowance:refuse_future_slot(crates/net/rpc/src/beacon/proposal.rs:465), called byproduceBlockV3(proposal.rs:552) andproduceBlockV4(crates/net/rpc/src/beacon/gloas_proposal.rs:264). The client never retries a proposal within its slot (crates/validator/src/proposal.rs:93-100), so the slot is lost. The log saysFailed to propose this slot's blockwith a 400 (lib.rs:938), and nothing points at the clock. - Ahead by a few seconds: attestations vote for the parent. They go out before the slot's block has arrived. Past the attester offset plus 500 ms (about 4.5 s before gloas, 3.5 s at gloas) the node refuses them as from the future (
crates/net/rpc/src/beacon/pool.rs:175-182), as every peer's gossip validation would. - Clock behind: nothing is refused. Blocks go out late, lose proposer boost and risk a late-block reorg. Attestations go out late and miss the head. Every series this client exports looks healthy. This is the direction an operator is least likely to notice.
When the client and its beacon node share a host they share the clock, so a comparison between the two only catches a split setup (separate machines or VMs). Catching a skewed host needs a reference outside it.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | Marks a beacon node unhealthy (CandidateError::TimeDiscrepancy) when the node's head_slot is more than one slot past the client's wall slot, and warns A connected beacon node errored during routine health check (validator_client/beacon_node_fallback/src/lib.rs:53, :314-318, :655-660). That catches a client clock behind by a slot or more. A clock that runs ahead only makes the node look further behind, which feeds node ranking (:319) |
unstable @ 11813cfb |
| Nimbus | The validator client used to measure its offset from the beacon node through Nimbus's own POST /nimbus/v1/timesync. status-im/nimbus-eth2#7406 ("VC: Remove timeoffset check function", 91581318, August 2025) removed the check without a stated reason. The node still serves the endpoint (beacon_chain/rpc/rest_nimbus_api.nim:454). The client does shut down when the wall clock moves back by more than one slot while it sleeps (beacon_chain/validator_client/common.nim:1511-1520) |
unstable @ 9bed158a |
Proposal
Two checks on calls the client already makes, plus a log line for clock steps:
- Slot-level comparison from
/node/syncing. Declarehead_slotandsync_distanceinSyncingDto.head_slotmore than one slot past the client's wall slot means the client's clock is behind (Lighthouse's check).head_slot + sync_distanceis the node's own wall slot whenever its head is not ahead of its clock. ethlambda's node computessync_distanceexactly that way (crates/net/rpc/src/beacon/node.rs:62-63). A difference from the client's wall slot, sampled on both sides of the request so a slot boundary inside the round trip is not counted, means the two clocks disagree by most of a slot or more.
- Second-level comparison from
Date. Read theDateheader ingetandpostand estimate the offset asDateminus the midpoint of send and receive. The Beacon API does not require the header, so a node that omits it gets only the slot-level check. The header has one-second resolution, so the estimate is good to about ±0.5 s plus half the round trip, which is enough to flag a skew of 2 s or more after a few samples. - Clock steps. In the duty loop, warn when the wall clock lands before the slot just served (a backward step) or more than one slot past it (a forward step, a suspended process or a paused VM).
- Surface it. A warning that names the offset and the node, and a gauge (
ethlambda_validator_beacon_node_clock_offset_seconds, plus a counter of detected steps), so it can be alerted on.
To decide
- Resolution. About 500 ms already gets proposals refused by an ethlambda node, and that is below what
Date(1 s) or a slot comparison can see. Is a 1-2 s warning enough, or is a finer source worth it: a timesync-style endpoint on ethlambda's own node, as Nimbus had, or an NTP query? - Which side is wrong. A client-vs-node difference does not say which clock is off. With several
--beacon-nodes, the majority can. - Warn only, or refuse. A warning seems right, since the node's clock may be the wrong one. Marking a node unhealthy for failover, as Lighthouse does, is a further step.
- A shared clock. Catching a skewed host where both processes run needs a signal from the chain, for example
headevents arriving consistently before the slot starts or several seconds after it. The client deliberately does not drive duties from the event stream (slot_clock.rs:3-6); reading it for this diagnostic alone would be new. - The beacon node side. A node with a skewed clock sees every gossip block as early or late. A node-side check is a separate issue.
Tests
MockBeaconNodeanswering/node/syncingwith ahead_slottwo slots past the client's clock: the refresh logs the discrepancy and sets the gauge. One slot past: no warning.- An offset estimator fed responses with a
Dateheader two seconds ahead and behind. next_slot_to_serve(or the loop around it) reporting a backward step and a multi-slot forward step.
Metadata
Metadata
Assignees
Labels
No labels
Activity