Skip to content

validator: warn when the local clock disagrees with the beacon node #834

Open

Description

@MegaRedHand

The validator client times every duty off the local wall clock and never checks that clock against anything. A clock that is off by a second or more makes duties early or late, and nothing the client logs or exports says why.

Code references are at bf66e86c.

What happens today

  • SlotClock is seeded once from the node's genesis time and /eth/v1/config/spec, then reads SystemTime::now() for the rest of the process (crates/validator/src/slot_clock.rs:1-6, slot_at at :96-101, built in run at crates/validator/src/lib.rs:253). The duty loop wakes on it at every slot boundary (lib.rs:381-382).
  • GET /eth/v1/node/syncing is already called once an epoch (refresh_epoch, lib.rs:979). SyncingDto declares only is_syncing, is_optimistic and el_offline (crates/validator/src/beacon_node/dto.rs:137-156), so the node's head_slot and sync_distance arrive and are dropped.
  • HttpBeaconNode::get decodes the body and discards the response headers, Date included (crates/validator/src/beacon_node/http.rs:192-206).
  • A wall-clock step is absorbed without a log line. After a forward step next_slot_to_serve serves the current slot at once, and after a backward step it moves on to served + 1 (slot_clock.rs:328-338). A clock stepped back by ten minutes makes the loop sleep ten minutes.
  • ethlambda_validator_publication_delay_seconds is measured against the client's own clock (lib.rs:756-759), so it reads normal however far that clock is off.

Why it matters

Against an ethlambda beacon node:

  • Clock ahead by more than about 500 ms: proposals are refused. The client asks for its block at its own slot boundary, after signing the RANDAO reveal (and, on an epoch's first slot, after the epoch refresh). The node refuses to produce a block for a slot that has not started by its clock, with MAXIMUM_GOSSIP_CLOCK_DISPARITY (500 ms) of allowance: refuse_future_slot (crates/net/rpc/src/beacon/proposal.rs:465), called by produceBlockV3 (proposal.rs:552) and produceBlockV4 (crates/net/rpc/src/beacon/gloas_proposal.rs:264). The client never retries a proposal within its slot (crates/validator/src/proposal.rs:93-100), so the slot is lost. The log says Failed to propose this slot's block with a 400 (lib.rs:938), and nothing points at the clock.
  • Ahead by a few seconds: attestations vote for the parent. They go out before the slot's block has arrived. Past the attester offset plus 500 ms (about 4.5 s before gloas, 3.5 s at gloas) the node refuses them as from the future (crates/net/rpc/src/beacon/pool.rs:175-182), as every peer's gossip validation would.
  • Clock behind: nothing is refused. Blocks go out late, lose proposer boost and risk a late-block reorg. Attestations go out late and miss the head. Every series this client exports looks healthy. This is the direction an operator is least likely to notice.

When the client and its beacon node share a host they share the clock, so a comparison between the two only catches a split setup (separate machines or VMs). Catching a skewed host needs a reference outside it.

What other clients do

Client Behavior Read at
Lighthouse Marks a beacon node unhealthy (CandidateError::TimeDiscrepancy) when the node's head_slot is more than one slot past the client's wall slot, and warns A connected beacon node errored during routine health check (validator_client/beacon_node_fallback/src/lib.rs:53, :314-318, :655-660). That catches a client clock behind by a slot or more. A clock that runs ahead only makes the node look further behind, which feeds node ranking (:319) unstable @ 11813cfb
Nimbus The validator client used to measure its offset from the beacon node through Nimbus's own POST /nimbus/v1/timesync. status-im/nimbus-eth2#7406 ("VC: Remove timeoffset check function", 91581318, August 2025) removed the check without a stated reason. The node still serves the endpoint (beacon_chain/rpc/rest_nimbus_api.nim:454). The client does shut down when the wall clock moves back by more than one slot while it sleeps (beacon_chain/validator_client/common.nim:1511-1520) unstable @ 9bed158a

Proposal

Two checks on calls the client already makes, plus a log line for clock steps:

  • Slot-level comparison from /node/syncing. Declare head_slot and sync_distance in SyncingDto.
    • head_slot more than one slot past the client's wall slot means the client's clock is behind (Lighthouse's check).
    • head_slot + sync_distance is the node's own wall slot whenever its head is not ahead of its clock. ethlambda's node computes sync_distance exactly that way (crates/net/rpc/src/beacon/node.rs:62-63). A difference from the client's wall slot, sampled on both sides of the request so a slot boundary inside the round trip is not counted, means the two clocks disagree by most of a slot or more.
  • Second-level comparison from Date. Read the Date header in get and post and estimate the offset as Date minus the midpoint of send and receive. The Beacon API does not require the header, so a node that omits it gets only the slot-level check. The header has one-second resolution, so the estimate is good to about ±0.5 s plus half the round trip, which is enough to flag a skew of 2 s or more after a few samples.
  • Clock steps. In the duty loop, warn when the wall clock lands before the slot just served (a backward step) or more than one slot past it (a forward step, a suspended process or a paused VM).
  • Surface it. A warning that names the offset and the node, and a gauge (ethlambda_validator_beacon_node_clock_offset_seconds, plus a counter of detected steps), so it can be alerted on.

To decide

  • Resolution. About 500 ms already gets proposals refused by an ethlambda node, and that is below what Date (1 s) or a slot comparison can see. Is a 1-2 s warning enough, or is a finer source worth it: a timesync-style endpoint on ethlambda's own node, as Nimbus had, or an NTP query?
  • Which side is wrong. A client-vs-node difference does not say which clock is off. With several --beacon-nodes, the majority can.
  • Warn only, or refuse. A warning seems right, since the node's clock may be the wrong one. Marking a node unhealthy for failover, as Lighthouse does, is a further step.
  • A shared clock. Catching a skewed host where both processes run needs a signal from the chain, for example head events arriving consistently before the slot starts or several seconds after it. The client deliberately does not drive duties from the event stream (slot_clock.rs:3-6); reading it for this diagnostic alone would be new.
  • The beacon node side. A node with a skewed clock sees every gossip block as early or late. A node-side check is a separate issue.

Tests

  • MockBeaconNode answering /node/syncing with a head_slot two slots past the client's clock: the refresh logs the discrepancy and sets the gauge. One slot past: no warning.
  • An offset estimator fed responses with a Date header two seconds ahead and behind.
  • next_slot_to_serve (or the loop around it) reporting a backward step and a multi-slot forward step.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions