Skip to content

validator: per-validator performance tracking #837

Open

Description

@MegaRedHand

Nothing in this client, or in an ethlambda beacon node, reports how each validator is doing on chain: whether its attestations were included with the right head, target and source and how late, whether its proposals became canonical, or whether its sync committee messages made it into blocks. Every other client has this somewhere. Most of them put it in the beacon node.

Code references are at bf66e86c.

What happens today

  • Validator client metrics (crates/validator/src/metrics.rs) are process-wide counters and gauges with no labels. They count what the client sent and what failed, per duty kind: ethlambda_validator_attestations_published_total (:38), ethlambda_validator_blocks_proposed_total (:94), ethlambda_validator_sync_committee_messages_published_total (:384), and the matching failure, refusal and deadline series. They do not say what the chain did with any of it, or for which validator.
  • Logs name the slot and validator index when a duty fails, for example Failed to propose this slot's block (crates/validator/src/lib.rs:938). Nothing summarizes outcomes per validator per epoch.
  • Nothing checks inclusion. A validator that publishes on time but is never included (a late or unpropagated attestation, say) looks healthy in every series. So does a proposer whose blocks are orphaned.
  • The ethlambda beacon node has no validator monitor and does not serve the standard rewards endpoints (/eth/v1/beacon/rewards/attestations/{epoch}, /rewards/blocks/{block_id}, /rewards/sync_committee/{block_id}). It serves POST /eth/v1/validator/liveness/{epoch} (crates/net/rpc/src/beacon/validator.rs:204-221), which also counts an attestation the node merely received or accepted through the API, so it does not measure inclusion.

Why it matters

A missed head or target vote is lost income, and a run of them is usually the first sign of a slow beacon node, a bad peer set or a skewed clock. Today an operator finds out from a block explorer, a day later, if they look.

What other clients do

Client Where and what Read at
Lighthouse Beacon node, by design ("Monitoring is in the Beacon Node", book/src/validator_monitoring.md:10-21). --validator-monitor-auto enrolls every validator that calls beacon_committee_subscriptions (beacon_node/beacon_chain/src/validator_monitor.rs:1102-1124). Per-validator head, target and source hits and misses, inclusion distance, and block and sync committee inclusion (validator_monitor.rs:790 onward; metric names in beacon_node/beacon_chain/src/metrics.rs:1036-1138). Per-validator series and logs stop at 64 validators unless the threshold is raised (book/src/validator_monitoring.md:60). The validator client exports signing counts by status and task timings, plus one per-validator gauge, vc_attestation_duty_slot (validator_client/validator_metrics/src/lib.rs:79, :276-282) unstable @ 11813cfb
Teku --validators-performance-tracking-mode (LOGGING, METRICS, ALL, NONE; default ALL) (teku/src/main/java/tech/pegasys/teku/cli/options/ValidatorOptions.java:75-82, validator/api/src/main/java/tech/pegasys/teku/validator/api/ValidatorPerformanceTrackingMode.java:34). The tracker is beacon/validator/src/main/java/tech/pegasys/teku/validator/coordinator/performance/DefaultPerformanceTracker.java, in the beacon/validator module. It reports attestation inclusion distance and correct target and head counts (:302-373), block production and sync committee participation master @ 62a54b96
Prysm Validator client. At each epoch's end it logs a Previous epoch voting summary per validator and sets per-key gauges such as validator_correctly_voted_source, _target, _head and validator_balance (validator/client/metrics.go:86-146, :248-301, :376). The data comes from Prysm's own POST /prysm/validators/performance, and the client skips it when the node's version string does not say Prysm (validator/client/beacon-api/beacon_api_beacon_chain_client.go:307-332) develop @ 0c1e3d24
Nimbus Beacon node. Logs Block proposal included and missing, Previous epoch attestation included and missing, and Attestation failed to match target and head per monitored validator (beacon_chain/validators/validator_monitor.nim:525-564), with automatic enrollment (addAutoMonitor, :298) unstable @ 9bed158a
Lodestar Beacon node, packages/beacon-node/src/chain/validatorMonitor.ts. Local validators are registered through registerLocalValidator (:346). Per-epoch logs such as Previous epoch attestation (:782) are at debug unless validatorMonitorLogs is set (:130, :305) unstable @ 77088eb7

Where it belongs

Mostly in the beacon node, with a small part in the client:

  • The node has the data. At each epoch transition its state holds every validator's timely source, target and head flags for the previous epoch. Each imported block says who proposed it and which attestations and sync committee bits it carries, which gives inclusion distance and sync participation. A client can only get this through endpoints the ethlambda node does not serve, or through a private one, as Prysm does.
  • A node-side monitor works for any client pointed at the node, and can watch validators that are not attached to it.
  • The client knows what it attempted, which the node does not: which duties it had, which it signed, and which failed before reaching any node.

Proposal

  • Beacon node: a validator monitor.
    • Enrollment: automatic from beacon_committee_subscriptions and prepare_beacon_proposer (crates/net/rpc/src/beacon/validator.rs:72, :76), plus an explicit pubkey list.
    • At each epoch transition: per monitored validator, previous-epoch source, target and head hits from the participation flags, and the effective balance and slashed flag.
    • At each block import: inclusion distance of monitored validators' attestations and their sync committee bits.
    • Per monitored proposer duty: whether the slot ended up with a canonical block, and whether the node saw a block for it at all, which separates a missed proposal from an orphaned one.
    • Output: Prometheus series with a validator label up to a cap (Lighthouse's 64) and aggregate series beyond it, plus one log line per validator per epoch at info.
  • Beacon node, separately: the standard rewards endpoints, so any client or tool can compute the same from ethlambda's node.
  • Validator client: per-validator outcomes of what it attempted.
    • Proposals published or failed, attestations published or failed, and sync committee messages, with a capped validator label.
    • A proposal check a few slots later through GET /eth/v1/beacon/headers/{slot}, which ethlambda's node serves, to see whether the published block is the canonical one at its slot.
    • Against a node that serves the rewards endpoints, attestation rewards for the client's own indices.
  • feat: write per-block telemetry rows to ClickHouse in Silver's block_events layout #765 (per-block ClickHouse telemetry, open, opt-in) could later carry per-validator rows, but Prometheus and logs should come first.

To decide

  • Scope and title. Most of the work lands in beacon node crates. Keep one issue, or split the node's monitor and its rewards endpoints into issues of their own and keep the client's part here?
  • Label cardinality. A validator label on thousands of keys (validator: test the client with a large number of validator keys #797) is a lot of series. Cap as Lighthouse does, or keep per-validator detail in logs only?
  • Enrollment. Automatic only, explicit only, or both.
  • Inclusion distance. Since altair the participation flags record timeliness only, so inclusion distance has to come from the attestations in each block.
  • Logs. One line per validator per epoch (Prysm, Nimbus) or an aggregate line with per-validator detail at debug (Lodestar).

Tests

  • Node: a fixture state with known participation flags gives the expected hits and misses for enrolled validators. A proposer duty whose slot has no canonical block counts as a miss.
  • Node: auto-enrollment after a beacon_committee_subscriptions request.
  • Client: MockBeaconNode answering /eth/v1/beacon/headers/{slot} with a 404, or with a root other than the published block's, marks the client's proposal as not canonical.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions