Repository navigation
Open
Description
Nothing in this client, or in an ethlambda beacon node, reports how each validator is doing on chain: whether its attestations were included with the right head, target and source and how late, whether its proposals became canonical, or whether its sync committee messages made it into blocks. Every other client has this somewhere. Most of them put it in the beacon node.
Code references are at bf66e86c.
What happens today
- Validator client metrics (
crates/validator/src/metrics.rs) are process-wide counters and gauges with no labels. They count what the client sent and what failed, per duty kind:ethlambda_validator_attestations_published_total(:38),ethlambda_validator_blocks_proposed_total(:94),ethlambda_validator_sync_committee_messages_published_total(:384), and the matching failure, refusal and deadline series. They do not say what the chain did with any of it, or for which validator. - Logs name the slot and validator index when a duty fails, for example
Failed to propose this slot's block(crates/validator/src/lib.rs:938). Nothing summarizes outcomes per validator per epoch. - Nothing checks inclusion. A validator that publishes on time but is never included (a late or unpropagated attestation, say) looks healthy in every series. So does a proposer whose blocks are orphaned.
- The ethlambda beacon node has no validator monitor and does not serve the standard rewards endpoints (
/eth/v1/beacon/rewards/attestations/{epoch},/rewards/blocks/{block_id},/rewards/sync_committee/{block_id}). It servesPOST /eth/v1/validator/liveness/{epoch}(crates/net/rpc/src/beacon/validator.rs:204-221), which also counts an attestation the node merely received or accepted through the API, so it does not measure inclusion.
Why it matters
A missed head or target vote is lost income, and a run of them is usually the first sign of a slow beacon node, a bad peer set or a skewed clock. Today an operator finds out from a block explorer, a day later, if they look.
What other clients do
| Client | Where and what | Read at |
|---|---|---|
| Lighthouse | Beacon node, by design ("Monitoring is in the Beacon Node", book/src/validator_monitoring.md:10-21). --validator-monitor-auto enrolls every validator that calls beacon_committee_subscriptions (beacon_node/beacon_chain/src/validator_monitor.rs:1102-1124). Per-validator head, target and source hits and misses, inclusion distance, and block and sync committee inclusion (validator_monitor.rs:790 onward; metric names in beacon_node/beacon_chain/src/metrics.rs:1036-1138). Per-validator series and logs stop at 64 validators unless the threshold is raised (book/src/validator_monitoring.md:60). The validator client exports signing counts by status and task timings, plus one per-validator gauge, vc_attestation_duty_slot (validator_client/validator_metrics/src/lib.rs:79, :276-282) |
unstable @ 11813cfb |
| Teku | --validators-performance-tracking-mode (LOGGING, METRICS, ALL, NONE; default ALL) (teku/src/main/java/tech/pegasys/teku/cli/options/ValidatorOptions.java:75-82, validator/api/src/main/java/tech/pegasys/teku/validator/api/ValidatorPerformanceTrackingMode.java:34). The tracker is beacon/validator/src/main/java/tech/pegasys/teku/validator/coordinator/performance/DefaultPerformanceTracker.java, in the beacon/validator module. It reports attestation inclusion distance and correct target and head counts (:302-373), block production and sync committee participation |
master @ 62a54b96 |
| Prysm | Validator client. At each epoch's end it logs a Previous epoch voting summary per validator and sets per-key gauges such as validator_correctly_voted_source, _target, _head and validator_balance (validator/client/metrics.go:86-146, :248-301, :376). The data comes from Prysm's own POST /prysm/validators/performance, and the client skips it when the node's version string does not say Prysm (validator/client/beacon-api/beacon_api_beacon_chain_client.go:307-332) |
develop @ 0c1e3d24 |
| Nimbus | Beacon node. Logs Block proposal included and missing, Previous epoch attestation included and missing, and Attestation failed to match target and head per monitored validator (beacon_chain/validators/validator_monitor.nim:525-564), with automatic enrollment (addAutoMonitor, :298) |
unstable @ 9bed158a |
| Lodestar | Beacon node, packages/beacon-node/src/chain/validatorMonitor.ts. Local validators are registered through registerLocalValidator (:346). Per-epoch logs such as Previous epoch attestation (:782) are at debug unless validatorMonitorLogs is set (:130, :305) |
unstable @ 77088eb7 |
Where it belongs
Mostly in the beacon node, with a small part in the client:
- The node has the data. At each epoch transition its state holds every validator's timely source, target and head flags for the previous epoch. Each imported block says who proposed it and which attestations and sync committee bits it carries, which gives inclusion distance and sync participation. A client can only get this through endpoints the ethlambda node does not serve, or through a private one, as Prysm does.
- A node-side monitor works for any client pointed at the node, and can watch validators that are not attached to it.
- The client knows what it attempted, which the node does not: which duties it had, which it signed, and which failed before reaching any node.
Proposal
- Beacon node: a validator monitor.
- Enrollment: automatic from
beacon_committee_subscriptionsandprepare_beacon_proposer(crates/net/rpc/src/beacon/validator.rs:72,:76), plus an explicit pubkey list. - At each epoch transition: per monitored validator, previous-epoch source, target and head hits from the participation flags, and the effective balance and
slashedflag. - At each block import: inclusion distance of monitored validators' attestations and their sync committee bits.
- Per monitored proposer duty: whether the slot ended up with a canonical block, and whether the node saw a block for it at all, which separates a missed proposal from an orphaned one.
- Output: Prometheus series with a
validatorlabel up to a cap (Lighthouse's 64) and aggregate series beyond it, plus one log line per validator per epoch at info.
- Enrollment: automatic from
- Beacon node, separately: the standard rewards endpoints, so any client or tool can compute the same from ethlambda's node.
- Validator client: per-validator outcomes of what it attempted.
- Proposals published or failed, attestations published or failed, and sync committee messages, with a capped
validatorlabel. - A proposal check a few slots later through
GET /eth/v1/beacon/headers/{slot}, which ethlambda's node serves, to see whether the published block is the canonical one at its slot. - Against a node that serves the rewards endpoints, attestation rewards for the client's own indices.
- Proposals published or failed, attestations published or failed, and sync committee messages, with a capped
- feat: write per-block telemetry rows to ClickHouse in Silver's block_events layout #765 (per-block ClickHouse telemetry, open, opt-in) could later carry per-validator rows, but Prometheus and logs should come first.
To decide
- Scope and title. Most of the work lands in beacon node crates. Keep one issue, or split the node's monitor and its rewards endpoints into issues of their own and keep the client's part here?
- Label cardinality. A
validatorlabel on thousands of keys (validator: test the client with a large number of validator keys #797) is a lot of series. Cap as Lighthouse does, or keep per-validator detail in logs only? - Enrollment. Automatic only, explicit only, or both.
- Inclusion distance. Since altair the participation flags record timeliness only, so inclusion distance has to come from the attestations in each block.
- Logs. One line per validator per epoch (Prysm, Nimbus) or an aggregate line with per-validator detail at debug (Lodestar).
Tests
- Node: a fixture state with known participation flags gives the expected hits and misses for enrolled validators. A proposer duty whose slot has no canonical block counts as a miss.
- Node: auto-enrollment after a
beacon_committee_subscriptionsrequest. - Client:
MockBeaconNodeanswering/eth/v1/beacon/headers/{slot}with a 404, or with a root other than the published block's, marks the client's proposal as not canonical.
Metadata
Metadata
Assignees
Labels
No labels
Activity