Skip to content

validator: make /health report whether the client can do its duties #833

Open

Description

@MegaRedHand

The validator client's /health always answers 200 {"status":"healthy","service":"ethlambda-validator"}. It says the process is up and nothing else. A client whose beacon nodes are all down, or that loaded no keys, or whose slashing-protection writes are failing, answers the same way. Kubernetes probes, load balancers and uptime checks that read it are told everything is fine while no duty is being done.

Code references are at main @ 5d8628e9.

What happens today

  • The handler is a constant. crates/validator/src/metrics.rs:664-669 returns the same body with no input. It is served on --metrics-port next to /metrics (metrics.rs:671-674).
  • It is bound early on purpose. crates/validator/src/lib.rs:217-232 binds it before any beacon node call, so that "an operator can tell 'still starting, node unreachable' apart from 'crashed with no HTTP surface at all'". It does tell those two apart. It cannot tell "still starting" from "running and attesting", because both answer healthy.
  • The code already knows it is not enough. run_until_stopped's doc (lib.rs:575-582) says that carrying on with a poisoned slashing-protection record "would leave a client that answers /health and does not attest, which nothing that supervises it has a reason to touch". That case is handled by exiting the process (lib.rs:594-612). The other ways to stop doing duties are not.
  • The state exists, as metrics.
    • ethlambda_validator_validators_loaded (set_validators_loaded, lib.rs:212).
    • ethlambda_validator_validators_resolved and duties_held, set in refresh_epoch (lib.rs:1026, lib.rs:1050).
    • ethlambda_validator_beacon_node_available (metrics.rs:308-314).
    • ethlambda_validator_slashing_protection_write_failures_total (metrics.rs:367, counted at slashing_protection/mod.rs:871).
  • beacon_node_available cannot simply be mirrored. Its help text says "1 when a beacon node answered the last duty refresh or attestation". In the code:
    • It is set to 1 only at the end of a successful refresh_epoch (lib.rs:1051), once per epoch.
    • It is set to 0 on any failed refresh, subscription, attestation, sync committee message or PTC vote (lib.rs:438, :497, :698, :807, :838).
    • So one failed attestation keeps it at 0 until the next epoch's refresh, up to 6.4 minutes on mainnet, even if every later attestation is accepted. A successful attestation never sets it back.
    • With no keys loaded it never becomes 1, because refresh_epoch returns before the line that sets it (lib.rs:1001-1002).
    • It is one value for all nodes. Nothing records which configured node is up.

What the client cannot report today

These are the states in which the process is up and no duty is done, or duties are done badly:

  1. No beacon node usable. Every configured node is unreachable, syncing or optimistic. refresh_epoch keeps the previous schedule and retries (lib.rs:436-439), and the attestation path fails each slot.
  2. No keys. store.is_empty() only logs "No validators loaded; the client will idle" once at startup (lib.rs:209-210).
  3. Keys loaded but none resolved. validators_resolved is 0 while validators_loaded is not. That is normal before activation, and wrong on a node that resolves nothing because it is on another network (see validator: check every beacon node is on the same chain, and follow config changes #831).
  4. Slashing protection refusing every signature.
    • A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (slashing_protection/mod.rs:860-879). The next slot tries again and fails the same way.
    • A record held past RECORD_WAIT (3 s, slashing_protection/mod.rs:118) refuses the duty with "a write may be stuck" (slashing_protection/mod.rs:171-178).
    • Neither stops the process. A poisoned record does, so /health does not need a state for it.
  5. Starting up. Between binding the listener and the first successful refresh, nothing has been checked yet.

What other clients report

Client Endpoint What it reports Read at
Lighthouse GET /lighthouse/health and GET /lighthouse/beacon/health on the validator client's HTTP API, behind its API token (validator_client/http_api/src/lib.rs:306-316, :433-466, :1425-1428) The first: host and process statistics. The second: every beacon node's health tier (sync distance, optimistic, execution client) or the reason it is unusable, the same state its failover ranks by. Both answer with a JSON body for the reader to interpret. unstable @ 11813cfb
Prysm GET /healthz on the monitoring port, which the validator client registers unless --disable-monitoring is set (validator/node/node.go:394-399, monitoring/prometheus/service.go:48, :61-91) 503 when any registered service reports an error. The validator service only reports whether it holds a connection object ("no connection to beacon RPC", validator/client/service.go:288-293). A connected node that is syncing passes. develop @ 0c1e3d24
Teku None. The validator client's REST API serves keymanager routes only (validator/client/src/main/java/tech/pegasys/teku/validator/client/restapi/ValidatorRestApi.java:122-146). master @ 62a54b96
Beacon API, for beacon nodes GET /eth/v1/node/health (ethereum/beacon-APIs @ 6bc8c7bd, apis/node/health.yaml:6-24) 200 ready, 206 syncing or optimistic, 503 not initialized or having issues. "Useful for load balancers."

No validator client read here reports keys or slashing-protection state through a health endpoint. Lighthouse's per-node view is the closest to the beacon_nodes check proposed below, and the Beacon API's status codes are the convention a probe already understands.

Proposal

Keep /health cheap and local: it reads state the duty loop already maintains and makes no beacon node request of its own. Answer 200 when the client can do its duties and 503 when it cannot, with a JSON body that says why.

{
  "status": "unhealthy",
  "service": "ethlambda-validator",
  "checks": {
    "beacon_nodes": { "usable": 0, "configured": 2 },
    "validators": { "loaded": 64, "resolved": 64 },
    "slashing_protection": { "last_write_ok": false },
    "started": true
  }
}
  • beacon_nodes. At least one configured node answered its last probe, is not syncing and not optimistic. This needs per-node state, which validator: rank beacon nodes by health and broadcast signed messages #830 adds. Until that lands, a smaller version can use the outcome of the last beacon node call of any kind, with a success setting it back.
  • validators. loaded and resolved, reported. resolved = 0 does not fail the check, since keys waiting for activation are normal. Whether loaded = 0 does is open (see "To decide").
  • slashing_protection. The last commit succeeded and the last access did not time out on RECORD_WAIT. A failed write fails the check until a later write succeeds.
  • started. false until the first successful refresh_epoch, so a probe can tell "starting" from "running".

Alongside it:

  • Fix beacon_node_available. Set it from the same per-node state, or at least to 1 on every successful duty call, so it matches its help text. Document the change in docs/metrics.md.
  • Add a ethlambda_validator_healthy gauge with the same value as the endpoint, so a Prometheus alert and a probe cannot disagree.
  • Document the endpoint in docs/cli.md's Metrics section, which does not mention /health today.

To decide

  • One endpoint or two. A liveness probe that fails while the beacon node is down makes Kubernetes restart the validator client, which does not help and adds a restart to the outage. Kubernetes separates liveness from readiness for this reason. Options: keep /health as liveness (always 200 while the process serves) and add /ready with the checks above, or make /health the checked one and tell operators not to use it as a liveness probe.
  • A degraded state. Some nodes down but one usable, or some keys unresolved. 200 with the detail in the body, or a separate status like the beacon API's 206.
  • Whether remote signers count. A Web3Signer that does not answer loses its keys' signatures (SIGNING_BUDGET, crates/validator/src/signing.rs:120). An upcheck per configured signer would put that in the body. It is a network call, so it would need its own cadence.
  • No keys. A client started empty to wait for a keymanager import is idle on purpose. Whether loaded = 0 answers 503, or 200 with the count in the body.
  • Doppelganger. feat(validator): doppelganger protection #807 adds keys that are loaded but withheld while the check runs. Whether those count as loaded for the check.

Tests

  • The handler returns 503 before the first refresh and 200 after one, driven through MockBeaconNode.
  • Every configured node failing turns it to 503; one recovering turns it back.
  • A slashing-protection write that fails (an unwritable directory) turns it to 503; a later successful write clears it.
  • An empty key store reports validators.loaded = 0.
  • beacon_node_available returns to 1 on the first successful duty after a failure.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions