Repository navigation
Description
The validator client's /health always answers 200 {"status":"healthy","service":"ethlambda-validator"}. It says the process is up and nothing else. A client whose beacon nodes are all down, or that loaded no keys, or whose slashing-protection writes are failing, answers the same way. Kubernetes probes, load balancers and uptime checks that read it are told everything is fine while no duty is being done.
Code references are at main @ 5d8628e9.
What happens today
- The handler is a constant.
crates/validator/src/metrics.rs:664-669returns the same body with no input. It is served on--metrics-portnext to/metrics(metrics.rs:671-674). - It is bound early on purpose.
crates/validator/src/lib.rs:217-232binds it before any beacon node call, so that "an operator can tell 'still starting, node unreachable' apart from 'crashed with no HTTP surface at all'". It does tell those two apart. It cannot tell "still starting" from "running and attesting", because both answerhealthy. - The code already knows it is not enough.
run_until_stopped's doc (lib.rs:575-582) says that carrying on with a poisoned slashing-protection record "would leave a client that answers/healthand does not attest, which nothing that supervises it has a reason to touch". That case is handled by exiting the process (lib.rs:594-612). The other ways to stop doing duties are not. - The state exists, as metrics.
ethlambda_validator_validators_loaded(set_validators_loaded,lib.rs:212).ethlambda_validator_validators_resolvedandduties_held, set inrefresh_epoch(lib.rs:1026,lib.rs:1050).ethlambda_validator_beacon_node_available(metrics.rs:308-314).ethlambda_validator_slashing_protection_write_failures_total(metrics.rs:367, counted atslashing_protection/mod.rs:871).
beacon_node_availablecannot simply be mirrored. Its help text says "1 when a beacon node answered the last duty refresh or attestation". In the code:- It is set to
1only at the end of a successfulrefresh_epoch(lib.rs:1051), once per epoch. - It is set to
0on any failed refresh, subscription, attestation, sync committee message or PTC vote (lib.rs:438,:497,:698,:807,:838). - So one failed attestation keeps it at
0until the next epoch's refresh, up to 6.4 minutes on mainnet, even if every later attestation is accepted. A successful attestation never sets it back. - With no keys loaded it never becomes
1, becauserefresh_epochreturns before the line that sets it (lib.rs:1001-1002). - It is one value for all nodes. Nothing records which configured node is up.
- It is set to
What the client cannot report today
These are the states in which the process is up and no duty is done, or duties are done badly:
- No beacon node usable. Every configured node is unreachable, syncing or optimistic.
refresh_epochkeeps the previous schedule and retries (lib.rs:436-439), and the attestation path fails each slot. - No keys.
store.is_empty()only logs "No validators loaded; the client will idle" once at startup (lib.rs:209-210). - Keys loaded but none resolved.
validators_resolvedis0whilevalidators_loadedis not. That is normal before activation, and wrong on a node that resolves nothing because it is on another network (see validator: check every beacon node is on the same chain, and follow config changes #831). - Slashing protection refusing every signature.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
slashing_protection/mod.rs:860-879). The next slot tries again and fails the same way. - A record held past
RECORD_WAIT(3 s,slashing_protection/mod.rs:118) refuses the duty with "a write may be stuck" (slashing_protection/mod.rs:171-178). - Neither stops the process. A poisoned record does, so
/healthdoes not need a state for it.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
- Starting up. Between binding the listener and the first successful refresh, nothing has been checked yet.
What other clients report
| Client | Endpoint | What it reports | Read at |
|---|---|---|---|
| Lighthouse | GET /lighthouse/health and GET /lighthouse/beacon/health on the validator client's HTTP API, behind its API token (validator_client/http_api/src/lib.rs:306-316, :433-466, :1425-1428) |
The first: host and process statistics. The second: every beacon node's health tier (sync distance, optimistic, execution client) or the reason it is unusable, the same state its failover ranks by. Both answer with a JSON body for the reader to interpret. | unstable @ 11813cfb |
| Prysm | GET /healthz on the monitoring port, which the validator client registers unless --disable-monitoring is set (validator/node/node.go:394-399, monitoring/prometheus/service.go:48, :61-91) |
503 when any registered service reports an error. The validator service only reports whether it holds a connection object ("no connection to beacon RPC", validator/client/service.go:288-293). A connected node that is syncing passes. |
develop @ 0c1e3d24 |
| Teku | None. The validator client's REST API serves keymanager routes only (validator/client/src/main/java/tech/pegasys/teku/validator/client/restapi/ValidatorRestApi.java:122-146). |
master @ 62a54b96 |
|
| Beacon API, for beacon nodes | GET /eth/v1/node/health (ethereum/beacon-APIs @ 6bc8c7bd, apis/node/health.yaml:6-24) |
200 ready, 206 syncing or optimistic, 503 not initialized or having issues. "Useful for load balancers." |
No validator client read here reports keys or slashing-protection state through a health endpoint. Lighthouse's per-node view is the closest to the beacon_nodes check proposed below, and the Beacon API's status codes are the convention a probe already understands.
Proposal
Keep /health cheap and local: it reads state the duty loop already maintains and makes no beacon node request of its own. Answer 200 when the client can do its duties and 503 when it cannot, with a JSON body that says why.
{
"status": "unhealthy",
"service": "ethlambda-validator",
"checks": {
"beacon_nodes": { "usable": 0, "configured": 2 },
"validators": { "loaded": 64, "resolved": 64 },
"slashing_protection": { "last_write_ok": false },
"started": true
}
}beacon_nodes. At least one configured node answered its last probe, is not syncing and not optimistic. This needs per-node state, which validator: rank beacon nodes by health and broadcast signed messages #830 adds. Until that lands, a smaller version can use the outcome of the last beacon node call of any kind, with a success setting it back.validators.loadedandresolved, reported.resolved = 0does not fail the check, since keys waiting for activation are normal. Whetherloaded = 0does is open (see "To decide").slashing_protection. The last commit succeeded and the last access did not time out onRECORD_WAIT. A failed write fails the check until a later write succeeds.started.falseuntil the first successfulrefresh_epoch, so a probe can tell "starting" from "running".
Alongside it:
- Fix
beacon_node_available. Set it from the same per-node state, or at least to1on every successful duty call, so it matches its help text. Document the change indocs/metrics.md. - Add a
ethlambda_validator_healthygauge with the same value as the endpoint, so a Prometheus alert and a probe cannot disagree. - Document the endpoint in
docs/cli.md's Metrics section, which does not mention/healthtoday.
To decide
- One endpoint or two. A liveness probe that fails while the beacon node is down makes Kubernetes restart the validator client, which does not help and adds a restart to the outage. Kubernetes separates liveness from readiness for this reason. Options: keep
/healthas liveness (always200while the process serves) and add/readywith the checks above, or make/healththe checked one and tell operators not to use it as a liveness probe. - A degraded state. Some nodes down but one usable, or some keys unresolved.
200with the detail in the body, or a separate status like the beacon API's206. - Whether remote signers count. A Web3Signer that does not answer loses its keys' signatures (
SIGNING_BUDGET,crates/validator/src/signing.rs:120). Anupcheckper configured signer would put that in the body. It is a network call, so it would need its own cadence. - No keys. A client started empty to wait for a keymanager import is idle on purpose. Whether
loaded = 0answers503, or200with the count in the body. - Doppelganger. feat(validator): doppelganger protection #807 adds keys that are loaded but withheld while the check runs. Whether those count as loaded for the check.
Tests
- The handler returns
503before the first refresh and200after one, driven throughMockBeaconNode. - Every configured node failing turns it to
503; one recovering turns it back. - A slashing-protection write that fails (an unwritable directory) turns it to
503; a later successful write clears it. - An empty key store reports
validators.loaded = 0. beacon_node_availablereturns to1on the first successful duty after a failure.
Activity