Repository navigation
Open
Description
ethlambda validator installs no signal handler. SIGINT and SIGTERM end it wherever it stands, and in the Docker image, where it runs as PID 1, both are ignored until Docker sends SIGKILL. #799 gave the node graceful SIGINT and SIGTERM handling and left the validator client unchanged, as its description says.
Code references are at 5d8628e9 (current main, with #799 merged).
What happens today
validator::runbuilds a runtime and awaitsethlambda_validator::run(bin/ethlambda/src/validator.rs:245-276). Neither listens for a signal. fix: shut down gracefully on SIGTERM #799'sShutdownSignals(bin/ethlambda/src/shutdown.rs:31) is used only by the node'swait_for_shutdownand the Hive test driver (bin/ethlambda/src/main.rs:1040,:1163).- With no handler, both signals keep their default action and the process ends at once. The image's entrypoint is the binary itself (
Dockerfile:96), so in a container it is PID 1, where the kernel drops a signal that has no handler.docker stopthen waits its timeout (10 s by default) and sends SIGKILL. - The client already has a stop path.
run_until_stoppedcancels the keymanager server and gives requests in flightKEYMANAGER_DRAIN(60 s) to finish (crates/validator/src/lib.rs:561-627). It runs only when the duty loop returns an error or the slashing-protection record is poisoned. No signal reaches it, andrunreturnsErron every path (lib.rs:558).
What an abrupt kill interrupts
Nothing it interrupts can get a validator slashed:
- Slashing-protection writes go to a
.tmpsibling, are flushed, renamed over the record, and the directory is flushed (crates/validator/src/secure_fs.rs:145-183, called fromcrates/validator/src/slashing_protection/file.rs:266-290). A kill leaves the old file or the new one, and a leftover.tmpis removed by the next write (secure_fs.rs:158-162). Every block and attestation is recorded before it is signed.validator_definitions.ymlis written the same way. - The record lock is an
flockon the validators directory (slashing_protection/file.rs:294-314). The kernel releases it when the process exits, SIGKILL included, so a restart is never refused for a stale lock.
What it does cost:
- A keymanager import cut between saving its keys and registering them. That leaves keys on disk with no slashing-protection record, and the next start refuses the whole client until those keys get a record (
ethlambda slashing-protection import, or the dangerous--init-slashing-protection) (crates/validator/src/http_api/mod.rs:211-219,crates/validator/src/http_api/keystores.rs:406-413). The drain inrun_until_stoppedexists to prevent exactly this, and a signal skips it. - A signed block whose publish is still in flight. The proposal is lost, and the restarted client cannot sign that slot again, because the record already holds it.
- A blinded block's publish runs on a detached task (
crates/validator/src/proposal.rs:755).docs/spec_deviations.md:806-808already notes that a SIGINT in that window ends it. - At gloas, a kill between the block's publish and its envelope's withholds the slot's payload (
proposal.rs:643).
- A blinded block's publish runs on a detached task (
- Attestations, aggregates and sync committee messages signed and not yet submitted are lost, for the same reason.
- Detached tasks are dropped mid-request: proposer preferences (
lib.rs:470), subnet subscriptions (lib.rs:492), builder registrations (lib.rs:512,:543) and the fee-recipient re-send (lib.rs:1191). Each is re-sent later, so this costs at most a slot or an epoch. - No shutdown line in the log, so a clean stop and a crash look the same. Every
docker stoptakes the full timeout.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | One handler for the beacon node and the validator client: SIGTERM, SIGINT and SIGHUP all start a shutdown (lighthouse/environment/src/lib.rs:365-388) |
unstable @ 11813cfb |
| Prysm | SIGINT or SIGTERM starts a "duty-aware" shutdown. It waits until the current slot's rewarded duties (attestation, sync committee message, proposal) are done, and through the next slot's too when that slot has one and less than 3 s would be left to restart. It gives up after 8 s, under Docker's default 10 s. A second signal stops at once, and ten more panic (validator/node/node.go:131-147, :156-172, validator/client/duty_aware_shutdown.go:14-27, :137) |
develop @ 0c1e3d24 |
Proposal
- Listen in
validator::runwith fix: shut down gracefully on SIGTERM #799'sShutdownSignals. It ispub(crate)in the same bin crate, sovalidator.rscan use it as it is. Passethlambda_validator::runa shutdown future or aCancellationToken; the library crate should not install handlers of its own. - Add a third arm to
run_until_stopped'sselect!for that signal. On it:- Stop the duty loop from starting another slot.
- Let the work in flight finish, bounded: the proposal (its own deadline is the attester offset), the gloas envelope, a blinded block's publish, and the slot's attestations.
- Cancel the keymanager server and drain it, as today.
- Log
Shutdown signal receivedwith the signal, thenShutdown complete, and returnOk(())so the process exits 0.
- Escalate on repeats with
ShutdownSignals::force_exit_on_repeats, so a stuck drain can still be ended by hand. - Track the detached tasks that matter, at least the blinded publish, in a
tokio_util::task::TaskTracker, so the shutdown can wait for them. - Docs. Drop the "handles no signals" sentence in
docs/spec_deviations.md:806-808, and say in the README's validator client section how the client stops under systemd and Docker.
To decide
- Drop the duties in flight, or wait. Dropping them at once is safe, since everything signed was recorded first. Waiting a few seconds saves a proposal or a slot's attestations. Prysm waits for the slot's rewarded duties, capped at 8 s.
KEYMANAGER_DRAINis 60 s, against Docker's 10 s default, Kubernetes' 30 s grace period and systemd's 90 s. Either document a longer stop timeout for the client, or shorten the drain when a signal started it. A cut-short import is the one costly outcome above, so shortening it trades one problem for another.- When to start listening. fix: shut down gracefully on SIGTERM #799 listens from the node's ready line, and a signal during startup keeps its default action. The client's startup decrypts every keystore, which can take a while with many keys (validator: test the client with a large number of validator keys #797), so listening from the top of
runwould need startup to check for a signal as it goes.
Tests
- Like fix: shut down gracefully on SIGTERM #799's
bin/ethlambda/tests/shutdown.rs: spawnethlambda validatorwith an empty validators directory against a stub HTTP server answering/eth/v1/beacon/genesis,/eth/v1/config/specand/eth/v1/node/syncing. Wait forValidator client starting, send SIGTERM, and assert exit code 0 and both shutdown lines. - Next to the existing
run_until_stoppedtests incrates/validator/src/lib.rs: a shutdown future that resolves while a keymanager request is in flight. The request finishes beforerun_until_stoppedreturns, and the result is a clean stop.
Metadata
Metadata
Assignees
Labels
No labels
Activity