Skip to content

validator: shut down gracefully on SIGINT and SIGTERM #835

Open

Description

@MegaRedHand

ethlambda validator installs no signal handler. SIGINT and SIGTERM end it wherever it stands, and in the Docker image, where it runs as PID 1, both are ignored until Docker sends SIGKILL. #799 gave the node graceful SIGINT and SIGTERM handling and left the validator client unchanged, as its description says.

Code references are at 5d8628e9 (current main, with #799 merged).

What happens today

  • validator::run builds a runtime and awaits ethlambda_validator::run (bin/ethlambda/src/validator.rs:245-276). Neither listens for a signal. fix: shut down gracefully on SIGTERM #799's ShutdownSignals (bin/ethlambda/src/shutdown.rs:31) is used only by the node's wait_for_shutdown and the Hive test driver (bin/ethlambda/src/main.rs:1040, :1163).
  • With no handler, both signals keep their default action and the process ends at once. The image's entrypoint is the binary itself (Dockerfile:96), so in a container it is PID 1, where the kernel drops a signal that has no handler. docker stop then waits its timeout (10 s by default) and sends SIGKILL.
  • The client already has a stop path. run_until_stopped cancels the keymanager server and gives requests in flight KEYMANAGER_DRAIN (60 s) to finish (crates/validator/src/lib.rs:561-627). It runs only when the duty loop returns an error or the slashing-protection record is poisoned. No signal reaches it, and run returns Err on every path (lib.rs:558).

What an abrupt kill interrupts

Nothing it interrupts can get a validator slashed:

  • Slashing-protection writes go to a .tmp sibling, are flushed, renamed over the record, and the directory is flushed (crates/validator/src/secure_fs.rs:145-183, called from crates/validator/src/slashing_protection/file.rs:266-290). A kill leaves the old file or the new one, and a leftover .tmp is removed by the next write (secure_fs.rs:158-162). Every block and attestation is recorded before it is signed. validator_definitions.yml is written the same way.
  • The record lock is an flock on the validators directory (slashing_protection/file.rs:294-314). The kernel releases it when the process exits, SIGKILL included, so a restart is never refused for a stale lock.

What it does cost:

  • A keymanager import cut between saving its keys and registering them. That leaves keys on disk with no slashing-protection record, and the next start refuses the whole client until those keys get a record (ethlambda slashing-protection import, or the dangerous --init-slashing-protection) (crates/validator/src/http_api/mod.rs:211-219, crates/validator/src/http_api/keystores.rs:406-413). The drain in run_until_stopped exists to prevent exactly this, and a signal skips it.
  • A signed block whose publish is still in flight. The proposal is lost, and the restarted client cannot sign that slot again, because the record already holds it.
    • A blinded block's publish runs on a detached task (crates/validator/src/proposal.rs:755). docs/spec_deviations.md:806-808 already notes that a SIGINT in that window ends it.
    • At gloas, a kill between the block's publish and its envelope's withholds the slot's payload (proposal.rs:643).
  • Attestations, aggregates and sync committee messages signed and not yet submitted are lost, for the same reason.
  • Detached tasks are dropped mid-request: proposer preferences (lib.rs:470), subnet subscriptions (lib.rs:492), builder registrations (lib.rs:512, :543) and the fee-recipient re-send (lib.rs:1191). Each is re-sent later, so this costs at most a slot or an epoch.
  • No shutdown line in the log, so a clean stop and a crash look the same. Every docker stop takes the full timeout.

What other clients do

Client Behavior Read at
Lighthouse One handler for the beacon node and the validator client: SIGTERM, SIGINT and SIGHUP all start a shutdown (lighthouse/environment/src/lib.rs:365-388) unstable @ 11813cfb
Prysm SIGINT or SIGTERM starts a "duty-aware" shutdown. It waits until the current slot's rewarded duties (attestation, sync committee message, proposal) are done, and through the next slot's too when that slot has one and less than 3 s would be left to restart. It gives up after 8 s, under Docker's default 10 s. A second signal stops at once, and ten more panic (validator/node/node.go:131-147, :156-172, validator/client/duty_aware_shutdown.go:14-27, :137) develop @ 0c1e3d24

Proposal

  • Listen in validator::run with fix: shut down gracefully on SIGTERM #799's ShutdownSignals. It is pub(crate) in the same bin crate, so validator.rs can use it as it is. Pass ethlambda_validator::run a shutdown future or a CancellationToken; the library crate should not install handlers of its own.
  • Add a third arm to run_until_stopped's select! for that signal. On it:
    1. Stop the duty loop from starting another slot.
    2. Let the work in flight finish, bounded: the proposal (its own deadline is the attester offset), the gloas envelope, a blinded block's publish, and the slot's attestations.
    3. Cancel the keymanager server and drain it, as today.
    4. Log Shutdown signal received with the signal, then Shutdown complete, and return Ok(()) so the process exits 0.
  • Escalate on repeats with ShutdownSignals::force_exit_on_repeats, so a stuck drain can still be ended by hand.
  • Track the detached tasks that matter, at least the blinded publish, in a tokio_util::task::TaskTracker, so the shutdown can wait for them.
  • Docs. Drop the "handles no signals" sentence in docs/spec_deviations.md:806-808, and say in the README's validator client section how the client stops under systemd and Docker.

To decide

  • Drop the duties in flight, or wait. Dropping them at once is safe, since everything signed was recorded first. Waiting a few seconds saves a proposal or a slot's attestations. Prysm waits for the slot's rewarded duties, capped at 8 s.
  • KEYMANAGER_DRAIN is 60 s, against Docker's 10 s default, Kubernetes' 30 s grace period and systemd's 90 s. Either document a longer stop timeout for the client, or shorten the drain when a signal started it. A cut-short import is the one costly outcome above, so shortening it trades one problem for another.
  • When to start listening. fix: shut down gracefully on SIGTERM #799 listens from the node's ready line, and a signal during startup keeps its default action. The client's startup decrypts every keystore, which can take a while with many keys (validator: test the client with a large number of validator keys #797), so listening from the top of run would need startup to check for a signal as it goes.

Tests

  • Like fix: shut down gracefully on SIGTERM #799's bin/ethlambda/tests/shutdown.rs: spawn ethlambda validator with an empty validators directory against a stub HTTP server answering /eth/v1/beacon/genesis, /eth/v1/config/spec and /eth/v1/node/syncing. Wait for Validator client starting, send SIGTERM, and assert exit code 0 and both shutdown lines.
  • Next to the existing run_until_stopped tests in crates/validator/src/lib.rs: a shutdown future that resolves while a keymanager request is in flight. The request finishes before run_until_stopped returns, and the result is a clean stop.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions