Repository navigation
fix: shut down gracefully on SIGTERM - #799
Conversation
The node only listened for SIGINT, so SIGTERM (what `docker stop`, systemd and Kubernetes send) killed it at once with exit 143: no shutdown log, actors and servers not stopped, nothing flushed. `shutdown_signal()` resolves on the first SIGINT or SIGTERM and names which one. Both the node's `wait_for_shutdown` and the Hive test driver wait on it. It is the same function #783 adds, so that PR rebases over this one. The 2nd-4th ctrl-c escalation stays ctrl-c only: once SIGTERM has a listener a repeated one no longer kills the process, and a hung shutdown ends with the SIGKILL that `docker stop` and systemd send after their timeout. The README keeps `--stop-signal SIGINT` on its `docker run` lines, since `:latest` is v1.0.0, which still exits on SIGTERM; the comments now say so.
🤖 Gemini Code ReviewThe changes look good.
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview: graceful shutdown on SIGTERMThe change is small and correct overall. Handling SIGTERM matters especially in containers: when the binary runs as PID 1, the kernel ignores signals that have no handler. Before this PR, Correctness
Behavior / design
Nits
SummaryThe logic is sound and the docs are updated consistently across the README, Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 Codex Code ReviewNo actionable findings. SIGTERM correctly enters the existing graceful shutdown path for both node and test-driver modes, with a non-Unix fallback. Repeated-signal behavior is documented. No consensus, cryptography, or SSZ logic changed. Review was static; tests were not run. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Grok Code ReviewI'll read the shutdown path around the new signal helper so the review covers the escalation loop, not just the diff hunk. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Kimi Code ReviewNo blocking issues found. The change is correct and appropriately scoped:
Optional nit: in the Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewReviewed the diff. No consensus-layer code is touched, so no fork choice / attestation / state transition / SSZ / XMSS concerns here. The SIGTERM handling is a reasonable improvement. Minor issues:
Documentation updates look accurate. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
pablodeymo
left a comment
There was a problem hiding this comment.
The signal handling looks right for a clean stop. Two things I'd change before merging: the README edits, which describe a release that doesn't exist yet, and the escalation, which leaves a hung shutdown with no way out short of SIGKILL. The rest are follow-ups. This is from reading the diff; I did not build or run it.
I also agree with the automated reviews about matching Ok(()) / Some(()) in the select!, so a failed SIGINT registration isn't logged as a received SIGINT.
| # Flushing its state can outlast Docker's default 10 s, after which Docker kills it and the | ||
| # unflushed state is lost, so `--stop-timeout` gives it a minute. | ||
| docker run -d --name ethlambda-beacon --network ethereum --stop-signal SIGINT --stop-timeout 60 \ | ||
| docker run -d --name ethlambda-beacon --network ethereum \ |
There was a problem hiding this comment.
ghcr.io/lambdaclass/ethlambda:latest and releases/latest/download/... are still v1.0.0, which has no SIGTERM handler (latest only moves when a release is promoted). Someone following this section today gets the old behavior: the binary is PID 1 in the image, so docker stop sends a SIGTERM the kernel discards, waits 10 s and SIGKILLs. Same for line 63: a systemd unit on the v1.0.0 binary without KillSignal=SIGINT is killed at once.
Could we keep --stop-signal SIGINT and the KillSignal=SIGINT note until a release carries this change, as the first commit of this PR did? The note could say that v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too.
There was a problem hiding this comment.
Restored in e3069ea. Both docker run lines keep --stop-signal SIGINT, and their comment now says v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too. Line 63 is back to KillSignal=SIGINT for a systemd unit running v1.0.0, with the same wording. Once a release carries this change, the flag and the note can go.
| # Flushing its state can outlast Docker's default 10 s, after which Docker kills it and the | ||
| # unflushed state is lost, so `--stop-timeout` gives it a minute. | ||
| docker run -d --name ethlambda-beacon --stop-signal SIGINT --stop-timeout 60 \ | ||
| docker run -d --name ethlambda-beacon \ |
There was a problem hiding this comment.
I'd keep --stop-timeout 60 on both docker run lines. Its reason isn't the signal: #727 added it because a clean shutdown during catch-up took 14.3 s, past Docker's 10 s default, and the state was lost to the SIGKILL. Handling SIGTERM doesn't make the shutdown any shorter, so docker stop during catch-up would hit that again.
There was a problem hiding this comment.
Kept in e3069ea, on both lines, with the comment that gives it its own reason: flushing the state can outlast Docker's 10 s default, whichever signal starts the shutdown.
| "Shutdown signal received, stopping actors and servers..." | ||
| ); | ||
|
|
||
| tokio::spawn(async move { |
There was a problem hiding this comment.
This task still counts only ctrl-c, so once the first signal has been handled a further SIGTERM does nothing and logs nothing. Two cases where that bites:
- ctrl-c, the shutdown hangs, the operator runs
kill <pid>. Onmainthat kills the process; here it is swallowed. run_test_driverhas no escalation task, so a hungrpc_handle.awaitthere now needskill -9.
The description says this matches #783, but #783 bounds the shutdown with SHUTDOWN_DEADLINE (8 s) and exit(1); here nothing does. Awaiting shutdown_signal() in place of ctrl_c() in this task would let either signal count, and it is a small change. The automated review raised this as optional; I think it's worth doing because of the kill <pid> regression.
There was a problem hiding this comment.
Done in 386f526. The escalation now counts either signal: after the first, three more ctrl-c or SIGTERM, in any mix, force exit(1). The 2nd and 3rd log a warning naming the signal and how many are left. So kill <pid> after a hung ctrl-c counts toward the exit, though one kill alone now warns rather than ending the process the way it did on main.
- No re-registration. I didn't call
shutdown_signal()again in the task: each call registered a fresh listener, and a signal landing between two calls was dropped.ShutdownSignals(newbin/ethlambda/src/shutdown.rs) registers both signals once, inlisten(). After the first signal it is moved into the escalation task (force_exit_on_repeats), so nothing is lost in between. - Test driver.
run_test_driveruses the same type and gets the same escalation. Itsrpc_handle.awaitwas already bounded onmain: fix(rpc): end event streams when the node shuts down #745 routesstart_test_driver_rpc_serverthroughserve(), which stops waitingHTTP_DRAIN_TIMEOUT(5 s) after cancellation. - Checked by hand in test-driver mode, with a request whose body never arrives so the drain holds the shutdown open:
- ctrl-c, then three SIGTERMs: both warnings, then
Forced ungraceful shutdown... signal="SIGTERM", exit 1. - One SIGTERM alone: waited out the 5 s drain, then
Shutdown complete, exit 0.
- ctrl-c, then three SIGTERMs: both warnings, then
- On feat: adopt Spawned 0.8.0 lifecycle, owned jobs and chain simulation #783: it is still open and none of it is on
main. The only boundmainhas is fix(rpc): end event streams when the node shuts down #745's 5 s drain on the HTTP side. The actor stop/join is unbounded, which is why the escalation matters here. The description no longer says this matches feat: adopt Spawned 0.8.0 lifecycle, owned jobs and chain simulation #783.
| Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers' | ||
| shutdown token, and three more force the process to exit if a graceful stop hangs. |
There was a problem hiding this comment.
With the escalation on ctrl-c only, this now reads as if three more SIGTERMs also force the exit. If the escalation stays as it is:
| Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers' | |
| shutdown token, and three more force the process to exit if a graceful stop hangs. | |
| Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers' | |
| shutdown token, and three more ctrl+c force the process to exit if a graceful stop hangs. |
No change needed if the escalation ends up counting either signal.
There was a problem hiding this comment.
The escalation now counts either signal (386f526), so the sentence was right. I made it explicit anyway: "three more, of either kind, force the process to exit". The wait_for_shutdown doc comment, which said a repeated SIGTERM does not escalate, is rewritten in the same commit.
| p2p_ref.join().await; | ||
| let _ = node.http.await; | ||
|
|
||
| info!("Shutdown complete"); |
There was a problem hiding this comment.
Not from this PR, but more visible with it: let _ = node.http.await; just above does not return while a client is connected to /eth/v1/events. The stream in events.rs ends only on RecvError::Closed, and the router's Extension(events) keeps a sender alive for as long as the server runs, so axum's graceful shutdown waits on that connection forever.
Interactively, three more ctrl-c got out of it. A supervisor sends none, so a stop with a subscriber attached would end in SIGKILL and never log Shutdown complete. The actors are already joined by then, so I read it as an exit-code and log problem, not lost state. I haven't reproduced it, but #783 names the same hang and bounds it with HTTP_SHUTDOWN_TIMEOUT, so it only matters while this PR is on main without #783.
There was a problem hiding this comment.
#745 is merged into this branch now, and it closes this. No change here. From the code:
- The token:
wait_for_shutdowncancelsnode.shutdown, and the HTTP task gets a clone of it (main.rs:844-845, passed at:901for the Beacon API and:912for lean). - The stream ends: every server goes through
serve()(crates/net/rpc/src/lib.rs:178): both API surfaces and metrics viastart_http_servers, and the test driver.serve()layers aShutdownSignal, a child of that token, onto the router (lib.rs:187)./eth/v1/eventsextracts it (beacon/events.rs:34-41), and so does/lean/v0/events.sse_responsethen ends the stream withtake_untilon it (events.rs:117-125), whatever the router'sEventBusclone does. - The drain is bounded:
serve()stops waitingHTTP_DRAIN_TIMEOUT(5 s,lib.rs:162) after cancellation for anything still in flight. A client that stops reading cannot hold it either. - Tests on
main:a_connected_event_stream_does_not_hold_up_shutdownand..._on_separate_ports(crates/net/rpc/tests/http_servers.rs:209,:217) keep/eth/v1/eventsopen, cancel, and require the servers to stop within 2 s.a_response_that_never_finishes_does_not_hold_up_shutdown(:274) covers the drain bound.
So the HTTP task finishes within HTTP_DRAIN_TIMEOUT of the cancel, subscriber or not, and node.http.await can no longer be what holds the shutdown. It still sits after the two actor joins, which are unbounded on main; the escalation covers those. I also saw the drain bound end to end: in test-driver mode with a stalled request, one SIGTERM logged the drain-timeout warning after 5 s, then Shutdown complete, exit 0.
| // 1st ctrl+c: start graceful shutdown | ||
| tokio::signal::ctrl_c().await.ok(); | ||
| // 1st ctrl+c or SIGTERM: start graceful shutdown | ||
| let signal = shutdown_signal().await; |
There was a problem hiding this comment.
Seconding the automated review's point about installing the listener earlier, with one addition for Docker: as PID 1 an unhandled SIGTERM is discarded, not fatal, so a docker stop during checkpoint sync or the DB open is lost entirely. The node finishes starting, never sees the signal, and is SIGKILLed at the timeout. Creating the Signal stream at the top of run_node and passing it down would latch an early signal. Follow-up material.
There was a problem hiding this comment.
Agreed it's worth doing. I've left it for a follow-up, because latching alone would trade one problem for another.
- The problem: with the listener installed at the top of
run_node, a ctrl-c during a long checkpoint sync would be held until startup finished. That sync can run up toMAX_CHECKPOINT_ATTEMPTSrounds with 15 s connect/read timeouts and a whole-state download. Today the same ctrl-c ends the process at once. - Doing it properly means startup has to give up on a signal: either race the startup half of
run_nodeagainstShutdownSignals::recv, or exit on a signal that comes beforeNode initialized. That touches most ofrun_node. - What this PR sets up:
ShutdownSignalsregisters synchronously and holds a signal untilrecvtakes it. The follow-up only has to build it at the top ofrun_nodeand pass it down. - Where it stands now: both signals are registered just before the ready line (
Node initialized,Test-driver RPC ready), not on first poll after it. The startup gap, PID 1 included, is written down in theshutdownmodule doc.
| /// | ||
| /// SIGTERM is what `docker stop`, systemd and Kubernetes send, so a container | ||
| /// no longer needs `--stop-signal SIGINT` to shut down gracefully. | ||
| async fn shutdown_signal() -> &'static str { |
There was a problem hiding this comment.
Optional: test-driver mode makes a cheap harness for a regression test. HIVE_LEAN_TEST_DRIVER=1 ethlambda node reaches shutdown_signal() before any keys are loaded or the chain starts, so a test could spawn the binary, send SIGTERM and assert exit 0 plus the shutdown log line. That would also protect the SIGTERM arm through #783's rewrite of wait_for_shutdown.
There was a problem hiding this comment.
Added in 0f9e250.
- What it does:
bin/ethlambda/tests/shutdown.rsspawns the built binary asHIVE_LEAN_TEST_DRIVER=1 ethlambda node, with the required flags pointing at files that need not exist and--api-port 0. It waits forTest-driver RPC readyand sends SIGTERM. It then asserts exit code 0, aShutdown signal receivedline naming"SIGTERM", andShutdown complete. - Why it can't race:
run_test_drivernow registers the signals before logging that ready line. Before, it registered them on first poll after the line, so a test could send SIGTERM before the listener existed. - Cost: it takes about 1.5 s; the driver is ready in under a second.
- It fails without the fix: run against
main'smain.rs(bf66e86), it fails withgot signal: 15 (SIGTERM), with no exit code. - CI: it is the bin crate's first integration test, so
test-nodenow also links the real binary, about 60 MB atrelease-faston macOS.
`latest` (the image and the release download) is still v1.0.0, which has no SIGTERM handler. In the image the binary is PID 1, so `docker stop`'s SIGTERM is discarded and the node is SIGKILLed at the timeout; a systemd unit without `KillSignal=SIGINT` kills it at once. Restore `--stop-signal SIGINT` on both `docker run` lines and the `KillSignal=SIGINT` note, saying that v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too. `--stop-timeout 60` stays for its own reason: a clean shutdown during catch-up can outlast Docker's 10 s default whichever signal starts it.
The escalation task counted ctrl-c only. Once tokio listens for SIGTERM it no longer kills the process, so after a ctrl-c whose shutdown hung, `kill <pid>` did nothing and logged nothing, where on main it killed the process. And `shutdown_signal()` matched `_` in its `select!`, so a failed SIGINT registration resolved at once and was logged as a received SIGINT; repeated, that would have been three instant "signals" and `exit(1)`. `ShutdownSignals` (new `shutdown` module) registers SIGINT and SIGTERM once, up front, and warns once for a signal it cannot listen for. Its `recv` returns only on a delivered signal (`Some(())` arms, with an `else` that never resolves), and a signal that arrives between two calls is held for the next. `force_exit_on_repeats` moves it into the escalation task, so the 2nd, 3rd and 4th signal of either kind force `exit(1)`, and no signal is lost between the first wait and the task. Both `wait_for_shutdown` and `run_test_driver` build it before their ready line (`Node initialized`, `Test-driver RPC ready`), so a signal sent once that line is out is handled. The test driver gets the same escalation; its HTTP drain is already bounded by `HTTP_DRAIN_TIMEOUT` since it goes through `serve()`. Listening still starts there, not at the top of `run_node`: latching a signal during startup would also hold a ctrl-c sent during a long checkpoint sync until startup finished, since nothing in startup gives up on a signal. The module doc says so. docs/architecture.md: three more signals "of either kind" force the exit.
`tests/shutdown.rs` spawns the built binary as `HIVE_LEAN_TEST_DRIVER=1 ethlambda node`, which reaches the shutdown wait without keys, a genesis or a database. Once it logs `Test-driver RPC ready` the test sends SIGTERM and asserts exit code 0, a `Shutdown signal received` line naming "SIGTERM", and `Shutdown complete`. It is race-free because `run_test_driver` registers its signals before that line. `--api-port 0` lets the OS pick the port, so parallel runs never collide, and a guard kills the child if the test fails first. Run against bf66e86's `main.rs`, which listens for ctrl-c only, it fails: the child dies by signal 15 and has no exit code. This is the bin crate's first integration test, so the test run now links the binary itself as well as its unit-test harness.
|
Pushed e3069ea, 386f526 and 0f9e250. The inline threads have their own replies. From the review body:
|
|
/review |
🤖 Codex Code ReviewNo actionable findings. The SIGINT/SIGTERM handling preserves graceful shutdown and repeat-signal escalation. No consensus, cryptography, or SSZ logic changes. The integration test covers graceful SIGTERM shutdown. I couldn’t run it because rustup attempted to write to its read-only toolchain directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview: graceful shutdown on SIGTERMThe change looks correct. Correctness
Test (
|
🤖 Gemini Code ReviewThis PR looks great. The signal handling implementation in
No issues found — LGTM. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewSIGTERM shutdown is the right shape and the 4-signal force-exit count matches the old ctrl-c path. Two concrete problems, plus a couple of smaller ones. No consensus-path changes in this diff.
warn!(
signal,
remaining,
"Graceful shutdown in progress. Send {remaining} more ctrl-c or SIGTERM to force ungraceful shutdown"
);
fn drop(&mut self) {
if self.0.try_wait().ok().flatten().is_none() {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
Unix Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Kimi Code ReviewNo blocking issues found. The implementation looks correct:
Optional test coverage improvement: No consensus-layer or state-transition impact. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewOverall this is a clean change and does not touch consensus logic, so fork choice, attestation processing, justification/finalization, state transition, XMSS, or SSZ code is unaffected. A few things to verify or tighten:
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🗒️ Description / Motivation
The node only listened for SIGINT. SIGTERM, which
docker stop, systemd and Kubernetes send, had no handler. Run directly, the node was killed at once (exit 143): noShutdown signal received, actors and servers not stopped, nothing flushed. As PID 1 in the image, the kernel discarded SIGTERM and Docker SIGKILLed the node at its timeout. The README worked around it with--stop-signal SIGINTandKillSignal=SIGINT.This is the small standalone fix #784 asks for. #783 adds a
shutdown_signal()as part of the spawned 0.8 upgrade. This PR replaces that function withShutdownSignals, so when #783 rebases it should keep this side and build its shutdown deadline on top. #783's version still matches_in itsselect!and escalates on ctrl-c only.What Changed
bin/ethlambda/src/shutdown.rs(new):ShutdownSignals.recvreturns only on a delivered signal (Some(())arms, anelsethat never resolves) and holds one that arrives between calls.force_exit_on_repeatsmoves it into the escalation task.bin/ethlambda/src/main.rswait_for_shutdown(both chains) andrun_test_driver(Hive driver mode) buildShutdownSignalsbefore their ready line (Node initialized/Test-driver RPC ready).Shutdown signal receivedline, then hand it to the escalation.README.md: keeps--stop-signal SIGINT,--stop-timeout 60and theKillSignal=SIGINTnote, becauselatestis still v1.0.0. They now say v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too.--stop-signaland the note can go once a release carries this change.--stop-timeoutstays for its own reason (a catch-up flush can outlast Docker's 10 s).docs/architecture.md,docs/cli.md: the shutdown step says ctrl-c or SIGTERM, and three more of either kind force the exit.Correctness / Behavior Guarantees
exit(1).kill <pid>three times after a hung ctrl-c ends the process. Onmainone SIGTERM killed it outright; here one alone warns.docker stop, systemd and Kubernetes send one SIGTERM and then SIGKILL, so they see no change.run_nodeis a follow-up, since startup would first have to give up on a signal; otherwise a ctrl-c during checkpoint sync would wait for startup to finish.main: fix(rpc): end event streams when the node shuts down #745'sHTTP_DRAIN_TIMEOUT(5 s) bounds the HTTP servers, a connected/eth/v1/eventssubscriber included. The actor stop/join is unbounded, which is what the escalation is for.ethlambda validator) is unchanged and still handles no signals.Tests Added / Run
bin/ethlambda/tests/shutdown.rs:HIVE_LEAN_TEST_DRIVER=1 ethlambda node, waits forTest-driver RPC ready, sends SIGTERM, and asserts exit code 0, aShutdown signal receivedline naming"SIGTERM", andShutdown complete. About 1.5 s.main'smain.rsit fails: the child dies by signal 15.Manual, test-driver mode, with a request whose body never arrives so the HTTP drain holds the shutdown open:
Forced ungraceful shutdown... signal="SIGTERM", exit 1.Shutdown complete, exit 0.Manual, first version of this PR, with a debug build on a beacon node anchored at mainnet genesis from a local network directory (mainnet
config.yaml, thegenesis.sszfixture, no bootnodes). The fullwait_for_shutdownpath ran with the P2P actor, chain actor and HTTP servers up:main(690f23b)Shutdown signal received ... signal="SIGTERM",Shutdown completesignal="SIGINT"The genesis state's
genesis_timewas patched to a minute before start. With the real 2020 genesis, the debug-build SIGTERM run still exited 0, but after 263 s, likely waiting on the chain actor's first tick across five years of slots (not investigated).Commands:
cargo test --profile release-fast -p ethlambda: 289 unit tests (4 ignored) and 1 integration test passed.cargo clippy -p ethlambda --all-targets -- -D warnings: clean.cargo fmt --all --check: clean.Related Issues / PRs
✅ Verification Checklist
make fmt— clean-D warningsonethlambda, the only crate touched — cleanmake test— not run in full;cargo test --profile release-fast -p ethlambda, the only crate touched, passes