Skip to content

fix: shut down gracefully on SIGTERM - #799

Merged
pablodeymo merged 10 commits into
mainfrom
fix/sigterm-graceful-shutdown
Nov 19, 2026
Merged

pablodeymo merged 10 commits into
mainfrom
fix/sigterm-graceful-shutdown

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

The node only listened for SIGINT. SIGTERM, which docker stop, systemd and Kubernetes send, had no handler. Run directly, the node was killed at once (exit 143): no Shutdown signal received, actors and servers not stopped, nothing flushed. As PID 1 in the image, the kernel discarded SIGTERM and Docker SIGKILLed the node at its timeout. The README worked around it with --stop-signal SIGINT and KillSignal=SIGINT.

This is the small standalone fix #784 asks for. #783 adds a shutdown_signal() as part of the spawned 0.8 upgrade. This PR replaces that function with ShutdownSignals, so when #783 rebases it should keep this side and build its shutdown deadline on top. #783's version still matches _ in its select! and escalates on ctrl-c only.

What Changed

  • bin/ethlambda/src/shutdown.rs (new): ShutdownSignals.
    • Registers SIGINT and SIGTERM once (Unix; ctrl-c only elsewhere), and warns once for a signal it cannot listen for.
    • recv returns only on a delivered signal (Some(()) arms, an else that never resolves) and holds one that arrives between calls.
    • force_exit_on_repeats moves it into the escalation task.
  • bin/ethlambda/src/main.rs
    • wait_for_shutdown (both chains) and run_test_driver (Hive driver mode) build ShutdownSignals before their ready line (Node initialized / Test-driver RPC ready).
    • They wait on it, log the signal on the Shutdown signal received line, then hand it to the escalation.
    • The test driver had no escalation before.
  • README.md: keeps --stop-signal SIGINT, --stop-timeout 60 and the KillSignal=SIGINT note, because latest is still v1.0.0. They now say v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too. --stop-signal and the note can go once a release carries this change. --stop-timeout stays for its own reason (a catch-up flush can outlast Docker's 10 s).
  • docs/architecture.md, docs/cli.md: the shutdown step says ctrl-c or SIGTERM, and three more of either kind force the exit.

Correctness / Behavior Guarantees

  • The first SIGINT or SIGTERM starts the same graceful shutdown the first ctrl-c always did.
  • Escalation: the 2nd, 3rd and 4th signal, ctrl-c or SIGTERM in any mix, escalate. The 2nd and 3rd warn with the count left; the 4th calls exit(1).
    • So kill <pid> three times after a hung ctrl-c ends the process. On main one SIGTERM killed it outright; here one alone warns.
    • docker stop, systemd and Kubernetes send one SIGTERM and then SIGKILL, so they see no change.
  • A signal that cannot be registered never counts as received. It keeps its default action, and a warning says so.
  • Startup: signals are listened for from the ready line on. During startup both keep their default action: they end the process at once, or, as PID 1, are ignored. Latching from the top of run_node is a follow-up, since startup would first have to give up on a signal; otherwise a ctrl-c during checkpoint sync would wait for startup to finish.
  • Shutdown bound on main: fix(rpc): end event streams when the node shuts down #745's HTTP_DRAIN_TIMEOUT (5 s) bounds the HTTP servers, a connected /eth/v1/events subscriber included. The actor stop/join is unbounded, which is what the escalation is for.
  • The validator client (ethlambda validator) is unchanged and still handles no signals.

Tests Added / Run

  • bin/ethlambda/tests/shutdown.rs:

    • Spawns HIVE_LEAN_TEST_DRIVER=1 ethlambda node, waits for Test-driver RPC ready, sends SIGTERM, and asserts exit code 0, a Shutdown signal received line naming "SIGTERM", and Shutdown complete. About 1.5 s.
    • Against main's main.rs it fails: the child dies by signal 15.
    • It is the bin crate's first integration test, so the test run now links the binary itself.
  • Manual, test-driver mode, with a request whose body never arrives so the HTTP drain holds the shutdown open:

    • ctrl-c, then three SIGTERMs: both warnings, then Forced ungraceful shutdown... signal="SIGTERM", exit 1.
    • One SIGTERM alone: the 5 s drain timeout, then Shutdown complete, exit 0.
  • Manual, first version of this PR, with a debug build on a beacon node anchored at mainnet genesis from a local network directory (mainnet config.yaml, the genesis.ssz fixture, no bootnodes). The full wait_for_shutdown path ran with the P2P actor, chain actor and HTTP servers up:

    binary signal exit time to exit shutdown log
    main (690f23b) SIGTERM 143 27 ms none
    this branch SIGTERM 0 27 ms Shutdown signal received ... signal="SIGTERM", Shutdown complete
    this branch SIGINT 0 26 ms same, signal="SIGINT"

    The genesis state's genesis_time was patched to a minute before start. With the real 2020 genesis, the debug-build SIGTERM run still exited 0, but after 263 s, likely waiting on the chain actor's first tick across five years of slots (not investigated).

  • Commands:

    • cargo test --profile release-fast -p ethlambda: 289 unit tests (4 ignored) and 1 integration test passed.
    • cargo clippy -p ethlambda --all-targets -- -D warnings: clean.
    • cargo fmt --all --check: clean.

Related Issues / PRs

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran clippy with -D warnings on ethlambda, the only crate touched — clean
  • Ran make test — not run in full; cargo test --profile release-fast -p ethlambda, the only crate touched, passes

The node only listened for SIGINT, so SIGTERM (what `docker stop`, systemd
and Kubernetes send) killed it at once with exit 143: no shutdown log,
actors and servers not stopped, nothing flushed.

`shutdown_signal()` resolves on the first SIGINT or SIGTERM and names which
one. Both the node's `wait_for_shutdown` and the Hive test driver wait on it.
It is the same function #783 adds, so that PR rebases over this one. The
2nd-4th ctrl-c escalation stays ctrl-c only: once SIGTERM has a listener a
repeated one no longer kills the process, and a hung shutdown ends with the
SIGKILL that `docker stop` and systemd send after their timeout.

The README keeps `--stop-signal SIGINT` on its `docker run` lines, since
`:latest` is v1.0.0, which still exits on SIGTERM; the comments now say so.
@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look good.

  • shutdown_signal cleanly handles both SIGINT and SIGTERM on Unix targets with a sensible fallback if SIGTERM registration fails, and covers both normal node operation (wait_for_shutdown) and run_test_driver.
  • The documentation and comments accurately describe the signal behavior (including Tokio's signal handler retention on repeated SIGTERMs and the v1.0.0 note in README.md).

Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: graceful shutdown on SIGTERM

The change is small and correct overall. Handling SIGTERM matters especially in containers: when the binary runs as PID 1, the kernel ignores signals that have no handler. Before this PR, docker stop effectively waited out the timeout and then sent SIGKILL. A few notes follow.

Correctness

  1. A ctrl_c() registration error triggers shutdown even when SIGTERM works (main.rs, shutdown_signal, the select! arms).

    • _ = tokio::signal::ctrl_c() matches both Ok and Err.
    • If SIGINT registration fails, that branch resolves immediately. The node then shuts down right after "Node initialized", even though the SIGTERM listener is healthy.
    • The old ctrl_c().await.ok() had the same flaw. Here it's easy to fix by disabling the branch on error:
    return tokio::select! {
        Ok(()) = tokio::signal::ctrl_c() => "SIGINT",
        Some(()) = terminate.recv() => "SIGTERM",
    };
    • If both patterns fail, select! panics because no else branch is present. You'd want else => "SIGINT" or similar, but in practice ctrl_c erroring together with the stream closing won't happen.
  2. Handlers are installed late.

    • shutdown_signal() is first called after Node initialized. A SIGTERM during startup still uses the default disposition and kills the process immediately. Startup includes DB open, checkpoint sync and state loading.
    • With Kubernetes or systemd restarts during a long boot, that can interrupt writes.
    • This isn't a regression, since SIGINT behaves the same. Still, consider creating the Signal stream at the top of main and passing it in. Tokio's handler is process-global, so registering it early is cheap.

Behavior / design

  1. Asymmetric escalation.

    • After the first SIGTERM, tokio's handler stays installed for the lifetime of the process, so later SIGTERMs are silently swallowed. The doc comment acknowledges this.
    • Only ctrl-c escalates to exit(1).
    • Consider passing the Signal stream into the spawned escalation task so it counts SIGTERMs too. It's a small change, and a hung stop()/join() would then respond to kill <pid> repeated, not only kill -9.
    • If you keep the current behavior, the doc is accurate.
  2. The Kubernetes grace period is not documented.

    • The new comment mentions Kubernetes, but its default terminationGracePeriodSeconds is 30s, shorter than the 60s the README budgets for docker stop.
    • A line in the README advising terminationGracePeriodSeconds: 60 would prevent the same "flush outlasts timeout" data loss the Docker section warns about.

Nits

  • info!(signal, ...): the local signal: &'static str shadows nothing problematic here. Inside shutdown_signal, though, signal is the imported tokio function. Consider renaming the local to sig or received for grep-ability.
  • README: the Docker examples still pass --stop-signal SIGINT. That's harmless and correct for v1.0.0. Once a release with this change ships, those examples could drop the flag, perhaps with a note pinning which tag needs it.

Summary

The logic is sound and the docs are updated consistently across the README, architecture.md and cli.md. Item 1 is a cheap robustness fix worth taking. Items 2–4 are optional improvements.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. SIGTERM correctly enters the existing graceful shutdown path for both node and test-driver modes, with a non-Unix fallback. Repeated-signal behavior is documented.

No consensus, cryptography, or SSZ logic changed. Review was static; tests were not run.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll read the shutdown path around the new signal helper so the review covers the escalation loop, not just the diff hunk.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found. The change is correct and appropriately scoped:

  • bin/ethlambda/src/main.rs:1076 registers SIGTERM only on Unix, retains the Ctrl+C fallback if registration fails, and uses cancellation-safe tokio::select! branches.
  • Existing repeated-Ctrl+C escalation remains unchanged. The documented behavior that subsequent SIGTERMs do not force termination matches Tokio’s Unix signal handling; supervisors must still enforce their SIGKILL timeout.
  • Both shutdown entry points use the new helper, and the README/architecture/CLI documentation is consistent with the implementation.
  • No performance or security concerns; the additional signal listener is negligible.

Optional nit: in the tokio::select!, the ctrl_c() result is discarded, so a listener error would initiate shutdown and be logged as "SIGINT". This preserves the previous fail-safe shutdown behavior, but logging the error would improve diagnostics.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Reviewed the diff. No consensus-layer code is touched, so no fork choice / attestation / state transition / SSZ / XMSS concerns here. The SIGTERM handling is a reasonable improvement.

Minor issues:

  • bin/ethlambda/src/main.rs:1081-1082: both select arms use _ =, so a failure to set up a signal stream is treated as if a signal arrived. tokio::signal::ctrl_c() returns io::Result<()>; if SIGINT registration fails, that future completes with Err, the select arm is immediately ready, and shutdown_signal returns "SIGINT" even though no Ctrl-C occurred. terminate.recv() has a similar issue if it returns None (closed stream). Match on Ok(())/Some(()) and log or fall back on errors instead.

  • bin/ethlambda/src/main.rs:1074-1097: returning &'static str is stringly typed. A small enum like ShutdownSignal::Interrupt / ShutdownSignal::Terminate would be clearer and make info!(signal, ...) less fragile. Optional.

  • bin/ethlambda/src/main.rs:1078-1082: SIGTERM is registered before the Ctrl-C future is created inside select!. This leaves a very small window where SIGTERM is handled but SIGINT still has its default disposition, so a Ctrl-C in that window would kill the process instead of starting graceful shutdown. Building both futures before awaiting the select would close that gap. Very minor, but easy to avoid.

Documentation updates look accurate.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@pablodeymo pablodeymo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signal handling looks right for a clean stop. Two things I'd change before merging: the README edits, which describe a release that doesn't exist yet, and the escalation, which leaves a hung shutdown with no way out short of SIGKILL. The rest are follow-ups. This is from reading the diff; I did not build or run it.

I also agree with the automated reviews about matching Ok(()) / Some(()) in the select!, so a failed SIGINT registration isn't logged as a received SIGINT.

Comment thread README.md Outdated
# Flushing its state can outlast Docker's default 10 s, after which Docker kills it and the
# unflushed state is lost, so `--stop-timeout` gives it a minute.
docker run -d --name ethlambda-beacon --network ethereum --stop-signal SIGINT --stop-timeout 60 \
docker run -d --name ethlambda-beacon --network ethereum \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ghcr.io/lambdaclass/ethlambda:latest and releases/latest/download/... are still v1.0.0, which has no SIGTERM handler (latest only moves when a release is promoted). Someone following this section today gets the old behavior: the binary is PID 1 in the image, so docker stop sends a SIGTERM the kernel discards, waits 10 s and SIGKILLs. Same for line 63: a systemd unit on the v1.0.0 binary without KillSignal=SIGINT is killed at once.

Could we keep --stop-signal SIGINT and the KillSignal=SIGINT note until a release carries this change, as the first commit of this PR did? The note could say that v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in e3069ea. Both docker run lines keep --stop-signal SIGINT, and their comment now says v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too. Line 63 is back to KillSignal=SIGINT for a systemd unit running v1.0.0, with the same wording. Once a release carries this change, the flag and the note can go.

Comment thread README.md Outdated
# Flushing its state can outlast Docker's default 10 s, after which Docker kills it and the
# unflushed state is lost, so `--stop-timeout` gives it a minute.
docker run -d --name ethlambda-beacon --stop-signal SIGINT --stop-timeout 60 \
docker run -d --name ethlambda-beacon \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep --stop-timeout 60 on both docker run lines. Its reason isn't the signal: #727 added it because a clean shutdown during catch-up took 14.3 s, past Docker's 10 s default, and the state was lost to the SIGKILL. Handling SIGTERM doesn't make the shutdown any shorter, so docker stop during catch-up would hit that again.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept in e3069ea, on both lines, with the comment that gives it its own reason: flushing the state can outlast Docker's 10 s default, whichever signal starts the shutdown.

Comment thread bin/ethlambda/src/main.rs Outdated
"Shutdown signal received, stopping actors and servers..."
);

tokio::spawn(async move {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This task still counts only ctrl-c, so once the first signal has been handled a further SIGTERM does nothing and logs nothing. Two cases where that bites:

  • ctrl-c, the shutdown hangs, the operator runs kill <pid>. On main that kills the process; here it is swallowed.
  • run_test_driver has no escalation task, so a hung rpc_handle.await there now needs kill -9.

The description says this matches #783, but #783 bounds the shutdown with SHUTDOWN_DEADLINE (8 s) and exit(1); here nothing does. Awaiting shutdown_signal() in place of ctrl_c() in this task would let either signal count, and it is a small change. The automated review raised this as optional; I think it's worth doing because of the kill <pid> regression.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 386f526. The escalation now counts either signal: after the first, three more ctrl-c or SIGTERM, in any mix, force exit(1). The 2nd and 3rd log a warning naming the signal and how many are left. So kill <pid> after a hung ctrl-c counts toward the exit, though one kill alone now warns rather than ending the process the way it did on main.

  • No re-registration. I didn't call shutdown_signal() again in the task: each call registered a fresh listener, and a signal landing between two calls was dropped. ShutdownSignals (new bin/ethlambda/src/shutdown.rs) registers both signals once, in listen(). After the first signal it is moved into the escalation task (force_exit_on_repeats), so nothing is lost in between.
  • Test driver. run_test_driver uses the same type and gets the same escalation. Its rpc_handle.await was already bounded on main: fix(rpc): end event streams when the node shuts down #745 routes start_test_driver_rpc_server through serve(), which stops waiting HTTP_DRAIN_TIMEOUT (5 s) after cancellation.
  • Checked by hand in test-driver mode, with a request whose body never arrives so the drain holds the shutdown open:
    • ctrl-c, then three SIGTERMs: both warnings, then Forced ungraceful shutdown... signal="SIGTERM", exit 1.
    • One SIGTERM alone: waited out the 5 s drain, then Shutdown complete, exit 0.
  • On feat: adopt Spawned 0.8.0 lifecycle, owned jobs and chain simulation #783: it is still open and none of it is on main. The only bound main has is fix(rpc): end event streams when the node shuts down #745's 5 s drain on the HTTP side. The actor stop/join is unbounded, which is why the escalation matters here. The description no longer says this matches feat: adopt Spawned 0.8.0 lifecycle, owned jobs and chain simulation #783.

Comment thread docs/architecture.md Outdated
Comment on lines 262 to 263
Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers'
shutdown token, and three more force the process to exit if a graceful stop hangs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the escalation on ctrl-c only, this now reads as if three more SIGTERMs also force the exit. If the escalation stays as it is:

Suggested change
Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers'
shutdown token, and three more force the process to exit if a graceful stop hangs.
Shutdown runs in reverse: the first ctrl+c or SIGTERM stops both actors and cancels the servers'
shutdown token, and three more ctrl+c force the process to exit if a graceful stop hangs.

No change needed if the escalation ends up counting either signal.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The escalation now counts either signal (386f526), so the sentence was right. I made it explicit anyway: "three more, of either kind, force the process to exit". The wait_for_shutdown doc comment, which said a repeated SIGTERM does not escalate, is rewritten in the same commit.

Comment thread bin/ethlambda/src/main.rs
p2p_ref.join().await;
let _ = node.http.await;

info!("Shutdown complete");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not from this PR, but more visible with it: let _ = node.http.await; just above does not return while a client is connected to /eth/v1/events. The stream in events.rs ends only on RecvError::Closed, and the router's Extension(events) keeps a sender alive for as long as the server runs, so axum's graceful shutdown waits on that connection forever.

Interactively, three more ctrl-c got out of it. A supervisor sends none, so a stop with a subscriber attached would end in SIGKILL and never log Shutdown complete. The actors are already joined by then, so I read it as an exit-code and log problem, not lost state. I haven't reproduced it, but #783 names the same hang and bounds it with HTTP_SHUTDOWN_TIMEOUT, so it only matters while this PR is on main without #783.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#745 is merged into this branch now, and it closes this. No change here. From the code:

  • The token: wait_for_shutdown cancels node.shutdown, and the HTTP task gets a clone of it (main.rs:844-845, passed at :901 for the Beacon API and :912 for lean).
  • The stream ends: every server goes through serve() (crates/net/rpc/src/lib.rs:178): both API surfaces and metrics via start_http_servers, and the test driver. serve() layers a ShutdownSignal, a child of that token, onto the router (lib.rs:187). /eth/v1/events extracts it (beacon/events.rs:34-41), and so does /lean/v0/events. sse_response then ends the stream with take_until on it (events.rs:117-125), whatever the router's EventBus clone does.
  • The drain is bounded: serve() stops waiting HTTP_DRAIN_TIMEOUT (5 s, lib.rs:162) after cancellation for anything still in flight. A client that stops reading cannot hold it either.
  • Tests on main:
    • a_connected_event_stream_does_not_hold_up_shutdown and ..._on_separate_ports (crates/net/rpc/tests/http_servers.rs:209, :217) keep /eth/v1/events open, cancel, and require the servers to stop within 2 s.
    • a_response_that_never_finishes_does_not_hold_up_shutdown (:274) covers the drain bound.

So the HTTP task finishes within HTTP_DRAIN_TIMEOUT of the cancel, subscriber or not, and node.http.await can no longer be what holds the shutdown. It still sits after the two actor joins, which are unbounded on main; the escalation covers those. I also saw the drain bound end to end: in test-driver mode with a stalled request, one SIGTERM logged the drain-timeout warning after 5 s, then Shutdown complete, exit 0.

Comment thread bin/ethlambda/src/main.rs Outdated
// 1st ctrl+c: start graceful shutdown
tokio::signal::ctrl_c().await.ok();
// 1st ctrl+c or SIGTERM: start graceful shutdown
let signal = shutdown_signal().await;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seconding the automated review's point about installing the listener earlier, with one addition for Docker: as PID 1 an unhandled SIGTERM is discarded, not fatal, so a docker stop during checkpoint sync or the DB open is lost entirely. The node finishes starting, never sees the signal, and is SIGKILLed at the timeout. Creating the Signal stream at the top of run_node and passing it down would latch an early signal. Follow-up material.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's worth doing. I've left it for a follow-up, because latching alone would trade one problem for another.

  • The problem: with the listener installed at the top of run_node, a ctrl-c during a long checkpoint sync would be held until startup finished. That sync can run up to MAX_CHECKPOINT_ATTEMPTS rounds with 15 s connect/read timeouts and a whole-state download. Today the same ctrl-c ends the process at once.
  • Doing it properly means startup has to give up on a signal: either race the startup half of run_node against ShutdownSignals::recv, or exit on a signal that comes before Node initialized. That touches most of run_node.
  • What this PR sets up: ShutdownSignals registers synchronously and holds a signal until recv takes it. The follow-up only has to build it at the top of run_node and pass it down.
  • Where it stands now: both signals are registered just before the ready line (Node initialized, Test-driver RPC ready), not on first poll after it. The startup gap, PID 1 included, is written down in the shutdown module doc.

Comment thread bin/ethlambda/src/main.rs Outdated
///
/// SIGTERM is what `docker stop`, systemd and Kubernetes send, so a container
/// no longer needs `--stop-signal SIGINT` to shut down gracefully.
async fn shutdown_signal() -> &'static str {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: test-driver mode makes a cheap harness for a regression test. HIVE_LEAN_TEST_DRIVER=1 ethlambda node reaches shutdown_signal() before any keys are loaded or the chain starts, so a test could spawn the binary, send SIGTERM and assert exit 0 plus the shutdown log line. That would also protect the SIGTERM arm through #783's rewrite of wait_for_shutdown.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 0f9e250.

  • What it does: bin/ethlambda/tests/shutdown.rs spawns the built binary as HIVE_LEAN_TEST_DRIVER=1 ethlambda node, with the required flags pointing at files that need not exist and --api-port 0. It waits for Test-driver RPC ready and sends SIGTERM. It then asserts exit code 0, a Shutdown signal received line naming "SIGTERM", and Shutdown complete.
  • Why it can't race: run_test_driver now registers the signals before logging that ready line. Before, it registered them on first poll after the line, so a test could send SIGTERM before the listener existed.
  • Cost: it takes about 1.5 s; the driver is ready in under a second.
  • It fails without the fix: run against main's main.rs (bf66e86), it fails with got signal: 15 (SIGTERM), with no exit code.
  • CI: it is the bin crate's first integration test, so test-node now also links the real binary, about 60 MB at release-fast on macOS.

`latest` (the image and the release download) is still v1.0.0, which has
no SIGTERM handler. In the image the binary is PID 1, so `docker stop`'s
SIGTERM is discarded and the node is SIGKILLed at the timeout; a systemd
unit without `KillSignal=SIGINT` kills it at once.

Restore `--stop-signal SIGINT` on both `docker run` lines and the
`KillSignal=SIGINT` note, saying that v1.0.0 shuts down gracefully on
SIGINT only and later versions handle SIGTERM too. `--stop-timeout 60`
stays for its own reason: a clean shutdown during catch-up can outlast
Docker's 10 s default whichever signal starts it.
The escalation task counted ctrl-c only. Once tokio listens for SIGTERM
it no longer kills the process, so after a ctrl-c whose shutdown hung,
`kill <pid>` did nothing and logged nothing, where on main it killed the
process. And `shutdown_signal()` matched `_` in its `select!`, so a
failed SIGINT registration resolved at once and was logged as a received
SIGINT; repeated, that would have been three instant "signals" and
`exit(1)`.

`ShutdownSignals` (new `shutdown` module) registers SIGINT and SIGTERM
once, up front, and warns once for a signal it cannot listen for. Its
`recv` returns only on a delivered signal (`Some(())` arms, with an
`else` that never resolves), and a signal that arrives between two calls
is held for the next. `force_exit_on_repeats` moves it into the
escalation task, so the 2nd, 3rd and 4th signal of either kind force
`exit(1)`, and no signal is lost between the first wait and the task.

Both `wait_for_shutdown` and `run_test_driver` build it before their
ready line (`Node initialized`, `Test-driver RPC ready`), so a signal
sent once that line is out is handled. The test driver gets the same
escalation; its HTTP drain is already bounded by `HTTP_DRAIN_TIMEOUT`
since it goes through `serve()`.

Listening still starts there, not at the top of `run_node`: latching a
signal during startup would also hold a ctrl-c sent during a long
checkpoint sync until startup finished, since nothing in startup gives
up on a signal. The module doc says so.

docs/architecture.md: three more signals "of either kind" force the exit.
`tests/shutdown.rs` spawns the built binary as
`HIVE_LEAN_TEST_DRIVER=1 ethlambda node`, which reaches the shutdown wait
without keys, a genesis or a database. Once it logs
`Test-driver RPC ready` the test sends SIGTERM and asserts exit code 0, a
`Shutdown signal received` line naming "SIGTERM", and
`Shutdown complete`.

It is race-free because `run_test_driver` registers its signals before
that line. `--api-port 0` lets the OS pick the port, so parallel runs
never collide, and a guard kills the child if the test fails first.

Run against bf66e86's `main.rs`, which listens for ctrl-c only, it
fails: the child dies by signal 15 and has no exit code. This is the bin
crate's first integration test, so the test run now links the binary
itself as well as its unit-test harness.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Pushed e3069ea, 386f526 and 0f9e250. The inline threads have their own replies. From the review body:

  1. Ok(()) / Some(()) in the select!.
    • ShutdownSignals::recv (bin/ethlambda/src/shutdown.rs) now has Some(()) = … arms over two Signal streams that listen() registers once. An else arm never resolves, so only a delivered signal returns.
    • A failed registration is warned about once, in listen(), and that signal keeps its default action. It no longer resolves the wait, and in the escalation loop it can't count as three instant signals.
    • SIGINT is registered with signal(SignalKind::interrupt()) rather than ctrl_c(). That is the same handler, so there's no Ok(()) arm left to match.
    • Both are registered together, which also closes the window the DeepSeek review pointed out, where SIGTERM had a handler and SIGINT did not yet.
  2. The README (before merge): --stop-signal SIGINT, --stop-timeout 60 and the KillSignal=SIGINT note are back. They now say v1.0.0 shuts down gracefully on SIGINT only and later versions handle SIGTERM too.
  3. The escalation (before merge): a 2nd, 3rd and 4th signal of either kind force exit(1), in the node and in the test driver.
  4. feat: adopt Spawned 0.8.0 lifecycle, owned jobs and chain simulation #783: it is still open, and none of it is on main.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The SIGINT/SIGTERM handling preserves graceful shutdown and repeat-signal escalation. No consensus, cryptography, or SSZ logic changes.

The integration test covers graceful SIGTERM shutdown. I couldn’t run it because rustup attempted to write to its read-only toolchain directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: graceful shutdown on SIGTERM

The change looks correct. ShutdownSignals is small, it keeps the old "three more signals force exit" behaviour, and the integration test exercises the real binary. Nothing here touches consensus logic. Notes below, most important first.

Correctness

  1. shutdown.rs recv (unix), lines ~54–60. The select! handles the edge cases properly:

    • A None from next() (handler not registered, or stream ended) disables that branch.
    • When both branches are disabled, else parks forever.
    • Because each recv is a one-shot select!, a failed registration cannot cause a busy loop.
  2. force_exit_on_repeats count. (1..3).rev() yields 2 and 1, then a final recv calls exit. That is 3 extra signals, with messages "2 more" and "1 more", matching the previous hand-unrolled version. A future change of SIGNALS_TO_FORCE_EXIT to 1 also works, since the range is empty and the next signal exits.

  3. Non-unix path. On Windows the doc comment on ShutdownSignals doesn't hold: "one that arrives before recv runs is held until it does".

    • listen() registers nothing, and ctrl_c() is only installed when recv is first polled.
    • So a ctrl-c between listen() and the first recv(), or between the graceful-shutdown recv and the spawned task's first poll, can be lost.
    • The warn! in the non-unix recv fires on every call, not "once, here" as listen's doc says.
    • Either install a tokio::signal::windows::ctrl_c() stream in listen() for parity, or scope the doc claims to Unix.
  4. run_test_driver behaviour change. It now also force-exits on repeats. That is an improvement, but worth calling out in the PR description because Hive drivers previously had no escalation path.

Test (tests/shutdown.rs)

  1. Dependencies. The test imports libc and tempfile, but Cargo.toml isn't in the diff. Confirm both are reachable from the integration test: libc as a normal or dev dependency of bin/ethlambda, and tempfile in [dev-dependencies].

    • Alternatively, Command::new("kill").args(["-TERM", &pid.to_string()]) would drop the unsafe and the libc dependency entirely.
  2. Brittle match on "SIGTERM" (line ~88). The assertion depends on tracing-subscriber's default formatter Debug-quoting &str field values (signal="SIGTERM"). A switch to a different formatter, or ANSI output if NO_COLOR isn't honoured by the log init, would break it silently. Matching contains("SIGTERM") on the Shutdown signal received line is sufficient.

  3. Debug-build runtime. READY_TIMEOUT is 120s because startup compiles the leanVM bytecode first. Under cargo test in debug mode on a loaded CI runner this could be slow or flaky. Check CI timing, or consider #[ignore] with an explicit CI job.

  4. Ports. --api-port 0 avoids collisions for the API. If the metrics server, or anything else bound before the test-driver branch, uses a fixed default port, parallel runs (for example, several cargo test invocations in CI) could still collide. Worth a quick check.

Docs / nits

  1. README docker examples. These still pass --stop-signal SIGINT, which is fine since both versions honour it. You could add a note that later versions don't need it, though keeping it for v1.0.0 compatibility is reasonable.

  2. shutdown.rs module doc. It clearly explains why listening starts late (it avoids latching ctrl-c during a long checkpoint sync). One small caveat is that this leaves a startup-time SIGTERM in Docker as PID 1 ignored until SIGKILL. That is acceptable and already documented.

Overall LGTM once item 5 is confirmed; items 3 and 6 are worth a small follow-up.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks great. The signal handling implementation in bin/ethlambda/src/shutdown.rs is clean and idiomatic:

  • tokio::signal::unix::Signal::recv is cancel-safe, so selecting across SIGINT and SIGTERM in ShutdownSignals::recv will not drop signals if both arrive close together.
  • Registering signal handlers via ShutdownSignals::listen() prior to logging "Node initialized" / "Test-driver RPC ready" eliminates any startup race window once readiness is reported.
  • The integration test in bin/ethlambda/tests/shutdown.rs properly guards against leaked child processes on failure via Drop, uses an ephemeral port (--api-port 0), and drains stdout/stderr on background threads to avoid pipe buffer deadlocks.

No issues found — LGTM.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

SIGTERM shutdown is the right shape and the 4-signal force-exit count matches the old ctrl-c path. Two concrete problems, plus a couple of smaller ones. No consensus-path changes in this diff.

  1. bin/ethlambda/src/shutdown.rs (~lines 79–82): {remaining} is not a tracing field. With fields present, warn! only interpolates declared fields, so this should fail to compile. Add the field (or a positional {}):
warn!(
    signal,
    remaining,
    "Graceful shutdown in progress. Send {remaining} more ctrl-c or SIGTERM to force ungraceful shutdown"
);
  1. bin/ethlambda/tests/shutdown.rs lines 24–28: wait_for_exit already try_waits, which reaps the child. Drop then kill()s child.id() anyway. After a successful exit that PID can be reused, so the test can SIGKILL an unrelated process. Kill only if it is still running:
fn drop(&mut self) {
    if self.0.try_wait().ok().flatten().is_none() {
        let _ = self.0.kill();
        let _ = self.0.wait();
    }
}
  1. The new test calls libc and tempfile, but the diff does not add them as dev-dependencies. Direct deps are required; transitive ones will not compile.

  2. README docker snippets still say “later versions handle SIGTERM” and keep --stop-signal SIGINT. This commit is that later version. SIGINT still works, but the wording will send operators to a workaround they no longer need. Architecture/cli notes are accurate.

Unix listen() before the ready log, buffering until recv, and counting either signal toward force-exit are correct. Tokio will not uninstall the handlers, so counting repeats in-process instead of relying on the default action is the right call.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found. The implementation looks correct:

  • bin/ethlambda/src/shutdown.rs preserves the existing escalation behavior: the first signal starts graceful shutdown, and the fourth forces exit(1).
  • Failed or closed signal streams do not disable handling of the other signal.
  • Listeners are installed before the readiness log messages, preventing the readiness-to-listener race on Unix.
  • The integration test has appropriate timeouts, captures both output streams, and reliably cleans up the child process.

Optional test coverage improvement: bin/ethlambda/tests/shutdown.rs:37-98 exercises only the initial SIGTERM. Consider adding a mixed SIGINT/SIGTERM escalation test to verify the repeat count and forced exit status, since that logic is newly shared.

No consensus-layer or state-transition impact.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a clean change and does not touch consensus logic, so fork choice, attestation processing, justification/finalization, state transition, XMSS, or SSZ code is unaffected.

A few things to verify or tighten:

  • bin/ethlambda/src/shutdown.rs:57-72: if both Unix signal listeners fail to register (listen_for returns None), recv() enters else => pending() and waits forever. The old code did tokio::signal::ctrl_c().await.ok() and would proceed with shutdown on a registration failure. The non-Unix ctrl_c error path has the same behavior: log, then pending(). That may be intentional, but it is a behavior change worth documenting or treating as fatal during startup.

  • bin/ethlambda/tests/shutdown.rs: the assertion looks for "\"SIGTERM\"" with quotes:

    line.contains("Shutdown signal received") && line.contains("\"SIGTERM\"")

    This depends on the tracing formatter using debug/JSON-style quoting. If log formatting changes, or a different formatter is configured, this test becomes brittle. Consider asserting on SIGTERM without requiring quotes, or explicitly launch the child with a known log format.

  • The diff does not show Cargo.toml changes, so make sure libc and tempfile are already in [dev-dependencies]; otherwise the new integration test will fail to build.

  • Minor: force_exit_on_repeats is only spawned after the first signal is received. If Signals are queued while unpolled this is fine, but if not, a burst of SIGINT/SIGTERM right at shutdown start could be coalesced or lost before the task begins. Worth confirming with a test that sends the first signal and immediate repeats, since the existing test only sends a single SIGTERM.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@pablodeymo
pablodeymo added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 86f57b7 Nov 19, 2026
14 checks passed
@pablodeymo
pablodeymo deleted the fix/sigterm-graceful-shutdown branch November 19, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGTERM is not handled: docker stop kills the node without a graceful shutdown

2 participants