Repository navigation
Description
Goal
Run the beacon follower through cskiraly/node-crashtest: it runs a node in a libvirt VM on a public network, cuts its power or network or fills its disk at a chosen moment, then checks that the node came back by itself, how long that took, and that its data is intact.
A VM power cut loses writes the kernel had not flushed yet, which kill -9 or docker kill never does. Our RocksDB backend writes every batch with set_sync(false), set_use_fsync(false) and wal_bytes_per_sync of 32 MiB (crates/storage/src/backend/rocksdb.rs), so a power cut drops the unsynced WAL tail. We have never checked that a resume from what is left works.
Scenarios and what each one exercises
| Scenario | Fault | What it tells us about ethlambda |
|---|---|---|
power-cuts.sh |
power cuts at the head | does a resume from a truncated WAL find a coherent head, finalized checkpoint and state (Store::from_db_state, BeaconUnrealizedJustifications from #628, envelopes); how far back each cut sets it |
sync-cuts.sh |
a power cut at the start of each sync phase | cuts during checkpoint sync, range sync and head following; needs our phase log lines (see below) |
net-cuts.sh |
link down 30 min, then partial cuts | discv5 re-discovery, range-sync catch-up after a long gap |
disk-full.sh |
disk full for 2 min | the storage layer panics on DB errors (#306); does the node restart cleanly once space is back |
write-cut.sh |
a power cut right after a file rewrite | nothing to target: at runtime the node writes no file outside RocksDB (without --node-key the key lives in memory) |
Known behavior to tell apart from new bugs (the role KNOWN_ERRORS plays for geth):
- Embedded ethrex restarts up to ~34 blocks below its pre-shutdown head and re-executes them within ~30 s (seen after a clean SIGINT; a power cut can only make it worse).
- ethrex answers every fcU with
-38006 Too deep reorgwhen its snap sync re-pivoted to the tip while the CL head was more than 128 blocks behind.sync-cuts.shandnet-cuts.sh' 30 min link-down (150 slots) are both able to produce that gap.
What has to be adapted
node-crashtest was built around geth, so it does not run ethlambda as is:
- Head and liveness checks use EL JSON-RPC (
eth_blockNumber,eth_syncingonRPC, compared with a public reference RPC). Options:- embedded ethrex (
feat/include-ethrex, not onmainyet): one process, so one systemd unit and oneSERVICE_USERfor the partial cuts, withRPCpointing at its in-process 8545. The EL head advances only while the CL drives it, so it is an end-to-end signal, andchecks/history.shworks unchanged. Preferred - ethlambda and a standalone ethrex on the VM: same signal, but two units, and node-crashtest assumes one
- or contribute a Beacon API probe upstream (
/eth/v1/node/syncing,/eth/v1/beacon/headers/head) with a reference beacon node
- embedded ethrex (
- A provisioner of our own (
provision/ethlambda.sh, the wayprovision/geth.shdoes it): VM, systemd unit, env file (lib/node.env.example). The binary must run on Ubuntu 24.04 (glibc 2.39). - The unit must set
KillSignal=SIGINT:wait_for_shutdown(bin/ethlambda/src/main.rs) only listens for ctrl-c, so systemd's default SIGTERM kills the node without a graceful shutdown, and every ordinary stop would already be an unclean one. Handling SIGTERM in the node is worth a separate fix. -
sync-cuts.shand the error check read geth's log lines (KNOWN_ERRORSinlib/common.sh): we need our own sync-phase markers and our own list of expected error lines after a crash.
Setup
- Network: Hoodi or Sepolia (the scenarios pick a reference RPC for mainnet, sepolia or hoodi). Lean has no public network, so this is beacon only.
- Host: Linux with KVM and libvirt (
virsh,virt-install), plus disk for the VM image.
Done when
- A provisioner for ethlambda with ethrex exists, here or upstream
- Each scenario has run at least once, with results linked here
- Every FAIL has its own issue
Activity