Skip to content

Crash-test the beacon follower with node-crashtest (power, network and disk faults in a VM) #709

Open

Description

@MegaRedHand

Goal

Run the beacon follower through cskiraly/node-crashtest: it runs a node in a libvirt VM on a public network, cuts its power or network or fills its disk at a chosen moment, then checks that the node came back by itself, how long that took, and that its data is intact.

A VM power cut loses writes the kernel had not flushed yet, which kill -9 or docker kill never does. Our RocksDB backend writes every batch with set_sync(false), set_use_fsync(false) and wal_bytes_per_sync of 32 MiB (crates/storage/src/backend/rocksdb.rs), so a power cut drops the unsynced WAL tail. We have never checked that a resume from what is left works.

Scenarios and what each one exercises

Scenario Fault What it tells us about ethlambda
power-cuts.sh power cuts at the head does a resume from a truncated WAL find a coherent head, finalized checkpoint and state (Store::from_db_state, BeaconUnrealizedJustifications from #628, envelopes); how far back each cut sets it
sync-cuts.sh a power cut at the start of each sync phase cuts during checkpoint sync, range sync and head following; needs our phase log lines (see below)
net-cuts.sh link down 30 min, then partial cuts discv5 re-discovery, range-sync catch-up after a long gap
disk-full.sh disk full for 2 min the storage layer panics on DB errors (#306); does the node restart cleanly once space is back
write-cut.sh a power cut right after a file rewrite nothing to target: at runtime the node writes no file outside RocksDB (without --node-key the key lives in memory)

Known behavior to tell apart from new bugs (the role KNOWN_ERRORS plays for geth):

  • Embedded ethrex restarts up to ~34 blocks below its pre-shutdown head and re-executes them within ~30 s (seen after a clean SIGINT; a power cut can only make it worse).
  • ethrex answers every fcU with -38006 Too deep reorg when its snap sync re-pivoted to the tip while the CL head was more than 128 blocks behind. sync-cuts.sh and net-cuts.sh' 30 min link-down (150 slots) are both able to produce that gap.

What has to be adapted

node-crashtest was built around geth, so it does not run ethlambda as is:

  • Head and liveness checks use EL JSON-RPC (eth_blockNumber, eth_syncing on RPC, compared with a public reference RPC). Options:
    • embedded ethrex (feat/include-ethrex, not on main yet): one process, so one systemd unit and one SERVICE_USER for the partial cuts, with RPC pointing at its in-process 8545. The EL head advances only while the CL drives it, so it is an end-to-end signal, and checks/history.sh works unchanged. Preferred
    • ethlambda and a standalone ethrex on the VM: same signal, but two units, and node-crashtest assumes one
    • or contribute a Beacon API probe upstream (/eth/v1/node/syncing, /eth/v1/beacon/headers/head) with a reference beacon node
  • A provisioner of our own (provision/ethlambda.sh, the way provision/geth.sh does it): VM, systemd unit, env file (lib/node.env.example). The binary must run on Ubuntu 24.04 (glibc 2.39).
  • The unit must set KillSignal=SIGINT: wait_for_shutdown (bin/ethlambda/src/main.rs) only listens for ctrl-c, so systemd's default SIGTERM kills the node without a graceful shutdown, and every ordinary stop would already be an unclean one. Handling SIGTERM in the node is worth a separate fix.
  • sync-cuts.sh and the error check read geth's log lines (KNOWN_ERRORS in lib/common.sh): we need our own sync-phase markers and our own list of expected error lines after a crash.

Setup

  • Network: Hoodi or Sepolia (the scenarios pick a reference RPC for mainnet, sepolia or hoodi). Lean has no public network, so this is beacon only.
  • Host: Linux with KVM and libvirt (virsh, virt-install), plus disk for the VM image.

Done when

  • A provisioner for ethlambda with ethrex exists, here or upstream
  • Each scenario has run at least once, with results linked here
  • Every FAIL has its own issue

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    beaconEthereum Beacon Chain clientenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions