Skip to content

plugin: Keep peer reconnects off the critical path - #767

Merged
cdecker merged 3 commits into
mainfrom
2026m11-reconnect-off-critical-path
Nov 18, 2026
Merged

cdecker merged 3 commits into
mainfrom
2026m11-reconnect-off-critical-path

Conversation

@cdecker

@cdecker cdecker commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Every successful connection is recorded in the greenlight/peerlist datastore and dialed again whenever a signer attaches. The reconnect held the shared RPC connection for the whole pass, so one stale entry whose address no longer answers queued every other caller of that connection — lsp_invoice in particular — behind a connect that only returns once the TCP connect times out (~2 min per pass, one pass per signer attach).

 stream_hsm_requests (signer attaches)
   spawn reconnect_peers
-    lock shared RPC connection
-    for peer in connected ∪ peerlist:      # sequential, no timeout
-      connect(peer)                         # dead address: ~135s
+    for peer in connected ∪ peerlist:      # up to 8 at a time
+      own RPC connection
+      timeout(30s, connect(peer))

 lsp_invoice
-  lock shared RPC connection                # waits for the whole pass
+  own RPC connection
   capacity check / LSP menus / ...
-  lsps-lsps2-invoice                        # no timeout
+  timeout(30s, lsps-lsps2-invoice)

The timeout needs the dedicated connection: cln_rpc reads the next frame off the socket without matching response ids, so abandoning a call on the shared connection would hand its late response to the next caller.

Also adds channels::reconnect_candidates: the peers with at least one channel passing is_channel_available, for callers that want to skip peers with nothing to restore (e.g. only an abandoned JIT channel). Peerlist entries are still dialed here.

Evidence

  • Before: production session logs: the reconnect starts dialing a stale peerlist entry, LspInvoice requests arrive 7s, ~2.5 min and ~3.5 min later and never log anything past the request (no capacity check, no LSP menus); each signer re-attach queues another full pass behind the dead connect.
    +0s      Calling connect: <stale peer>          (holds shared RPC)
    +7s      Got a request for /greenlight.Node/LspInvoice   → silence
    +136s    Connect returned: ... Connection timed out
    +160s    Reconnecting all peers (plugin)        (signer re-attached)
    ...      next pass dials the stale peer again
    
  • After: cargo test -p gl-plugin --lib: 36 passed, including the new test_reconnect_needs_one_available_channel. An integration test downstream plants a peerlist entry that accepts TCP but never answers the handshake, re-attaches the signer, and asserts lsp_invoice returns within 15s while that connect hangs.

Merge Danger

Door: two-way

Blast Radius: reconnects

Reconnect attempts now run concurrently (≤8) on their own connections, and give up after 30s instead of waiting for the OS timeout. lsp_invoice now fails with DEADLINE_EXCEEDED after 30s if the LSP doesn't complete the negotiation, where it used to wait indefinitely.

cdecker added 3 commits November 18, 2026 14:24
Every successful connection is recorded in the `greenlight/peerlist`
datastore, and every entry is dialed again whenever a signer attaches.
The reconnect held the shared RPC connection for the whole pass and
dialed peers one after the other, so a single stale entry whose address
no longer answers stalled it for minutes per pass, until the TCP connect
timed out. Everything else on that connection, `lsp_invoice` in
particular, queued behind it: invoice creation hung while the app polled
on.

Reconnecting is best effort, so take it off the critical path: collect
the peers and dial each of them on their own RPC connection, with a 30s
timeout, up to 8 at a time. Peerlist entries are still dialed, but a dead
one now costs at most 30s in the background and blocks nobody.
`lsp_invoice` held the shared RPC connection across the whole LSPS2
negotiation, including the `lsps-lsps2-invoice` call that waits on the
remote LSP and had no timeout. An LSP that never answers would hang the
invoice, and every other caller of the shared connection, indefinitely.

Use a dedicated connection and bound the negotiation to 30s. The timeout
is only safe on a dedicated connection: `cln_rpc` reads the next frame
off the socket without matching response ids, so abandoning a call on a
shared connection would hand its late response to the next caller.
A reconnect only helps to bring channels back, so a peer is worth dialing
if at least one of its channels is available by the existing rule in
`is_channel_available`. One with only unavailable channels, e.g. an
abandoned JIT channel the LSP server may have forgotten, is not.

`reconnect_candidates` exposes that set so callers can skip such peers.
@cdecker
cdecker enabled auto-merge (rebase) November 18, 2026 14:33
@cdecker
cdecker merged commit d96fb9d into main Nov 18, 2026
16 checks passed
@cdecker
cdecker deleted the 2026m11-reconnect-off-critical-path branch November 18, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant