Repository navigation
plugin: Keep peer reconnects off the critical path - #767
Merged
Merged
Conversation
added 3 commits
November 18, 2026 14:24
Every successful connection is recorded in the `greenlight/peerlist` datastore, and every entry is dialed again whenever a signer attaches. The reconnect held the shared RPC connection for the whole pass and dialed peers one after the other, so a single stale entry whose address no longer answers stalled it for minutes per pass, until the TCP connect timed out. Everything else on that connection, `lsp_invoice` in particular, queued behind it: invoice creation hung while the app polled on. Reconnecting is best effort, so take it off the critical path: collect the peers and dial each of them on their own RPC connection, with a 30s timeout, up to 8 at a time. Peerlist entries are still dialed, but a dead one now costs at most 30s in the background and blocks nobody.
`lsp_invoice` held the shared RPC connection across the whole LSPS2 negotiation, including the `lsps-lsps2-invoice` call that waits on the remote LSP and had no timeout. An LSP that never answers would hang the invoice, and every other caller of the shared connection, indefinitely. Use a dedicated connection and bound the negotiation to 30s. The timeout is only safe on a dedicated connection: `cln_rpc` reads the next frame off the socket without matching response ids, so abandoning a call on a shared connection would hand its late response to the next caller.
A reconnect only helps to bring channels back, so a peer is worth dialing if at least one of its channels is available by the existing rule in `is_channel_available`. One with only unavailable channels, e.g. an abandoned JIT channel the LSP server may have forgotten, is not. `reconnect_candidates` exposes that set so callers can skip such peers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every successful connection is recorded in the
greenlight/peerlistdatastore and dialed again whenever a signer attaches. The reconnect held the shared RPC connection for the whole pass, so one stale entry whose address no longer answers queued every other caller of that connection —lsp_invoicein particular — behind aconnectthat only returns once the TCP connect times out (~2 min per pass, one pass per signer attach).The timeout needs the dedicated connection:
cln_rpcreads the next frame off the socket without matching response ids, so abandoning a call on the shared connection would hand its late response to the next caller.Also adds
channels::reconnect_candidates: the peers with at least one channel passingis_channel_available, for callers that want to skip peers with nothing to restore (e.g. only an abandoned JIT channel). Peerlist entries are still dialed here.Evidence
LspInvoicerequests arrive 7s, ~2.5 min and ~3.5 min later and never log anything past the request (no capacity check, no LSP menus); each signer re-attach queues another full pass behind the deadconnect.cargo test -p gl-plugin --lib: 36 passed, including the newtest_reconnect_needs_one_available_channel. An integration test downstream plants a peerlist entry that accepts TCP but never answers the handshake, re-attaches the signer, and assertslsp_invoicereturns within 15s while that connect hangs.Merge Danger
Door: two-way
Blast Radius: reconnects
Reconnect attempts now run concurrently (≤8) on their own connections, and give up after 30s instead of waiting for the OS timeout.
lsp_invoicenow fails withDEADLINE_EXCEEDEDafter 30s if the LSP doesn't complete the negotiation, where it used to wait indefinitely.