Repository navigation
fix(p2p): keep a column lookup open until every column has arrived - #803
Conversation
A DataColumnsByRoot lookup is split across the peers that custody its columns, with the columns no known custodian covers going to one random peer, but it was retired on the first non-empty answer. Every column asked of another peer in the same round then waited for the chain actor's next per-slot re-drive, ~12 s on mainnet timing, however soon it could have come. An answer now takes its columns off the lookup, and the lookup is retired only once nothing is left. A short answer ends its request the way a failure does, so the rest is asked again on the existing backoff ladder; the peer is charged only if it left out a column it was itself asked for. Found on a kurtosis devnet (gloas at epoch 2): one ethlambda node had a known custodian for 2 of its 8 custody columns; about one envelope in twenty waited ~11.7 s for its columns, which made its PTC vote payload_present=false for payloads the network had on time and orphaned its next proposal, built on the EMPTY parent.
🤖 Grok Code ReviewI'll inspect the column-fetch response path around the new settle/retry logic so the review is grounded in the surrounding control flow. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Claude Code ReviewReview: settle partial column answers instead of retiring the lookupSummary: Before this change, the lookup was retired on the first non-empty answer. Any column asked of a sibling peer, or merged in later, then waited for the chain actor's per-slot re-drive (about 12 s). This PR keeps the lookup open and reuses the same in-flight round and retry ladder as the failure path. The change makes sense, and the factoring is clean: Correctness
Observability
Minor
Overall this is a good fix for the 11.7 s envelope stall. Item 1 is the main thing I'd want addressed or explicitly ruled out before merging. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 Gemini Code ReviewThis PR cleanly fixes a real latency bug where the first peer's partial response in a split The logic in One minor cleanup note:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Codex Code ReviewNo blocking findings. The change correctly retains incomplete column lookups, retries only missing columns, and avoids charging peers that delivered their full requested share. Minor documentation nit: handlers.rs:2490 still describes partial responses as completing the lookup. Update it to match the new retry behavior. Tests could not run because rustup attempted to write outside the permitted filesystem. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Kimi Code ReviewNo blocking issues found. The new settlement logic correctly:
The tests in Non-blocking test gap: consider a handler-level regression test around Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewReviewed the PR. The core change looks correct: A few minor suggestions:
One thing worth confirming: ensure sidecar payloads are validated before Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
Review of
|
…assed the checks The lookup counted its open requests (`in_flight`) and acted only once the last one had reported. A short answer's leftover waited out a sibling's request timeout, productive rounds were charged as failures, a custodian that answered short at the tip was excluded for the rest of the lookup, and a replaced lookup's late report closed one of the new lookup's requests. - Each lookup records which column is on the wire with which peer (`asked`), so a request settles only its own columns and whatever it left out is asked again at once. A retry asks only what nothing open covers, so sibling failures still cannot multiply into a fan-out each; they coalesce into the one retry already due (`retry_due`). - Only a request that left out a column it was asked for spends an attempt and doubles the backoff. A whole share spends nothing, and a column merged in afterwards is asked at once, from the merge itself too. - A peer that answers without a column is not written off. It is recorded in `missed`. While the block is at the tip (it may not have the column yet) a retry asks every other candidate for that column first, then comes back to it; past the tip it is not asked for that column again. A failed request or a refused sidecar still writes the peer off. - Columns no known custodian covers go to a peer of unknown custody, as the range path already does, rather than to a uniformly random one. Only when none is connected are they asked of a peer of known custody, once per lookup. - A lookup with nobody left to ask, or whose ladder ran out on columns no connected peer is known to custody, ends at debug level and counts as `no_custodian` on `lean_data_column_fetch_failures_total`, not as an error and `max_retries`. - An answer is settled after the chain checks, on what passed them (`column_checks::check_and_forward_then` reports each sidecar's verdict back to the actor), not on unverified sidecars. - Before asking again, the columns the store already holds are dropped. - Requests and retry timers carry their lookup's id, so a replaced lookup's report or timer settles nothing of its successor. - `ColumnAttemptOutcome::Complete` replaces `Option<Option<_>>`, and `retire_column_lookup` is the one place a lookup ends. - Docs: the by-root response doc, `pending_column_requests`, `PendingRequestKind::Columns`, `STALE_COLUMN_LOOKUP`, the outcome enum, and the new metric reason in docs/metrics.md. - Tests for the arithmetic, and for the wiring on a beacon test server: completion after the checks, the no-entry path still forwarding, giving up from an answer, the store check, the merge in `fetch_missing_columns` and the stale replacement, the stale report and timer, a short peer asked again at the tip and once past it, and the once-per-lookup ask beyond custody.
Each request settles on its own after the checks, a peer that answers short is asked again only while the block is at the tip, the store is read before asking again, and the columns no known custodian covers go to a peer of unknown custody, or once per lookup to one of known custody when none is connected.
|
Thanks for this. You were right that the round counter was the wrong shape. I reworked the PR's core rather than patching it: ce1dc39 (code, tests,
I haven't re-run the devnet. What would show the effect is the hold distribution of held envelopes and |
|
/review |
🤖 Claude Code ReviewPR Review: per-request settlement for
|
🤖 Codex Code Review
Tests could not run because rustup attempted to write to its read-only temporary directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Grok Code ReviewI'll inspect the column-lookup settlement paths and the surrounding fetch/retry code for correctness bugs.The lookup-id and retry paths look sensitive to re-entrancy. I'll read those functions next. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Gemini Code ReviewThis PR looks solid. The transition from retiring a
The unit test coverage across Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 DeepSeek Code ReviewNo consensus-layer state-transition/fork-choice logic is touched by this PR; the changes are confined to p2p data-column lookup bookkeeping.
The rest of the refactor is well-structured, especially the stale-lookup generation id preventing late old requests from spending the new lookup’s attempts. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewThe lookup-state redesign is generally coherent, and the tests cover late reports, retries, stale lookups, and callback failures well. Two edge cases are worth addressing:
No other blocking correctness or security issues stood out. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
…ndelivered keep The completion callback ran from the end of the check task, so a panic outside the blocking checks or a task ended early left the lookup with its columns on the wire until it went stale. Verdicts now report when dropped, with anything not judged as undecided. A kept sidecar counted as delivered even when handing it to the chain actor failed, so the lookup could retire as complete for a column the chain never got. Those are now reported undecided, and asked again.
Past the tip, a peer of unknown custody that had answered without one uncovered column was passed over for all of them, so a column merged in after its short answer went to nobody and waited for the next slot's re-ask. Uncovered columns are now assigned per column: a peer takes the ones it may still be asked for, and another peer the rest. The once-per- lookup ask beyond custody follows the same rule. Also from the third review round: - A replaced lookup's report that carried an invalid sidecar writes its sender off for the new lookup too; a failure from it still does not. - The plan's `excluded` counts only connected peers. - `settle_column_report` documents that a short report riding on a zero-backoff retry spends nothing, and why that is bounded. - The unreachable `ColumnIndices` error says so, and a missing blank line. - Tests: the per-column assignment past the tip, the invalid stale report, and a lookup reading the tip from its held block in the store.
…usted A lookup that had written off every connected peer (a failed request or an invalid sidecar) and had nobody left to ask counted as no_custodian, which says nobody holds the columns. Nobody was left to say. It now counts as peers_exhausted on lean_data_column_fetch_failures_total; the next slot's re-ask starts a fresh lookup that asks them again.
|
Round three (the automated review of c03ef90). Fixed in fd24089, 2c51d42, 4bae913 and f31e521. Fixed
Not changed
Gemini found nothing to change, and Grok's comment ended after its preamble with no findings. |
|
/review |
🤖 Codex Code ReviewNo actionable findings. The changes preserve sidecar validation before completing lookups, isolate replies from replaced lookups, and retry missing columns without duplicating active requests. Tests could not run: Rustup attempted to write to a read-only directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview:
|
🤖 Grok Code ReviewI'll read the settlement and lookup paths in context to verify a few behaviors the diff only shows partially. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Gemini Code ReviewThis PR is well-designed and thoroughly tested. Moving lookup settlement to a post-verification callback ( What looks good
No issues found — LGTM. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 DeepSeek Code ReviewThe rework is thorough, with good docs and test coverage around the new verdict/retry model. A few issues worth addressing:
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewOverall, the lookup-ID fencing, post-validation settlement, and partial-response handling look sound. The tests cover the important stale-request, invalid-sidecar, and retry cases. I found no blocking consensus or security issue, but there are a few non-blocking problems:
Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
# Conflicts: # crates/net/p2p/src/beacon/column_checks.rs # docs/metrics.md
|
Re-read the merge of main at 597306c. The approval stands; two points from my earlier pass were not written down with it, so they are here. Neither blocks the merge. The mergeOnly
One nit: Still open1. A stale lookup is replaced just before its hung request would fail over.
The constant's doc treats "shorter than the request timeout" as harmless because the old request's late report settles nothing of the new lookup ( Either of these would close it:
2. The devnet was not re-run on the rewritten design. The PR's goal is fewer long holds, and that is still unmeasured since the rework. A follow-up issue for the first is fine by me. |
🗒️ Description / Motivation
A
DataColumnsByRootlookup for a held block's or envelope's missing columns is split across peers: each column goes to a peer known to custody it, and the columns no known custodian covers go to another peer. But the lookup was retired on the first non-empty answer. When the custodian of one column answered first, the request for the other columns lost its lookup: its empty answer found no entry to retry, and its partial answer was not retried either. Those columns waited for the chain actor's next per-slot re-drive (redrive_envelopes_awaiting_columns, once a slot), about 12 s, however soon they could have come.Found on a kurtosis devnet (mainnet preset, gloas at epoch 2, 4 CLs). One ethlambda node had a known custodian for only 2 of its 8 custody columns (
lean_custody_column_peers):payload_present=falsefor payloads the network had on time. The node's next proposal was built on the parent's EMPTY branch while everyone else built on FULL, so it was orphaned. Example: slot 422's envelope was held at 04:07:01.30 and verified at 04:07:13.057; the node's block 423 was built on 422 EMPTY and orphaned by 424 on 422 FULL.What Changed
Each request of a lookup now settles on its own, on what passed the checks, and what it left out is asked again at once.
p2p/src/lib.rsPendingColumnRequesttracks which column is on the wire with which peer (asked), what each peer failed to deliver (missed), whether a retry is due, the lookup's id and the block's slot.PendingRequestKind::ColumnsandRetryDataColumnFetchcarry the lookup id. A newColumnAnswerCheckedmessage. A merge infetch_missing_columnsasks at once for anything nothing open covers.p2p/src/req_resp/handlers.rssettle_column_report: a request settles only its own columns, only a request that left something out spends an attempt, a short answer never writes its peer off (a failed request or a refused sidecar does), and one retry is due at a time for every leftover. A replaced lookup's report closes nothing and spends nothing, but its invalid sidecars still count against its peer.plan_column_requests: a peer that missed a column is asked again only while the block is at the tip, after the others; uncovered columns go to peers of unknown custody, per column, or once per lookup to one of known custody when none can take them.ask_unasked_columnsdrops what the store holds first.retire_column_lookupis the one place a lookup ends.p2p/src/beacon/column_checks.rscheck_and_forward_thenreports a verdict per sidecar once the checks are done, on every path. The per-block groups (#754) share oneVerdictsguard that reports when the last group's task is done; a kept sidecar counts as delivered only once the chain actor has taken it.p2p/src/metrics.rs,docs/metrics.mdlean_data_column_fetch_failures_total{reason="no_custodian"}: a lookup that found nobody connected holding the missing columns, now ended at debug level instead oferror!+max_retries.reason="peers_exhausted": a lookup whose every candidate peer failed or sent a sidecar the checks refused.docs/beacon_wire.mdCorrectness / Behavior Guarantees
MAX_FETCH_RETRIESattempts, with the doubling backoff from 5 ms, spent only by requests that left something out. One retry is due at a time, and it asks only what no open request covers, so failures cannot multiply into fan-outs. Productive rounds spend nothing, but each removes at least one column. Past the tip each candidate is asked for a column once.no_custodian(orpeers_exhaustedwhen every candidate failed) at debug level and waits for the next per-slot re-ask.Tests Added / Run
failures_across_a_split_lookup_share_one_retry,a_short_answer_has_its_leftover_asked_without_waiting_for_its_siblings,a_custodian_answering_without_its_own_column_is_not_written_off,a_request_that_delivered_its_whole_share_spends_no_attempt,a_report_from_a_replaced_lookup_settles_nothing_of_the_new_one,a_sidecar_the_checks_refuse_stays_missing_and_is_held_against_its_sender,an_undecided_sidecar_stays_missing_without_blaming_its_sender,a_custodian_that_missed_a_column_is_asked_after_the_othersan_answer_settles_its_lookup_once_its_sidecars_are_checked,an_answer_no_lookup_waits_on_is_still_checked_and_forwarded,an_answer_at_the_end_of_the_ladder_gives_the_lookup_up,a_ladder_spent_on_columns_nobody_keeps_ends_as_no_custodian,a_column_no_known_custodian_covers_goes_only_to_a_peer_that_has_said_nothing,a_column_nobody_is_known_to_keep_is_asked_beyond_custody_once_per_lookup,at_the_tip_a_peer_that_answered_short_is_asked_again,past_the_tip_each_peer_is_asked_for_a_column_once,a_retry_asks_only_for_what_the_store_does_not_have,a_retry_for_a_lookup_that_was_replaced_does_nothing,a_column_merged_into_a_lookup_is_asked_at_once,a_re_ask_that_merges_a_column_asks_for_it_at_once,a_re_ask_replaces_a_lookup_that_stopped_asking,a_lookup_reads_the_tip_from_its_held_block_in_the_store,past_the_tip_a_peer_is_still_asked_for_the_columns_it_has_not_missed,a_lookup_every_peer_failed_ends_as_peers_exhaustedthe_callback_reports_what_the_checks_made_of_each_sidecar,the_callback_runs_when_nothing_can_be_checked,a_repeated_column_is_reported_once,verdicts_dropped_early_report_the_rest_as_undecided,a_kept_sidecar_is_delivered_only_once_the_chain_has_itcargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,cargo test -p ethlambda-p2p --profile release-fast(406 passed).Related Issues / PRs
✅ Verification Checklist
make fmt— clean-D warningson the workspace — cleanmake test(test-consensusplustest-node, atrelease-fast) —ethlambda-p2ptests only; CI runs the rest