Skip to content

fix(p2p): keep a column lookup open until every column has arrived - #803

Merged
MegaRedHand merged 12 commits into
mainfrom
fix/column-lookup-partial-answer
Nov 19, 2026
Merged

MegaRedHand merged 12 commits into
mainfrom
fix/column-lookup-partial-answer

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

A DataColumnsByRoot lookup for a held block's or envelope's missing columns is split across peers: each column goes to a peer known to custody it, and the columns no known custodian covers go to another peer. But the lookup was retired on the first non-empty answer. When the custodian of one column answered first, the request for the other columns lost its lookup: its empty answer found no entry to retry, and its partial answer was not retried either. Those columns waited for the chain actor's next per-slot re-drive (redrive_envelopes_awaiting_columns, once a slot), about 12 s, however soon they could have come.

Found on a kurtosis devnet (mainnet preset, gloas at epoch 2, 4 CLs). One ethlambda node had a known custodian for only 2 of its 8 custody columns (lean_custody_column_peers):

  • Envelope holds were bimodal: sub-second, or ~11.7 s with release 40–80 ms after the next slot started. That happened to 37 of 758 held envelopes (4.9%) on one node and 29 of 754 (3.8%) on the other.
  • While held, the node's PTC members voted payload_present=false for payloads the network had on time. The node's next proposal was built on the parent's EMPTY branch while everyone else built on FULL, so it was orphaned. Example: slot 422's envelope was held at 04:07:01.30 and verified at 04:07:13.057; the node's block 423 was built on 422 EMPTY and orphaned by 424 on 422 FULL.
  • Lighthouse and prysm had no orphans.

What Changed

Each request of a lookup now settles on its own, on what passed the checks, and what it left out is asked again at once.

Where Change
p2p/src/lib.rs PendingColumnRequest tracks which column is on the wire with which peer (asked), what each peer failed to deliver (missed), whether a retry is due, the lookup's id and the block's slot. PendingRequestKind::Columns and RetryDataColumnFetch carry the lookup id. A new ColumnAnswerChecked message. A merge in fetch_missing_columns asks at once for anything nothing open covers.
p2p/src/req_resp/handlers.rs settle_column_report: a request settles only its own columns, only a request that left something out spends an attempt, a short answer never writes its peer off (a failed request or a refused sidecar does), and one retry is due at a time for every leftover. A replaced lookup's report closes nothing and spends nothing, but its invalid sidecars still count against its peer. plan_column_requests: a peer that missed a column is asked again only while the block is at the tip, after the others; uncovered columns go to peers of unknown custody, per column, or once per lookup to one of known custody when none can take them. ask_unasked_columns drops what the store holds first. retire_column_lookup is the one place a lookup ends.
p2p/src/beacon/column_checks.rs check_and_forward_then reports a verdict per sidecar once the checks are done, on every path. The per-block groups (#754) share one Verdicts guard that reports when the last group's task is done; a kept sidecar counts as delivered only once the chain actor has taken it.
p2p/src/metrics.rs, docs/metrics.md lean_data_column_fetch_failures_total{reason="no_custodian"}: a lookup that found nobody connected holding the missing columns, now ended at debug level instead of error! + max_retries. reason="peers_exhausted": a lookup whose every candidate peer failed or sent a sidecar the checks refused.
docs/beacon_wire.md The by-root lookup as it now runs.

Correctness / Behavior Guarantees

  • Bounded. At most MAX_FETCH_RETRIES attempts, with the doubling backoff from 5 ms, spent only by requests that left something out. One retry is due at a time, and it asks only what no open request covers, so failures cannot multiply into fan-outs. Productive rounds spend nothing, but each removes at least one column. Past the tip each candidate is asked for a column once.
  • Settled on verified data. A request's columns count as on the wire until its sidecars have been through the chain checks. Only what passed and reached the chain actor (or was already stored) leaves the lookup, and a refused sidecar writes its sender off.
  • Still forwarded. Sidecars are checked and forwarded whether or not a lookup still waits on them.
  • Replacement-safe. A report or retry timer of a replaced lookup settles nothing of its successor.
  • Quiet when nobody holds a column. It counts as no_custodian (or peers_exhausted when every candidate failed) at debug level and waits for the next per-slot re-ask.
  • Known limit. A column on the wire with a hung peer still waits out the request timeout (10 s); nothing hedges an open request.
  • Fulu benefits too. The same lookup serves held fulu blocks.

Tests Added / Run

  • Arithmetic: failures_across_a_split_lookup_share_one_retry, a_short_answer_has_its_leftover_asked_without_waiting_for_its_siblings, a_custodian_answering_without_its_own_column_is_not_written_off, a_request_that_delivered_its_whole_share_spends_no_attempt, a_report_from_a_replaced_lookup_settles_nothing_of_the_new_one, a_sidecar_the_checks_refuse_stays_missing_and_is_held_against_its_sender, an_undecided_sidecar_stays_missing_without_blaming_its_sender, a_custodian_that_missed_a_column_is_asked_after_the_others
  • Wiring, on a beacon test server: an_answer_settles_its_lookup_once_its_sidecars_are_checked, an_answer_no_lookup_waits_on_is_still_checked_and_forwarded, an_answer_at_the_end_of_the_ladder_gives_the_lookup_up, a_ladder_spent_on_columns_nobody_keeps_ends_as_no_custodian, a_column_no_known_custodian_covers_goes_only_to_a_peer_that_has_said_nothing, a_column_nobody_is_known_to_keep_is_asked_beyond_custody_once_per_lookup, at_the_tip_a_peer_that_answered_short_is_asked_again, past_the_tip_each_peer_is_asked_for_a_column_once, a_retry_asks_only_for_what_the_store_does_not_have, a_retry_for_a_lookup_that_was_replaced_does_nothing, a_column_merged_into_a_lookup_is_asked_at_once, a_re_ask_that_merges_a_column_asks_for_it_at_once, a_re_ask_replaces_a_lookup_that_stopped_asking, a_lookup_reads_the_tip_from_its_held_block_in_the_store, past_the_tip_a_peer_is_still_asked_for_the_columns_it_has_not_missed, a_lookup_every_peer_failed_ends_as_peers_exhausted
  • Column checks: the_callback_reports_what_the_checks_made_of_each_sidecar, the_callback_runs_when_nothing_can_be_checked, a_repeated_column_is_reported_once, verdicts_dropped_early_report_the_rest_as_undecided, a_kept_sidecar_is_delivered_only_once_the_chain_has_it
  • Ran cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p ethlambda-p2p --profile release-fast (406 passed).

Related Issues / PRs

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran clippy with -D warnings on the workspace — clean
  • Ran make test (test-consensus plus test-node, at release-fast) — ethlambda-p2p tests only; CI runs the rest
  • Devnet re-run of the hold distribution — not done

A DataColumnsByRoot lookup is split across the peers that custody its
columns, with the columns no known custodian covers going to one random
peer, but it was retired on the first non-empty answer. Every column asked
of another peer in the same round then waited for the chain actor's next
per-slot re-drive, ~12 s on mainnet timing, however soon it could have come.

An answer now takes its columns off the lookup, and the lookup is retired
only once nothing is left. A short answer ends its request the way a failure
does, so the rest is asked again on the existing backoff ladder; the peer is
charged only if it left out a column it was itself asked for.

Found on a kurtosis devnet (gloas at epoch 2): one ethlambda node had a
known custodian for 2 of its 8 custody columns; about one envelope in twenty
waited ~11.7 s for its columns, which made its PTC vote payload_present=false
for payloads the network had on time and orphaned its next proposal, built on
the EMPTY parent.
@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the column-fetch response path around the new settle/retry logic so the review is grounded in the surrounding control flow.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: settle partial column answers instead of retiring the lookup

Summary: Before this change, the lookup was retired on the first non-empty answer. Any column asked of a sibling peer, or merged in later, then waited for the chain actor's per-slot re-drive (about 12 s). This PR keeps the lookup open and reuses the same in-flight round and retry ladder as the failure path. The change makes sense, and the factoring is clean: close_column_request is shared, and follow_column_attempt is extracted. The unit tests cover the main shapes well. A few concerns follow.

Correctness

  1. Stale answers can now corrupt a newer lookup's in_flight (handlers.rs, ~2603–2618).

    • Some(None) removes the entry as soon as pending.columns is empty, even if in_flight > 0. This happens when a column merged into the lookup lands, or when one peer returns columns from a sibling's share.
    • If fetch_missing_columns then creates a fresh entry for the same root, the old sibling's late answer is applied to the new entry. Its settle_column_answer → close_column_request decrements the new round's in_flight. That can end the new round early with a Retry while its real requests are still on the wire. The result is a double dispatch and an extra attempts charge.
    • The failure path already had this exposure, but answers didn't touch in_flight before, so this PR widens it.
    • Suggested fix: carry a round or generation id in PendingRequestKind::Columns and ignore outcomes whose id doesn't match. At minimum, remove the entry only when in_flight reaches 0, and otherwise just mark it complete.
  2. Columns count as delivered before they are validated (handlers.rs, ~2607).

    • delivered comes straight from sidecars.iter().map(|s| s.index()). A peer could return a junk sidecar with a valid index (bad KZG proof or inclusion proof). That column is then dropped from pending.columns, the peer is not charged, and no retry happens.
    • The old behavior was worse here, since it retired everything. Still, please confirm the earlier filtering in this handler limits sidecars to block_root and to indices in requested. Ideally, verification failures downstream should re-add the column or charge the peer.
    • Relatedly, if unrequested indices can get through, a peer can "complete" a sibling's share. That is what triggers the in_flight > 0 removal described in Item 1.
  3. Charging rule (handlers.rs, ~1672).

    • requested.iter().any(|c| !delivered.contains(c)) charges a peer for omitting a column that a sibling already delivered. That is arguably fair, since the peer was asked for it.
    • However, charging adds the peer to failed_peers, which excludes it from the retry round for columns it may well custody. Consider charging only for omitted columns that are still in pending.columns after the retain. The test a_split_lookup_waits_for_every_share_and_charges_only_the_short_one would still pass.

Observability

  1. Metric meaning has changed.
    • inc_data_column_fetch_failure("max_retries") now also fires when a lookup gives up after repeated short (but non-empty) answers.
    • Consider a distinct label (e.g. "incomplete") so dashboards can tell unresponsive peers apart from partial custody.

Minor

  1. Small contains scans. delivered.contains inside retain and any is O(n·m), which is fine at 128 columns max. No change needed.

  2. Wordy doc comment (lib.rs, 390–394). The comment on PendingRequestKind::Columns is a single long sentence that is hard to parse. Suggested wording: "Carries the subset this request asked for: the answer is judged against it, and it may arrive after a sibling already retired the shared entry."

  3. Missing tests.

    • Nothing covers the handler wiring: the None entry path, which should still forward sidecars, and Some(None) removal while in_flight > 0.
    • Nothing covers short answers reaching GiveUp once attempts is at the max.
    • A test for Item 1 (stale answer arriving after re-creation) would be valuable.

Overall this is a good fix for the 11.7 s envelope stall. Item 1 is the main thing I'd want addressed or explicitly ruled out before merging.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR cleanly fixes a real latency bug where the first peer's partial response in a split DataColumnsByRoot lookup retired the entire PendingColumnRequest, stranding any remaining columns if a sibling request failed or came back short.

The logic in settle_column_answer and the unit tests covering full answers, short answers, split custody lookups, and mid-flight merged columns look solid.

One minor cleanup note:

  1. Possibly stale doc comment on handle_data_column_sidecars_response (crates/net/p2p/src/req_resp/handlers.rs):
    The removed inline comment at line 2553 previously noted:

    see this function's own doc comment for why a partial answer still retires the entry rather than triggering a same-lookup retry for the rest.

    Please check if the doc comment on handle_data_column_sidecars_response (or PendingColumnRequest) still describes the old retire-on-first-partial-answer behavior and update it to match settle_column_answer.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No blocking findings. The change correctly retains incomplete column lookups, retries only missing columns, and avoids charging peers that delivered their full requested share.

Minor documentation nit: handlers.rs:2490 still describes partial responses as completing the lookup. Update it to match the new retry behavior.

Tests could not run because rustup attempted to write outside the permitted filesystem.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found. The new settlement logic correctly:

  • Keeps partially satisfied lookups alive while preserving in_flight accounting.
  • Retries only columns still missing.
  • Avoids penalizing peers that delivered their complete assigned share.
  • Handles late responses after the pending entry has been removed without dropping their sidecars.

The tests in handlers.rs cover the important short-answer, split-request, and merged-column cases.

Non-blocking test gap: consider a handler-level regression test around handlers.rs:2603-2617 for a response arriving when pending_column_requests no longer contains the root, verifying that no retry is scheduled and the sidecars are still forwarded. The current tests exercise settle_column_answer directly but not that integration branch.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Reviewed the PR. The core change looks correct: settle_column_answer removes only delivered indices from pending.columns, leaves the lookup open when columns remain, and only records peer in failed_peers when it omitted a column it was directly asked for. The unit tests cover the important split/merged cases.

A few minor suggestions:

  • settle_column_answer uses delivered.contains(...) inside both retain and any (around lines 1657-1665). This is O(|pending| * |delivered| + |requested| * |delivered|). Fine for normal data-column counts, but a HashSet<u64> built once would be simpler and avoids repeated linear scans.

  • In handle_data_column_sidecars_response (around lines 2603-2620), the settle/retry logic runs before the server.blockchain.is_none() guard. If no blockchain handler is available, a short answer can schedule a retry even though the sidecars won't be processed. Consider moving the guard earlier or making the retry path no-op when server.blockchain is None.

  • The Option<Option<ColumnAttemptOutcome>> from get_mut(...).map(...) is correct but slightly confusing. An explicit enum for Complete vs Incomplete(outcome) would make the match clearer.

  • Tests look good; adding a split case where the second sibling delivers all remaining columns and gets None would lock in the completion path when in_flight > 1.

One thing worth confirming: ensure sidecar payloads are validated before settle_column_answer uses sidecar.index() to mutate pending_column_requests. If validation happens later, a malicious peer could remove columns from the pending set by sending responses with the right indices but invalid payloads. That exposure predates this PR, but this change makes the retry decision depend more directly on those delivered indices.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@pablodeymo

Copy link
Copy Markdown
Collaborator

Review of 45ec390b

This is a read-through review: nothing was built or run. Line numbers are for the PR head, and everything is in crates/net/p2p/src/req_resp/handlers.rs unless noted.

The main concern is that in several timings the new retry path is slower or noisier than the behaviour it replaces. Items 1, 2 and 8 are the ones I would look at first: 1 and 2 undercut the PR's goal, and fixing 8 would remove most of the short answers the retry ladder exists to work around.

Introduced by this PR

1. Leftover columns wait for the whole round (line 1684)

A short answer's leftover is only re-asked once every sibling request has reported, so one hung peer holds it for the 10 s libp2p request timeout (fetch_protocol_config keeps the default). The per-slot re-drive no longer rescues it, because an entry younger than STALE_COLUMN_LOOKUP (8 s) is merged and nothing is sent.

Example: a round starts 5 s into a slot. Random peer R answers short at +20 ms (RoundIncomplete, in_flight = 1) and custodian A never answers. The 12 s tick finds a 7 s old entry and merges. A's request times out at 15 s, and only then is a retry scheduled. Before this PR, R's answer removed the entry and the 12 s tick started a fresh lookup, so this timing is about 3 s worse.

Tracking which columns each open request covers, instead of an in_flight count, would let the leftover be re-asked at once.

2. A custodian that answers short is excluded for the rest of the lookup (line 1674)

It goes into failed_peers, so every retry for the column it left out goes to random non-custodians and the ladder runs out without asking it again.

Example: a gloas envelope is held at the tip and hold_envelope_awaiting_columns asks immediately, racing gossip. Custodian C, asked [c1, c2], has c1 but not yet c2 and returns [c1]. C is charged. The retry 5 ms later excludes C, finds no other known custodian for c2 and picks a random peer, which answers empty and is charged too. With more than about 10 connected peers the pool never empties, failed_peers is never cleared, and after 10 rounds (about 2.6 s of backoff) the lookup gives up with error!. c2 then waits for the next slot re-drive, which is the ~12 s hold this PR set out to remove, plus 9 futile requests. The doc on hold_block_for_columns already notes that peers asked at the tip usually do not have the columns yet.

3. Columns no connected peer holds now cost 10 rounds per slot (line 1603)

Every held root walks the full ladder each slot and ends in error! plus the max_retries metric, where the first custodian answer used to end the lookup silently after one round.

Example, from the mainnet case in the doc on redrive_held_blocks: a held block where 6 of 8 sampled columns have custodians and 2 have none. Custodians deliver their shares, the random peer for the 2 uncovered columns answers empty, and rounds 2 to 10 each ask another random peer for the same 2 columns. Then GiveUp logs "Data column fetch failed after max retries, giving up" and bumps lean_data_column_fetch_failures_total{reason=max_retries}. That is 9 extra requests and one error line per held root per slot until peers churn, and the error names a peer that may have just delivered a valid share.

4. Productive rounds are charged as failures (line 1698)

close_column_request spends an attempt and doubles the backoff for every round that leaves anything, including rounds where every peer delivered its whole share.

Example: a column is merged in after the request went out (the PR's own test). Nobody failed, yet attempts goes from 1 to 2 and the next backoff doubles. With 8 uncovered columns delivered one per round, the wait before round 8 is 640 ms and cumulative backoff is about 1.3 s with no failed request. A node with a high custody count and many uncovered columns hits MAX_FETCH_RETRIES after 10 productive rounds and logs the max-retries error.

5. Stale docs (line 2496, and crates/net/p2p/src/lib.rs:1380)

  • The doc block for the by-root response (which already sits above the range handler) still says a non-empty answer is the lookup's success and that the function does not retry the remainder. The PR deleted the inline comment that pointed at it but left the doc.
  • lib.rs:1380 says handle_column_fetch_failure is the only place an entry is retired.
  • The doc on ColumnAttemptOutcome says "What a failed request means", though answers now produce it.
  • The new PendingRequestKind::Columns doc and the inline comment at line 2603 say a sibling's answer may already have retired the entry. That cannot happen while this request's columns are still listed, since per-round subsets are disjoint.

6. Option<Option<ColumnAttemptOutcome>> (line 2612)

Some(None) means complete and None means no entry, which is easy to swap, and entry removal is now split between this match and the GiveUp arm of follow_column_attempt. A ColumnAttemptOutcome::Complete variant returned by settle_column_answer and handled in follow_column_attempt would leave a single Option for "no entry" and one place that retires a lookup.

7. Tests cover only the pure settle_column_answer (line 3144)

Nothing exercises the wiring in handle_data_column_sidecars_response. Untested paths:

  • removal on a complete answer
  • the no-entry path still forwarding sidecars
  • a GiveUp reached from an answer (attempts = MAX_FETCH_RETRIES)
  • settle running before forwarding
  • a report arriving with in_flight already 0

Dropping the Some(None) removal, which would leave an empty-columns entry that swallows re-drives for 8 s, would pass the suite. Separately, the new column_lookup helper duplicates the struct literals in the tests at lines 3096 and 3128, which could use it.

Pre-existing, but this PR leans on them

8. The fallback peer choice ignores known custody (line 1156)

Columns with no known custodian go to a uniformly random non-failed peer, including peers whose recorded custody excludes those columns. The new ladder converges only by eliminating them one round at a time.

Example, in the PR's devnet shape: 3 peers, two with known custody covering only c1 and c2. The 6 uncovered columns go to pool.choose(), which picks one of the two known-futile peers 2 times in 3. Each such pick is a wasted round with doubled backoff, and the peer is then charged for columns it was known not to hold.

The range path already restricts its speculative ask to peers_of_unknown_custody. The by-root fallback should prefer the same set.

9. delivered is built before the chain checks (line 2607)

It is built from sidecars that have only passed the root and index filter, before check_and_forward runs. A sidecar that later fails the checks still takes its column off the lookup and counts as the peer's whole share.

Example: a peer answers [c3..c8] with gloas sidecars carrying the right beacon_block_root and index but bad cells or proofs (gloas sidecars are unsigned). settle_column_answer removes all six columns, the lookup completes, the peer is not charged, and the checks drop everything. The retry path never engages. The columns wait for the per-slot re-drive, which starts with an empty failed_peers and picks custodians deterministically by min (load, peer id), so the same peer can be chosen every slot. The new "sent its whole share, did nothing wrong" judgment is made on unverified data.

10. Retries do not consult the store (crates/net/p2p/src/lib.rs:1635)

A retry resends pending.columns without checking what the store now holds, so the longer-lived lookup re-requests columns gossip has already delivered.

Example: an envelope is held at the tip and the lookup asks at once. Peers answer short because gossip is still propagating, and within about 300 ms gossip delivers the rest and the envelope is released. The lookup still lists those columns and keeps running rounds until peers return them, each a full sidecar download that chain_checks then drops as AlreadyStored. Related: a slot-tick FetchRequest that arrives while delivered sidecars are still in the check task merges them back and causes one redundant round.

Filtering with store.has_data_column in handle_retry_data_column_fetch would avoid both.

11. Requests carry no round or generation tag (line 1682)

A late report from a lookup that fetch_missing_columns replaced as stale is settled against the new entry and decrements its in_flight. saturating_sub hides the mismatch.

Example: STALE_COLUMN_LOOKUP is 8 s but the request timeout is 10 s. A round with a hung peer starts 3 s into a slot. The 12 s tick sees a 9 s old entry, replaces it and sends n new requests. At 13 s the old request times out or answers and goes through settle_column_answer / close_column_request on the new entry. in_flight drops to n-1, the round closes one report early, a retry fans out while a request is still open, and each later round is off by one. The replacement also discards the failed_peers the old lookup had built up. This predates the PR for failures; the PR routes non-empty answers through the same counter.

…assed the checks

The lookup counted its open requests (`in_flight`) and acted only once the
last one had reported. A short answer's leftover waited out a sibling's
request timeout, productive rounds were charged as failures, a custodian
that answered short at the tip was excluded for the rest of the lookup,
and a replaced lookup's late report closed one of the new lookup's requests.

- Each lookup records which column is on the wire with which peer
  (`asked`), so a request settles only its own columns and whatever it left
  out is asked again at once. A retry asks only what nothing open covers, so
  sibling failures still cannot multiply into a fan-out each; they coalesce
  into the one retry already due (`retry_due`).
- Only a request that left out a column it was asked for spends an attempt
  and doubles the backoff. A whole share spends nothing, and a column merged
  in afterwards is asked at once, from the merge itself too.
- A peer that answers without a column is not written off. It is recorded
  in `missed`. While the block is at the tip (it may not have the column
  yet) a retry asks every other candidate for that column first, then comes
  back to it; past the tip it is not asked for that column again. A failed
  request or a refused sidecar still writes the peer off.
- Columns no known custodian covers go to a peer of unknown custody, as the
  range path already does, rather than to a uniformly random one. Only when
  none is connected are they asked of a peer of known custody, once per
  lookup.
- A lookup with nobody left to ask, or whose ladder ran out on columns no
  connected peer is known to custody, ends at debug level and counts as
  `no_custodian` on `lean_data_column_fetch_failures_total`, not as an
  error and `max_retries`.
- An answer is settled after the chain checks, on what passed them
  (`column_checks::check_and_forward_then` reports each sidecar's verdict
  back to the actor), not on unverified sidecars.
- Before asking again, the columns the store already holds are dropped.
- Requests and retry timers carry their lookup's id, so a replaced lookup's
  report or timer settles nothing of its successor.
- `ColumnAttemptOutcome::Complete` replaces `Option<Option<_>>`, and
  `retire_column_lookup` is the one place a lookup ends.
- Docs: the by-root response doc, `pending_column_requests`,
  `PendingRequestKind::Columns`, `STALE_COLUMN_LOOKUP`, the outcome enum,
  and the new metric reason in docs/metrics.md.
- Tests for the arithmetic, and for the wiring on a beacon test server:
  completion after the checks, the no-entry path still forwarding, giving
  up from an answer, the store check, the merge in `fetch_missing_columns`
  and the stale replacement, the stale report and timer, a short peer asked
  again at the tip and once past it, and the once-per-lookup ask beyond
  custody.
Each request settles on its own after the checks, a peer that answers
short is asked again only while the block is at the tip, the store is read
before asking again, and the columns no known custodian covers go to a
peer of unknown custody, or once per lookup to one of known custody when
none is connected.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Thanks for this. You were right that the round counter was the wrong shape. I reworked the PR's core rather than patching it: ce1dc39 (code, tests, metrics.md) and c03ef90 (beacon_wire.md). In order:

  1. Leftover waits for the round. in_flight is gone. Each lookup records which column is on the wire with which peer (PendingColumnRequest::asked). A report settles only its own request's columns, and what it left out is asked again after the backoff without waiting for siblings. A retry asks only what no open request covers, so sibling failures still can't fan out once each. They share the one retry already due (retry_due). The merge in fetch_missing_columns now also asks right away for anything nothing open covers. Your example (R short at +20 ms, A hung) re-asks R's leftover at about +25 ms. What remains: a column still on the wire with a hung peer waits out that request's timeout. I didn't add hedging. Tests: a_short_answer_has_its_leftover_asked_without_waiting_for_its_siblings, failures_across_a_split_lookup_share_one_retry, a_re_ask_that_merges_a_column_asks_for_it_at_once (in lib.rs, through fetch_missing_columns).

  2. Custodian excluded after a short answer. An answer without a column no longer writes the peer off, whether it was asked as a custodian or speculatively. It goes into missed. While the block is at the tip (its slot is the wall clock's), a retry tries every other candidate for that column first, then comes back to it. Past the tip it doesn't come back, since nothing is still arriving (the same reasoning as hold_block_for_columns). Only a failed request or a refused sidecar puts a peer in failed_peers. Tests: a_custodian_answering_without_its_own_column_is_not_written_off, a_custodian_that_missed_a_column_is_asked_after_the_others, at_the_tip_a_peer_that_answered_short_is_asked_again (the devnet shape).

  3. Uncovered columns spin the ladder. Past the tip, which covers every slot re-drive (your example), each candidate is asked for a column once: each peer of unknown custody, or one peer of known custody if none is connected (see 8). Then the lookup ends at debug! with a new no_custodian reason, not error! + max_retries. Your 2-uncovered-columns case goes from 10 requests per held root per slot to one per connected peer of unknown custody. At the tip, a peer that answered short is still re-asked along the ladder, because there "answered without it" and "doesn't have it yet" look the same. If nobody holds the column there, that is still up to MAX_FETCH_RETRIES requests, but it ends as no_custodian too. Tests: past_the_tip_each_peer_is_asked_for_a_column_once, a_column_nobody_is_known_to_keep_is_asked_beyond_custody_once_per_lookup, a_ladder_spent_on_columns_nobody_keeps_ends_as_no_custodian.

  4. Productive rounds charged. Only a request that left out a column it was asked for spends an attempt and doubles the backoff. A whole share schedules the retry for merged-in columns at zero backoff and spends nothing. a_request_that_delivered_its_whole_share_spends_no_attempt runs 20 such rounds and stays at attempt 1.

  5. Stale docs. Fixed: the by-root response doc (moved back above handle_data_column_sidecars_response and rewritten), the pending_column_requests field doc, the ColumnAttemptOutcome doc and the PendingRequestKind::Columns doc. The inline comment went with the code. I also updated STALE_COLUMN_LOOKUP, the PendingColumnRequest fields, the merge comment in fetch_missing_columns, columns_custodied_by, and the by-root section of docs/beacon_wire.md.

  6. Option<Option<_>>. Now ColumnAttemptOutcome::Complete, with Option meaning only "no lookup". Every retirement goes through retire_column_lookup, which logs and counts by reason.

  7. Wiring tests. The answer path is split into a ctx-free take_column_answer and report_column_request, and the tests drive them on unconnected_beacon_server with real request ids and the column checks' recording chain:

    • Complete removes the entry: an_answer_settles_its_lookup_once_its_sidecars_are_checked.
    • No entry still forwards: an_answer_no_lookup_waits_on_is_still_checked_and_forwarded.
    • GiveUp from an answer: an_answer_at_the_end_of_the_ladder_gives_the_lookup_up.

    Two of your cases changed meaning. "Settle before forward" no longer holds by design (see 9): settling now follows the checks, and the test asserts that nothing settles and the column stays on the wire until the verdict comes back. "A report with in_flight already 0" became "a report from a request the lookup doesn't have", covered by a_report_from_a_replaced_lookup_settles_nothing_of_the_new_one, a_retry_for_a_lookup_that_was_replaced_does_nothing and a_re_ask_replaces_a_lookup_that_stopped_asking. The older tests use column_lookup too.

  8. Fallback ignores custody. Uncovered columns now go to a peer of unknown custody, as the range path does. Only when none is connected does the lookup ask one peer of known custody, once per lookup. That keeps main's floor of one ask per slot for a peer that may hold more than it custodies, without walking every known peer a round at a time. Test: a_column_no_known_custodian_covers_goes_only_to_a_peer_that_has_said_nothing.

  9. Settling on unverified sidecars. column_checks::check_and_forward_then reports a verdict per sidecar once the checks are done: delivered (kept or already stored), rejected, or undecided (parked, ignored, or not checked). The callback runs on every path, including no chain actor, a permit error and a panicked check. It messages the actor, which settles on what passed. A refused sidecar stays missing and writes its sender off. Tests: the_callback_reports_what_the_checks_made_of_each_sidecar, the_callback_runs_when_nothing_can_be_checked, a_sidecar_the_checks_refuse_stays_missing_and_is_held_against_its_sender.

  10. Retries ignore the store. Before asking again (retry or merge), the lookup drops what store.has_data_column finds, and retires as complete if nothing is left. The slot comes from block_entry once per lookup and is cached, and it also decides "at the tip" for 2. Test: a_retry_asks_only_for_what_the_store_does_not_have.

  11. No generation tag. Each lookup has an id, carried by PendingRequestKind::Columns and by the retry timer. A late report from a replaced lookup only takes off columns it actually delivered; it closes nothing and spends nothing. A stale timer does nothing. The replacement still starts with empty failed_peers, which I kept: it's a fresh lookup a slot later. Tests as in 7.

I haven't re-run the devnet. What would show the effect is the hold distribution of held envelopes and lean_data_column_fetch_failures_total by reason, compared with the 37/758 and 29/754 in the description.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

PR Review: per-request settlement for DataColumnsByRoot lookups

The redesign is sound. Settling each request on its own, on post-check verdicts, with lookup ids to fence off replaced lookups, fixes real problems:

  • Partial answers used to retire the whole lookup.
  • Sibling failures could fan out.
  • A sidecar with the right root and index was trusted without checking its cells and proofs.

Test coverage is thorough. The points below are mostly robustness and observability concerns.

Correctness / robustness

  1. done is not guaranteed to run (column_checks.rs, check_and_forward_then).

    • The doc promises "done runs on every path". That only holds if the spawned task reaches run_done.
    • A panic outside spawn_blocking would skip it. That includes the forwarding calls into blockchain, or anything else in the async body.
    • If it is skipped, the request's columns stay in asked until STALE_COLUMN_LOOKUP.
    • Suggested fix: a small drop guard holding Option<Done> plus the indices. On Drop it reports anything not yet reported as Undecided. That makes the invariant structural instead of path-by-path.
  2. Delivered is reported even if forwarding failed (column_checks.rs, around L140–152).

    • ChainVerdict::Keep maps to Delivered before new_data_column_sidecars(...) runs.
    • If that send fails (only a warn!), the lookup still retires as Complete, yet the column never reached the chain actor.
    • The per-slot re-ask recovers, but the lookup state lies in the meantime.
    • Consider downgrading the kept batch to Undecided when the forward errors.
  3. The at_tip window may almost never be open (handlers.rs, plan_column_requests / drop_stored_columns).

    • at_tip requires pending.slot to be Some and slot >= wall_slot.
    • slot is only filled by drop_stored_columns, and only if store.block_entry finds the block.
    • Two cases close the window:
      • If held (not-yet-imported) blocks are not in Store, slot stays None forever. at_tip is then always false, and a peer that answered short is never re-asked.
      • Even when slot is set, hold_block_for_columns waits a slot before asking (per the docs), so slot >= now is usually already false.
    • The tests set pending.slot by hand, so they don't exercise this.
    • Please confirm the block is in Store at this point. Also consider slot + 1 >= now, or seeding slot from the caller, who knows it.
  4. Zero-backoff retry can carry a missed column past the ladder (settle_column_report).

    • Sequence:
      • A whole-share report sets retry_due with Duration::ZERO.
      • A later short report hits the early pending.retry_due return.
      • It therefore spends no attempt and gets no backoff.
      • Its leftover is re-asked immediately on the zero-delay retry.
    • This is still bounded (failed/missed filtering, staleness), so it is harmless. It is worth a comment, since the doc says "Only a request that left out a column … spends an attempt".
  5. Replaced-lookup reports never penalize the peer (settle_column_report, the request.lookup != pending.lookup branch).

    • A Rejected verdict from an old lookup's request is dropped on the floor.
    • That is probably fine, since gossip-level scoring should cover it. Just flagging that invalid-sidecar senders escape failed_peers in this window.

Observability

  1. no_custodian is misattributed (ask_unasked_columns).

    • When every connected peer is in failed_peers, !sent && !open retires with Retirement::NobodyToAsk and increments no_custodian.
    • That case is "all peers failed", not "nobody holds the columns".
    • The old code cleared failed_peers and retried the full set; that fallback is now gone, so this path will fire more often.
    • Consider a distinct reason, e.g. peers_exhausted, or reuse max_retries.
  2. excluded now counts failed.len() (plan_column_requests).

    • That set can include disconnected peers, so the log field no longer means "connected peers excluded". Minor.

Nits

  • handlers.rs: there is no blank line between the closing } of drop_stored_columns and the next /// The beacon wire… doc comment.
  • fetch_data_columns_from_peer: the ColumnIndices::try_from error changed from return false to continue. That is fine, but the comment still says "this does not retry". Those columns now stay in unasked and will be re-planned on the next report or retry, hitting the same error. The error is unreachable in practice (≤128 columns), so a debug_assert! might be clearer.
  • ColumnReport::delivered and settle_column_report use Vec::contains in loops. That is fine at ≤128 columns.
  • Good catch fixing the previously duplicated/misplaced doc comment above the DataColumnsByRange handler.

Overall this is a clear improvement. Item 1 and Item 3 are the ones I'd want addressed or confirmed before merge.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Filter missed columns per peer, rather than excluding the peer for the entire batch — handlers.rs:1308. Past the tip, a peer that previously omitted any uncovered column is excluded from serving all uncovered columns. For example, if the only peer omitted column 1 and column 2 is merged during backoff, the retry excludes that peer from serving column 2 despite never having requested it. The lookup then retires, delaying block availability until the next slot’s re-ask. Select eligible peers per column, requesting only columns each peer has not already missed; apply the same correction to the fallback at line 1319.

Tests could not run because rustup attempted to write to its read-only temporary directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the column-lookup settlement paths and the surrounding fetch/retry code for correctness bugs.The lookup-id and retry paths look sensitive to re-entrancy. I'll read those functions next.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks solid. The transition from retiring a DataColumnsByRoot lookup on the first non-empty response to settling each split request individually after column_checks completes is well-designed and addresses several subtle failure modes:

  • Post-verification settlement (column_checks.rs, handlers.rs): Settling lookups via the ColumnAnswerChecked callback ensures columns remain marked as in-flight (pending.asked) during KZG/chain verification and that invalid (Rejected) sidecars neither satisfy the lookup nor exonerate the sender.
  • Lookup generation tracking (PendingColumnRequest::lookup): Tagging outbound requests and scheduled retries with a monotonically increasing lookup ID cleanly prevents late responses or timers from a replaced stale lookup (STALE_COLUMN_LOOKUP) from mutating the replacement lookup's retry ladder or in-flight state.
  • Per-request retry & peer selection (settle_column_report, plan_column_requests): Separating missed (peer, column) pairs from failed_peers, gating re-asks of peers that answered short on at_tip, capping beyond_custody fallbacks to once per lookup, and pruning gossip-delivered columns via drop_stored_columns before retrying all make sense and avoid burning retries on uncustodied columns.

The unit test coverage across column_checks.rs, lib.rs, and handlers.rs is thorough, and the documentation/metrics updates in beacon_wire.md and metrics.md are clear. LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

No consensus-layer state-transition/fork-choice logic is touched by this PR; the changes are confined to p2p data-column lookup bookkeeping.

  • Bug: check_and_forward_then can report Delivered for sidecars that were never forwarded.
    In crates/net/p2p/src/beacon/column_checks.rs, verdicts is populated with ColumnVerdict::Delivered for ChainVerdict::Keep before the batch is sent to the chain actor. If blockchain.send_checked_columns(keep).await fails, the callback still receives Delivered, so settle_column_report removes those columns from the lookup even though the chain actor never got them. The verdicts should be adjusted to Undecided for those sidecars on send failure, or the Delivered verdict should be produced only after a successful send.

  • Bug: no_peers metric is no longer incremented.
    retire_column_lookup in crates/net/p2p/src/req_resp/handlers.rs has a Retirement::NoPeers arm that only logs; it does not call metrics::inc_data_column_fetch_failure("no_peers"). Since ask_unasked_columns now retires lookups with Retirement::NoPeers, the documented no_peers reason will go uncounted. Add the metric there.

  • Unused variable/dead field.
    let excluded = plan.excluded; in fetch_data_columns_from_peer does not appear to be used, and the excluded field on ColumnPlan may be entirely dead. This will at least produce an unused-variable warning; remove it or log it if it was intended for diagnostics.

  • All-peers-failed may be classified as no_custodian.
    In ask_unasked_columns, the if !sent && !open branch retires with NobodyToAsk whenever server.connected_peers is non-empty. That includes the case where the lookup has connected peers but they are all in failed_peers. The old code recognized this and retried with the full peer set after the pool was exhausted; the new code never asks a failed peer again within the same lookup. If this is intentional, it is fine behaviorally, but the metric/reason should not call it no_custodian, since a custodian may exist but have failed. Consider returning GaveUp/max_retries or clearing failed peers when no other candidate remains.

  • Possible duplicate fetches for AwaitParent sidecars.
    ColumnVerdict::from maps ChainVerdict::AwaitParent to Undecided, so a response containing only sidecars parked waiting on a parent will leave those columns missing and cause the lookup to retry them. If the same peer is selected again, it may download and park the same sidecars multiple times. This may be acceptable but is worth considering whether AwaitParent should be treated as “not missing for fetching purposes” or recorded more strongly in missed to avoid duplicate parked sidecars.

  • Minor: verdict ordering can change on semaphore failure.
    In check_and_forward_then, if semaphore acquisition fails mid-batch, it pushes Undecided verdicts for the current and remaining sidecars before later pushing verdicts for already-spawned checks. The callback’s Vec order therefore may not match the input order. The callback uses column indices, so this is likely harmless, but it should either be documented as unordered or sorted.

The rest of the refactor is well-structured, especially the stale-lookup generation id preventing late old requests from spending the new lookup’s attempts.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

The lookup-state redesign is generally coherent, and the tests cover late reports, retries, stale lookups, and callback failures well. Two edge cases are worth addressing:

  1. Keep is reported as Delivered before the chain-actor handoff succeeds
    crates/net/p2p/src/beacon/column_checks.rs:132-155 records ColumnVerdict::Delivered while collecting verdicts, but forwarding errors afterward are only logged. settle_column_report can consequently remove those columns and retire the lookup as Complete even though the sidecars never reached the chain actor. If the batch send fails, downgrade that batch’s Keep verdicts to Undecided so the lookup remains retryable. A test with a closed RecordingChain channel would cover this.

  2. Parked sidecars can be downloaded repeatedly without making progress
    ColumnVerdict::from maps ChainVerdict::AwaitParent to Undecided, and settle_column_report therefore includes the column in left_out and retries it. However, check_and_forward_then has already handed that sidecar to the chain actor to park until its parent is ready. Until then, every retry is likely to produce the same outcome, consuming bandwidth and retry attempts on each subsequent lookup. Consider treating a successfully parked sidecar as delivered for lookup purposes, or otherwise suppressing re-fetches while it remains parked.

No other blocking correctness or security issues stood out.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

…ndelivered keep

The completion callback ran from the end of the check task, so a panic
outside the blocking checks or a task ended early left the lookup with its
columns on the wire until it went stale. Verdicts now report when dropped,
with anything not judged as undecided.

A kept sidecar counted as delivered even when handing it to the chain
actor failed, so the lookup could retire as complete for a column the
chain never got. Those are now reported undecided, and asked again.
Past the tip, a peer of unknown custody that had answered without one
uncovered column was passed over for all of them, so a column merged in
after its short answer went to nobody and waited for the next slot's
re-ask. Uncovered columns are now assigned per column: a peer takes the
ones it may still be asked for, and another peer the rest. The once-per-
lookup ask beyond custody follows the same rule.

Also from the third review round:
- A replaced lookup's report that carried an invalid sidecar writes its
  sender off for the new lookup too; a failure from it still does not.
- The plan's `excluded` counts only connected peers.
- `settle_column_report` documents that a short report riding on a
  zero-backoff retry spends nothing, and why that is bounded.
- The unreachable `ColumnIndices` error says so, and a missing blank line.
- Tests: the per-column assignment past the tip, the invalid stale report,
  and a lookup reading the tip from its held block in the store.
…usted

A lookup that had written off every connected peer (a failed request or
an invalid sidecar) and had nobody left to ask counted as no_custodian,
which says nobody holds the columns. Nobody was left to say. It now
counts as peers_exhausted on lean_data_column_fetch_failures_total; the
next slot's re-ask starts a fresh lookup that asks them again.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round three (the automated review of c03ef90). Fixed in fd24089, 2c51d42, 4bae913 and f31e521. make fmt and make lint are clean, and cargo test -p ethlambda-p2p passes (398 passed, 0 failed, 1 ignored).

Fixed

  • Claude 1, the verdict callback running on every path: the verdicts are now a drop guard (Verdicts in column_checks.rs). The callback fires when it drops, so a panic outside the blocking checks or a task ended early still settles the lookup, with anything not judged as undecided. Tests: verdicts_dropped_early_report_the_rest_as_undecided; the existing the_callback_runs_when_nothing_can_be_checked still passes.
  • Claude 2, DeepSeek 1, Kimi 1, a keep that never reached the chain actor: if handing the kept batch over fails, those verdicts go back to undecided, so the lookup asks again rather than retiring as complete. The guard's side is covered by a_kept_sidecar_the_chain_never_got_is_not_delivered. The hand-off failure itself is not exercised: no test here can build a sidecar the checks keep (that takes KZG-valid cells and a parent state).
  • Codex, uncovered columns past the tip: a peer that had missed one uncovered column was passed over for all of them, so a column merged in after its short answer went to nobody. They are now assigned per column: a peer takes the ones it may still be asked for, and another peer the rest. The once-per-lookup ask beyond custody follows the same rule. Test: past_the_tip_a_peer_is_still_asked_for_the_columns_it_has_not_missed.
  • Claude 6, DeepSeek 4, all peers failed counted as no_custodian: that case now counts as a new peers_exhausted reason (metrics.md updated). I didn't bring back clearing failed_peers inside the lookup; that was the spin round two removed, and the next slot's re-ask starts fresh. Test: a_lookup_every_peer_failed_ends_as_peers_exhausted.
  • Claude 5, a replaced lookup's invalid sidecar: its sender is now written off for the new lookup too. A failure from the old lookup still isn't held against the peer, since it may be what got the old lookup replaced. Covered in a_report_from_a_replaced_lookup_settles_nothing_of_the_new_one.
  • Claude 4, the zero-backoff retry: documented on settle_column_report. A short report riding on a whole share's zero-backoff retry spends nothing, which is bounded: each such retry takes a whole share, and the next short report after it spends an attempt.
  • Claude 7 and nits: excluded counts only connected peers; the unreachable ColumnIndices error says so; the missing blank line is fixed.

Not changed

  • Claude 3, the at-tip window: it is open where it matters. A held envelope's block is imported (its signature check reads that block's state) and the envelope is asked at once, at the tip. A held fulu block is in the store as pending and is asked past the tip by design. slot + 1 would re-ask a peer a slot later, with nothing still in flight. a_lookup_reads_the_tip_from_its_held_block_in_the_store now covers reading the slot from the store with nothing set by hand.
  • DeepSeek 2, no_peers: still counted once, in fetch_data_columns_from_peer, on the same condition that picks NoPeers.
  • DeepSeek 3, excluded: it is logged on every send.
  • DeepSeek 5, Kimi 2, parked sidecars: a lookup's block has a post-state by construction, so its sidecars don't park short of a race. Counting a parked, unverified sidecar as delivered would undo settling on what passed.
  • DeepSeek 6, verdict order: documented as unordered; the guard no longer has a separate semaphore path.
  • Claude nit, Vec::contains: kept, at most 128 columns.

Gemini found nothing to change, and Grok's comment ended after its preamble with no findings.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The changes preserve sidecar validation before completing lookups, isolate replies from replaced lookups, and retry missing columns without duplicating active requests.

Tests could not run: Rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: DataColumnsByRoot lookups settled per request on checked verdicts

The direction is right. Settling each request on what passed the chain checks, rather than on what the peer sent, fixes a real data-availability hole: a matching root and index used to be enough to take a column off the lookup. The Drop-based Verdicts guard is a good way to make sure every path reports back. Tagging requests with a lookup id also closes the stale-lookup race that the 8 s window versus the 10 s request timeout made possible. Test coverage is thorough.

Remaining points, roughly by importance:

Correctness / robustness

  1. A peer can avoid blame by sending sidecars that hit an Ignore (column_checks.rs, impl From<&ChainVerdict> for ColumnVerdict)

    • Every Drop(Outcome::Ignore(_)) except AlreadyStored maps to Undecided, so the sender is never added to failed_peers.
    • A gloas sidecar carries no signature, so a peer can return the requested block_root/index with, say, a future or finalized slot. That lands on Ignore, and the peer is only recorded in missed.
    • At the tip, missed peers are re-asked (sorted last, but still re-asked), so such a peer can keep burning rungs of the ladder.
    • The cost is bounded by MAX_FETCH_RETRIES, so this is not critical. Consider treating ignores that are impossible for an honest answer to a by-root request as Rejected, e.g. a slot that disagrees with the held block's pending.slot when it is known.
  2. retry_due can stick (handlers.rs, settle_column_report / retry_column_lookup)

    • Once retry_due = true, only retry_column_lookup clears it.
    • If the RetryDataColumnFetch timer never fires (actor shutting down, mailbox closed), every later short answer returns Waiting.
    • fetch_missing_columns' merge path also skips ask_unasked_columns, so the lookup sits idle until STALE_COLUMN_LOOKUP.
    • The outage is bounded at 8 s. Still, worth a comment, or having the merge path also ask when retry_due has been pending longer than the expected backoff.
  3. GiveUp retires the lookup while sibling requests are still open (settle_column_report, around the attempts >= MAX_FETCH_RETRIES branch)

    • Later reports from those siblings find no entry. Their sidecars still reach the chain actor, but the next slot's re-ask starts a fresh lookup.
    • That fresh lookup will re-request columns that are already in flight or being verified, so the same data is downloaded twice.
    • Consider deferring GiveUp until pending.asked is empty, or at least have the new lookup consult the store first, which ask_unasked_columns does on retries but the initial fetch_data_columns_from_peer does not.
  4. The "at the tip" test is strict (plan_column_requests, slot >= now)

    • A block from wall_slot - 1 that arrived late, with columns still propagating, counts as past the tip. Its short-answering peers are then never re-asked.
    • A lookup whose block is not in the store (slot == None) is treated the same way.
    • Consider a one-slot tolerance (slot + 1 >= now). This matches the reasoning in hold_block_for_columns, which waits a slot.
  5. Verdicts::not_delivered can flip the wrong entry for duplicate indices (column_checks.rs)

    • If a response carries the same index twice, one AlreadyStored and one Keep, a failed forward flips both to Undecided.
    • This is harmless; the column is just re-asked. Only worth fixing if retain_requested_columns doesn't already dedupe.

Performance

  1. Verdicts::new allocates on the gossip path (column_checks.rs)

    • It collects unjudged even when done is None, which is the case for every gossip and range call through check_and_forward.
    • The cost is small, but trivially avoidable: let unjudged = if done.is_some() { ... } else { Vec::new() };.
  2. drop_stored_columns runs synchronous DB reads on the actor thread (handlers.rs)

    • It does up to 128 has_data_column lookups per retry or merge, plus one full beacon-block decode per lookup.
    • Probably fine, but note that fetch_missing_columns' merge path now does this on every re-ask, not only on retries.

Tests

  1. Callback test asserts an exact order (the_callback_reports_what_the_checks_made_of_each_sidecar)

    • It compares a Vec in exact order, while the doc for Done says "in no particular order".
    • The order is deterministic today (sequential JoinHandle awaits, unjudged entries appended last), but sorting before comparing would match the documented contract.
  2. Metric assertions can pass because of other tests (several tests using metrics::data_column_fetch_failures_total(..) > before)

    • The counters are process-global, so a parallel test bumping the same label can make the assertion pass even if the code under test didn't increment it.
    • Low risk, but a scoped or registry-local counter would make these assertions meaningful.
  3. Test helpers exposed across modules (column_checks.rs)

    • pub(crate) mod tests exists to share RecordingChain and Forwarded. Moving them into test_support would keep the tests module private.

Nits

  • server.blockchain.clone().filter(|_| !sidecars.is_empty()) reads oddly. An explicit if sidecars.is_empty() { return; } ahead of the let Some(...) is clearer; verdicts still reports on drop either way.
  • ColumnAttemptOutcome::Retry { attempts: pending.attempts, backoff: ZERO } reports the current attempt rather than a spent one. That is fine, but the debug log "scheduling retry" with that count may confuse whoever reads it. Consider logging "free retry" for the zero-backoff case.

Nothing here touches fork choice or the state transition. The changes are confined to data-availability fetching, and the semantics are a clear improvement. Items 1 and 3 are the ones I'd address before merge.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll read the settlement and lookup paths in context to verify a few behaviors the diff only shows partially.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR is well-designed and thoroughly tested. Moving lookup settlement to a post-verification callback (check_and_forward_then with the Verdicts drop guard) and tracking in-flight columns per (column, peer) with lookup IDs cleanly solves premature lookup retirement on partial responses, unverified sidecar acceptance, stale-lookup races, and retry-ladder waste on uncustodied columns.

What looks good

  1. Drop-safe verification reporting (crates/net/p2p/src/beacon/column_checks.rs): Verdicts guarantees via Drop that done is always invoked (defaulting unjudged columns to ColumnVerdict::Undecided on early exit/panic, and downgrading Delivered to Undecided via not_delivered if forwarding to the chain actor fails), preventing columns from getting stuck in pending.asked.
  2. Per-column settlement and retry coalescing (crates/net/p2p/src/req_resp/handlers.rs): settle_column_report and PendingColumnRequest properly separate wire/validation faults (failed_peers) from missing columns (missed), coalesce concurrent failures onto a single scheduled retry (retry_due), and avoid burning retry attempts when a request delivers its full requested subset.
  3. Lookup generation IDs (PendingColumnRequest::lookup): Tagging outbound requests and retry messages with the lookup ID ensures late responses/timeouts from a lookup replaced after STALE_COLUMN_LOOKUP still deliver valid columns without perturbing the new lookup's state or retry budget.
  4. Tip-aware peer selection and store pre-checks (plan_column_requests, drop_stored_columns): Checking the store before retrying avoids re-fetching columns that arrived over gossip, and bounding beyond-custody / past-tip retries prevents walking the full retry ladder every slot when no connected peer holds a column.

No issues found — LGTM.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The rework is thorough, with good docs and test coverage around the new verdict/retry model. A few issues worth addressing:

  • Compile/ownership issue in crates/net/p2p/src/req_resp/handlers.rs
    In follow_column_attempt, the diff shows:

    match *outcome { ... }

    where outcome: &ColumnAttemptOutcome and ColumnAttemptOutcome is only #[derive(Debug, PartialEq, Eq)], not Copy. This attempts to move out of a shared reference and should fail to compile. Match on outcome directly, clone it, or derive Copy.
    Similarly, ColumnReport::peer_at_fault uses matches!(self.reply, ColumnReply::Failed), which reads like a move from &self; prefer matches!(&self.reply, ColumnReply::Failed) unless ColumnReply becomes Copy.

  • Retirement::NoPeers no longer increments the documented metric
    In retire_column_lookup, the Retirement::NoPeers arm only logs:

    Retirement::NoPeers => {
        debug!(%block_root, "No connected peers to ask for this lookup's columns");
    }

    docs/metrics.md still lists reason=no_peers, and metrics.rs still documents it. This path should call metrics::inc_data_column_fetch_failure("no_peers"), otherwise the metric silently stops being emitted.

  • Unused excluded bookkeeping
    ColumnPlan has an excluded field, and fetch_data_columns_from_peer immediately binds it with:

    let excluded = plan.excluded;

    but the variable is never used. This will warn under #![deny(warnings)] and is dead code. Either use it in a debug/trace line or remove the field.

  • Duplicate sidecar indices can produce duplicate verdict entries
    Verdicts::record removes an index from unjudged but does not prevent the same index from being recorded twice. A malicious or buggy peer can send multiple sidecars for the same column, resulting in duplicate (index, verdict) reports from the completion callback. Downstream delivered() / sent_invalid() are mostly idempotent, but the report shape is untidy and can mask a peer sending duplicates. Consider a debug_assert or skipping an already-recorded index.

  • Stale requests do not record missed for short answers
    In settle_column_report, the request.lookup != pending.lookup branch removes delivered columns but only records the peer in failed_peers for invalid sidecars. It does not insert (peer, column) into pending.missed for non-delivered requested columns. That may be intentional as a fresh replacement lookup, but it means a replacement lookup past the tip can immediately re-ask a peer that just said it did not have the column. Worth confirming this is the desired behavior.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

Overall, the lookup-ID fencing, post-validation settlement, and partial-response handling look sound. The tests cover the important stale-request, invalid-sidecar, and retry cases. I found no blocking consensus or security issue, but there are a few non-blocking problems:

  1. Panic paths can falsely report a column as delivered
    crates/net/p2p/src/beacon/column_checks.rs:176-193 records ChainVerdict::Keep as ColumnVerdict::Delivered before new_data_column_sidecars succeeds. If the task panics after recording but before or during forwarding, Verdicts::drop reports Delivered, potentially completing the lookup even though the chain actor never received the column. This contradicts the guard’s panic-safety guarantee.
    Consider recording Keep as Undecided initially and changing it to Delivered only after forwarding returns Ok; AlreadyStored can still be recorded as delivered immediately.

  2. Some no_peers abandonments are not counted
    In handlers.rs, retire_column_lookup’s Retirement::NoPeers arm only logs. If peers disconnect after a lookup has started and a subsequent ask_unasked_columns finds nobody connected, the lookup is retired without incrementing lean_data_column_fetch_failures_total{reason="no_peers"}. The initial-send path does not cover this case. Add the metric increment here, ideally with a disconnect-between-retries test.

  3. Unused planning value
    In handlers.rs around fetch_data_columns_from_peer (let excluded = plan.excluded;, approximately line 1174), excluded is never read. This produces a compiler warning and makes ColumnPlan::excluded dead bookkeeping. Remove it or restore the diagnostic that was meant to consume it.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

# Conflicts:
#	crates/net/p2p/src/beacon/column_checks.rs
#	docs/metrics.md
@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 19, 2026
@pablodeymo

Copy link
Copy Markdown
Collaborator

Re-read the merge of main at 597306c. The approval stands; two points from my earlier pass were not written down with it, so they are here. Neither blocks the merge.

The merge

Only column_checks.rs and docs/metrics.md conflicted, and column_checks.rs was a rewrite rather than a pick of one side: the lookup's verdicts now go through #754's per-block group tasks, which share one Verdicts. I read it for the following and found nothing wrong. I did not run the tests locally.

  • Every way out still reports to the lookup: an empty call, no chain actor, a precheck panic, a KZG sub-batch panic and a closed semaphore all end as Undecided.
  • A kept sidecar is Delivered only once the chain actor's mailbox took it, which replaces not_delivered.
  • A column repeated within one answer is now judged once, on its first copy, where before the merge each copy was judged. That matches perf(p2p): verify one KZG batch per block for fetched data columns #754's rule.

One nit: Verdicts keeps one verdict per column index, while group_by_block dedups per (block, index). That is safe today because retain_requested_columns keeps only the requested block before the checks, so a by-root call is always one block. A line on check_and_forward_then saying done assumes a single block would keep a later caller from getting the first block's verdict for an index two blocks share.

Still open

1. A stale lookup is replaced just before its hung request would fail over.

STALE_COLUMN_LOOKUP is 8 s and the request timeout is 10 s. In fetch_missing_columns, a re-drive that arrives when last_asked is 8 to 10 s old removes the entry and starts a new lookup with an empty failed_peers. With per-request tracking, an entry in that state is one whose request is still on the wire and at most 2 s from timing out and marking its peer.

The constant's doc treats "shorter than the request timeout" as harmless because the old request's late report settles nothing of the new lookup (a_report_from_a_replaced_lookup_settles_nothing_of_the_new_one). The cost is on the other side: that late failure also marks nobody, so the new lookup can pick the hung peer again. plan_column_requests picks at random among the peers not failed, so it does whenever that peer is the column's only custodian, and the column then waits another full timeout.

Either of these would close it:

  • Call an entry stale only when nothing is on the wire and no retry is due (asked.is_empty() && !retry_due).
  • Raise the constant above the request timeout.

2. The devnet was not re-run on the rewritten design. The PR's goal is fewer long holds, and that is still unmeasured since the rework.

A follow-up issue for the first is fine by me.

Merged via the queue into main with commit 68a8fc0 Nov 19, 2026
14 checks passed
@MegaRedHand
MegaRedHand deleted the fix/column-lookup-partial-answer branch November 19, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants