Skip to content

fix: account for pending handshake messages when streams stop (#3690) - #3702

Open
He-Pin wants to merge 1 commit into
mainfrom
fix-3690-handshake-pending-message
Open

He-Pin wants to merge 1 commit into
mainfrom
fix-3690-handshake-pending-message

Conversation

@He-Pin

@He-Pin He-Pin commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Motivation

OutboundHandshake can retain an envelope after removing it from the send queue. Stopping the stage must account for that envelope without quarantining an orderly local shutdown, attributing the message to a newer remote UID, or sending quarantine notifications through the control stream that is stopping.

Modification

  • Pass held envelopes and the UID observed when they were held to an Association callback. Keep the existing constructor and transport method signatures available.
  • Share pending-system-message accounting between the control queue and handshake stage. Wait for both cleanup callbacks, install the replacement control queue, and then apply quarantine policy for the matching known UID.
  • If SystemMessageDelivery already quarantined the same peer during teardown, send the notification through the replacement control ingress. Ordinary and large lanes only report their held envelopes to dead letters.
  • Always remove the handshake listener and run superclass cleanup in finally. Ensure test subscriptions are also removed in finally.
  • Exercise real Association, SendQueue, handshake, delivery, control-junction and encoding stages with controlled transport demand. Cover orderly and unexpected stops, unknown and changing UIDs, first-handshake completion, notification output, and initialization failure.
  • Narrow the documentation to the control stream and the queue/handshake buffers this change accounts for.

Result

Held envelopes reach dead letters. Pending system messages in these buffers trigger quarantine after an unexpected control-stream stop for the matching known remote UID; their cleanup during local shutdown does not quarantine. Notifications use the replacement control stream. The wire protocol is unchanged.

Tests

  • JDK 17: sbt "remote / Test / testOnly org.apache.pekko.remote.artery.OutboundHandshakeSpec org.apache.pekko.remote.artery.OutboundHandshakePendingMessageSpec org.apache.pekko.remote.artery.OutboundHandshakeAssociationSpec" — all 21 passed on the final shared integration tree; the standalone Association suite also passed all 11 tests.
  • Adjacent SystemMessageDeliverySpec, OutboundIdleShutdownSpec and HarmlessQuarantineSpec — 17 passed after the lifecycle/notification changes, before the final held-UID refinement.
  • Running the 11 Association regressions against the exact previous head (10ca75f) produced 8 passes and 3 expected failures: orderly shutdown quarantined, an old held message quarantined a newer UID, and delayed completion observed an earlier old-UID notification. The last assertion does not establish quarantine of the new UID.
  • A separate new-incarnation regression failed when the UID was captured only during postStop and passed after capturing it when the handshake stage holds the envelope.
  • Removing the listener-cleanup finally caused the exception-path regression to fail; restoring it passed all 3 pending-message tests.
  • A delivery-plus-handshake regression failed before the replacement-stream re-notification fix and passed after it. This failure was established against the intermediate callback implementation; the exact original-head run did not deterministically lose that notification.
  • sbt "project remote" headerCreateAll sortImports "+mimaReportBinaryIssues" "+headerCheckAll", native scalafmt --mode diff-ref=origin/main, and git diff --check passed. Remote MiMa and headers passed on Scala 2.13.18 and 3.3.8.
  • sbt docs/paradox passed on the preceding shared integration tree (d0863b); the documentation is identical in the final integration tree (b6f826f1).
  • Historical shared sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequest predates this follow-up. Its ScalaTest summary was 8,600 passed/3 failed (two TCP timeouts and a Netty address-bind failure); four persistence modules could not run because the LevelDB JNI binary was incompatible with Darwin arm64, and the Docker DNS test was pending. The two TCP cases passed when independently repeated on that earlier final artifact. This historical run is not validation of the revised implementation.
  • Final full-diff Qoder review — No must-fix findings.
  • No local full or broad validatePullRequest rerun, as requested.

References

Fixes #3690

@pjfanning pjfanning added this to the 2.1.0 milestone Oct 25, 2026
@He-Pin
He-Pin force-pushed the fix-3690-handshake-pending-message branch from cab0464 to 10ca75f Compare October 25, 2026 10:44
@He-Pin He-Pin changed the title Do not drop the pending message in OutboundHandshake.postStop fix: account for pending handshake messages on shutdown (#3690) Oct 25, 2026

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few concerns with the new postStop handling, plus some test and doc notes. Details are inline.

}

override def postStop(): Unit = {
pendingMessage match {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This repeats the policy in Association.sendQueuePostStop, and the copies already behave differently: SendQueue quarantines only for the control stream, but this quarantines for every lane (ordinary and large streams too), and the reason strings and counting differ. Could OutboundHandshake take a postStop callback, like SendQueue's postStopAction, so there is one place that decides between dead letters and quarantine?

More broadly, this covers only the one envelope held by this stage. System messages already pulled into other buffered or async stages upstream of SystemMessageDelivery can still be dropped silently when the stream stops. A general check when the outbound control stream terminates might be a more complete fix for #3690.

@He-Pin
He-Pin force-pushed the fix-3690-handshake-pending-message branch from 10ca75f to b59b25a Compare October 26, 2026 03:58
@He-Pin He-Pin changed the title fix: account for pending handshake messages on shutdown (#3690) fix: account for pending handshake messages when streams stop (#3690) Oct 26, 2026
@He-Pin
He-Pin force-pushed the fix-3690-handshake-pending-message branch from b59b25a to 6bf3475 Compare November 11, 2026 11:26
Motivation:
OutboundHandshake can retain an envelope after removing it from the send queue. Stopping the stage must account for that envelope without quarantining an orderly local shutdown, attributing the message to a newer remote UID, or sending quarantine notifications through the control stream that is stopping.

Modification:
- Pass held envelopes and the UID observed when they were held to an Association callback. Keep the existing constructor and transport method signatures available.
- Share pending-system-message accounting between the control queue and handshake stage. Wait for both cleanup callbacks, install the replacement control queue, and then apply quarantine policy for the matching known UID.
- If SystemMessageDelivery already quarantined the same peer during teardown, send the notification through the replacement control ingress. Ordinary and large lanes only report their held envelopes to dead letters.
- Always remove the handshake listener and run superclass cleanup in finally. Ensure test subscriptions are also removed in finally.
- Exercise real Association, SendQueue, handshake, delivery, control-junction and encoding stages with controlled transport demand. Cover orderly and unexpected stops, unknown and changing UIDs, first-handshake completion, notification output, and initialization failure.
- Narrow the documentation to the control stream and the queue/handshake buffers this change accounts for.

Result:
Held envelopes reach dead letters. Pending system messages in these buffers trigger quarantine after an unexpected control-stream stop for the matching known remote UID; their cleanup during local shutdown does not quarantine. Notifications use the replacement control stream. The wire protocol is unchanged.

Historical validation before the latest main rebase:
- JDK 17: `sbt "remote / Test / testOnly org.apache.pekko.remote.artery.OutboundHandshakeSpec org.apache.pekko.remote.artery.OutboundHandshakePendingMessageSpec org.apache.pekko.remote.artery.OutboundHandshakeAssociationSpec"` — all 21 passed on the final shared integration tree; the standalone Association suite also passed all 11 tests.
- Adjacent SystemMessageDeliverySpec, OutboundIdleShutdownSpec and HarmlessQuarantineSpec — 17 passed after the lifecycle/notification changes, before the final held-UID refinement.
- Running the 11 Association regressions against the exact previous head (10ca75f) produced 8 passes and 3 expected failures: orderly shutdown quarantined, an old held message quarantined a newer UID, and delayed completion observed an earlier old-UID notification. The last assertion does not establish quarantine of the new UID.
- A separate new-incarnation regression failed when the UID was captured only during postStop and passed after capturing it when the handshake stage holds the envelope.
- Removing the listener-cleanup finally caused the exception-path regression to fail; restoring it passed all 3 pending-message tests.
- A delivery-plus-handshake regression failed before the replacement-stream re-notification fix and passed after it. This failure was established against the intermediate callback implementation; the exact original-head run did not deterministically lose that notification.
- `sbt "project remote" headerCreateAll sortImports "+mimaReportBinaryIssues" "+headerCheckAll"`, native `scalafmt --mode diff-ref=origin/main`, and `git diff --check` passed. Remote MiMa and headers passed on Scala 2.13.18 and 3.3.8.
- `sbt docs/paradox` passed on the preceding shared integration tree (d0863b); the documentation is identical in the final integration tree (b6f826f1).
- Historical shared `sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequest` predates this follow-up. Its ScalaTest summary was 8,600 passed/3 failed (two TCP timeouts and a Netty address-bind failure); four persistence modules could not run because the LevelDB JNI binary was incompatible with Darwin arm64, and the Docker DNS test was pending. The two TCP cases passed when independently repeated on that earlier final artifact. This historical run is not validation of the revised implementation.
- Final full-diff Qoder review — No must-fix findings.
- No local full or broad `validatePullRequest` rerun, as requested.

Tests (current review update, 2026-11-11):
- Rebased onto main ede48bd without conflicts; git range-diff confirms the patch is unchanged.
- git diff --check: passed.
- This rebase-only update does not rerun the historical test suites listed above.

References:
Fixes #3690
@He-Pin
He-Pin force-pushed the fix-3690-handshake-pending-message branch from 6bf3475 to a3d9c34 Compare November 11, 2026 18:33

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earlier review points all look addressed, thanks. A few remaining things, details inline.

if (!current.quarantinedButHarmless(uid))
sendControl(Quarantined(localAddress, peer))
} else quarantine(reason, Some(uid), harmless = false)
case _ =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fall-through silently drops cases that quarantine(reason) used to log, now that quarantineAfterFailure replaces it on control stream restarts:

  • unknown UID: previously a WARN "Quarantine of [...] ignored because unknown UID"; now nothing, since stoppedUid is None and the foreach never runs
  • UID mismatch: previously an INFO log plus ClearSystemMessageDelivery(incarnation - 1); now neither

Skipping the clear is probably harmless because the replacement stream builds a fresh SystemMessageDelivery, but the behaviour change isn't mentioned in the PR and the lost logging makes these restarts harder to diagnose. Could this case log at debug at least?

// A transport sink can complete before the upstream stages have run postStop.
// Account for both buffers before replacing the control queue and notifying the peer.
implicit val ec = materializer.executionContext
val stopped = completed.transformWith { result =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The restart (and the completed that transport shutdown waits on) now also depends on both queueStopped and handshakeStopped, with no timeout. I couldn't find a path where either postStop is skipped (synchronous materialization failure throws from run() before this point, abrupt termination still runs postStop, and both callbacks complete their promise in finally), and the initialization-failure test covers the obvious case. But it does make restarts depend on two internal stages always running postStop; a short comment stating that assumption would help the next person touching this.

val queueStopped = Promise[Done]()
val handshakeStopped = Promise[Done]()
val pendingSystemMessages = new AtomicReference(Map.empty[Long, Int])
val stoppedUid = new AtomicReference[Option[Long]]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: using null as "not yet pinned" in an AtomicReference[Option[Long]] (with compareAndSet(null, ...)) works but is easy to misread. Starting at None with a separate flag, or a Promise[Option[Long]], would make the intent clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OutboundHandshake.postStop drops an already-dequeued pendingMessage, bypassing the pending-system-message quarantine safeguard

2 participants