Repository navigation
Conversation
cab0464 to
10ca75f
Compare
pjfanning
left a comment
There was a problem hiding this comment.
A few concerns with the new postStop handling, plus some test and doc notes. Details are inline.
| } | ||
|
|
||
| override def postStop(): Unit = { | ||
| pendingMessage match { |
There was a problem hiding this comment.
This repeats the policy in Association.sendQueuePostStop, and the copies already behave differently: SendQueue quarantines only for the control stream, but this quarantines for every lane (ordinary and large streams too), and the reason strings and counting differ. Could OutboundHandshake take a postStop callback, like SendQueue's postStopAction, so there is one place that decides between dead letters and quarantine?
More broadly, this covers only the one envelope held by this stage. System messages already pulled into other buffered or async stages upstream of SystemMessageDelivery can still be dropped silently when the stream stops. A general check when the outbound control stream terminates might be a more complete fix for #3690.
10ca75f to
b59b25a
Compare
b59b25a to
6bf3475
Compare
Motivation: OutboundHandshake can retain an envelope after removing it from the send queue. Stopping the stage must account for that envelope without quarantining an orderly local shutdown, attributing the message to a newer remote UID, or sending quarantine notifications through the control stream that is stopping. Modification: - Pass held envelopes and the UID observed when they were held to an Association callback. Keep the existing constructor and transport method signatures available. - Share pending-system-message accounting between the control queue and handshake stage. Wait for both cleanup callbacks, install the replacement control queue, and then apply quarantine policy for the matching known UID. - If SystemMessageDelivery already quarantined the same peer during teardown, send the notification through the replacement control ingress. Ordinary and large lanes only report their held envelopes to dead letters. - Always remove the handshake listener and run superclass cleanup in finally. Ensure test subscriptions are also removed in finally. - Exercise real Association, SendQueue, handshake, delivery, control-junction and encoding stages with controlled transport demand. Cover orderly and unexpected stops, unknown and changing UIDs, first-handshake completion, notification output, and initialization failure. - Narrow the documentation to the control stream and the queue/handshake buffers this change accounts for. Result: Held envelopes reach dead letters. Pending system messages in these buffers trigger quarantine after an unexpected control-stream stop for the matching known remote UID; their cleanup during local shutdown does not quarantine. Notifications use the replacement control stream. The wire protocol is unchanged. Historical validation before the latest main rebase: - JDK 17: `sbt "remote / Test / testOnly org.apache.pekko.remote.artery.OutboundHandshakeSpec org.apache.pekko.remote.artery.OutboundHandshakePendingMessageSpec org.apache.pekko.remote.artery.OutboundHandshakeAssociationSpec"` — all 21 passed on the final shared integration tree; the standalone Association suite also passed all 11 tests. - Adjacent SystemMessageDeliverySpec, OutboundIdleShutdownSpec and HarmlessQuarantineSpec — 17 passed after the lifecycle/notification changes, before the final held-UID refinement. - Running the 11 Association regressions against the exact previous head (10ca75f) produced 8 passes and 3 expected failures: orderly shutdown quarantined, an old held message quarantined a newer UID, and delayed completion observed an earlier old-UID notification. The last assertion does not establish quarantine of the new UID. - A separate new-incarnation regression failed when the UID was captured only during postStop and passed after capturing it when the handshake stage holds the envelope. - Removing the listener-cleanup finally caused the exception-path regression to fail; restoring it passed all 3 pending-message tests. - A delivery-plus-handshake regression failed before the replacement-stream re-notification fix and passed after it. This failure was established against the intermediate callback implementation; the exact original-head run did not deterministically lose that notification. - `sbt "project remote" headerCreateAll sortImports "+mimaReportBinaryIssues" "+headerCheckAll"`, native `scalafmt --mode diff-ref=origin/main`, and `git diff --check` passed. Remote MiMa and headers passed on Scala 2.13.18 and 3.3.8. - `sbt docs/paradox` passed on the preceding shared integration tree (d0863b); the documentation is identical in the final integration tree (b6f826f1). - Historical shared `sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequest` predates this follow-up. Its ScalaTest summary was 8,600 passed/3 failed (two TCP timeouts and a Netty address-bind failure); four persistence modules could not run because the LevelDB JNI binary was incompatible with Darwin arm64, and the Docker DNS test was pending. The two TCP cases passed when independently repeated on that earlier final artifact. This historical run is not validation of the revised implementation. - Final full-diff Qoder review — No must-fix findings. - No local full or broad `validatePullRequest` rerun, as requested. Tests (current review update, 2026-11-11): - Rebased onto main ede48bd without conflicts; git range-diff confirms the patch is unchanged. - git diff --check: passed. - This rebase-only update does not rerun the historical test suites listed above. References: Fixes #3690
6bf3475 to
a3d9c34
Compare
pjfanning
left a comment
There was a problem hiding this comment.
The earlier review points all look addressed, thanks. A few remaining things, details inline.
| if (!current.quarantinedButHarmless(uid)) | ||
| sendControl(Quarantined(localAddress, peer)) | ||
| } else quarantine(reason, Some(uid), harmless = false) | ||
| case _ => |
There was a problem hiding this comment.
This fall-through silently drops cases that quarantine(reason) used to log, now that quarantineAfterFailure replaces it on control stream restarts:
- unknown UID: previously a WARN "Quarantine of [...] ignored because unknown UID"; now nothing, since
stoppedUidisNoneand theforeachnever runs - UID mismatch: previously an INFO log plus
ClearSystemMessageDelivery(incarnation - 1); now neither
Skipping the clear is probably harmless because the replacement stream builds a fresh SystemMessageDelivery, but the behaviour change isn't mentioned in the PR and the lost logging makes these restarts harder to diagnose. Could this case log at debug at least?
| // A transport sink can complete before the upstream stages have run postStop. | ||
| // Account for both buffers before replacing the control queue and notifying the peer. | ||
| implicit val ec = materializer.executionContext | ||
| val stopped = completed.transformWith { result => |
There was a problem hiding this comment.
The restart (and the completed that transport shutdown waits on) now also depends on both queueStopped and handshakeStopped, with no timeout. I couldn't find a path where either postStop is skipped (synchronous materialization failure throws from run() before this point, abrupt termination still runs postStop, and both callbacks complete their promise in finally), and the initialization-failure test covers the obvious case. But it does make restarts depend on two internal stages always running postStop; a short comment stating that assumption would help the next person touching this.
| val queueStopped = Promise[Done]() | ||
| val handshakeStopped = Promise[Done]() | ||
| val pendingSystemMessages = new AtomicReference(Map.empty[Long, Int]) | ||
| val stoppedUid = new AtomicReference[Option[Long]] |
There was a problem hiding this comment.
nit: using null as "not yet pinned" in an AtomicReference[Option[Long]] (with compareAndSet(null, ...)) works but is easy to misread. Starting at None with a separate flag, or a Promise[Option[Long]], would make the intent clearer.
Motivation
OutboundHandshake can retain an envelope after removing it from the send queue. Stopping the stage must account for that envelope without quarantining an orderly local shutdown, attributing the message to a newer remote UID, or sending quarantine notifications through the control stream that is stopping.
Modification
Result
Held envelopes reach dead letters. Pending system messages in these buffers trigger quarantine after an unexpected control-stream stop for the matching known remote UID; their cleanup during local shutdown does not quarantine. Notifications use the replacement control stream. The wire protocol is unchanged.
Tests
sbt "remote / Test / testOnly org.apache.pekko.remote.artery.OutboundHandshakeSpec org.apache.pekko.remote.artery.OutboundHandshakePendingMessageSpec org.apache.pekko.remote.artery.OutboundHandshakeAssociationSpec"— all 21 passed on the final shared integration tree; the standalone Association suite also passed all 11 tests.sbt "project remote" headerCreateAll sortImports "+mimaReportBinaryIssues" "+headerCheckAll", nativescalafmt --mode diff-ref=origin/main, andgit diff --checkpassed. Remote MiMa and headers passed on Scala 2.13.18 and 3.3.8.sbt docs/paradoxpassed on the preceding shared integration tree (d0863b); the documentation is identical in the final integration tree (b6f826f1).sbt -Dpekko.test.tags.exclude=performance,timing,long-running -Dpekko.test.multi-in-test=false validatePullRequestpredates this follow-up. Its ScalaTest summary was 8,600 passed/3 failed (two TCP timeouts and a Netty address-bind failure); four persistence modules could not run because the LevelDB JNI binary was incompatible with Darwin arm64, and the Docker DNS test was pending. The two TCP cases passed when independently repeated on that earlier final artifact. This historical run is not validation of the revised implementation.validatePullRequestrerun, as requested.References
Fixes #3690