Repository navigation
Conversation
added 3 commits
November 6, 2026 21:34
…ation can leave threads waiting indefinitely * Complete the NIO session creation future with the registration error (ClosedChannelException or the IOException of the registration) when a channel is closed before or while it is being registered with the selector, so the caller observes the failure and can retry instead of the establishment hanging on a future that stays pending forever. The registration error handling is scoped to the registration call itself, so it cannot observe a state in which the channel was already registered: a failure before the registration (resolving the socket addresses) and a failure after it (the filter chain) keep their original handling, and the latter still closes the registered session. * Add a narrow test-only seam to reproduce the condition: the NioChannelRegistrar interface of the internal org.apache.ignite.internal.util.nio package (not part of the product API) and the public GridNioServer#setChannelRegistrar setter let a test intercept the channel registration; the null default keeps the production behavior unchanged. The seam is public because the tests that use it live in a different build module, where package-private access is not possible. * Add a regression test in GridNioSelfTest that closes the channel via the registrar and asserts that the session creation future completes with the ClosedChannelException.
…ation can leave threads waiting indefinitely * Apply a single connection establishment deadline covering the TCP handshake and the NIO session registration, so that even after a successful handshake a thread is not left blocked while waiting for the NIO session registration (previously the handshake timeout covered the TCP handshake only). The socket connection is bounded by its own connect timeout, and the deadline is armed after it with the handshake window, so the handshake keeps the full window the backoff strategy allocated to it instead of sharing it with the connection. * When the deadline elapses the channel is closed, which aborts the in-progress handshake and, under the precondition that the responsible NIO worker keeps processing, makes the pending NIO session registration fail with a ClosedChannelException as well. * The deadline bounds the wait for the NIO session registration only indirectly: when it elapses the channel is closed, and the registration then fails with the resulting ClosedChannelException, but only once the responsible NIO worker processes the queued registration operation. If that worker itself stops making progress, the thread waiting on the session creation future is not released by the deadline; a strictly bounded registration wait (a timed future wait plus cancelling the still-queued registration operation) is a separate follow-up. * Write the re-verified handshake window back to the attempt's timeout, the pre-existing form, so the timeout logged on failure is the clamped value that actually bounded the handshake; the backoff progression itself is driven by the strategy's no-arg nextTimeout() call at the start of each attempt and is unaffected. * Make the deadline outcome explicit: HandshakeTimeoutObject tracks it as a single atomic state, so the attempt is only reported as timed out if the deadline actually fired and closed the channel - a lost cancel never leaves the ambiguity whether the deadline elapsed or the timeout task merely could no longer be cancelled. * Simplify HandshakeTimeoutObject to hold the SelectableChannel directly: after the handshake timeout moved out of safeTcpHandshake, no caller passes a GridCommunicationClient anymore. * Treat a peer-requested NEED_WAIT discovery-lag delay separately from the connection establishment: the deadline is cancelled before the wait and the next attempt arms a fresh one, so the delay does not consume the connection timeout budget. * Cover the deadline with regression tests, including a deliberately delayed NIO session registration, a deliberately slow socket connect that must not consume the deadline of the handshake and registration that follow it, the deadline object itself (a cancelled deadline is never reported as fired while a fired one is), and the exponential connection backoff across the retries (the establishment windows of three consecutive deadline-failed attempts must increase), and execute the tests also with SSL enabled. * Register the new test classes in the communication SPI test suite, as the orphaned-test check requires every test class to be in a suite. * Raise the reconnect-attempt bound of the simultaneous-reconnect regression test so the scenario is observed robustly instead of probabilistically.
…ation can leave threads waiting indefinitely * Bound the total time spent reserving a connection in ConnectionClientPool.reserveClient() with an overall reservation timeout, so threads waiting on the shared connection future are not blocked indefinitely when the thread completing that future gets stuck during connection establishment; the remaining budget is recomputed on every iteration of the wait loop and caps each individual wait, so the total budget is enforced without a single wait overshooting it. The budget is checked only while no client is available yet — including right before the wait loop gives up — so that an exhausted budget never prevents returning a client that a concurrent thread has established in the meantime: it bounds waiting for a connection, not the availability of an existing client. * Complete all pending connection futures when the pool is stopped so that waiting threads are released. * Base the reservation budget on the same total timeout that bounds the repeated connection attempts to a node, computed once in CommunicationTcpUtils and shared by both call sites; the budget bounds only the wait on an in-progress connection future, not the inverse-connection protocol. Clarify the JavaDoc of that total timeout: it is a budget over the repeated connection attempts, not a single end-to-end deadline of one attempt, whose socket connect is bounded by its own connect timeout and whose handshake and NIO session registration deadline is armed after the connect. * After a failed direct connection attempt, trigger the inverse-connection protocol and wait for its result with a dedicated window (the failure detection timeout, or the connection timeout when the failure detection timeout is disabled) instead of the remaining reservation budget, which the failed direct attempts have already consumed, so the protocol still runs after the budget is exhausted; the synchronous connection-request call precedes the window and is not covered by it either, so the path relies on the requestor returning promptly — a precondition documented in the ConnectionRequestor contract. * Let the thread that establishes a connection itself return the client it created directly instead of re-entering the wait/budget loop, so a valid client is not rejected once the reservation budget has elapsed; the fast path applies to both the direct establishment and the inverse-connection result and is taken only when that thread actually completed the shared future with its own client, otherwise the client is removed and closed and the thread is released with the stop error like any other waiter. * Cover the reservation timeout, the waiter that must return a client established by a concurrent thread even after its reserve budget has elapsed (the client is installed only after the waiter is verified from its stack trace to be blocked in the shared-future wait), the variant with the system worker blocked timeout disabled, the stop-releases-waiters scenario, the stop-beats-the-creator completion race, the inverse connection established within its own bounded window after the reservation budget is exhausted, and a waiter released by its own budget while the producer is still in the inverse-connection phase, with regression tests.
mroeppis
force-pushed
the
ignite-29251
branch
from
November 6, 2026 20:39
c8ed978 to
89412a2
Compare
There was a problem hiding this comment.
🟡 Changes recommended
An unbounded connection wait can block past its deadline, a shutdown race can expose a client whose future failed, and timeout handling can mask recorded failures.
3 open issues
What changed in this PR
This PR hardens Ignite’s TCP communication connection handling to reduce indefinite waits during connection establishment, client reservation, and shutdown.
Changes:
- Completes failed NIO registration futures and extends handshake deadlines through session registration.
- Adds bounded reservation waits and shutdown handling for pending connection futures.
- Adds regression coverage for connection failures, timeout behavior, and shutdown races, including SSL variants.
| File | Description |
|---|---|
modules/nio/src/main/java/org/apache/ignite/internal/util/nio/NioChannelRegistrar.java |
Adds a registration hook for fault-injection tests. |
modules/nio/src/main/java/org/apache/ignite/internal/util/nio/GridNioServer.java |
Completes session futures on registration failures. |
modules/core/src/test/java/org/apache/ignite/testsuites/IgniteSpiCommunicationSelfTestSuite.java |
Registers the new communication tests. |
modules/core/src/test/java/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpiConnectClosureSslSelfTest.java |
Runs connection regression scenarios with SSL. |
modules/core/src/test/java/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpiConnectClosureSelfTest.java |
Tests deadlines, reservation waits, recovery, and shutdown races. |
modules/core/src/test/java/org/apache/ignite/internal/util/nio/GridNioSelfTest.java |
Tests future completion when registration encounters a closed channel. |
modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/HandshakeTimeoutObject.java |
Distinguishes cancelled deadlines from expired deadlines. |
modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/GridNioServerWrapper.java |
Extends the handshake deadline through NIO session registration. |
modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/ConnectionRequestor.java |
Documents the synchronous requestor’s timeout responsibilities. |
modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/ConnectionClientPool.java |
Bounds reservation waits and handles shutdown completion races. |
modules/core/src/main/java/org/apache/ignite/spi/communication/tcp/internal/CommunicationTcpUtils.java |
Centralizes connection timeout budget calculation. |
Review effort: Balanced
mroeppis
force-pushed
the
ignite-29251
branch
from
November 8, 2026 15:14
2e3eb49 to
c2339aa
Compare
added 3 commits
November 9, 2026 00:02
…ation can leave threads waiting indefinitely * Coordinate the publication of a newly established client with the completion of the shared connect future in ConnectionClientPool.reserveClient(): the future is completed first, and the client is published to the node client slot only if that completion wins the race. A thread that lost the race to stop() (or to any other completion) neither publishes nor returns its client and closes it instead, so the node client slot never holds a client whose shared future carries an error. Previously the client was published before the future was completed and only removed on a lost race, so a concurrent waiter reading the slot between publication and removal could take a client whose future had already been completed with the stop error, and the removal left a stale client array (and its metrics) for the node behind. * Move the publication decision — whether the client may be handed out through the shared future at all — before the future is completed, and coordinate the decision, the completion, and the publication with stop() by the pool's stop lock: stop() sets the stopping flag and completes the still pending connect futures under the same lock, so the decision and the shutdown are strictly ordered — either the decision sees no stop (the future is completed with the client, which is then published) or the stop is ordered first (the future is never completed with the client, which is dropped and closed, and the stop is reported). A concurrent waiter can therefore never take a client that the creating thread closes afterwards, and the node client slot is never published to after the shutdown has started. * Cover the stop-wins-the-race scenario with a concurrent waiter on the same shared future: the session registration is held stalled until stop() completes the future and then released, so the creating thread deterministically loses the completion race while the waiter is verified from its stack trace to be blocked in the shared-future wait; the creating thread must not publish or return its client, the concurrent waiter must be released by the stop() completion, and no client array may be left for the node, with a regression test. * Cover the interleaving in which the stop starts after the client creation but before the publication decision with a regression test that holds the pool's stop lock while the creating thread finishes the client creation and stops the pool while still holding the lock, with a concurrent waiter blocked in the shared-future wait, so that the stop's future completion is deterministically ordered before the publication decision: the future must be completed with the stop error, not with the client, both the waiter and the creating thread must be released with the stop error, no client array may be published, and the client the creating thread established must be closed. * Order the stop check and the connect-future insertion of ConnectionClientPool.reserveClient() with the pool's stop(): both happen under the stop lock, so either the reserving thread sees the stopping flag and bails out before its future is inserted, or stop() — which sets the flag and completes the still pending connect futures under the same lock — sees the inserted future and completes it with the stop error. Previously a thread that passed the unsynchronized stop check could insert its future after stop() had completed the futures it knew about, leaving that future unreleased by the stop; covered by a regression test that holds the stop lock while a competing thread enters reserveClient, so the stop is deterministically ordered before the future insertion: the competing thread must be rejected at the stop check before its future reaches the pool.
…ation can leave threads waiting indefinitely * Bound the wait for the NIO session registration in GridNioServerWrapper.createNioSession() by the part of the connection establishment deadline that is still left when the wait starts: the registration completes only when the responsible NIO worker processes the queued operation, so when that worker is stuck, the deadline closing the channel does not make the registration complete and the unbounded wait outlived the deadline. When the wait times out, the channel is closed so that the queued registration fails, a session the worker manages to create anyway is closed through a listener on the registration future, and the attempt is reported as a handshake timeout like any other deadline breach. * Cover the stalled-worker scenario with a regression test: the session registration of a new outgoing connection is held on a latch the deadline does not touch, so the worker stays stalled even after the deadline closes the channel, and the send must finish within a bounded time (the connection establishment deadline) instead of waiting for the stalled worker. * Cover the late-session scenario with a regression test: the registration is held on a latch only after the responsible NIO worker has registered the channel, so that when the latch is released after the deadline has fired, the worker completes the registration future with the session after the bounded wait has already given up, and that late session must be closed through the listener and removed from the NIO server. * Make the channel-close and backoff-window observations of the deadline tests robust to the attempt now completing at the deadline itself, up to the stalled worker's polling interval before the stalled worker records its observation: wait for that observation (the channel close offset in the deadline-covers-registration test, the third backoff window in the backoff progression test) with a bounded condition instead of reading it right after the send thread finished, and record the backoff windows in a synchronized list.
…ation can leave threads waiting indefinitely * Propagate the outcome carried by a completed connect future when the reserve budget is exhausted in the wait loop of ConnectionClientPool.reserveClient(): when the creating thread's connection attempt fails, the shared future is already completed with the error the attempt produced, and a thread entering the wait loop with the budget used up must be released with that error instead of the generic "Failed to reserve a client" timeout that masked the real cause. * Cover the scenario with a regression test: the session registration of a new outgoing connection is held stalled on a latch the connection establishment deadline does not touch, and the reserve budget is collapsed (connect timeout = maximum connect timeout = the whole budget), so the single stalled attempt consumes the budget and the creating thread must fail with the connection establishment error, not the reserve timeout.
mroeppis
force-pushed
the
ignite-29251
branch
from
November 8, 2026 23:27
c2339aa to
0a17254
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
Communication connection establishment can leave threads waiting indefinitely when an operation fails without completing its associated future or when a timeout does not cover the entire operation. Threads waiting for an ongoing connection attempt can be affected as well, potentially leading to communication pool starvation.
This PR addresses these issues to ensure that failed connection attempts are reported, connection establishment is time-bounded, and waiting threads are released when their reservation expires or the communication SPI stops.
Changes
NIO session creation: Ensure that failures during channel registration complete the associated session creation future exceptionally. This prevents callers from waiting indefinitely when a session can no longer be established.
Connection establishment timeout: Apply a single deadline to the complete connection establishment process. This prevents individual stages of an attempt from extending its duration beyond the configured connection establishment timeout.
Connection client reservation: Bound waits for an ongoing connection attempt by the remaining reservation timeout. This prevents threads from waiting indefinitely for a connection future that is never completed.
Communication SPI shutdown: Complete pending connection futures when the communication SPI stops, ensuring that waiting threads are released.
Concurrent client creation and shutdown: Prevent a newly created client from being returned or published if shutdown has already completed the corresponding connection future.
Tests
Regression tests cover failed NIO channel registration, connection establishment deadlines, simultaneous connection recovery, bounded client reservation waits, shutdown handling, concurrent client creation and shutdown, and inverse connection budget handling. Relevant scenarios are also covered with SSL.
Scope
The connection establishment deadline applies to each individual connection attempt. The reservation timeout bounds threads waiting for an ongoing connection attempt; it does not impose a single overall deadline on the creator's connection process across multiple addresses. The synchronous connection requestor call is also outside the reservation budget.
Recovery after an unexpected NIO worker termination while the Ignite node remains operational is not covered by this change.