Repository navigation
Conversation
The HPC-X plugin shipped in the base image (4c7fc63) does not reset the handshake counter when rejecting a connection with an invalid magic. The next valid handshake is then decoded from the wrong offset and NCCL aborts with Error 3, which is what an HTTP probe on the training port triggered in the 512-GPU job. Rebuild the plugin from upstream 0b5eeb9, which ports the socket cleanup from NCCL, and install it over the HPC-X copy in /opt/hpcx.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root Cause
The base image ships the HPC-X NCCL RDMA SHARP net plugin built from
4c7fc63. InsocketFinalizeAccept, rejecting a connection with an invalid magic (an HTTP probe hitting the training port produces this) closes the socket but leavesfinalizeCounterat the handshake size. The next valid connection then skips the magic check, and the low 4 bytes of the valid magic are decoded as the socket type (wrong type -62153364 != 4); NCCL aborts with Error 3 and other ranks follow with Error 6.Fix
Mellanox/nccl-rdma-sharp-pluginsat0b5eeb9("Clean up the socket code", which ports NCCL's socket cleanup that resets the handshake counter on rejection). IB, UCX and SHARP support are enabled against the HPC-X runtime already in the image./opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so.0.0.0in the final image and refresh the loader cache./etc/ld.so.conf.d/hpcx.confalready resolveslibnccl-net.sofrom that directory, so NCCL picks up the fixed plugin without any environment change.image_build.shand add the matching build arg/label.Test Plan
/opt/hpcx/{ucx,sharp}/lib, andlddreports no missing libraries.IBext_v10, the Muon step matched the reference bitwise, and the job exited 0 with no NCCL WARN/Error 3/6.docker-innerworkflow will build it.The old workaround of setting
NCCL_NET_PLUGIN=noneis no longer needed with this image.