Skip to content

[Fix] Rebuild NCCL RDMA SHARP plugin to recover from invalid handshakes - #2306

Open
jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:fix/nccl-rdma-sharp-plugin
Open

jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:fix/nccl-rdma-sharp-plugin

Conversation

@jayhenry

Copy link
Copy Markdown
Collaborator

Root Cause

The base image ships the HPC-X NCCL RDMA SHARP net plugin built from 4c7fc63. In socketFinalizeAccept, rejecting a connection with an invalid magic (an HTTP probe hitting the training port produces this) closes the socket but leaves finalizeCounter at the handshake size. The next valid connection then skips the magic check, and the low 4 bytes of the valid magic are decoded as the socket type (wrong type -62153364 != 4); NCCL aborts with Error 3 and other ranks follow with Error 6.

Fix

  • Add a build stage that builds Mellanox/nccl-rdma-sharp-plugins at 0b5eeb9 ("Clean up the socket code", which ports NCCL's socket cleanup that resets the handshake counter on rejection). IB, UCX and SHARP support are enabled against the HPC-X runtime already in the image.
  • Replace /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so.0.0.0 in the final image and refresh the loader cache. /etc/ld.so.conf.d/hpcx.conf already resolves libnccl-net.so from that directory, so NCCL picks up the fixed plugin without any environment change.
  • Pin the plugin URL in image_build.sh and add the matching build arg/label.

Test Plan

  • Built the plugin with exactly the Dockerfile commands in the same image environment: it exports net/collnet v10, its RUNPATH resolves to /opt/hpcx/{ucx,sharp}/lib, and ldd reports no missing libraries.
  • Ran the 8-GPU HTTP-replay repro against the rebuilt library: 7 invalid-magic connections were rejected and each handshake reset, NCCL selected IBext_v10, the Muon step matched the reference bitwise, and the job exited 0 with no NCCL WARN/Error 3/6.
  • A full image build was not run here (no Docker daemon in this environment); the docker-inner workflow will build it.

The old workaround of setting NCCL_NET_PLUGIN=none is no longer needed with this image.

The HPC-X plugin shipped in the base image (4c7fc63) does not reset the
handshake counter when rejecting a connection with an invalid magic. The
next valid handshake is then decoded from the wrong offset and NCCL aborts
with Error 3, which is what an HTTP probe on the training port triggered in
the 512-GPU job.

Rebuild the plugin from upstream 0b5eeb9, which ports the socket cleanup
from NCCL, and install it over the HPC-X copy in /opt/hpcx.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant