Repository navigation
Conversation
Essoz
marked this pull request as ready for review
November 17, 2026 21:14
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fresh matched pairs assessed readiness of a two-GPU DDP MNIST pipeline with dropout, AdamW, cosine scheduling, shuffled per-rank data and persistent DataLoader workers. Both arms got the same neutral task and smoke run. Treatment added only the unchanged generic TrainCheck kit. No task-specific checker or bug hint was shared.
The ordinary smoke passed on Ring18. The starter still diverged silently in four of six private continuation scenarios. Two agents reproduced the concrete failure themselves, one per arm. Their checkpoint stored offset 112 after training on 48 samples per rank, skipping 64 on resume. A separate fresh-process Ring18 restart confirmed all six prefix rank-steps matched, the restore exited normally, and all ten continuation rank-step losses diverged.
Outcome: All 3 controls and all 3 tool-available agents identified the cause from source before their first GPU request. All six delivered repairs that passed all six independent scenarios on both ranks. All optional agents read the guide and all six wrote ordinary PyTorch diagnostics. No executed solver code used TrainCheck. There were no control misses and no detections unique to the optional arm. The measured comparison is optional access in this fixture.
Four agents first submitted correct source in turn 1. The two who requested evidence on the unmodified pipeline first submitted correct source in turn 2. Per-agent times, raw/cached token counters, prior GPU cost and validation history are in the results. Solver trials used 402.553 allocated GPU-seconds. Including native qualification, eight private replays and the separate process restart proof, the study used 753.363 GPU-seconds. Both GPUs were released.
Validation: 23 existing harness tests passed, plus scoped strict type checking and configured hooks. The exact launcher and staged file hashes, accepted source revisions and holdout receipts are archived. The fixture uses real MNIST examples and native DDP. It represents a realistic engineered failure mode at reduced scale. Private replay creates a new trainer and worker pool after reading the checkpoint. The separate process proof exercises a true torchrun restart.
Stacked on #174. Full report · Per-agent and per-scenario results.