Skip to content

Evaluate optional TrainCheck on multi-worker DDP checkpoint readiness - #175

Open
Essoz wants to merge 3 commits into
codex/ddp-window-sync-agent-studyfrom
codex/ddp-prefetch-resume-agent-study
Open

Essoz wants to merge 3 commits into
codex/ddp-window-sync-agent-studyfrom
codex/ddp-prefetch-resume-agent-study

Conversation

@Essoz

@Essoz Essoz commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Three fresh matched pairs assessed readiness of a two-GPU DDP MNIST pipeline with dropout, AdamW, cosine scheduling, shuffled per-rank data and persistent DataLoader workers. Both arms got the same neutral task and smoke run. Treatment added only the unchanged generic TrainCheck kit. No task-specific checker or bug hint was shared.

The ordinary smoke passed on Ring18. The starter still diverged silently in four of six private continuation scenarios. Two agents reproduced the concrete failure themselves, one per arm. Their checkpoint stored offset 112 after training on 48 samples per rank, skipping 64 on resume. A separate fresh-process Ring18 restart confirmed all six prefix rank-steps matched, the restore exited normally, and all ten continuation rank-step losses diverged.

Outcome: All 3 controls and all 3 tool-available agents identified the cause from source before their first GPU request. All six delivered repairs that passed all six independent scenarios on both ranks. All optional agents read the guide and all six wrote ordinary PyTorch diagnostics. No executed solver code used TrainCheck. There were no control misses and no detections unique to the optional arm. The measured comparison is optional access in this fixture.

Four agents first submitted correct source in turn 1. The two who requested evidence on the unmodified pipeline first submitted correct source in turn 2. Per-agent times, raw/cached token counters, prior GPU cost and validation history are in the results. Solver trials used 402.553 allocated GPU-seconds. Including native qualification, eight private replays and the separate process restart proof, the study used 753.363 GPU-seconds. Both GPUs were released.

Validation: 23 existing harness tests passed, plus scoped strict type checking and configured hooks. The exact launcher and staged file hashes, accepted source revisions and holdout receipts are archived. The fixture uses real MNIST examples and native DDP. It represents a realistic engineered failure mode at reduced scale. Private replay creates a new trainer and worker pool after reading the checkpoint. The separate process proof exercises a true torchrun restart.

Stacked on #174. Full report · Per-agent and per-scenario results.

@Essoz
Essoz marked this pull request as ready for review November 17, 2026 21:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant