Skip to content

Evaluate optional TrainCheck checks for DDP window synchronization - #174

Open
Essoz wants to merge 3 commits into
codex/ddp-token-budget-agent-studyfrom
codex/ddp-window-sync-agent-study
Open

Essoz wants to merge 3 commits into
codex/ddp-token-budget-agent-studyfrom
codex/ddp-window-sync-agent-study

Conversation

@Essoz

@Essoz Essoz commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Three fresh matched pairs on Ring18 delayed gradient synchronization until the token window ends. Both arms received the same working model, neutral task, and budgets. The optional arm could call a new TrainCheck adapter that profiles native NCCL communication and checks numerical agreement, including the partial window.

All six agents passed all six independent scenarios on their first submitted source and delivered it unchanged. No solver delivered a silent numerical or communication-contract failure. The independent missing-tail-sync control failed all three partial-tail scenarios. The original starter passed the numerical comparisons and failed the new communication contract where it synchronized early.

Pair Diagnostic lines control / optional First correct wall s control / optional First correct total tokens control / optional
1 174 / 7 72.26 / 41.35 116,693 / 122,357
2 169 / 7 69.31 / 34.40 100,290 / 106,146
3 198 / 164 35.59 / 63.21 74,408 / 100,482

Two of three optional agents ran the provided diagnostic, passing four public cases on both ranks. Their submitted diagnostic sources were about 96% shorter than their paired controls. The third wrote its own diagnosis after no observed guide read. Independently selected diagnostics cover different numbers of cases. Every first correct submission preceded GPU execution. Every solver subsequently made two requests, with 296.231 aggregate allocated GPU-seconds. Total tokens include cached input once. The measured benefit is less handwritten diagnostic scaffolding in the two trials that used the adapter. Time and token measurements vary across pairs.

Actual communications: In the held-out overshoot-and-tail scenario every solver reduced 2,175 gradient elements per rank compared with 4,350 for the starter. Gradient all-reduce launches increased from 6 to 9 because every solver reduced parameters individually. Packed qualification control made 3 launches for the same 2,175 elements. The task allowed separate collectives at window boundaries. This pilot measures launches and reduced tensor elements. Throughput remains unmeasured.

Validation: 23 focused tests, scoped strict mypy and configured checks passed. 18 frozen decisions and sampled edits formed eight independent source/environment replays, 48 scenarios and 96 rank outcomes. Qualification including profiler probe cost 125.137 GPU-seconds and private grading 200.303. GPU release was verified for every allocation. Six conversations used independent packets; recorded external reads were of host policy documentation.

Stacked on #173. See the full report and per-agent and per-scenario results.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant