Skip to content

Evaluate three fresh agent pairs implementing token-budget DDP accumulation - #173

Open
Essoz wants to merge 6 commits into
codex/model-snapshot-agent-workflowfrom
codex/ddp-token-budget-agent-study
Open

Essoz wants to merge 6 commits into
codex/model-snapshot-agent-workflowfrom
codex/ddp-token-budget-agent-study

Conversation

@Essoz

@Essoz Essoz commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Can agents deliver a correct token-budget gradient accumulation feature on top of healthy DDP training, and does optional TrainCheck access change the outcome? Three fresh matched pairs implemented the same feature on Ring18. The neutral task specified the global-batch objective and budget semantics. Agents chose their own diagnostics.

All six first submissions passed all six held-out scenarios. Each agent requested one two-GPU run of its own PyTorch diagnostic, then delivered the identical production source. We observed 0/6 silent numerical errors at delivery and 0/6 cases of an agent finding and repairing an earlier numerical defect. All three optional-tool agents read the TrainCheck guide. No TrainCheck use was observed in the executed diagnostics or production code.

Pair Control elapsed / total tokens Optional TrainCheck elapsed / total tokens
1 61.95 s / 109,666 56.19 s / 90,894
2 54.37 s / 74,898 68.78 s / 128,224
3 60.55 s / 94,550 64.25 s / 95,300

These are costs to the first independently correct frozen submission. All six reached it before their first GPU request, hence 0 GPU-seconds at that boundary. Subsequent solver checks used 148.844 GPU-seconds in total. Qualification used 128.585 and independent replays 210.974 GPU-seconds. All 20 GPU allocations have release receipts. Total token counts include cached input once. The report gives uncached input plus output, elapsed and active CLI clocks by agent.

The new tools provide a working fixed-mode task, a separately implemented concatenated-batch oracle, frozen native solver sessions and submitted sources, and an independently checked cost join. Both ranks were checked at every update for gradients, parameters, momentum and microbatch boundaries. A normally completing wrong-denominator control failed four of six scenarios. The six solver sources plus the starter in both environments yielded eight native replays of six scenarios each. All outcomes are known.

Validation: 21 focused study tests passed. Scoped type checks and configured hooks ran on the study implementation. The report's six rows and eight raw evidence hashes were rechecked. The older training framework and its full suite were outside the change.

The public fixed-mode baseline already contains a correct single-microbatch normalization example. The specification gives agents a direct numerical reference to implement. Three pairs on this task cannot establish general silent-error prevalence or the efficiency of actually using TrainCheck. Three solvers read a generic host policy skill outside their packets. No access to case-private grading or a peer solution was observed. Intermediate edit sampling can miss short-lived changes.

See the full report and per-agent data. Stacked on #172 to share its frozen model observation implementation. Signed checkpoints preserve the prospective endpoints, implementation, qualification and final results.

@Essoz
Essoz marked this pull request as ready for review November 17, 2026 19:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant