Repository navigation
Conversation
Essoz
marked this pull request as ready for review
November 17, 2026 19:31
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Can agents deliver a correct token-budget gradient accumulation feature on top of healthy DDP training, and does optional TrainCheck access change the outcome? Three fresh matched pairs implemented the same feature on Ring18. The neutral task specified the global-batch objective and budget semantics. Agents chose their own diagnostics.
All six first submissions passed all six held-out scenarios. Each agent requested one two-GPU run of its own PyTorch diagnostic, then delivered the identical production source. We observed 0/6 silent numerical errors at delivery and 0/6 cases of an agent finding and repairing an earlier numerical defect. All three optional-tool agents read the TrainCheck guide. No TrainCheck use was observed in the executed diagnostics or production code.
These are costs to the first independently correct frozen submission. All six reached it before their first GPU request, hence 0 GPU-seconds at that boundary. Subsequent solver checks used 148.844 GPU-seconds in total. Qualification used 128.585 and independent replays 210.974 GPU-seconds. All 20 GPU allocations have release receipts. Total token counts include cached input once. The report gives uncached input plus output, elapsed and active CLI clocks by agent.
The new tools provide a working fixed-mode task, a separately implemented concatenated-batch oracle, frozen native solver sessions and submitted sources, and an independently checked cost join. Both ranks were checked at every update for gradients, parameters, momentum and microbatch boundaries. A normally completing wrong-denominator control failed four of six scenarios. The six solver sources plus the starter in both environments yielded eight native replays of six scenarios each. All outcomes are known.
Validation: 21 focused study tests passed. Scoped type checks and configured hooks ran on the study implementation. The report's six rows and eight raw evidence hashes were rechecked. The older training framework and its full suite were outside the change.
The public fixed-mode baseline already contains a correct single-microbatch normalization example. The specification gives agents a direct numerical reference to implement. Three pairs on this task cannot establish general silent-error prevalence or the efficiency of actually using TrainCheck. Three solvers read a generic host policy skill outside their packets. No access to case-private grading or a peer solution was observed. Intermediate edit sampling can miss short-lived changes.
See the full report and per-agent data. Stacked on #172 to share its frozen model observation implementation. Signed checkpoints preserve the prospective endpoints, implementation, qualification and final results.