Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
925 changes: 925 additions & 0 deletions docs/measurements/data/window-sync-agent-study.json

Large diffs are not rendered by default.

142 changes: 142 additions & 0 deletions docs/measurements/window-sync-agent-study.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Delaying DDP gradient synchronization to token-window boundaries

Three new matched agent pairs start with the same working streaming DDP loop.
The requested improvement is to delay gradient communication until each token
window ends. The task preserves the numeric trajectory and streaming input
contract, including epoch tails. One arm receives a documented optional
two-line TrainCheck diagnostic. The adapter runs public examples, profiles
native NCCL all-reduces and uses TrainCheck fingerprints for final replica
agreement. This compares access to the complete optional diagnostic workflow,
including its new task adapter. Both arms may write diagnostics and receive
the same GPU request, CLI and decision budgets. The task does not hint at a
pre-existing bug or a preferred implementation.

The operator stores each submitted production file and samples intermediate
edits. Native CLI input, cached input, output and elapsed time are kept apart
from GPU usage. First valid submission means the first frozen decision whose
code later passes every held-out check. Its cost ends before any GPU run
requested by that decision. Agent verification and independent replay are
separate events and GPU costs. Report time, token use and GPU cost for each
pair. Count submitted diagnostic source lines for each agent, including short
entry points that call the optional checker. Audit execution using actual
check output, not filename or documentation reads alone.

The prior [token budget study](token-budget-agent-study-20261118.md) supplies
the workload, independent concatenated-batch reference and frozen numerical
scenarios. The augmented scorer checks pre-step gradients, post-step parameters
and momentum, window boundaries, and repeated or early gradient communication.
It uses CPU profiler records emitted by the underlying NCCL process group with
input shape recording active. The workload uses single-element all-reduces for
token counts. Larger all-reduces carry gradient data. Compare the aggregate
gradient element count with the full trainable gradient payload at each step.
Separate buckets or parameters may be reduced individually. Each reduction
must occur following the final backward marker of that window and precede the
optimizer step. The measured quantities are NCCL launches and reduced tensor
elements for this workload's DDP reducer and ordinary tensor all-reduce
implementations.

Native qualification compares the working per-microbatch-sync starter, a
correct once-per-window implementation, and a private variant omitting
synchronization in the final partial window. The optional public check uses
separately specified public cases and prints evidence on each rank. Private
replay uses all six frozen scenarios, and is withheld until the solvers finish.
The low-level collective parser is shared, so controls verify its positive
and negative behavior. This is a small workflow pilot, with no model-training
performance claim.

## Results

All six agents first submitted a valid implementation in turn one and
delivered the same production file in turn three. Both ranks passed all six
held-out scenarios for every solver implementation. No solver delivered
a silent numerical failure or an early-gradient-communication violation.
The sampled edit histories contain only the working starter and one solver
implementation per trial, and no agent made a production edit after seeing a
GPU diagnostic result. Source samples can miss brief intermediate edits.

| Pair | Arm | Diagnostic lines | TrainCheck cases passed (both ranks) | First correct elapsed s | First correct total tokens | First correct uncached input + output | Delivery elapsed s | Delivery GPU-s |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | Control | 174 | 0 | 72.26 | 116,693 | 14,549 | 157.50 | 49.256 |
| 1 | Optional TrainCheck | 7 | 4 | 41.35 | 122,357 | 16,501 | 97.39 | 49.436 |
| 2 | Control | 169 | 0 | 69.31 | 100,290 | 26,818 | 126.37 | 49.622 |
| 2 | Optional TrainCheck | 7 | 4 | 34.40 | 106,146 | 13,986 | 83.56 | 49.623 |
| 3 | Control | 198 | 0 | 35.59 | 74,408 | 9,640 | 136.80 | 49.465 |
| 3 | Optional TrainCheck | 164 | 0 | 63.21 | 100,482 | 24,322 | 122.89 | 48.829 |

All six correct submissions predate the first GPU request, so all first
correct GPU costs are zero. Total tokens count cached input once. Each agent
used one diagnostic request and one ordinary example run. Timings and token
counts reflect three pairs with independently chosen diagnostics and queue
effects.

Two of three optional-arm agents opened the guide and ran the provided check.
Their seven-line entry points replaced 174 and 169 lines of handwritten
diagnostic code in their paired controls, a reduction of about 96% in
submitted diagnostic lines. Each executed all four public cases successfully
on both ranks, including the partial tail. The third optional agent did not
open the guide in the recorded operations and wrote 164 lines for its own
diagnostic. That pair’s control wrote 198. The custom diagnostics chose their
own inputs and coverage. The line counts describe authored scaffolding,
with equal frozen private scoring across all six implementations.

The two adopters reached first correct submission faster in their pairs.
The third available-tool run was slower to first correct submission. All
three optional runs had higher first-correct total token counts than their
controls. Uncached input plus output moved in both directions. GPU usage
through delivery was close because all six made the same number of requests.

## Communication results

All six agents used no_sync during token-mode accumulation, then issued
one all-reduce per parameter at each window boundary, including the partial
tail. In the held-out overshoot-and-tail case, at each rank the six
microbatches became updates ending at microbatches 2, 4, and 6. Under the
working starter, the rank reduced 4,350 gradient elements in six NCCL calls.
Every solver reduced 2,175 elements in nine NCCL calls. The operator’s
packed positive control reduced the same 2,175 elements in three calls.
Scalar reductions for token counts are separate from these numbers.

The new task allowed separate all-reduces per parameter or bucket at a
window boundary. This yielded correct values and halved the gradient
payload in that example, while dispatching 50% more gradient collectives.
The launch count and reduced tensor-element count are actual native
profiler observations. Throughput, link traffic and collective latency
were outside the experiment.

The private tail-drop control produced numerical errors in all three
scenarios with a partial tail. With one epoch-wide partial window it
emitted zero gradient reductions. The original per-microbatch starter
matched the numerical reference and failed the communication contract
where a window spanned multiple microbatches. These controls test distinct
ways the optimization can appear correct under a narrow check.

## Reproduction and evidence

Ring18 used devices 4 and 5 (NVIDIA A16, driver 535.309.01), Python 3.10.14
and PyTorch 2.2.2+cu121. Both arms ran the same native Codex 0.160.0 model
settings, requested gpt-6-astra with high reasoning and priority service.
Two agents worked concurrently per pair. Their GPU requests were serialized.
The scope relied on instructions. All six read a generic host policy skill and
four also read its generic schema. No reads of peer solutions or private
grading code were observed in the recorded commands.

Evidence is stored in outputs/ring18-window-sync-study/ in the canonical
checkout and in /home/yuxuan/experiments/traincheck-window-sync-study on
Ring18. The 18 decision submissions and sampled source states required
eight replays, counting the unchanged starter separately in each arm.
Across those replays the evaluator ran 48 two-rank scenarios and recorded
96 rank outcomes. The GPU watcher verified release after each allocation.

| GPU use | Allocated GPU-seconds |
| --- | ---: |
| Native qualification and profiler probe | 125.137 |
| Six solver trials | 296.231 |
| Independent grading | 200.303 |
| Total | 621.671 |

The [per-trial results](data/window-sync-agent-study.json) preserve exact
costs, tool adoption evidence, communication profiles and receipt
hashes. The private replay and cost join run with the repository’s
tools.window_sync_study.replay and tools.token_budget_study.operator.score_submissions.
Twenty-three focused local tests passed, along with the scoped mypy and
configured pre-commit checks.
17 changes: 13 additions & 4 deletions tools/token_budget_study/operator/agent_trials.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,9 @@ class Arm:
packet: Path


def prepare(evidence: Path, packets: Path, optional: Path, manifest: Path) -> None:
def prepare(
evidence: Path, packets: Path, optional: Path, manifest: Path, public: Path = PUBLIC
) -> None:
packet_root = packets.resolve()
packet_root.mkdir(parents=True, exist_ok=False)
evidence.mkdir(parents=True, exist_ok=True)
Expand All @@ -116,7 +118,7 @@ def prepare(evidence: Path, packets: Path, optional: Path, manifest: Path) -> No
packet = packet_root / secrets.token_hex(8)
packet.mkdir()
for filename in PUBLIC_FILES:
shutil.copyfile(PUBLIC / filename, packet / filename)
shutil.copyfile(public / filename, packet / filename)
if treatment:
shutil.copytree(
optional,
Expand Down Expand Up @@ -149,7 +151,7 @@ def prepare(evidence: Path, packets: Path, optional: Path, manifest: Path) -> No
"max_turns": MAX_TURNS,
"cli_seconds": CLI_BUDGET,
"worker_seconds": 120,
"public_hashes": {f: digest(PUBLIC / f) for f in PUBLIC_FILES},
"public_hashes": {f: digest(public / f) for f in PUBLIC_FILES},
"optional_manifest_sha256": digest(manifest),
"entry_sha256": digest(Path(__file__)),
"service_sha256": digest(SERVICE),
Expand Down Expand Up @@ -394,6 +396,7 @@ def main() -> None:
parser.add_argument("--packets", type=Path)
parser.add_argument("--optional", type=Path)
parser.add_argument("--optional-manifest", type=Path)
parser.add_argument("--public", type=Path, default=PUBLIC)
args = parser.parse_args()
if args.action == "prepare":
if (
Expand All @@ -404,7 +407,13 @@ def main() -> None:
parser.error(
"prepare requires --packets, --optional and --optional-manifest"
)
prepare(args.evidence, args.packets, args.optional, args.optional_manifest)
prepare(
args.evidence,
args.packets,
args.optional,
args.optional_manifest,
args.public,
)
else:
run(args.evidence)

Expand Down
14 changes: 12 additions & 2 deletions tools/token_budget_study/operator/ring_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@
from pathlib import Path
from typing import TypedDict

REMOTE = Path("/home/yuxuan/experiments/traincheck-token-budget-20261118")
DEFAULT_REMOTE = Path("/home/yuxuan/experiments/traincheck-token-budget-20261118")
REMOTE = DEFAULT_REMOTE
PYTHON = (
"/home/yuxuan/experiments/traincheck-token-normalization-20261102/venv/bin/python"
)
Expand Down Expand Up @@ -185,6 +186,9 @@ def run_worker(req: Request, run: Path, source: Path) -> dict[str, object]:
]
if req["with_traincheck"]:
command.extend(["--optional-package", str(REMOTE / "optional/traincheck")])
collective_audit = REMOTE / "private/collectives.py"
if collective_audit.is_file():
command.extend(["--collective-audit", str(collective_audit)])
env = os.environ.copy()
env.update(
TRAINCHECK_STUDY_E_REQUEST=req["id"],
Expand Down Expand Up @@ -299,6 +303,11 @@ def client() -> None:
parser.add_argument("--args-json", default="[]")
parser.add_argument("--with-traincheck", action="store_true")
args = parser.parse_args()
remote = Path(os.environ.get("TRAINCHECK_STUDY_ROOT", str(DEFAULT_REMOTE)))
if remote.parent != DEFAULT_REMOTE.parent or not re.fullmatch(
r"traincheck-[a-z0-9-]+", remote.name
):
raise ValueError("Expected a named TrainCheck experiment directory")
files: dict[str, str] = {}
for name in ("training.py", "diagnostic.py"):
path = args.source / name
Expand All @@ -322,7 +331,7 @@ def client() -> None:
(args.output / "request.json").write_bytes(payload)
since = time.monotonic()
proc = subprocess.run(
[*SSH, PYTHON, str(REMOTE / "ring_service.py"), "serve"],
[*SSH, PYTHON, str(remote / "ring_service.py"), "serve"],
input=payload,
capture_output=True,
timeout=1800,
Expand Down Expand Up @@ -353,6 +362,7 @@ def client() -> None:

if __name__ == "__main__":
if sys.argv[1:] == ["serve"]:
REMOTE = Path(__file__).resolve().parent
print(json.dumps(serve(parse_request(json.load(sys.stdin)))))
else:
client()
1 change: 1 addition & 0 deletions tools/token_budget_study/operator/score_submissions.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
"one-microbatch-updates",
}
OUTCOMES = {
"communication_contract",
"pass",
"silent_wrong",
"unsupported",
Expand Down
9 changes: 9 additions & 0 deletions tools/token_budget_study/private/grade.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
"nonfinite",
"runtime_error",
"unsupported",
"communication_contract",
}


Expand All @@ -39,6 +40,7 @@ def grade(
device: str,
timeout: float,
optional_package: Path | None,
collective_audit: Path | None = None,
) -> None:
output.parent.mkdir(parents=True, exist_ok=True)
# Reserve the path before allocating any worker time.
Expand All @@ -49,6 +51,8 @@ def grade(
for name in ("training.py", "workload.py"):
shutil.copyfile(source / name, scratch / name)
shutil.copyfile(Path(__file__).with_name("worker.py"), scratch / "worker.py")
if collective_audit is not None:
shutil.copyfile(collective_audit, scratch / "collectives.py")
package_files: dict[str, str] = {}
if optional_package is not None:
if (
Expand Down Expand Up @@ -80,6 +84,8 @@ def grade(
"--device",
device,
]
if collective_audit is not None:
command.append("--audit-communication")
with (scratch / "worker.log").open("wb") as log:
proc = subprocess.Popen(
command,
Expand Down Expand Up @@ -141,6 +147,7 @@ def grade(
"nonfinite",
"silent_wrong",
"memory_contract",
"communication_contract",
)
if x in statuses
),
Expand Down Expand Up @@ -173,13 +180,15 @@ def main() -> None:
parser.add_argument("--device", choices=["cpu", "cuda"], default="cuda")
parser.add_argument("--timeout", type=float, default=90)
parser.add_argument("--optional-package", type=Path)
parser.add_argument("--collective-audit", type=Path)
args = parser.parse_args()
grade(
args.source.resolve(),
args.output.resolve(),
args.device,
args.timeout,
args.optional_package,
args.collective_audit,
)


Expand Down
Loading