Repository navigation
Description
Feature area: Capture, instrumentation, and framework adapters; Numerical determinism and numerical alignment
Debugging problem
Inference engines such as vLLM run decode inside CUDA Graphs and compile the model with torch.compile(fullgraph=True). Divergences that exist only on these paths (a branch frozen at capture time, a fused kernel that differs from eager) cannot be reproduced in eager mode, and OpGuard cannot observe them today.
Official workflow (eager preflight → instrument → run) on vLLM (Qwen2.5-0.5B):
| Mode | Callsites inside the compiled region | Callsites outside |
|---|---|---|
enforce_eager |
Recorded (62/70 callsites, 4203 records) | Recorded |
| torch.compile only | vLLM fails to start | Recorded |
| torch.compile + CUDA Graph | vLLM fails to start | Recorded |
- Graph replay produces no records. The XOR kernel is captured and recomputes correct fingerprints on every replay, but the record is built by the Python wrapper, which replay does not run.
- The capture-time record is not a snapshot. It references a fingerprint tensor in the graph memory pool, so its value depends on when the dump thread reads it (
[0,0,0,0]before the first replay, the latest replay afterwards). It never corresponds to a specific execution. fullgraph=Truefails. Dynamo cannot trace the wrapper: it stops atContextVar.getin_in_tracer()(runtime_tracer.py:213), and counter reservation, timestamps, stack capture and buffering are untraceable as well. Withoutfullgraphthis presumably degrades to graph breaks (not verified), which engines requiring full-graph compilation do not allow.
Root cause. Records are produced by Python side effects on every call. Replay runs only captured GPU work, and compiled code runs only the ops in the compiled graph, not the original Python. Any recording that must survive these paths has to be GPU work inside an op the compiler can see.
Minimal CUDA Graph reproducer: https://github.com/Jayden-Xu/OpGuard/tree/repro/cuda-graph-replay
Desired behavior
One record per execution, holding that execution's fingerprints, in eager, compiled and graph-replayed runs alike. Records use the existing JSONL format (gpu_XXX/ops/<func>/<func>_line<N>.jsonl), so opguard compare pairs eager and graph/compiled runs and reports the first divergent operator as it does today. Origin (eager / replay) and capture/replay identity are stored under __-prefixed keys, which comparison already ignores (utils.py:42).
Workload and execution context
- Any CUDA Graph capture (manual
torch.cuda.CUDAGraph; vLLM FULL / PIECEWISE / FULL_AND_PIECEWISE;reduce-overhead) andtorch.compilewith or withoutfullgraph. - Validation order: vLLM full graphs without compilation first (isolates replay from compile), vLLM V1's default (compile + FULL_AND_PIECEWISE) last.
- Observed with OpGuard main
372d645on RTX 3090 (torch 2.11 + cu128) and L20 (torch 2.13 + cu130), with the native XOR extension and with the tensor-op fallback.
Numerical or alignment contract
- Comparison stays bitwise on fingerprints (
tensor_equality: not_establishedunchanged). Alignment logic is unchanged. counteris the global commit order. Records are sorted bycounterbefore alignment (build.py), the Perfetto timeline is ordered by it (schedule.py), andfirst_diffwalks that timeline (trace_reader.py). Per-callsite numbering would misplace the first divergence. For a callsite on a single stream, global commit order equals execution order.- Records match the existing mode field by field, because
align_recordsuses the fullinputsas anchors. Fingerprints followsummarize_tensor: device-side XOR for CUDA tensors (including the existingcontiguous()/resolve_conj/resolve_negpath); the existing static summaries for large CPU tensors, non-viewable tensors and meta/fake tensors. Small CPU tensors, fingerprinted with numpy today, get a static summary in this mode; this is a documented difference. Stacks reuse the existing per-(thread_id, tag_prefix, lineno)cache (runtime_tracer.py), so they are already static per callsite. - Instrumentation must not change what it observes. Under compile, probing an intermediate tensor can block fusion and change numerics. Instrumented and uninstrumented final outputs must be bitwise equal; boundaries that are materialized anyway are preferred.
- Padding. vLLM pads graph batches to the capture size. Pairs with different shapes are reported as not comparable at first. Later, for token-major tensors, the kernel fingerprints only valid rows (count read from a device scalar written before each replay), without copying or modifying model tensors. Valid rows can still differ when kernels pick different algorithms for different M; that is a real difference, expected to vanish under batch-invariant kernels.
- Extra runs. Warmup and capture add records with no eager counterpart (4 / 9 / 22 records for one callsite across the three modes).
Capture and agent workflow
Each design choice follows from one constraint:
| Constraint | Design choice |
|---|---|
| Replay and compiled code run only GPU work and compiler-visible ops | Each callsite becomes a torch.library custom op, opguard::probe, that only launches GPU work. The Python wrapper outside it becomes thin. |
| Compilers may remove or reorder ops whose side effects they cannot see | Mutated buffers are op arguments: probe(Tensor[] tensors, int site, Tensor(a!) ring, Tensor(b!) cursor, Tensor(c!) counters, Tensor enable). Global tensors referenced inside the op would hide the mutation. |
| A replayed kernel reuses its captured launch arguments but reads device memory fresh | Slots are allocated with atomicAdd on a device-side global cursor, so every replay writes a new slot. |
| One call must be one record | Native XOR kernels fingerprint the call's CUDA tensors, then one small commit kernel writes the whole call into one slot of a global ring. |
| Work issued during capture is replayed | Ring, cursor, counters and enable flags are allocated outside capture. A ring initialized during capture is reset on every replay. |
| Readback can race with ring overflow or an in-flight write | Each slot stores its global sequence number, written last after __threadfence(). Readback accepts a slot only if the stored number matches the expected one; otherwise it is a gap. |
| Lost entries must not shift later pairing | A per-callsite call number (per-callsite atomicAdd in the commit kernel, used only for numbering) locates gaps. Gaps become placeholder records, and compare reports them as undetermined, not as diffs. |
- Entry layout: callsite id, per-callsite call number, fingerprints of up to T tensors (4 × uint32 each), global sequence number; about 24 B + 16 B per tensor. The tensor count is fixed at registration. Because the serializer expands args, kwargs,
selfand outputs recursively, counts can be large; callsites over T are rejected in this mode with a clear error. - Host side: static metadata (function, file, line, cached stack, tensor layout per capture) is registered once. The ring is read back asynchronously at safe points (never during capture) and materialized into JSONL.
- Streams:
atomicAddkeeps slots unique across streams. A callsite's stream is recorded at registration; if the callsite later runs on another stream (e.g. dual-batch overlap), it is reported with a warning and its records are marked order-uncertain. - Enable flags: per-callsite flags live on the device. A disabled callsite returns at the start of every kernel, so narrowing an investigation needs no recompilation and adds almost no cost.
Compatibility
- Opt-in mode. The existing wrapper path stays the default and is unchanged.
- Requires the native extension (XOR and commit kernels); there is no tensor-op fallback. Enabling the mode without the extension fails with installation guidance.
- Both runs in a comparison must use the same mode; mixing modes is rejected, not paired.
- Not available in this mode: full tensor dumps, statistics that need
.item()(NaN/Inf counts), timestamps and durations, callsites over T tensors or with varying tensor counts. - Non-tensor arguments are recorded statically at registration/capture time.
- Input fingerprints are taken after the operator runs, as today (in-place operators record post-mutation inputs).
- Parity test: for the same eager run, both modes give identical fingerprints and pairings, with small CPU tensors documented as the expected difference.
First milestone and follow-up scope
- Eager + manual
torch.cuda.CUDAGraph: probe op, native XOR + commit, readback with sequence validation, materialization, gap placeholders, cross-stream detection, parity test. Validated on vLLM full graphs without compilation. torch.compile(fullgraph=True)on the same op, plus runtime enable flags. Validated on vLLM's default configuration.- Valid-row fingerprints for padded graph batches.
Open questions:
- Ring size and readback. The ring must hold at least one replay's calls. Reading back after every replay minimizes memory but needs a stall or double buffering (2× memory); less frequent readback needs a larger ring. Size is derived from the capture template and the enabled callsites and capped by a user budget; overflow is reported as gaps. An optional stop-when-full policy keeps the earliest records, which usually contain the first divergence.
- Overhead. Extra kernels can slow small decode graphs, and materializing many JSON records per step may cost more than the GPU side. Plan: measure replay time (none / fingerprints only / fingerprints + commit), store raw ring data during the run, materialize offline. Possible optimization: two launches per call regardless of tensor count (one kernel computes partials for all tensors, with pointers passed by value as kernel arguments; one finishes and commits).
- Registration. Python
torch.library.custom_opvs C++TORCH_LIBRARY. The latter also covers models run from C++ (TorchScript, libtorch, AOTInductor). - Compile risks. Whether Inductor preserves probe order and the ring mutation, how much probing changes fusion, and behavior on torch 2.4 (minimum supported). CUDA Graph trees (
reduce-overhead) may skip graphs with mutated inputs; vLLM uses its own graph wrapper (to confirm before milestone 2), andtorch._dynamo.mark_static_addresson the ring buffers is the expected mitigation.
Acceptance and validation
- N replays produce N records per callsite, with fingerprints equal to an independent numpy computation.
- Eager vs graph runs of the reproducer pair 100% by
callsite_sequence_orderwith all fingerprints equal. - Parity between modes on the same eager run (small CPU tensors documented).
- A toy model under
torch.compile(fullgraph=True), with and without CUDA Graph, records every call; instrumented and uninstrumented outputs are bitwise equal. - An injected graph-only divergence is reported at the correct operator by
first_diff. - Ring overflow and partially written slots are detected by sequence validation and become gap placeholders; later records at those callsites still pair correctly.
- A callsite on two streams is reported as order-uncertain; a callsite over T tensors is rejected with a clear error.
- Enabling the mode without the native extension fails with a clear message.
- Padded-shape pairs are reported as not comparable (milestone 1) and are equal after valid-row fingerprinting under batch-invariant kernels (milestone 3).
- Overhead measured on a small vLLM decode graph: none / fingerprints only / fingerprints + commit.
Related roadmap items and evidence
- Tracking:
- Milestone 1: eager + CUDA Graph
- Milestone 2: torch.compile (
fullgraph=True) - Milestone 3: padded batches
- Prototype (hand-written wrapper with tensor-op commit, single stream, RTX 3090, torch 2.13; observed locally): a captured commit writes a new slot on every replay; a ring initialized inside capture is reset on every replay; eager and graph records align after ordering by commit sequence.
- vLLM experiment above: official flow, observed locally.
- The existing CUPTI replay mapping (
opguard_cupti.cpp) tracks kernel activity during replay but not tensor values. - PyTorch's prototype
DebugMode(2.10+) logs per-op tensor hashes, including Inductor-generated Triton kernels, and compares eager and compiled runs within one process. It does not appear to cover CUDA Graph replay, cross-run or cross-framework alignment, or distributed capture; the two could complement each other.
Activity
[-][Feature]: Record operator fingerprints during CUDA Graph replay[/-][+][Feature]: Device-side fingerprint recording for CUDA Graph replay and torch.compile[/+]Jayden-Xu commentedon Nov 15, 2026
Update: prototype results and design changes
A standalone prototype (outside OpGuard) tested the core mechanism. Where this comment and the issue text differ, this comment supersedes it.
Environment: RTX 3090, torch 2.14.1+cu132, triton 3.8.0, reference OpGuard main
74b6807.Prototype results
xor_signature(t, regional=4, by="words")torch.cuda.CUDAGraphtorch.compile(fullgraph=True)mmoutput prevents fusingmm+addintoaddmm, adding one bf16 roundingDesign changes
torch.library.custom_op. Replay runs no host code, and per-call host cost is far below the current wrapper. A C++ registration with the same name and schema can be added later if models run from C++ (e.g. AOTInductor) need it.torch.cuda.CUDAGraph.replay, gives each replayed record its replay identity and the meta in effect at that replay. Pure-CPU calls inside a captured region still commit on every replay and are flagged__frozen_at_capture. Small CPU tensors therefore keep their numpy fingerprints, and the parity test has no expected differences.Plan
Milestone 1 is split into two PRs:
torch.cuda.CUDAGraph. Not wired into instrumentation.Milestones 2 (compile) and 3 (padding) are unchanged.