Skip to content

[Feature]: Device-side fingerprint recording for CUDA Graph replay and torch.compile #8

Open

Description

@Jayden-Xu

Feature area: Capture, instrumentation, and framework adapters; Numerical determinism and numerical alignment

Debugging problem

Inference engines such as vLLM run decode inside CUDA Graphs and compile the model with torch.compile(fullgraph=True). Divergences that exist only on these paths (a branch frozen at capture time, a fused kernel that differs from eager) cannot be reproduced in eager mode, and OpGuard cannot observe them today.

Official workflow (eager preflight → instrument → run) on vLLM (Qwen2.5-0.5B):

Mode Callsites inside the compiled region Callsites outside
enforce_eager Recorded (62/70 callsites, 4203 records) Recorded
torch.compile only vLLM fails to start Recorded
torch.compile + CUDA Graph vLLM fails to start Recorded
  • Graph replay produces no records. The XOR kernel is captured and recomputes correct fingerprints on every replay, but the record is built by the Python wrapper, which replay does not run.
  • The capture-time record is not a snapshot. It references a fingerprint tensor in the graph memory pool, so its value depends on when the dump thread reads it ([0,0,0,0] before the first replay, the latest replay afterwards). It never corresponds to a specific execution.
  • fullgraph=True fails. Dynamo cannot trace the wrapper: it stops at ContextVar.get in _in_tracer() (runtime_tracer.py:213), and counter reservation, timestamps, stack capture and buffering are untraceable as well. Without fullgraph this presumably degrades to graph breaks (not verified), which engines requiring full-graph compilation do not allow.

Root cause. Records are produced by Python side effects on every call. Replay runs only captured GPU work, and compiled code runs only the ops in the compiled graph, not the original Python. Any recording that must survive these paths has to be GPU work inside an op the compiler can see.

Minimal CUDA Graph reproducer: https://github.com/Jayden-Xu/OpGuard/tree/repro/cuda-graph-replay

Desired behavior

One record per execution, holding that execution's fingerprints, in eager, compiled and graph-replayed runs alike. Records use the existing JSONL format (gpu_XXX/ops/<func>/<func>_line<N>.jsonl), so opguard compare pairs eager and graph/compiled runs and reports the first divergent operator as it does today. Origin (eager / replay) and capture/replay identity are stored under __-prefixed keys, which comparison already ignores (utils.py:42).

Workload and execution context

  • Any CUDA Graph capture (manual torch.cuda.CUDAGraph; vLLM FULL / PIECEWISE / FULL_AND_PIECEWISE; reduce-overhead) and torch.compile with or without fullgraph.
  • Validation order: vLLM full graphs without compilation first (isolates replay from compile), vLLM V1's default (compile + FULL_AND_PIECEWISE) last.
  • Observed with OpGuard main 372d645 on RTX 3090 (torch 2.11 + cu128) and L20 (torch 2.13 + cu130), with the native XOR extension and with the tensor-op fallback.

Numerical or alignment contract

  • Comparison stays bitwise on fingerprints (tensor_equality: not_established unchanged). Alignment logic is unchanged.
  • counter is the global commit order. Records are sorted by counter before alignment (build.py), the Perfetto timeline is ordered by it (schedule.py), and first_diff walks that timeline (trace_reader.py). Per-callsite numbering would misplace the first divergence. For a callsite on a single stream, global commit order equals execution order.
  • Records match the existing mode field by field, because align_records uses the full inputs as anchors. Fingerprints follow summarize_tensor: device-side XOR for CUDA tensors (including the existing contiguous() / resolve_conj / resolve_neg path); the existing static summaries for large CPU tensors, non-viewable tensors and meta/fake tensors. Small CPU tensors, fingerprinted with numpy today, get a static summary in this mode; this is a documented difference. Stacks reuse the existing per-(thread_id, tag_prefix, lineno) cache (runtime_tracer.py), so they are already static per callsite.
  • Instrumentation must not change what it observes. Under compile, probing an intermediate tensor can block fusion and change numerics. Instrumented and uninstrumented final outputs must be bitwise equal; boundaries that are materialized anyway are preferred.
  • Padding. vLLM pads graph batches to the capture size. Pairs with different shapes are reported as not comparable at first. Later, for token-major tensors, the kernel fingerprints only valid rows (count read from a device scalar written before each replay), without copying or modifying model tensors. Valid rows can still differ when kernels pick different algorithms for different M; that is a real difference, expected to vanish under batch-invariant kernels.
  • Extra runs. Warmup and capture add records with no eager counterpart (4 / 9 / 22 records for one callsite across the three modes).

Capture and agent workflow

Each design choice follows from one constraint:

Constraint Design choice
Replay and compiled code run only GPU work and compiler-visible ops Each callsite becomes a torch.library custom op, opguard::probe, that only launches GPU work. The Python wrapper outside it becomes thin.
Compilers may remove or reorder ops whose side effects they cannot see Mutated buffers are op arguments: probe(Tensor[] tensors, int site, Tensor(a!) ring, Tensor(b!) cursor, Tensor(c!) counters, Tensor enable). Global tensors referenced inside the op would hide the mutation.
A replayed kernel reuses its captured launch arguments but reads device memory fresh Slots are allocated with atomicAdd on a device-side global cursor, so every replay writes a new slot.
One call must be one record Native XOR kernels fingerprint the call's CUDA tensors, then one small commit kernel writes the whole call into one slot of a global ring.
Work issued during capture is replayed Ring, cursor, counters and enable flags are allocated outside capture. A ring initialized during capture is reset on every replay.
Readback can race with ring overflow or an in-flight write Each slot stores its global sequence number, written last after __threadfence(). Readback accepts a slot only if the stored number matches the expected one; otherwise it is a gap.
Lost entries must not shift later pairing A per-callsite call number (per-callsite atomicAdd in the commit kernel, used only for numbering) locates gaps. Gaps become placeholder records, and compare reports them as undetermined, not as diffs.
  • Entry layout: callsite id, per-callsite call number, fingerprints of up to T tensors (4 × uint32 each), global sequence number; about 24 B + 16 B per tensor. The tensor count is fixed at registration. Because the serializer expands args, kwargs, self and outputs recursively, counts can be large; callsites over T are rejected in this mode with a clear error.
  • Host side: static metadata (function, file, line, cached stack, tensor layout per capture) is registered once. The ring is read back asynchronously at safe points (never during capture) and materialized into JSONL.
  • Streams: atomicAdd keeps slots unique across streams. A callsite's stream is recorded at registration; if the callsite later runs on another stream (e.g. dual-batch overlap), it is reported with a warning and its records are marked order-uncertain.
  • Enable flags: per-callsite flags live on the device. A disabled callsite returns at the start of every kernel, so narrowing an investigation needs no recompilation and adds almost no cost.

Compatibility

  • Opt-in mode. The existing wrapper path stays the default and is unchanged.
  • Requires the native extension (XOR and commit kernels); there is no tensor-op fallback. Enabling the mode without the extension fails with installation guidance.
  • Both runs in a comparison must use the same mode; mixing modes is rejected, not paired.
  • Not available in this mode: full tensor dumps, statistics that need .item() (NaN/Inf counts), timestamps and durations, callsites over T tensors or with varying tensor counts.
  • Non-tensor arguments are recorded statically at registration/capture time.
  • Input fingerprints are taken after the operator runs, as today (in-place operators record post-mutation inputs).
  • Parity test: for the same eager run, both modes give identical fingerprints and pairings, with small CPU tensors documented as the expected difference.

First milestone and follow-up scope

  1. Eager + manual torch.cuda.CUDAGraph: probe op, native XOR + commit, readback with sequence validation, materialization, gap placeholders, cross-stream detection, parity test. Validated on vLLM full graphs without compilation.
  2. torch.compile(fullgraph=True) on the same op, plus runtime enable flags. Validated on vLLM's default configuration.
  3. Valid-row fingerprints for padded graph batches.

Open questions:

  • Ring size and readback. The ring must hold at least one replay's calls. Reading back after every replay minimizes memory but needs a stall or double buffering (2× memory); less frequent readback needs a larger ring. Size is derived from the capture template and the enabled callsites and capped by a user budget; overflow is reported as gaps. An optional stop-when-full policy keeps the earliest records, which usually contain the first divergence.
  • Overhead. Extra kernels can slow small decode graphs, and materializing many JSON records per step may cost more than the GPU side. Plan: measure replay time (none / fingerprints only / fingerprints + commit), store raw ring data during the run, materialize offline. Possible optimization: two launches per call regardless of tensor count (one kernel computes partials for all tensors, with pointers passed by value as kernel arguments; one finishes and commits).
  • Registration. Python torch.library.custom_op vs C++ TORCH_LIBRARY. The latter also covers models run from C++ (TorchScript, libtorch, AOTInductor).
  • Compile risks. Whether Inductor preserves probe order and the ring mutation, how much probing changes fusion, and behavior on torch 2.4 (minimum supported). CUDA Graph trees (reduce-overhead) may skip graphs with mutated inputs; vLLM uses its own graph wrapper (to confirm before milestone 2), and torch._dynamo.mark_static_address on the ring buffers is the expected mitigation.

Acceptance and validation

  • N replays produce N records per callsite, with fingerprints equal to an independent numpy computation.
  • Eager vs graph runs of the reproducer pair 100% by callsite_sequence_order with all fingerprints equal.
  • Parity between modes on the same eager run (small CPU tensors documented).
  • A toy model under torch.compile(fullgraph=True), with and without CUDA Graph, records every call; instrumented and uninstrumented outputs are bitwise equal.
  • An injected graph-only divergence is reported at the correct operator by first_diff.
  • Ring overflow and partially written slots are detected by sequence validation and become gap placeholders; later records at those callsites still pair correctly.
  • A callsite on two streams is reported as order-uncertain; a callsite over T tensors is rejected with a clear error.
  • Enabling the mode without the native extension fails with a clear message.
  • Padded-shape pairs are reported as not comparable (milestone 1) and are equal after valid-row fingerprinting under batch-invariant kernels (milestone 3).
  • Overhead measured on a small vLLM decode graph: none / fingerprints only / fingerprints + commit.

Related roadmap items and evidence

  • Tracking:
    • Milestone 1: eager + CUDA Graph
    • Milestone 2: torch.compile (fullgraph=True)
    • Milestone 3: padded batches
  • Prototype (hand-written wrapper with tensor-op commit, single stream, RTX 3090, torch 2.13; observed locally): a captured commit writes a new slot on every replay; a ring initialized inside capture is reset on every replay; eager and graph records align after ordering by commit sequence.
  • vLLM experiment above: official flow, observed locally.
  • The existing CUPTI replay mapping (opguard_cupti.cpp) tracks kernel activity during replay but not tensor values.
  • PyTorch's prototype DebugMode (2.10+) logs per-op tensor hashes, including Inductor-generated Triton kernels, and compares eager and compiled runs within one process. It does not appear to cover CUDA Graph replay, cross-run or cross-framework alignment, or distributed capture; the two could complement each other.

Activity

  1. self-assigned this
    on Nov 14, 2026
  2. changed the title [-][Feature]: Record operator fingerprints during CUDA Graph replay[/-] [+][Feature]: Device-side fingerprint recording for CUDA Graph replay and torch.compile[/+] on Nov 15, 2026
  3. Jayden-Xu commented on Nov 15, 2026

    @Jayden-Xu
    CollaboratorAuthor

    Update: prototype results and design changes

    A standalone prototype (outside OpGuard) tested the core mechanism. Where this comment and the issue text differ, this comment supersedes it.

    Environment: RTX 3090, torch 2.14.1+cu132, triton 3.8.0, reference OpGuard main 74b6807.

    Prototype results

    Check Result
    Device-side fingerprints vs xor_signature(t, regional=4, by="words") Bitwise equal on 411 cases (11 dtypes, empty, scalar, conj/neg views, non-contiguous and unaligned views); in-place ops record post-mutation inputs
    Manual torch.cuda.CUDAGraph Capture writes nothing; 50 replays give one record per callsite per replay (350/350), each matching that replay's inputs
    Ring overflow and readback concurrent with replay 0 false accepts over >1M calls; overwritten slots become gap placeholders
    torch.compile(fullgraph=True) 0 graph breaks; every probe kept, order preserved; no ring clone/copy left after functionalization
    Instrumented vs uninstrumented compiled outputs Equal in fp32, different in bf16 (2,788 of 10,240 elements, max abs 0.03125): probing an mm output prevents fusing mm + add into addmm, adding one bf16 rounding

    Design changes

    1. Readback (snap). Reading the cursor on the copy stream after an event can include commits still in flight (observed in 398 of 401 readbacks with a delayed copy). Readback now snapshots the cursor and per-callsite counters with a device-to-device copy on the recording stream, then records the event; the host copies the snapshot, the ring, and the cursor again. A slot is accepted only if its sequence number matches and the re-read cursor shows no later lap reached it. Correctness depends on stream order, not DMA byte order.
    2. Per-callsite scratch. Fingerprint partials are accumulated per callsite, so callsites on different streams can run concurrently. A callsite seen on two streams at once (e.g. dual-batch overlap) is rejected in this mode.
    3. Compile contract. Probing a value the compiler would not otherwise materialize can change fusion and, in low precision, numerics. Probes record exactly what the instrumented program computes, so comparing two runs instrumented the same way stays valid. The remaining risk is that instrumentation masks or moves the divergence; the workflow checks that the instrumented runs still diverge in their final outputs and narrows probes otherwise.
    4. Registration. Python torch.library.custom_op. Replay runs no host code, and per-call host cost is far below the current wrapper. A C++ registration with the same name and schema can be added later if models run from C++ (e.g. AOTInductor) need it.
    5. Host-side record joined by a call id. Whenever the probe's Python implementation runs (eager, compiled without graphs, once during capture), it records shapes, non-tensor arguments, meta, timestamps and CPU-tensor fingerprints (numpy, as today), and passes a call id to the commit kernel as an argument. Eager calls get unique ids; during capture the id is frozen into the graph, so each replayed slot points at its capture-time template, which is what the replay executes. A device-side replay counter, incremented before each torch.cuda.CUDAGraph.replay, gives each replayed record its replay identity and the meta in effect at that replay. Pure-CPU calls inside a captured region still commit on every replay and are flagged __frozen_at_capture. Small CPU tensors therefore keep their numpy fingerprints, and the parity test has no expected differences.
    6. Ring sizing without gaps. The ring's address and capacity live in a device-side cell read by the commit kernel, not in kernel arguments frozen at capture. The ring is allocated after capture from the exact number of probe calls counted during capture (plus host-counted eager calls), can be swapped for double buffering without stalls, and can be replaced by a larger one without recapture. The host checks the remaining space before issuing work, so overflow does not occur in normal operation; sequence validation and gap placeholders remain as a safety net. The per-call tensor limit T is taken per callsite from the discovery run (overridable), and estimated memory is printed at startup.
    7. Entry layout. Global sequence number (written last), callsite id, tensor count, per-callsite call number, call id, replay number (about 40 B), then up to T fingerprints of 4 × uint32.

    Plan

    Milestone 1 is split into two PRs:

    1. Device layer: probe op, kernels, ring and readback, tested on synthetic workloads under eager and manual torch.cuda.CUDAGraph. Not wired into instrumentation.
    2. Integration: wrapper, callsite registration, host-side records, JSONL materialization, parity test; validated on vLLM full graphs without compilation, including an overhead measurement.

    Milestones 2 (compile) and 3 (padding) are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions