[GrinZero]

Summary

This PR supersedes #62915 with the same feature work rebased onto current main.

It adds builtin http/https request-body support to network inspection, so Network.getRequestPostData can return text request bodies while preserving the existing rejection behavior for binary request bodies.

It also moves builtin http response-body tracking to a raw-byte hook before IncomingMessage decoding, so response inspection remains correct even when user code calls response.setEncoding(...).

In addition, this PR lets Network.requestWillBeSent carry JS-provided initiator data captured at request construction time. The C++ inspector side validates structured initiator.stack objects against the inspector protocol schema before forwarding them to DevTools.

Problem

Builtin http/https network inspection currently emits request and response lifecycle events, but the builtin client path does not expose request-body bytes to the inspector. As a result, Network.getRequestPostData cannot return POST data for builtin http/https requests.

There are two related gaps:

• IncomingMessage 'data' events are not a stable raw-byte source. If user code calls response.setEncoding('utf8'), chunks observed through userland become strings, while the inspector protocol expects byte-oriented payloads.
• Delaying requestWillBeSent until request-body state is known means a C++ stack captured at emission time would point into Node.js internals rather than the user call site. The initiator needs to be captured when the request is created and validated before being emitted.

Approach

1. Reuse the existing Network.dataSent(...) -> NetworkAgent::getRequestPostData(...) buffering pipeline instead of changing the CDP schema.
2. Add builtin HTTP diagnostics events for request body chunks and request body completion:
   • http.client.request.bodyChunkSent
   • http.client.request.bodySent
3. Capture builtin HTTP response body bytes through a parser-side raw-byte diagnostics event before stream decoding:
   • http.client.response.bodyChunkReceived
4. Defer requestWillBeSent until body state is known, so hasPostData is accurate for both body and body-less requests.
5. Capture the initiator stack in JS at request creation time, pass it through the inspector binding, and validate initiator.stack with generated inspector protocol schema support in C++.
6. Document the new diagnostics channels and message shapes.

Behavior

After this change:

• builtin http and https POST requests with UTF-8 text bodies are available through Network.getRequestPostData
• binary request bodies still reject with the existing inspector error behavior
• builtin http response inspection continues to work even if user code calls response.setEncoding('utf8')
• Network.requestWillBeSent reports accurate hasPostData
• Network.requestWillBeSent can carry JS-captured initiator stack data from the request construction site
• malformed initiator stack input is rejected by C++ schema validation

Tests

This PR adds and extends coverage in:

• test/parallel/test-diagnostics-channel-http.js
• test/parallel/test-inspector-network-http.js
• test/parallel/test-inspector-emit-protocol-event.js
• test/parallel/test-inspector-emit-protocol-event-errors.js

The updated tests cover:

• request body chunk and request body finished diagnostics events
• response body raw-byte diagnostics events
• text request bodies split across write() and end()
• binary request bodies
• http and https Network.getRequestPostData
• binary request-body rejection semantics
• response inspection when the client calls response.setEncoding('utf8')
• accurate hasPostData for body-less requests
• JS-provided initiator stack parsing and forwarding
• malformed initiator stack error handling

Verification

Validated with the targeted test suite:

python3 tools/test.py \
  parallel/test-diagnostics-channel-http \
  parallel/test-inspector-network-http \
  parallel/test-inspector-emit-protocol-event \
  parallel/test-inspector-emit-protocol-event-errors

Also validated manually with an end-to-end inspector run:

./out/Release/node --inspect --experimental-network-inspection ./test.js

Refs

• Replaces inspector,http: support builtin http request bodies #62915
• Refs Tracking issue: Stabilizing network inspection in Node.js #53946

[GrinZero]

e2e same like: #62915 (comment)

[nodejs-github-bot]

Review requested:

• @nodejs/http
• @nodejs/inspector
• @nodejs/net

[codecov]

Codecov Report

❌ Patch coverage is 96.09756% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.70%. Comparing base (7b4c9e3) to head (7f96de8).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
src/inspector/network_agent.cc	88.57%	1 Missing and 3 partials ⚠️
src/inspector/network_agent.h	25.00%	3 Missing ⚠️
src/inspector_agent.cc	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #64369      +/-   ##
==========================================
- Coverage   90.71%   90.70%   -0.01%     
==========================================
  Files         801      803       +2     
  Lines      281889   282296     +407     
  Branches    54315    54391      +76     
==========================================
+ Hits       255709   256066     +357     
- Misses      16307    16375      +68     
+ Partials     9873     9855      -18     

Files with missing lines	Coverage Δ
lib/internal/http.js	92.56% <100.00%> (+0.04%)	⬆️
lib/internal/http/client.js	97.20% <100.00%> (+0.05%)	⬆️
lib/internal/http/common.js	98.61% <100.00%> (+0.06%)	⬆️
lib/internal/http/outgoing.js	98.01% <100.00%> (+0.02%)	⬆️
lib/internal/inspector/network_http.js	97.15% <100.00%> (+1.44%)	⬆️
src/inspector_agent.h	100.00% <ø> (ø)
src/inspector_js_api.cc	77.55% <100.00%> (+0.26%)	⬆️
src/inspector_agent.cc	83.41% <75.00%> (+0.21%)	⬆️
src/inspector/network_agent.h	25.00% <25.00%> (ø)
src/inspector/network_agent.cc	88.02% <88.57%> (-0.02%)	⬇️

... and 46 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[GrinZero]

I think it's time to conduct a re-review. Thank you. @metcoder95 @ShogunPanda @mcollina

[GrinZero]

push

[GrinZero]

push

[GrinZero]

Friendly ping on the next step.

The existing review feedback has been addressed, @metcoder95 has approved, and the PR is currently mergeable. It is still labeled "needs-ci" — could a collaborator please trigger the full CI?

If CI is green, could @nodejs/http and @nodejs/inspector give it a final look or let me know if anything else is needed before landing?

Happy to address any follow-up feedback promptly. Thanks!

[GrinZero]

push

[GrinZero]

push... @jasnell Can you help me add the label to run the CI

[GrinZero]

Hi @mcollina , I’ve rebased #64369 onto the latest main and resolved the HTTP outgoing conflict. The PR is mergeable again, and CI is running. The targeted HTTP/Inspector tests pass locally. When you have a chance, could you review the HTTP integration, particularly the request-body tracking and raw response-body hook? I’d appreciate your feedback and will address any comments promptly. Thanks!
And, can you help me add the label to run the CI...

[mcollina]

lgtm

[mcollina]

Can you do a run of the benchmarks?

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/78032/

[GrinZero]

Can you do a run of the benchmarks?

I tried comparing the main branch and found that the newly added features resulted in a performance drop of around 18%. Due to the collection of body data, this decrease is actually within expectations. But did CI just fail? I think while investigating, I can also see if there are any ways to optimize.

GrinZero is missing the Overall/Read permission

I can’t access the Jenkins results due to the CI read permissions. Both node-test-commit/92841 and node-test-pull-request/78032 failed, although the reported platform checks appear green. Could someone with CI access share the failing job/platform, test name, and relevant console output? If the failure is unrelated to this change, could you please rerun the affected job? Thanks! @mcollina

[GrinZero]

I reduced the inspection overhead by encoding initiator stacks directly into CBOR, removing a redundant request-body copy, and simplifying the body-completion notification. Async caller frames and validation checks are preserved. With Network inspection enabled on both sides, local benchmarks showed an 8.1%–9.7% throughput improvement over the previous implementation; throughput remains approximately 10%–11% below the main baseline. Regression tests and the remote Inspector E2E passed locally.

[GrinZero]

@mcollina plz cc again.

[GrinZero]

I re-ran Jenkins and it was successful. It seems that the linting failure was indeed a flake.

Note that GitHub is currently suffering from an incident https://www.githubstatus.com/incidents/smtdswd1h2w4 (and caused the first Jenkins re-run to fail with "fatal: Could not read from remote repository.")

The next step is for a collaborator familiar with this subject matter to review the latest commit, and land it, if it's approved.

Thank you, sir. I have been troubled by this for a long time, and I never expected to make such rapid progress after meeting you. Thank you

[MikeMcC399]

Correction: The linting error was due to a bug in #66837 which has been corrected. Jenkins CI rebases on main before it runs, so the repeat run was rebased on the fix, and that is why it worked.

[GrinZero]

Correction: The linting error was due to a bug in #66837 which has been corrected. Jenkins CI rebases on main before it runs, so the repeat run was rebased on the fix, and that is why it worked.

That sounds like good news - it means the issue isn't due to this PR, so we can move forward more smoothly

[MikeMcC399]

The gating factor is the availability of a collaborator to review, approve and land your PR.

[GrinZero]

@mcollina @legendecas @jasnell The CI (including Jenkins) for this PR has been completed. Please help review the code. Thank you

[legendecas]

I tried comparing the main branch and found that the newly added features resulted in a performance drop of around 18%. Due to the collection of body data, this decrease is actually within expectations.

This seems non-trivial. Which benchmark did you run?

[GrinZero]

I tried comparing the main branch and found that the newly added features resulted in a performance drop of around 18%. Due to the collection of body data, this decrease is actually within expectations.

This seems non-trivial. Which benchmark did you run?

@legendecas

Hello~

Thank you for taking the time to review this PR!

Let me explain why I consider some additional overhead expected, and how I ran the benchmarks.

First, the approximately 18% figure in my earlier comment predates the optimizations. The latest recorded local benchmark results show a 10.0%–11.0% throughput reduction compared with the main baseline, with Network inspection enabled on both sides.

The additional overhead is expected because the PR extends the work performed during inspection: it retains request-body data for Network.getRequestPostData and captures and serializes initiator stacks at request construction time. These operations add CPU and memory work to each inspected request. The benchmark repeatedly sends small requests, making that per-request work visible in throughput.

I used Node’s existing benchmark/http/client-request-body.js through benchmark/compare.js:

• Environment: Apple M2 Max, macOS arm64, Release builds.
• Baseline: main at cff7b12df1f9.
• Workload: local HTTP POST requests with 1 KiB bodies, using UTF-8 strings and Buffers, through both write() and end().
• Sampling: 12 runs per binary and configuration, lasting 3 seconds each.
• Inspection: both binaries used --experimental-network-inspection, with a preload script connecting an Inspector Session and calling Network.enable before measurement. Logs confirmed Network events on both sides.

The results after optimization were:

Body type	Method	Main (requests/s)	Optimized PR (requests/s)	Change
UTF-8 string	write()	9,777	8,703	−10.99%
UTF-8 string	end()	9,812	8,734	−10.99%
Buffer	write()	9,848	8,860	−10.03%
Buffer	end()	9,887	8,843	−10.55%

The optimizations improved throughput by 8.1%–9.7% over the previous implementation, through direct CBOR encoding of initiator stacks, removal of a redundant request-body copy, and simplification of body-completion notification.

Separately, the inspection-disabled HTTP benchmarks in the original run showed no statistically significant regression.

These measurements were taken on September 30, before the later rebases. I have retained the benchmark commands, preload script, and raw CSV results for reproducibility.

[GrinZero]

@marcastben-hash Thanks for reviewing the HTTP integration. These are useful points. My preference would be to address them in focused follow-up PRs rather than make all four prerequisites for landing, if the HTTP and Inspector maintainers are comfortable with that.

1. Request timing: I agree that taking the timestamps when the deferred event is emitted changes the timing shown by the inspector. A body-less request can lose time spent waiting in the Agent queue, whereas a request with an early non-empty write is reported from that write. Capturing the timestamps at request creation would preserve the previous timing origin. That alone would not make queued requests appear earlier in DevTools, so earlier visibility needs separate consideration. I would propose addressing these in a focused inspector timing follow-up.
2. Channel semantics: These events describe the application writing body data and ending the body, not the bytes reaching the peer or necessarily being flushed to the socket. I agree the documentation should make this explicit. If we decide to rename the channels instead, we should settle the names before they ship to avoid a compatibility issue later.
3. Buffer ownership: Diagnostics subscribers receive the existing buffers and should treat them as read-only. For example, mutating a response chunk before it reaches the stream can affect what the application reads. The inspector’s own request-body cache copies the bytes, but that does not make the buffers exposed through diagnostics channels immutable. I would document this contract in a follow-up.
4. Server performance: The default server path does not perform the inspector’s body copying or initiator-stack collection. The shared HTTP paths do gain a small number of field initializations, symbol reads/writes, and conditional checks, but the existing sending and parsing flow remains in place.

My expectation is that these additions are unlikely to cause a substantial regression with past test.


[legendecas]

Sorry if I was not clear. I was curious about the performance cost on regular mode, that is when inspector not enabled, and how much cost do these new changes incur?

[GrinZero]

Sorry if I was not clear. I was curious about the performance cost on regular mode, that is when inspector not enabled, and how much cost do these new changes incur?

@legendecas

Please don't worry! With inspection disabled, we do not expect a measurable performance regression. Across 19 benchmark configurations, mean throughput changes ranged from −1.62% to +1.83%, with none statistically significant—consistent with normal run-to-run variation. Body buffering and initiator-stack capture remain inactive in this case

This is my test conclusion on 9.29 with benchmark/compare.js ->

benchmark/http/client-request-body.js
benchmark/http/create-clientrequest.js
benchmark/http/simple.js
benchmark/http/end-string.js

[GrinZero]

@legendecas
Thanks for the feedback. I've updated the PR in 68d81263b5 to remove handwritten JS stack construction and CBOR serialization.(e2e done)

Summary of Changes

• V8-native stack capture: Requests now capture a V8-native stack at creation in an internal GC-managed handle. The deferred event builds its native initiator from that snapshot, preserving the caller and scriptIds across sessions.
• Unchanged semantics: The existing initiator assertion and event timing remain unchanged. Response bytes are captured before application decoding.
• Docs: Clarified that body channels describe application-level write()/end() operations (including buffered data) rather than socket transmission.

Benchmarks

Benchmarked the full PR against main (7b4c9e30ee15) with Release builds on an Apple M2 Max (macOS arm64):

• Inspector disabled: Across 4 HTTP benchmarks (19 configs, 30 samples each), changes ranged from -1.93% to +1.19% with no statistically significant regressions after Holm correction.
• Inspector connected (Debugger/Network enabled): External CDP consumer, loopback HTTP, 1 request in flight, default Network buffers, no DevTools UI (1 persistent process, 131,072 warmup requests, 30 × 5s windows in balanced alternating order):

Scenario	Throughput vs main	Holm-adjusted p
GET, 1 KiB response, depth 1	+6.65%	0.091
Text POST, 1 KiB, end, depth 1	-0.37%	0.236
Text POST, 1 KiB, split, depth 1	-2.70%	9.19e-19
Buffer POST, 1 KiB, end, depth 1	-0.43%	0.082
Text POST, 1 KiB, end, depth 20	-3.02%	0.746
Text POST, 1 KiB, end, depth 100	-1.07%	0.746
Text POST, 64 KiB, end, depth 20	-4.15%	1.04e-15
GET, 16 KiB raw response, depth 20	+8.00%	5.84e-48

(Note: The PR emits two fewer stack frames per event, so these numbers reflect end-to-end throughput rather than isolated stack-capture cost.)

Mean window-end RSS increased in a few scenarios (e.g., 313 → 419 MiB for 64 KiB POST and 207 → 248 MiB for 16 KiB GET) due to retaining the GC-managed stack handles and buffered response bytes until event emission. Raw results, CIs, CPU/RSS data, and reproduction scripts are available in the benchmark archives.

Local tests and lint pass (including coverage for script-source lookup, delayed callers, multi-session handling, body edge cases, and snapshot GC). Waiting on cross-platform CI.

[jasnell]

This should describe what each of these are. It's not clear what chunk actually is.

I assume it's the actual chunk of data... and I'd have to question if this is actually wise. It means this is either a copy of the chunk, in which case we're copying every chunk of data in memory (doubling the memory usage) or we're exposing a mutable reference to the chunk.

There's also likely a range of security/privacy concerns with this.

[GrinZero]

Thanks for raising this—the concern around mutable body data makes total sense. To help align on the right design here, could you clarify what makes this HTTP/1 integration different from the existing pattern?

For context, Undici #4289 (motivated by #4166 specifically for Node’s Network Inspector) already publishes raw request and response chunks via Diagnostics Channel, and Node consumes them directly. Core HTTP/2 took the same approach after review feedback explicitly recommended preserving the original chunk type and leaving copying/conversion to subscribers to avoid extra allocation overhead.

This PR follows that exact producer/consumer split: the publish path stays zero-copy (and is completely skipped when there are no subscribers), while the Inspector handles copying on its own when body retention is active.

Is there something specific to HTTP/1's lifecycle or buffer handling that makes this risky here, or are we reconsidering this zero-copy DC pattern more broadly?

[GrinZero]

To verify the current baseline behavior, I ran a control test on main (without this PR and with the Inspector disabled) across the existing body DC paths:

Existing Path	Observed Behavior
HTTP/2 body DC	Mutating the published chunk buffer alters the actual request payload received by the server.
Undici response DC	Mutating the published chunk buffer alters the response body read by the application.
Undici streaming request (5 MiB)	Even when published after socket.write(), ~4.69 MiB of the payload was still mutated by the subscriber before being flushed over the socket.

This confirms that publishing raw, mutable chunks over DC—and trusting subscribers not to mutate them—is already the established runtime contract across both core HTTP/2 and Undici, rather than a new invariant introduced by HTTP/1.

[GrinZero]

I have done some research, and indeed, the early documentation for the diagnostic channels of http and http2 only listed field names and types. However, the newly incorporated core channels (such as crypto.fips.indicator, sqlite.db.query) all require the semantics of each field to be specified. So, is the focus of this review on whether I need to complete this section?

[jasnell]

Not at all convinced the diagnostic channel integration here is a good idea.