[rursprung]

(full disclosure: AI-generated, reviewed by me)

What

Adds opt-in metrics for the declared request and response entity lengths to
httpclient5-observation, for classic and async clients:

Meter	Type	Unit
<prefix>.request.content.length	DistributionSummary	bytes
<prefix>.response.content.length	DistributionSummary	bytes

Enable with the new ObservingOptions.MetricSet.CONTENT_LENGTH:

ObservingOptions opts = ObservingOptions.builder()
        .metrics(EnumSet.of(MetricSet.BASIC, MetricSet.CONTENT_LENGTH))
        .build();
HttpClientObservationSupport.enable(builder, obsReg, meterReg, opts);

The classic and async enable(...) variants install the new
ContentLengthExec / AsyncContentLengthExec interceptors. The interceptors
are public, so they can also be added by hand. The caching enable(...)
variants contain the same wiring as for BASIC and IO, but the caching
builders currently add CACHING after custom interceptors are processed, so
none of these interceptors are installed there yet. That is a pre-existing
issue that will be fixed separately; this PR does not claim caching support.

Why the declared length

Many servers limit the size of a request after decoding, not its size on
the wire. Our use case is OpenSearch, which is used through HttpClient 5 by
opensearch-java:

• OpenSearch rejects a request with 413 Request Entity Too Large if its
  body exceeds http.max_content_length (default 100mb)
  (setting).
• In the default Netty HTTP transport the pipeline is
  decoder -> HttpContentDecompressor -> HttpObjectAggregator(maxContentLength).
  The aggregator enforces the limit after decompression, so it counts the
  inflated bytes. A small gzip body that inflates past the limit is rejected
  (pipeline).

So to see how close large _bulk requests get to the limit, users need the
uncompressed size. When the client handles content coding, the declared
entity length that the client sees before compression is exactly that. The
wire size would not show this.

Background:

• Support HTTP request/response size micrometer-metrics/micrometer#880 asks for HTTP request/response size
  metrics. The Micrometer team said
  (comment)
  that this must be done per HTTP library, because Content-Length is
  missing for chunked bodies and compression makes body size differ from
  bytes on the network.
• Deprecate Apache HTTP client instrumentation in favor of httpclient5-observation micrometer-metrics/micrometer#7158 deprecates Micrometer's own hc5 binder
  (since 1.17.0) in favor of httpclient5-observation, so new HttpClient 5
  instrumentation belongs here.

Design

• Additive and opt-in. The existing IO counters
  (<prefix>.request.bytes / <prefix>.response.bytes) are unchanged.
  ObservingOptions.DEFAULT (BASIC only) is unchanged. Note:
  ObservingOptions.allMetricSets() now includes CONTENT_LENGTH. japicmp
  reports the enum change as compatible.
• Names. <prefix>.request.content.length /
  <prefix>.response.content.length. These are deliberately not the
  OpenTelemetry names http.client.request.body.size /
  http.client.response.body.size: OTel defines those as bytes transferred,
  compressed if a content coding is used. There is no current OTel name for
  the uncompressed size (the old http.*_content_length_uncompressed
  attributes are deprecated).
• Type. DistributionSummary, base unit bytes. MetricConfig.percentiles
  are published if configured.
• Tags. Same low-cardinality tags as the request timer: method,
  status, and with TagLevel.EXTENDED also protocol and target.
  MetricConfig.commonTags and ObservingOptions.tagCustomizer apply.
• Source. Classic: HttpEntity#getContentLength() of the request and
  response entities. Async: AsyncEntityProducer#getContentLength() and
  EntityDetails#getContentLength(). The interceptors sit at the start of
  the chain, so they see the request before the client compresses it and the
  response after the client decompresses it. The value is therefore
  uncompressed when the client handles content coding. It is the encoded
  length if the application sends an already encoded entity (for example
  with its own Content-Encoding: gzip) or if content compression is
  disabled.
• No overhead on the data path. Entities and data streams are never
  wrapped or buffered. spanSampling is honored.

What is and is not recorded

• Recorded only if a response was received. If the exchange fails before
  that (for example connection refused), nothing is recorded.
• HTTP error responses (4xx/5xx) are recorded with their status.
• Not recorded: entities of unknown length (Transfer-Encoding: chunked,
  getContentLength() == -1), including responses that the client
  decompresses transparently (the decompressing wrappers report -1), and
  messages without an entity.
• Over HTTP/2, HttpCore reports the response entity length as unknown even
  if Content-Length is present, so no response length is recorded for
  HTTP/2 exchanges.
• The value is the declared length. It is not the number of bytes
  transferred: it excludes headers and framing, and is
  recorded even if the body was not fully sent or received (for example a
  server that rejects a request before reading its body).

Possible follow-ups (separate PRs)

• Transferred body size under the OTel names
  http.client.request.body.size / http.client.response.body.size:
  interceptor just before MAIN_TRANSPORT (after compression) plus counting
  of the bytes actually written and read. This also covers chunked bodies.
• Counting the uncompressed size of chunked or unknown-length bodies by
  wrapping entities.

Tests

• ContentLengthExecTest (classic) and AsyncContentLengthExecTest
  (async): known length, chunked/unknown length, no entity (GET / 204),
  GET with response entity, error response (500), no response
  (connection refused, nothing recorded), extended and common tags,
  spanSampling skip, default options not installing the meters.
• ./mvnw -pl httpclient5-observation verify javadoc:javadoc passes: tests,
  checkstyle, japicmp (against 5.6), Javadoc and apache-rat.

Docs

• The module guide (observation.md) lives in apache/httpcomponents-website
  and has no 5.7.x version yet. I'll raise a follow-up PR there once this PR here is approved/merged.

[arturobernalg]

I do not think getContentLength() is sufficient for these metrics. It reports the declared entity length, not necessarily the number of bytes transferred.

The tests even record a positive request body size when the connection is refused and no body is sent.

If we use the OpenTelemetry metric names, the recorded values should follow their semantics. Otherwise, these should be exposed as declared entity lengths under different names.

[rursprung]

Thanks, you are right. getContentLength() is the declared entity length,
and the OpenTelemetry *.body.size names are defined as transferred bytes
(compressed if a content coding is used). I have changed the PR so that it
no longer claims those names:

• Meters renamed to <prefix>.request.content.length and
  <prefix>.response.content.length. The opt-in switch is now
  MetricSet.CONTENT_LENGTH, the interceptors are ContentLengthExec and
  AsyncContentLengthExec.
• Javadoc now says these are declared, uncompressed entity lengths and not
  the number of bytes transferred.
• Nothing is recorded any more if no response was received. The
  connection-refused test now asserts that no meter exists.

Why the declared, uncompressed length is still useful: servers often limit
the decoded request size, not the bytes on the wire. OpenSearch, for
example, rejects requests with 413 when the decoded body exceeds
http.max_content_length; the limit is checked after decompression. With
this metric, users can see how close their (for example _bulk) requests
get to that limit. Details are in the PR description.

If you wish, I can add transferred body sizes under the OpenTelemetry names in a
separate PR. They need a different chain position (after compression, just
before MAIN_TRANSPORT) and byte counting on the data path, so I think it
is easier to review on its own.

[arturobernalg]

Thanks for the update. The metric semantics are now clear and the interceptors themselves look fine.

There are still a couple of limitations worth addressing.

The caching builder overloads currently do not install the interceptors because CACHING is added after the custom interceptor entries are processed. This is a pre-existing issue, which I will address separately. We should not claim caching support until it is covered by integration tests.

Also, HttpCore reports an unknown response entity length over HTTP/2 even when Content-Length is present. This should be documented.

Finally, getContentLength() does not necessarily represent the uncompressed entity length. I would avoid making that claim unconditionally.

[rursprung]

Thanks! I have addressed the points:

• Caching builders: I no longer claim caching support in the PR
  description. I kept the CONTENT_LENGTH wiring in the caching
  enable(...) overloads so that it matches BASIC and IO and starts
  working once the CACHING ordering is fixed. Integration tests for the
  caching builders can come with that fix. If you prefer, I can remove the
  caching wiring from this PR instead.
• HTTP/2: Documented in the Javadoc of AsyncContentLengthExec and in
  the PR description: HttpCore reports the response entity length as
  unknown over HTTP/2, so no response length is recorded there.
• "Uncompressed": The Javadoc and commit message no longer say the
  value is always uncompressed. It is the declared entity length as the
  application sees it. It is uncompressed when the client handles content
  coding, and it is the encoded length if the application sends an already
  encoded entity or if content compression is disabled.

[arturobernalg]

@rursprung Could you demonstrate how this metric captures the uncompressed request size when using opensearch-java with gzip compression enabled? ContentCompressingEntity#getContentLength() appears to report the compressed length, whereas OpenSearch enforces its limit after decompression. I am not convinced the proposed metric addresses the stated use case.

[rursprung]

oh, i didn't even notice that because opensearch-java by default doesn't enable gzip compression and we're running it with the default. i tested it and indeed, in this case it doesn't report anything.

so for our setup (no gzip enabled on our side) the functionality from this PR is still useful and i presume there are many setups out there where this would also be the case. does this PR still make sense to you? (i'd of course change the javadoc & commit-msg to reflect the new facts)

[arturobernalg]

I am not convinced this feature justifies the additional public API and maintenance overhead.

It only records known entity lengths, not actual content sizes or bytes transferred. It does not cover compressed or streaming requests, and the original OpenSearch use case is only partially addressed.

Applications can already obtain declared lengths through getContentLength().

I would prefer not to merge this in its current form. I think we should reconsider whether HttpClient should expose declared entity lengths or actual content size metrics.

[shinseongsu]

Hi @rursprung, thanks for working on this! 🙂 I was curious how the new meters behave with gzip and HTTP/2, so I tried the PR (74d9bc1) against a small local server. I'm sharing the numbers in case they help.

Setup: HttpClientObservationSupport.enable(...) with CONTENT_LENGTH, and a 100,000 byte body (133 bytes after gzip).

Case	Wire	Recorded
plain request	100000	100000 ✅
request via EntityBuilder.gzipCompressed()	133 (chunked)	nothing
pre-gzipped request entity	133	133
gzip response, decompressed by the client	133	nothing
gzip response, content compression off	133	133
async HTTP/2 response with Content-Length: 100000	100000	nothing

So I couldn't find a case where the uncompressed size shows up 😅 When something is recorded, it's the same as the Content-Length on the wire. The existing IO counters gave me the same numbers, since they also read getContentLength().

For the OpenSearch case I think this means the decoded _bulk size isn't visible once gzip is on. And since opensearch-java compresses the request itself before the exec chain, I'm not sure the client can see the decoded request size at all. Counting the bytes that actually go through (like Arturo mentioned) might be the way to get real numbers.

If you want to try it yourself, drop the test below into httpclient5-observation/src/test/java/org/apache/hc/client5/http/observation/interceptors/ and run:

./mvnw -pl httpclient5-observation -am test -Dtest=ContentLengthSemanticsTest -Dsurefire.failIfNoSpecifiedTests=false

All 4 tests pass on the PR branch. The assertions just describe what happens today.

Hope this is useful! 🙏

ContentLengthSemanticsTest.java

package org.apache.hc.client5.http.observation.interceptors;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.concurrent.TimeUnit;
import java.util.zip.GZIPOutputStream;

import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import org.apache.hc.client5.http.async.methods.SimpleHttpResponse;
import org.apache.hc.client5.http.async.methods.SimpleRequestBuilder;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.entity.EntityBuilder;
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient;
import org.apache.hc.client5.http.impl.async.HttpAsyncClientBuilder;
import org.apache.hc.client5.http.impl.async.HttpAsyncClients;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.observation.HttpClientObservationSupport;
import org.apache.hc.client5.http.observation.MetricConfig;
import org.apache.hc.client5.http.observation.ObservingOptions;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.EntityDetails;
import org.apache.hc.core5.http.HttpException;
import org.apache.hc.core5.http.HttpRequest;
import org.apache.hc.core5.http.Message;
import org.apache.hc.core5.http.URIScheme;
import org.apache.hc.core5.http.impl.bootstrap.HttpAsyncServer;
import org.apache.hc.core5.http.impl.bootstrap.HttpServer;
import org.apache.hc.core5.http.impl.bootstrap.ServerBootstrap;
import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.message.BasicHttpResponse;
import org.apache.hc.core5.http.nio.AsyncRequestConsumer;
import org.apache.hc.core5.http.nio.AsyncServerRequestHandler;
import org.apache.hc.core5.http.nio.entity.BasicAsyncEntityConsumer;
import org.apache.hc.core5.http.nio.entity.BasicAsyncEntityProducer;
import org.apache.hc.core5.http.nio.support.BasicRequestConsumer;
import org.apache.hc.core5.http.nio.support.BasicResponseProducer;
import org.apache.hc.core5.http.protocol.HttpContext;
import org.apache.hc.core5.http2.HttpVersionPolicy;
import org.apache.hc.core5.http2.impl.nio.bootstrap.H2ServerBootstrap;
import org.apache.hc.core5.io.CloseMode;
import org.apache.hc.core5.reactor.ListenerEndpoint;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

/**
 * What CONTENT_LENGTH records when compression or HTTP/2 is involved.
 * Body: 100_000 bytes of 'a' (gzips to 133 bytes). Assertions describe the observed behaviour.
 */
class ContentLengthSemanticsTest {

    private static final String REQUEST = "t.request.content.length";
    private static final String RESPONSE = "t.response.content.length";
    private static final byte[] BODY = new byte[100_000];

    static {
        Arrays.fill(BODY, (byte) 'a');
    }

    private byte[] gzippedBody;
    private HttpServer server;
    private String base;
    private volatile int serverReceivedBytes;

    private static ObservingOptions options() {
        return ObservingOptions.builder().metrics(EnumSet.of(ObservingOptions.MetricSet.CONTENT_LENGTH)).build();
    }

    private static MetricConfig config() {
        return MetricConfig.builder().prefix("t").build();
    }

    private static Double total(final MeterRegistry meters, final String name) {
        final DistributionSummary s = meters.find(name).summary();
        return s != null ? s.totalAmount() : null;
    }

    @BeforeEach
    void setUp() throws IOException {
        final ByteArrayOutputStream out = new ByteArrayOutputStream();
        try (GZIPOutputStream gz = new GZIPOutputStream(out)) {
            gz.write(BODY);
        }
        gzippedBody = out.toByteArray();
        server = ServerBootstrap.bootstrap()
                .setLocalAddress(InetAddress.getLoopbackAddress())
                .setListenerPort(0)
                .register("localhost", "/echo", (request, response, context) -> {
                    serverReceivedBytes = EntityUtils.toByteArray(request.getEntity()).length;
                    response.setCode(200);
                })
                .register("localhost", "/gzip", (request, response, context) -> {
                    response.setCode(200);
                    response.setEntity(new ByteArrayEntity(gzippedBody, ContentType.TEXT_PLAIN, "gzip"));
                })
                .create();
        server.start();
        base = "http://localhost:" + server.getLocalPort();
    }

    @AfterEach
    void tearDown() {
        server.close(CloseMode.IMMEDIATE);
    }

    private MeterRegistry classic(final HttpPost post) throws IOException {
        final MeterRegistry meters = new SimpleMeterRegistry();
        final HttpClientBuilder builder = HttpClients.custom();
        HttpClientObservationSupport.enable(builder, null, meters, options(), config());
        try (CloseableHttpClient client = builder.build()) {
            client.execute(post, r -> {
                EntityUtils.consume(r.getEntity());
                return null;
            });
        }
        return meters;
    }

    @Test
    void requestCompressedWithEntityBuilderIsNotRecorded() throws Exception {
        final HttpPost post = new HttpPost(base + "/echo");
        post.setEntity(EntityBuilder.create().setBinary(BODY)
                .setContentType(ContentType.APPLICATION_JSON).gzipCompressed().build());

        final MeterRegistry meters = classic(post);

        assertEquals(gzippedBody.length, serverReceivedBytes, "server received the gzipped body");
        assertNull(total(meters, REQUEST), "neither the decoded nor the wire size is recorded");
    }

    @Test
    void preEncodedRequestRecordsCompressedLength() throws Exception {
        final HttpPost post = new HttpPost(base + "/echo");
        post.setEntity(new ByteArrayEntity(gzippedBody, ContentType.APPLICATION_JSON, "gzip"));

        final MeterRegistry meters = classic(post);

        assertEquals(gzippedBody.length, serverReceivedBytes);
        assertEquals((double) gzippedBody.length, total(meters, REQUEST), "compressed length, not 100000");
    }

    @Test
    void transparentlyDecompressedResponseIsNotRecorded() throws Exception {
        final MeterRegistry meters = classic(new HttpPost(base + "/gzip"));

        assertNull(total(meters, RESPONSE), "Content-Length: 133 on the wire, 100000 read by the app, nothing recorded");
    }

    @Test
    void http2ResponseWithContentLengthIsNotRecorded() throws Exception {
        final HttpAsyncServer h2 = H2ServerBootstrap.bootstrap()
                .setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2)
                .register("*", new AsyncServerRequestHandler<Message<HttpRequest, byte[]>>() {
                    @Override
                    public AsyncRequestConsumer<Message<HttpRequest, byte[]>> prepare(
                            final HttpRequest request, final EntityDetails details, final HttpContext context) {
                        return new BasicRequestConsumer<>(details != null ? new BasicAsyncEntityConsumer() : null);
                    }

                    @Override
                    public void handle(final Message<HttpRequest, byte[]> message, final ResponseTrigger trigger,
                                       final HttpContext context) throws HttpException, IOException {
                        final BasicHttpResponse head = new BasicHttpResponse(200);
                        head.addHeader("Content-Length", Integer.toString(BODY.length));
                        trigger.submitResponse(new BasicResponseProducer(head,
                                new BasicAsyncEntityProducer(BODY, ContentType.APPLICATION_OCTET_STREAM)), context);
                    }
                })
                .create();
        h2.start();
        final ListenerEndpoint endpoint = h2.listen(
                new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), URIScheme.HTTP).get();
        final int port = ((InetSocketAddress) endpoint.getAddress()).getPort();
        final MeterRegistry meters = new SimpleMeterRegistry();
        final HttpAsyncClientBuilder builder = HttpAsyncClients.custom().setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2);
        HttpClientObservationSupport.enable(builder, null, meters, options(), config());
        try (CloseableHttpAsyncClient client = builder.build()) {
            client.start();

            final SimpleHttpResponse response = client.execute(
                    SimpleRequestBuilder.get("http://localhost:" + port + "/").build(), null).get(20, TimeUnit.SECONDS);

            assertEquals("100000", response.getFirstHeader("Content-Length").getValue());
            assertNull(total(meters, RESPONSE), "HTTP/2 response with Content-Length: 100000, nothing recorded");
        } finally {
            h2.close(CloseMode.IMMEDIATE);
        }
    }
}