[rursprung]

(full disclosure: AI-generated, reviewed by me)

What

Adds opt-in metrics for the declared request and response entity lengths to
httpclient5-observation, for classic and async clients:

Meter	Type	Unit
<prefix>.request.content.length	DistributionSummary	bytes
<prefix>.response.content.length	DistributionSummary	bytes

Enable with the new ObservingOptions.MetricSet.CONTENT_LENGTH:

ObservingOptions opts = ObservingOptions.builder()
        .metrics(EnumSet.of(MetricSet.BASIC, MetricSet.CONTENT_LENGTH))
        .build();
HttpClientObservationSupport.enable(builder, obsReg, meterReg, opts);

The classic and async enable(...) variants install the new
ContentLengthExec / AsyncContentLengthExec interceptors. The interceptors
are public, so they can also be added by hand. The caching enable(...)
variants contain the same wiring as for BASIC and IO, but the caching
builders currently add CACHING after custom interceptors are processed, so
none of these interceptors are installed there yet. That is a pre-existing
issue that will be fixed separately; this PR does not claim caching support.

Why the declared length

Many servers limit the size of a request after decoding, not its size on
the wire. Our use case is OpenSearch, which is used through HttpClient 5 by
opensearch-java:

• OpenSearch rejects a request with 413 Request Entity Too Large if its
  body exceeds http.max_content_length (default 100mb)
  (setting).
• In the default Netty HTTP transport the pipeline is
  decoder -> HttpContentDecompressor -> HttpObjectAggregator(maxContentLength).
  The aggregator enforces the limit after decompression, so it counts the
  inflated bytes. A small gzip body that inflates past the limit is rejected
  (pipeline).

So to see how close large _bulk requests get to the limit, users need the
uncompressed size. When the client handles content coding, the declared
entity length that the client sees before compression is exactly that. The
wire size would not show this.

Background:

• Support HTTP request/response size micrometer-metrics/micrometer#880 asks for HTTP request/response size
  metrics. The Micrometer team said
  (comment)
  that this must be done per HTTP library, because Content-Length is
  missing for chunked bodies and compression makes body size differ from
  bytes on the network.
• Deprecate Apache HTTP client instrumentation in favor of httpclient5-observation micrometer-metrics/micrometer#7158 deprecates Micrometer's own hc5 binder
  (since 1.17.0) in favor of httpclient5-observation, so new HttpClient 5
  instrumentation belongs here.

Design

• Additive and opt-in. The existing IO counters
  (<prefix>.request.bytes / <prefix>.response.bytes) are unchanged.
  ObservingOptions.DEFAULT (BASIC only) is unchanged. Note:
  ObservingOptions.allMetricSets() now includes CONTENT_LENGTH. japicmp
  reports the enum change as compatible.
• Names. <prefix>.request.content.length /
  <prefix>.response.content.length. These are deliberately not the
  OpenTelemetry names http.client.request.body.size /
  http.client.response.body.size: OTel defines those as bytes transferred,
  compressed if a content coding is used. There is no current OTel name for
  the uncompressed size (the old http.*_content_length_uncompressed
  attributes are deprecated).
• Type. DistributionSummary, base unit bytes. MetricConfig.percentiles
  are published if configured.
• Tags. Same low-cardinality tags as the request timer: method,
  status, and with TagLevel.EXTENDED also protocol and target.
  MetricConfig.commonTags and ObservingOptions.tagCustomizer apply.
• Source. Classic: HttpEntity#getContentLength() of the request and
  response entities. Async: AsyncEntityProducer#getContentLength() and
  EntityDetails#getContentLength(). The interceptors sit at the start of
  the chain, so they see the request before the client compresses it and the
  response after the client decompresses it. The value is therefore
  uncompressed when the client handles content coding. It is the encoded
  length if the application sends an already encoded entity (for example
  with its own Content-Encoding: gzip) or if content compression is
  disabled.
• No overhead on the data path. Entities and data streams are never
  wrapped or buffered. spanSampling is honored.

What is and is not recorded

• Recorded only if a response was received. If the exchange fails before
  that (for example connection refused), nothing is recorded.
• HTTP error responses (4xx/5xx) are recorded with their status.
• Not recorded: entities of unknown length (Transfer-Encoding: chunked,
  getContentLength() == -1), including responses that the client
  decompresses transparently (the decompressing wrappers report -1), and
  messages without an entity.
• Over HTTP/2, HttpCore reports the response entity length as unknown even
  if Content-Length is present, so no response length is recorded for
  HTTP/2 exchanges.
• The value is the declared length. It is not the number of bytes
  transferred: it excludes headers and framing, and is
  recorded even if the body was not fully sent or received (for example a
  server that rejects a request before reading its body).

Possible follow-ups (separate PRs)

• Transferred body size under the OTel names
  http.client.request.body.size / http.client.response.body.size:
  interceptor just before MAIN_TRANSPORT (after compression) plus counting
  of the bytes actually written and read. This also covers chunked bodies.
• Counting the uncompressed size of chunked or unknown-length bodies by
  wrapping entities.

Tests

• ContentLengthExecTest (classic) and AsyncContentLengthExecTest
  (async): known length, chunked/unknown length, no entity (GET / 204),
  GET with response entity, error response (500), no response
  (connection refused, nothing recorded), extended and common tags,
  spanSampling skip, default options not installing the meters.
• ./mvnw -pl httpclient5-observation verify javadoc:javadoc passes: tests,
  checkstyle, japicmp (against 5.6), Javadoc and apache-rat.

Docs

• The module guide (observation.md) lives in apache/httpcomponents-website
  and has no 5.7.x version yet. I'll raise a follow-up PR there once this PR here is approved/merged.

[arturobernalg]

I do not think getContentLength() is sufficient for these metrics. It reports the declared entity length, not necessarily the number of bytes transferred.

The tests even record a positive request body size when the connection is refused and no body is sent.

If we use the OpenTelemetry metric names, the recorded values should follow their semantics. Otherwise, these should be exposed as declared entity lengths under different names.

[rursprung]

Thanks, you are right. getContentLength() is the declared entity length,
and the OpenTelemetry *.body.size names are defined as transferred bytes
(compressed if a content coding is used). I have changed the PR so that it
no longer claims those names:

• Meters renamed to <prefix>.request.content.length and
  <prefix>.response.content.length. The opt-in switch is now
  MetricSet.CONTENT_LENGTH, the interceptors are ContentLengthExec and
  AsyncContentLengthExec.
• Javadoc now says these are declared, uncompressed entity lengths and not
  the number of bytes transferred.
• Nothing is recorded any more if no response was received. The
  connection-refused test now asserts that no meter exists.

Why the declared, uncompressed length is still useful: servers often limit
the decoded request size, not the bytes on the wire. OpenSearch, for
example, rejects requests with 413 when the decoded body exceeds
http.max_content_length; the limit is checked after decompression. With
this metric, users can see how close their (for example _bulk) requests
get to that limit. Details are in the PR description.

If you wish, I can add transferred body sizes under the OpenTelemetry names in a
separate PR. They need a different chain position (after compression, just
before MAIN_TRANSPORT) and byte counting on the data path, so I think it
is easier to review on its own.

[arturobernalg]

Thanks for the update. The metric semantics are now clear and the interceptors themselves look fine.

There are still a couple of limitations worth addressing.

The caching builder overloads currently do not install the interceptors because CACHING is added after the custom interceptor entries are processed. This is a pre-existing issue, which I will address separately. We should not claim caching support until it is covered by integration tests.

Also, HttpCore reports an unknown response entity length over HTTP/2 even when Content-Length is present. This should be documented.

Finally, getContentLength() does not necessarily represent the uncompressed entity length. I would avoid making that claim unconditionally.

[rursprung]

Thanks! I have addressed the points:

• Caching builders: I no longer claim caching support in the PR
  description. I kept the CONTENT_LENGTH wiring in the caching
  enable(...) overloads so that it matches BASIC and IO and starts
  working once the CACHING ordering is fixed. Integration tests for the
  caching builders can come with that fix. If you prefer, I can remove the
  caching wiring from this PR instead.
• HTTP/2: Documented in the Javadoc of AsyncContentLengthExec and in
  the PR description: HttpCore reports the response entity length as
  unknown over HTTP/2, so no response length is recorded there.
• "Uncompressed": The Javadoc and commit message no longer say the
  value is always uncompressed. It is the declared entity length as the
  application sees it. It is uncompressed when the client handles content
  coding, and it is the encoded length if the application sends an already
  encoded entity or if content compression is disabled.

[arturobernalg]

@rursprung Could you demonstrate how this metric captures the uncompressed request size when using opensearch-java with gzip compression enabled? ContentCompressingEntity#getContentLength() appears to report the compressed length, whereas OpenSearch enforces its limit after decompression. I am not convinced the proposed metric addresses the stated use case.

[rursprung]

oh, i didn't even notice that because opensearch-java by default doesn't enable gzip compression and we're running it with the default. i tested it and indeed, in this case it doesn't report anything.

so for our setup (no gzip enabled on our side) the functionality from this PR is still useful and i presume there are many setups out there where this would also be the case. does this PR still make sense to you? (i'd of course change the javadoc & commit-msg to reflect the new facts)