Skip to content

fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks - #7815

Merged
copybara-service[bot] merged 1 commit into
mainfrom
copybara/1009687738
Nov 2, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
copybara/1009687738

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks

The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.

Both yield the whole answer as a partial=False aggregate after the
partial=True chunks. Telemetry now skips the partial=True chunks and
records that aggregate.

A BaseLlm that leaves partial unset ran no aggregator, so its chunks are
still recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
text parts in the output messages.

Addresses the LiteLlm duplicate reported in
#7694.

Co-authored-by: Baixin Guo 141702260+GBX-Max1220@users.noreply.github.com
Co-authored-by: Max Ind maxind@google.com

@GBX-Max1220

Copy link
Copy Markdown
Contributor

Thanks for taking the broader fix forward! I tried the supplemental tests from ishan-1010#1 against this PR (2200585e).

The completed EVENT_ONLY aggregate test passes. The interrupted-stream test behaves differently: if a partial=True response arrives and the stream raises before the aggregate, the original exception propagates and the span is marked ERROR, but gen_ai.output.messages is absent, so the partial text isn't recorded. The same test passes on #7694's head (2c5773f0).

Is dropping partial output on an interrupted stream the intended behavior here, or should it be retained when no aggregate arrives?

… not its chunks

The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.

Both yield the whole answer as a `partial=False` aggregate after the
`partial=True` chunks. Telemetry now skips the `partial=True` chunks and
records that aggregate.

A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are
still recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
`text` parts in the output messages.

Addresses the LiteLlm duplicate reported in
#7694.

Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com>
Co-authored-by: Max Ind <maxind@google.com>
PiperOrigin-RevId: 1012105751
@copybara-service
copybara-service Bot merged commit 80a4411 into main Nov 2, 2026
2 of 3 checks passed
@copybara-service
copybara-service Bot deleted the copybara/1009687738 branch November 2, 2026 13:53
@RKest

RKest commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thanks @GBX-Max1220 for the further review, I missed this prior to submitting,

I'll have the fix for the dropping output messages on interrupt in #7938

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants