Repository navigation
fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks - #7815
Conversation
8a03d9d to
2200585
Compare
|
Thanks for taking the broader fix forward! I tried the supplemental tests from ishan-1010#1 against this PR ( The completed Is dropping partial output on an interrupted stream the intended behavior here, or should it be retained when no aggregate arrives? |
… not its chunks The inference span recorded every streamed chunk as its own output message, ending at the first response with a finish reason. Model adapters put that finish reason in different places: Gemini on an empty partial chunk before its aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the span recorded the chunks and then the whole answer again. Both yield the whole answer as a `partial=False` aggregate after the `partial=True` chunks. Telemetry now skips the `partial=True` chunks and records that aggregate. A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are still recorded up to the finish reason, but now as one output message whose parts are the chunks' parts, as opentelemetry-instrumentation-google-genai reports a stream. Text parts are not joined: thought and answer text are both `text` parts in the output messages. Addresses the LiteLlm duplicate reported in #7694. Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com> Co-authored-by: Max Ind <maxind@google.com> PiperOrigin-RevId: 1012105751
2200585 to
80a4411
Compare
|
Thanks @GBX-Max1220 for the further review, I missed this prior to submitting, I'll have the fix for the dropping output messages on interrupt in #7938 |
fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks
The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.
Both yield the whole answer as a
partial=Falseaggregate after thepartial=Truechunks. Telemetry now skips thepartial=Truechunks andrecords that aggregate.
A
BaseLlmthat leavespartialunset ran no aggregator, so its chunks arestill recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
textparts in the output messages.Addresses the LiteLlm duplicate reported in
#7694.
Co-authored-by: Baixin Guo 141702260+GBX-Max1220@users.noreply.github.com
Co-authored-by: Max Ind maxind@google.com