Repository navigation
LiteLlm with SSE streaming records the answer twice in gen_ai.output.messages #7630
Description
🔴 Required Information
Describe the Bug:
with StreamingMode.SSE and a LiteLlm model the generate_content span ends up with every streamed chunk in gen_ai.output.messages and then the full answer again at the end. so the answer shows up twice and all the chunks have an empty finish_reason. without streaming its one message like expected. only visible with message content capture on (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT)
looks like the text chunks get yielded as partial responses with no finish reason (lite_llm.py#L4475-L4485) and only the final aggregated response gets one (#L4556-L4558). record_llm_response keeps recording until it sees a finish reason (_instrumentation.py#L568-L595) so every chunk plus the aggregate goes in. since its in the LiteLlm wrapper i'd expect real providers to do the same but i only tested with litellm's mock_response
Steps to Reproduce:
pip install google-adk litellm- run this (no api key needed, it uses litellm's
mock_response)
import asyncio
import json
import os
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "SPAN_ONLY"
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
spans = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(spans))
trace.set_tracer_provider(provider)
from google.adk.agents import Agent
from google.adk.agents.run_config import RunConfig, StreamingMode
from google.adk.models.lite_llm import LiteLlm
from google.adk.runners import InMemoryRunner
from google.genai import types
async def run(mode):
spans.clear()
model = LiteLlm(model="openai/gpt-4o", mock_response="Oslo is 4C and cloudy today.", api_key="unused")
runner = InMemoryRunner(agent=Agent(name="a", model=model), app_name="p")
session = await runner.session_service.create_session(app_name="p", user_id="u")
msg = types.Content(role="user", parts=[types.Part(text="weather?")])
async for _ in runner.run_async(user_id="u", session_id=session.id, new_message=msg,
run_config=RunConfig(streaming_mode=mode)):
pass
await runner.close()
span = next(s for s in spans.get_finished_spans() if s.name.startswith("generate_content"))
messages = json.loads(span.attributes["gen_ai.output.messages"])
print(f"{mode.name}: {len(messages)} output message(s)")
for m in messages:
print(" ", repr(m.get("finish_reason")), "".join(p.get("content", "") for p in m["parts"]))
asyncio.run(run(StreamingMode.NONE))
asyncio.run(run(StreamingMode.SSE))Expected Behavior:
one output message with the full answer, same as without streaming
Observed Behavior:
NONE: 1 output message(s)
'stop' Oslo is 4C and cloudy today.
SSE: 11 output message(s)
'' Osl
'' o i
'' s 4
'' C a
'' nd
'' clo
'' udy
'' to
'' day
'' .
'stop' Oslo is 4C and cloudy today.
Environment Details:
- ADK Library Version (pip show google-adk): 2.12.0 (main at 77a74f0)
- Desktop OS: macOS
- Python Version (python -V): 3.12.15
Model Information:
- Are you using LiteLLM: Yes
- Which model is being used: openai/gpt-4o via litellm
mock_response
🟡 Optional Information
Regression:
havent checked older versions
maybe skip the partial chunks when recording output messages and keep just the aggregate? happy to send a PR if that sounds right
Activity
GBX-Max1220 commentedon Oct 20, 2026
Thanks for the clear reproduction. I looked into the telemetry path and noticed that
set_operation_details_attributes_from_response()deliberately appends each response togen_ai.output.messages, while LiteLLM streaming also emits a final aggregated response.Before proposing a fix, could we clarify the intended contract here?
Should a streaming model call record one canonical completed output message, or preserve individual streamed chunks without duplicating the final aggregate?
I think the distinction matters for incomplete streams, finish reasons, and usage metadata.
I'm happy to help with focused offline regression tests if useful. Since the reporter has already offered to prepare a fix, I wouldn't open a competing PR.
ishan-1010 commentedon Oct 21, 2026
i think one completed message per call. thats what you already get with streaming off and the partial chunks have no finish reason anyway. if the stream gets cut off before the final response i'd record whatever text came in so far with no finish reason. would like a maintainer to confirm before i send the PR
GBX-Max1220 commentedon Oct 21, 2026
Thanks, that makes sense as a proposed contract. I agree we should wait for maintainer confirmation, especially on interrupted streams.
Once confirmed, I’m happy to help with offline regression tests covering completed streams without duplicate text, interrupted streams, and finish-reason and usage metadata. I’ll leave the implementation PR with you.
surajksharma07 commentedon Oct 22, 2026
Reproduced this on 2.12.0 and main (facd7dd) @ishan-1010. 11 messages with SSE vs 1 without. LiteLlm’s chunks carry no finish_reason and only the final aggregate has one so record_llm_response keeps recording until the aggregate and adds the whole answer again. Gemini doesn’t hit this because its last chunk already has the finish_reason so the aggregate is never recorded.
Something that worked as a workaround: in InferenceScope.record_llm_response when a non-partial response with a finish_reason follows only partial chunks that have none pop gen_ai.output.messages from span.operation_details_attributes before recording it. LiteLlm SSE then gives one 'stop' message and Gemini-style streams keep their per-chunk messages which the existing telemetry tests expect.
Could you check if that holds up for you with a real provider too plus a tool-call stream and a stream cut off mid-way (those should keep their chunks)? Please test it thoroughly before raising the PR. @GBX-Max1220’s regression tests would fit well alongside it.
Will also ask a telemetry maintainer to confirm the contract since this gives LiteLlm one completed message while Gemini stays per-chunk.
ishan-1010 commentedon Oct 22, 2026
@surajksharma07 the pop holds up.
gen_ai.output.messageson main (facd7dd) vs with the change:openai/gpt-oss-20b, text answeropenai/gpt-oss-20b, tool calltool_callwitharguments: nulltool_call({"city": "Oslo"})arguments: null)same in the completion log with
EVENT_ONLY. a stream that raises before its aggregate keeps its chunks, and the existing telemetry tests incl. the functional goldens don't change (1023 passed).two things:
finish_reason="stop", so telemetry can't tell it was cut off. that's in lite_llm.py, not this change.separately, on groq the call after the tool fails because the history sends
reasoning_contentback (property 'reasoning_content' is unsupported). i'll open an issue for that one.GBX-Max1220 commentedon Oct 22, 2026
Thanks for reproducing this and spelling out the workaround. I’d be happy to coordinate on the regression cases, particularly completed responses, tool-call streams, and interrupted streams.
It would help to get the maintainer’s view on the intended contract first, since the proposed behavior differs between LiteLlm and Gemini. I haven’t validated the workaround against a real provider.
GBX-Max1220 commentedon Oct 22, 2026
Thanks, Ishan—your provider results are helpful. I put together a small offline regression-preparation pack against main
6c67240.Seven isolated checks pass using extracted source functions with lightweight test doubles. These aren’t native ADK tests or a reproduction of the network cutoff.
The pack includes cases for duplicate aggregates, partial tool arguments, and missing terminal responses. Your cutoff finding is especially useful: removing duplicate messages won’t recover completion information that was already lost upstream.
Happy to coordinate these cases with your fix once the telemetry contract is confirmed.
ADK_7630_Offline_Regression_Preparation_20261023.zip
surajksharma07 commentedon Oct 23, 2026
That’s a solid set of runs @ishan-1010. On the tool call: keeping only the finished call is fine. The partial chunks only carry partial_args which is why they show up as arguments: null so dropping them loses nothing. The cut-off one I can confirm too: when the stream ends without a finish chunk lite_llm.py falls back to last_finish_reason or "stop" (still the case on main, 2de5876) so telemetry has no way to tell it was cut off. Please open that one separately same as the groq reasoning_content one and keep this PR to the telemetry change.
Feel free to open the PR as a draft now so the telemetry contract can be reviewed on the actual diff. @GBX-Max1220 it would help to add your cases as native tests under tests/unittests/telemetry in that PR (completed stream, tool-call stream, a stream that raises before the aggregate, Gemini unchanged) instead of keeping them in the zip.
GBX-Max1220 commentedon Oct 23, 2026
Thanks, Suraj—that gives me a clear scope. I’m happy to adapt the offline cases into native tests under
tests/unittests/telemetry, covering completed streams, tool-call streams, an exception before the aggregate, and unchanged Gemini behavior.@ishan-1010 please link the draft PR when it’s open so I can coordinate against your branch. I’ll keep the silent-cutoff and Groq history issues separate.
ishan-1010 commentedon Oct 23, 2026
draft PR is #7694. the four cases you listed are already native tests in it (completed stream, tool-call stream, a stream that raises before the aggregate, Gemini unchanged). @GBX-Max1220 if you have cases beyond those, open a PR against my branch or paste them here. cut-off issue: #7693
GBX-Max1220 commentedon Oct 23, 2026
Thanks, Ishan! I see the four native cases in #7694. I’ll review those against the offline cases and check for any additional gaps, rather than duplicate what you’ve already covered. If I find something useful, I’ll share a focused reproducer here or a test patch against your branch.
surajksharma07 commentedon Oct 26, 2026
Went through #7694 (2c5773f) @ishan-1010. The guard is tight in the right places. It only pops gen_ai.output.messages when the response is non-partial, has a reported finish reason and every earlier response was a partial chunk with none. So non-streamed calls (one response), Gemini-shaped streams (span already ended at the last chunk), streams that raise before the aggregate and models that leave partial unset all stay as they are. The LiteLlm aggregate also carries the reasoning/thought parts and the finished tool call so nothing is lost by dropping the chunks and token metrics still come from every response. _instrumentation.py is the same on main and 2.13.0 so the bug is still there and the PR merges cleanly.
One thing for the PR description, not a blocker: this only covers the experimental semconv path (the v2 default). With genai_semconv_stability_opt_in="stable" (or v1) trace_inference_result still emits one gen_ai.choice log per chunk plus one for the aggregate. Those are already emitted by the time the aggregate arrives so that would need a different fix. Worth mentioning as out of scope or opening a separate issue.
Feel free to mark it ready for review. @jawoszek this is the telemetry contract question from above: LiteLlm gets one completed message, Gemini stays per chunk. @GBX-Max1220 could you push your two extra tests (exception propagation with ERROR status, EVENT_ONLY once) as a PR against ishan’s branch or paste them here? A zip can’t be reviewed on the diff.