Skip to content

feat: report stream events the parsers skip (DUX-14977) - #45

Merged
BadassBison merged 1 commit into
mainfrom
dux-14977-report-skipped-stream-events
Nov 13, 2026
Merged

BadassBison merged 1 commit into
mainfrom
dux-14977-report-skipped-stream-events

Conversation

@staging-supernova-dx-appf-io

Copy link
Copy Markdown

What this does

Streaming parsers used to rescue events they could not decode and drop them with only a debug log. A dropped event, such as a tool call the model actually sent, never surfaced anywhere.

This PR makes every skip on the streamed path visible:

  • Adds RubyLLM::SkippedStreamEvent, a value object describing an event that could not be decoded.
  • Adds config.on_stream_parse_error, a proc called with a SkippedStreamEvent for each skipped event, so applications can count or alert on them.
  • Skipped events are always logged at warn, regardless of log_stream_debug.
  • Skipped events are collected on the final message as Message#skipped_stream_events, via StreamAccumulator.
  • Applies the same reporting to the Converse event stream and the Anthropic (invoke) streaming parser.
  • Documents the behavior in the streaming guide, configuration connection guide, and configuration reference.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Required for new features

  • I opened an issue before writing code and received maintainer approval
  • Linked issue: #___

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
    • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
    • All tests pass: bundle exec rspec
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

Notes for reviewers

  • New public surface: RubyLLM::SkippedStreamEvent, Message#skipped_stream_events, and config.on_stream_parse_error. Other APIs are unchanged.
  • Converse and Anthropic streaming specs and the streaming spec were extended to cover skipped events.
  • The "Required for new features" section is left unchecked and no issue link is filled in. Confirm whether this change needs an approved issue before merging. Jira DUX-14977 is the tracking reference.

The streaming parsers rescued events they could not decode and dropped them
with only a debug log, so a lost event (for example part of a tool call the
model really sent) never surfaced. Every skip on the streamed path now logs
at warn (provider, model, site, event type, payload size; never the payload),
calls the new config.on_stream_parse_error hook with a frozen
RubyLLM::SkippedStreamEvent, and is counted on the final message as
Message#skipped_stream_events. A hook that raises is logged and never breaks
the stream.

Converse now decodes a string delta once per event so a bad one counts once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: Shawn Scott <shawn.scott@appfolio.com>
@kermitapp

kermitapp Bot commented Nov 13, 2026

Copy link
Copy Markdown

@BadassBison
BadassBison marked this pull request as ready for review November 13, 2026 01:57
@BadassBison
BadassBison requested a review from a team as a code owner November 13, 2026 01:57

@BadassBison BadassBison left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: every streamed-path skip now warns (no payload text), calls on_stream_parse_error (hook errors contained), and counts on the final message. CI green across the matrix.

@BadassBison
BadassBison merged commit aa95f61 into main Nov 13, 2026
24 of 25 checks passed
@staging-supernova-dx-appf-io

Copy link
Copy Markdown
Author

🤖 Supernova Code Review — View trace

🔄 Re-run this review

— via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com)

@staging-supernova-dx-appf-io staging-supernova-dx-appf-io Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review of the skipped-stream-event reporting. Things I checked that hold up: protocol instances are built per call, so @stream_accumulator doesn't race between requests. @provider/@model/@config resolve correctly on Protocol. Hook exceptions are contained. The SSE handle_data non-Hash path now reports skips correctly.

One inline finding: in lib/ruby_llm/protocols/converse/event_stream.rb, Bedrock eventstream payloads that are valid JSON but not objects still raise and end the stream instead of being reported as :event_payload skips. The docs promise these get reported.

🔄 Re-run this review

— via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com)

Comment on lines 130 to 140
@@ -134,8 +136,7 @@ def decode_event_payload(payload)
outer
end
rescue JSON::ParserError => e
RubyLLM.logger.debug { "Failed to decode Bedrock stream event payload: #{e.message}" }
nil
skip_stream_event(:event_payload, payload, event_type: event_type, error: e)
end

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A payload that is valid JSON but not an object still crashes the stream instead of being reported as a skip. JSON.parse returns nil, an Integer, or an Array for null, 5, or [...]. Then outer['bytes'] raises NoMethodError or TypeError, and the rescue only catches JSON::ParserError. The same thing happens when bytes decodes to a non-object: Converse's decode_events passes it on to nest_event_under_type / build_chunk, which call event.key? / event.dig and raise. The streaming guide says :event_payload covers payloads that were "not valid JSON or not a JSON object", so the Converse path doesn't do what the docs say. The non-Hash branch added in InvokeAnthropic#decode_invoke_event only catches part of these cases. The fix is to make the guard here cover both protocols:

outer = JSON.parse(payload)
decoded = outer.is_a?(Hash) && outer['bytes'].is_a?(String) ? JSON.parse(Base64.decode64(outer['bytes'])) : outer
return decoded if decoded.is_a?(Hash)

skip_stream_event(:event_payload, payload, event_type: event_type)

With that in place, the separate unless data.nil? skip in invoke_anthropic/streaming.rb can go. Add a Converse spec for a null or array payload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants