Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Archspec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
lib/ruby_llm/message.rb
lib/ruby_llm/moderation.rb
lib/ruby_llm/search_results.rb
lib/ruby_llm/skipped_stream_event.rb
lib/ruby_llm/stream_accumulator.rb
lib/ruby_llm/streaming.rb
lib/ruby_llm/thinking.rb
Expand Down
45 changes: 45 additions & 0 deletions docs/_core_features/streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ After reading this guide, you will know:
* How the final accumulated message is handled.
* How to integrate streaming with web frameworks like Rails and Sinatra.
* How streaming interacts with Tools.
* How to detect stream events RubyLLM could not decode.
* Performance considerations for streaming.

## Basic Streaming
Expand Down Expand Up @@ -195,6 +196,50 @@ end

Refer to the [Error Handling Guide]({% link _advanced/error-handling.md %}) for details on specific error types.

## Skipped Stream Events

Occasionally a provider sends a stream event RubyLLM cannot decode, such as truncated JSON. Instead of failing the whole response, RubyLLM drops that event and keeps streaming. Each dropped event is:

* Logged at `warn` level with the provider, model, site, event type and payload size. The payload itself is never logged.
* Passed to `config.on_stream_parse_error`, if you set one.
* Counted on the final message as `message.skipped_stream_events`, which is `0` when nothing was dropped.

A dropped event can carry content the model really sent, such as part of a tool call, so a non-zero count means the turn may be incomplete:

```ruby
RubyLLM.configure do |config|
config.on_stream_parse_error = ->(event) do
StatsD.increment('llm.stream.skipped_event', tags: ["provider:#{event.provider}", "site:#{event.site}"])
end
end

response = chat.ask("What's the weather in Berlin?") { |chunk| print chunk.content }
if response.skipped_stream_events.positive?
# The reply may be missing content the model sent: retry the turn or flag it for review.
end
```

The hook receives one frozen `RubyLLM::SkippedStreamEvent`:

| Field | Type | Description |
| --- | --- | --- |
| `provider` | String | Provider slug, e.g. `"bedrock"` or `"openai"` |
| `model` | String | Model id the request was sent to |
| `site` | Symbol | Where the event was dropped (see below) |
| `event_type` | String or nil | The SSE `event:` name or Bedrock `:event-type` header, when known |
| `byte_size` | Integer | Size of the dropped payload in bytes |
| `error_class` | String or nil | Name of the decoding error, e.g. `"JSON::ParserError"`; `nil` when the payload decoded but was not a JSON object |

| Site | Meaning |
| --- | --- |
| `:sse_data` | An SSE `data:` payload was not valid JSON or not a JSON object. The `[DONE]` sentinel is never reported. |
| `:event_payload` | A Bedrock eventstream payload (or its base64 `bytes`) was not valid JSON or not a JSON object. |
| `:event_type_header` | A Bedrock ConverseStream message's type header could not be read. |
| `:exception_type_header` | A Bedrock InvokeModel stream message's exception header could not be read. |
| `:content_block_delta` | A Bedrock ConverseStream delta sent as a JSON string was not valid JSON. |

The hook runs on the streaming thread, so keep it fast. If it raises, RubyLLM logs the error and carries on with the stream. `RubyLLM::SkippedStreamEvent::SITES` lists every site.

## Streaming with Tools

When a chat interaction involves [Tools]({% link _core_features/tools.md %}), the streaming behavior has distinct phases:
Expand Down
2 changes: 2 additions & 0 deletions docs/_getting_started/configuration-connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@ end
- Invaluable for diagnosing streaming/provider parsing issues
- Can also be enabled with `RUBYLLM_STREAM_DEBUG=true`

Stream events RubyLLM could not decode are always logged at `warn`, whatever `log_stream_debug` is set to. To count them, set `config.on_stream_parse_error`; see [Skipped Stream Events]({% link _core_features/streaming.md %}#skipped-stream-events).

`log_regexp_timeout` notes:
- Available in `v1.13.0+`
- Applies to regex filters used in request/response debug logging
Expand Down
1 change: 1 addition & 0 deletions docs/_getting_started/configuration-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ RubyLLM.configure do |config|
config.log_file = String
config.log_level = Symbol
config.log_stream_debug = Boolean
config.on_stream_parse_error = Proc # Called with a RubyLLM::SkippedStreamEvent for each stream event that couldn't be decoded
config.log_regexp_timeout = Numeric # v1.13.0+ (Ruby 3.2+ support)
end
```
Expand Down
1 change: 1 addition & 0 deletions lib/ruby_llm/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ def defaults = @defaults ||= {}
option :log_level, -> { ENV['RUBYLLM_DEBUG'] ? Logger::DEBUG : Logger::INFO }
option :log_stream_debug, -> { ENV['RUBYLLM_STREAM_DEBUG'] == 'true' }
option :log_regexp_timeout, -> { Regexp.respond_to?(:timeout) ? (Regexp.timeout || 1.0) : nil }
option :on_stream_parse_error, nil

def initialize
self.class.send(:defaults).each do |key, default|
Expand Down
4 changes: 3 additions & 1 deletion lib/ruby_llm/message.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ class Message
:CONTENT_FILTERED_FINISH_REASONS

attr_reader :role, :model_id, :tool_calls, :tool_call_id, :raw, :thinking, :tokens, :citations,
:finish_reason, :provider_data
:finish_reason, :provider_data, :skipped_stream_events
attr_writer :content

def initialize(options = {})
Expand All @@ -38,6 +38,8 @@ def initialize(options = {})
# Response fields a protocol passes through without modeling them, as plain string-keyed
# hashes (e.g. InvokeAnthropic's context_management and usage iterations).
@provider_data = options[:provider_data] || {}
# Streamed events the parser could not decode and dropped (see SkippedStreamEvent).
@skipped_stream_events = options[:skipped_stream_events] || 0

ensure_valid_role
end
Expand Down
15 changes: 8 additions & 7 deletions lib/ruby_llm/protocols/converse/event_stream.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ class Converse
# messages. What each endpoint puts INSIDE the payload differs, so event interpretation
# stays with the including protocol.
module EventStream
include SkippedStreamEvent::Reporting

ErrorResponse = Struct.new(:body, :status)

# Bedrock stream exception type => HTTP status, so ErrorMiddleware raises the same error
Expand Down Expand Up @@ -113,19 +115,19 @@ def event_type_header(message)
header = headers[':event-type'] || headers[':exception-type']
value = header.respond_to?(:value) ? header.value : header
value.is_a?(String) && !value.empty? ? value : nil
rescue StandardError
nil
rescue StandardError => e
skip_stream_event(:event_type_header, message.payload, error: e)
end

def exception_type_header(message)
header = message.headers[':exception-type']
value = header.respond_to?(:value) ? header.value : header
value.is_a?(String) && !value.empty? ? value : nil
rescue StandardError
nil
rescue StandardError => e
skip_stream_event(:exception_type_header, message.payload, error: e)
end

def decode_event_payload(payload)
def decode_event_payload(payload, event_type: nil)
outer = JSON.parse(payload)

if outer['bytes'].is_a?(String)
Expand All @@ -134,8 +136,7 @@ def decode_event_payload(payload)
outer
end
rescue JSON::ParserError => e
RubyLLM.logger.debug { "Failed to decode Bedrock stream event payload: #{e.message}" }
nil
skip_stream_event(:event_payload, payload, event_type: event_type, error: e)
end
Comment on lines 130 to 140

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A payload that is valid JSON but not an object still crashes the stream instead of being reported as a skip. JSON.parse returns nil, an Integer, or an Array for null, 5, or [...]. Then outer['bytes'] raises NoMethodError or TypeError, and the rescue only catches JSON::ParserError. The same thing happens when bytes decodes to a non-object: Converse's decode_events passes it on to nest_event_under_type / build_chunk, which call event.key? / event.dig and raise. The streaming guide says :event_payload covers payloads that were "not valid JSON or not a JSON object", so the Converse path doesn't do what the docs say. The non-Hash branch added in InvokeAnthropic#decode_invoke_event only catches part of these cases. The fix is to make the guard here cover both protocols:

outer = JSON.parse(payload)
decoded = outer.is_a?(Hash) && outer['bytes'].is_a?(String) ? JSON.parse(Base64.decode64(outer['bytes'])) : outer
return decoded if decoded.is_a?(Hash)

skip_stream_event(:event_payload, payload, event_type: event_type)

With that in place, the separate unless data.nil? skip in invoke_anthropic/streaming.rb can go. Add a Converse spec for a null or array payload.


def raise_event_stream_exception(exception_type, message)
Expand Down
57 changes: 29 additions & 28 deletions lib/ruby_llm/protocols/converse/streaming.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ def stream_url

def stream_response(payload, additional_headers = {}, &block)
accumulator = StreamAccumulator.new
@stream_accumulator = accumulator
decoder = event_stream_decoder
thinking_state = {}

Expand All @@ -45,10 +46,11 @@ def parse_stream_chunk(decoder, raw_chunk, accumulator, thinking_state)

def decode_events(decoder, raw_chunk)
decode_event_messages(decoder, raw_chunk).filter_map do |message|
event = decode_event_payload(message.payload.read)
type = event_type_header(message)
event = decode_event_payload(message.payload.read, event_type: type)
next unless event

event = nest_event_under_type(event, message)
event = nest_event_under_type(event, type)
RubyLLM.logger.debug { "Bedrock stream event keys: #{event.keys}" } if RubyLLM.config.log_stream_debug
event
end
Expand All @@ -62,8 +64,7 @@ def decode_events(decoder, raw_chunk)
# payload under its header type here. A payload already nested under its type (the
# shape older specs fabricate) or a message with no type header passes through
# unchanged.
def nest_event_under_type(event, message)
type = event_type_header(message)
def nest_event_under_type(event, type)
return event unless type
return event if event.key?(type)

Expand All @@ -73,19 +74,20 @@ def nest_event_under_type(event, message)
def build_chunk(event, thinking_state = {})
raise_stream_error(event) if stream_error_event?(event)

track_thinking_delta(event, thinking_state)
delta = normalized_delta(event)
track_thinking_delta(event, delta, thinking_state)
metadata_usage, usage = event_usage(event)

Chunk.new(
role: :assistant,
model_id: event['modelId'] || @model&.id,
content: extract_content_delta(event),
content: delta['text'],
thinking: Thinking.build(
text: extract_thinking_delta(event),
signature: extract_thinking_signature(event),
text: extract_thinking_delta(delta),
signature: extract_thinking_signature(event, delta),
blocks: extract_finalized_thinking_blocks(event, thinking_state)
),
tool_calls: extract_tool_calls(event),
tool_calls: extract_tool_calls(event, delta),
input_tokens: extract_input_tokens(metadata_usage, usage),
output_tokens: extract_output_tokens(metadata_usage, usage),
cached_tokens: extract_cached_tokens(metadata_usage, usage),
Expand All @@ -103,8 +105,8 @@ def build_chunk(event, thinking_state = {})
# block — shaped exactly like the blocks Converse::Chat.parse_thinking captures from
# the non-streaming response — on that index's contentBlockStop
# (see extract_finalized_thinking_blocks).
def track_thinking_delta(event, thinking_state)
index, reasoning_content = thinking_delta_source(event)
def track_thinking_delta(event, delta, thinking_state)
index, reasoning_content = thinking_delta_source(event, delta)
return unless index && reasoning_content.is_a?(Hash)

state = (thinking_state[index] ||= { text: +'', signature: nil, redacted: nil })
Expand All @@ -123,12 +125,12 @@ def apply_reasoning_content(state, reasoning_content)
state[:redacted] = state[:redacted].to_s + redacted if redacted
end

def thinking_delta_source(event)
def thinking_delta_source(event, delta)
start_index = event.dig('contentBlockStart', 'contentBlockIndex')
return [start_index, event.dig('contentBlockStart', 'start', 'reasoningContent')] if start_index

delta_index = event.dig('contentBlockDelta', 'contentBlockIndex')
[delta_index, normalized_delta(event)['reasoningContent']]
[delta_index, delta['reasoningContent']]
end

# A thinking block finalizes on its own content_block_stop. As a safety net, messageStop
Expand Down Expand Up @@ -231,19 +233,15 @@ def raise_stream_error(event)
raise_event_stream_exception(key, event[key]['message'])
end

def extract_content_delta(event)
normalized_delta(event)['text']
end

def extract_thinking_delta(event)
reasoning_content = normalized_delta(event)['reasoningContent'] || {}
def extract_thinking_delta(delta)
reasoning_content = delta['reasoningContent'] || {}

reasoning_text = reasoning_content['reasoningText'] || {}
reasoning_text['text'] || reasoning_content['text']
end

def extract_thinking_signature(event)
signature = extract_signature_from_delta(event)
def extract_thinking_signature(event, delta)
signature = extract_signature_from_delta(delta)
return signature if signature

signature = extract_signature_from_start(event)
Expand All @@ -252,8 +250,8 @@ def extract_thinking_signature(event)
nil
end

def extract_signature_from_delta(event)
reasoning_content = normalized_delta(event)['reasoningContent'] || {}
def extract_signature_from_delta(delta)
reasoning_content = delta['reasoningContent'] || {}
reasoning_text = reasoning_content['reasoningText'] || {}
reasoning_text['signature'] || reasoning_content['signature']
end
Expand All @@ -269,9 +267,9 @@ def extract_signature_from_start(event)
nil
end

def extract_tool_calls(event)
def extract_tool_calls(event, delta)
return extract_tool_call_start(event) if tool_call_start_event?(event)
return extract_tool_call_delta(event) if tool_call_delta_event?(event)
return extract_tool_call_delta(delta) if tool_call_delta_event?(event)

nil
end
Expand All @@ -298,13 +296,15 @@ def extract_tool_call_start(event)
}
end

def extract_tool_call_delta(event)
input = normalized_delta(event).dig('toolUse', 'input')
def extract_tool_call_delta(delta)
input = delta.dig('toolUse', 'input')
return nil unless input

{ nil => ToolCall.new(id: nil, name: nil, arguments: input) }
end

# Decoded once per event: a delta sent as a JSON string that fails to parse is reported as
# one skip, not once per extractor.
def normalized_delta(event)
delta = event.dig('contentBlockDelta', 'delta') || event['delta'] || {}
return delta if delta.is_a?(Hash)
Expand All @@ -314,7 +314,8 @@ def normalized_delta(event)
else
{}
end
rescue JSON::ParserError
rescue JSON::ParserError => e
skip_stream_event(:content_block_delta, delta, event_type: 'contentBlockDelta', error: e)
{}
end
end
Expand Down
8 changes: 6 additions & 2 deletions lib/ruby_llm/protocols/invoke_anthropic/streaming.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ def stream_url

def stream_response(payload, additional_headers = {}, &block)
accumulator = StreamAccumulator.new
@stream_accumulator = accumulator
decoder = event_stream_decoder
thinking_state = {}
block_state = {}
Expand Down Expand Up @@ -65,12 +66,15 @@ def decode_invoke_event(message)
payload = message.payload.read
exception_type = exception_type_header(message)
if exception_type
parsed = decode_event_payload(payload) || {}
parsed = decode_event_payload(payload, event_type: exception_type) || {}
raise_event_stream_exception(exception_type, parsed['message'] || parsed['Message'])
end

data = decode_event_payload(payload)
return nil unless data.is_a?(Hash)
unless data.is_a?(Hash)
skip_stream_event(:event_payload, payload) unless data.nil?
return nil
end

RubyLLM.logger.debug { "Bedrock invoke stream event: #{data['type']}" } if RubyLLM.config.log_stream_debug
raise_anthropic_stream_error(data) if data['type'] == 'error'
Expand Down
46 changes: 46 additions & 0 deletions lib/ruby_llm/skipped_stream_event.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# frozen_string_literal: true

module RubyLLM
SkippedStreamEvent = Struct.new(:provider, :model, :site, :event_type, :byte_size, :error_class,
keyword_init: true)

# A streamed event a parser could not decode and dropped. Passed to config.on_stream_parse_error;
# its fields are documented in docs/_core_features/streaming.md and must stay stable.
class SkippedStreamEvent
SITES = %i[sse_data event_payload event_type_header exception_type_header content_block_delta].freeze

def self.report(event, config)
RubyLLM.logger.warn("RubyLLM skipped a stream event it could not parse: #{event.log_fields}")
config.on_stream_parse_error&.call(event)
rescue StandardError => e
RubyLLM.logger.warn("RubyLLM on_stream_parse_error hook raised #{e.class}: #{e.message}")
end

def log_fields
to_h.map { |key, value| "#{key}=#{value.nil? ? '-' : value}" }.join(' ')
end

# Mixed into the streaming parsers. A skip never logs the payload itself, only its size.
module Reporting
private

def skip_stream_event(site, payload, event_type: nil, error: nil)
event = SkippedStreamEvent.new(
provider: @provider&.slug,
model: @model&.id,
site: site,
event_type: event_type,
byte_size: skipped_payload_bytesize(payload),
error_class: error&.class&.name
).freeze
@stream_accumulator&.skip_event
SkippedStreamEvent.report(event, @config || RubyLLM.config)
nil
end

def skipped_payload_bytesize(payload)
payload.respond_to?(:bytesize) ? payload.bytesize : payload&.size
end
end
end
end
Loading
Loading