From 9d6ebb9b097e8cbc4fe25db856da123a8f66db13 Mon Sep 17 00:00:00 2001 From: Supernova Date: Fri, 13 Nov 2026 01:52:16 +0000 Subject: [PATCH] DUX-14977: Report stream events the parsers skip The streaming parsers rescued events they could not decode and dropped them with only a debug log, so a lost event (for example part of a tool call the model really sent) never surfaced. Every skip on the streamed path now logs at warn (provider, model, site, event type, payload size; never the payload), calls the new config.on_stream_parse_error hook with a frozen RubyLLM::SkippedStreamEvent, and is counted on the final message as Message#skipped_stream_events. A hook that raises is logged and never breaks the stream. Converse now decodes a string delta once per event so a bad one counts once. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Shawn Scott --- Archspec.rb | 1 + docs/_core_features/streaming.md | 45 +++++++ .../configuration-connection.md | 2 + .../configuration-reference.md | 1 + lib/ruby_llm/configuration.rb | 1 + lib/ruby_llm/message.rb | 4 +- .../protocols/converse/event_stream.rb | 15 ++- lib/ruby_llm/protocols/converse/streaming.rb | 57 ++++---- .../protocols/invoke_anthropic/streaming.rb | 8 +- lib/ruby_llm/skipped_stream_event.rb | 46 +++++++ lib/ruby_llm/stream_accumulator.rb | 8 +- lib/ruby_llm/streaming.rb | 16 ++- spec/ruby_llm/configuration_spec.rb | 2 + spec/ruby_llm/message_spec.rb | 12 ++ .../protocols/converse/streaming_spec.rb | 124 +++++++++++++++++- .../protocols/invoke_anthropic_spec.rb | 87 ++++++++++++ spec/ruby_llm/skipped_stream_event_spec.rb | 102 ++++++++++++++ spec/ruby_llm/stream_accumulator_spec.rb | 11 ++ spec/ruby_llm/streaming_spec.rb | 98 ++++++++++++++ 19 files changed, 595 insertions(+), 45 deletions(-) create mode 100644 lib/ruby_llm/skipped_stream_event.rb create mode 100644 spec/ruby_llm/skipped_stream_event_spec.rb diff --git a/Archspec.rb b/Archspec.rb index e933e194a3..5a636201be 100644 --- a/Archspec.rb +++ b/Archspec.rb @@ -24,6 +24,7 @@ lib/ruby_llm/message.rb lib/ruby_llm/moderation.rb lib/ruby_llm/search_results.rb + lib/ruby_llm/skipped_stream_event.rb lib/ruby_llm/stream_accumulator.rb lib/ruby_llm/streaming.rb lib/ruby_llm/thinking.rb diff --git a/docs/_core_features/streaming.md b/docs/_core_features/streaming.md index 3cbac02d83..0b1972b844 100644 --- a/docs/_core_features/streaming.md +++ b/docs/_core_features/streaming.md @@ -29,6 +29,7 @@ After reading this guide, you will know: * How the final accumulated message is handled. * How to integrate streaming with web frameworks like Rails and Sinatra. * How streaming interacts with Tools. +* How to detect stream events RubyLLM could not decode. * Performance considerations for streaming. ## Basic Streaming @@ -195,6 +196,50 @@ end Refer to the [Error Handling Guide]({% link _advanced/error-handling.md %}) for details on specific error types. +## Skipped Stream Events + +Occasionally a provider sends a stream event RubyLLM cannot decode, such as truncated JSON. Instead of failing the whole response, RubyLLM drops that event and keeps streaming. Each dropped event is: + +* Logged at `warn` level with the provider, model, site, event type and payload size. The payload itself is never logged. +* Passed to `config.on_stream_parse_error`, if you set one. +* Counted on the final message as `message.skipped_stream_events`, which is `0` when nothing was dropped. + +A dropped event can carry content the model really sent, such as part of a tool call, so a non-zero count means the turn may be incomplete: + +```ruby +RubyLLM.configure do |config| + config.on_stream_parse_error = ->(event) do + StatsD.increment('llm.stream.skipped_event', tags: ["provider:#{event.provider}", "site:#{event.site}"]) + end +end + +response = chat.ask("What's the weather in Berlin?") { |chunk| print chunk.content } +if response.skipped_stream_events.positive? + # The reply may be missing content the model sent: retry the turn or flag it for review. +end +``` + +The hook receives one frozen `RubyLLM::SkippedStreamEvent`: + +| Field | Type | Description | +| --- | --- | --- | +| `provider` | String | Provider slug, e.g. `"bedrock"` or `"openai"` | +| `model` | String | Model id the request was sent to | +| `site` | Symbol | Where the event was dropped (see below) | +| `event_type` | String or nil | The SSE `event:` name or Bedrock `:event-type` header, when known | +| `byte_size` | Integer | Size of the dropped payload in bytes | +| `error_class` | String or nil | Name of the decoding error, e.g. `"JSON::ParserError"`; `nil` when the payload decoded but was not a JSON object | + +| Site | Meaning | +| --- | --- | +| `:sse_data` | An SSE `data:` payload was not valid JSON or not a JSON object. The `[DONE]` sentinel is never reported. | +| `:event_payload` | A Bedrock eventstream payload (or its base64 `bytes`) was not valid JSON or not a JSON object. | +| `:event_type_header` | A Bedrock ConverseStream message's type header could not be read. | +| `:exception_type_header` | A Bedrock InvokeModel stream message's exception header could not be read. | +| `:content_block_delta` | A Bedrock ConverseStream delta sent as a JSON string was not valid JSON. | + +The hook runs on the streaming thread, so keep it fast. If it raises, RubyLLM logs the error and carries on with the stream. `RubyLLM::SkippedStreamEvent::SITES` lists every site. + ## Streaming with Tools When a chat interaction involves [Tools]({% link _core_features/tools.md %}), the streaming behavior has distinct phases: diff --git a/docs/_getting_started/configuration-connection.md b/docs/_getting_started/configuration-connection.md index 72c555c06e..3f81327c4b 100644 --- a/docs/_getting_started/configuration-connection.md +++ b/docs/_getting_started/configuration-connection.md @@ -144,6 +144,8 @@ end - Invaluable for diagnosing streaming/provider parsing issues - Can also be enabled with `RUBYLLM_STREAM_DEBUG=true` +Stream events RubyLLM could not decode are always logged at `warn`, whatever `log_stream_debug` is set to. To count them, set `config.on_stream_parse_error`; see [Skipped Stream Events]({% link _core_features/streaming.md %}#skipped-stream-events). + `log_regexp_timeout` notes: - Available in `v1.13.0+` - Applies to regex filters used in request/response debug logging diff --git a/docs/_getting_started/configuration-reference.md b/docs/_getting_started/configuration-reference.md index 19acf77f19..cacf7acad5 100644 --- a/docs/_getting_started/configuration-reference.md +++ b/docs/_getting_started/configuration-reference.md @@ -123,6 +123,7 @@ RubyLLM.configure do |config| config.log_file = String config.log_level = Symbol config.log_stream_debug = Boolean + config.on_stream_parse_error = Proc # Called with a RubyLLM::SkippedStreamEvent for each stream event that couldn't be decoded config.log_regexp_timeout = Numeric # v1.13.0+ (Ruby 3.2+ support) end ``` diff --git a/lib/ruby_llm/configuration.rb b/lib/ruby_llm/configuration.rb index 3af732bcfb..8de130071a 100644 --- a/lib/ruby_llm/configuration.rb +++ b/lib/ruby_llm/configuration.rb @@ -70,6 +70,7 @@ def defaults = @defaults ||= {} option :log_level, -> { ENV['RUBYLLM_DEBUG'] ? Logger::DEBUG : Logger::INFO } option :log_stream_debug, -> { ENV['RUBYLLM_STREAM_DEBUG'] == 'true' } option :log_regexp_timeout, -> { Regexp.respond_to?(:timeout) ? (Regexp.timeout || 1.0) : nil } + option :on_stream_parse_error, nil def initialize self.class.send(:defaults).each do |key, default| diff --git a/lib/ruby_llm/message.rb b/lib/ruby_llm/message.rb index abd92d1a05..39e870100b 100644 --- a/lib/ruby_llm/message.rb +++ b/lib/ruby_llm/message.rb @@ -15,7 +15,7 @@ class Message :CONTENT_FILTERED_FINISH_REASONS attr_reader :role, :model_id, :tool_calls, :tool_call_id, :raw, :thinking, :tokens, :citations, - :finish_reason, :provider_data + :finish_reason, :provider_data, :skipped_stream_events attr_writer :content def initialize(options = {}) @@ -38,6 +38,8 @@ def initialize(options = {}) # Response fields a protocol passes through without modeling them, as plain string-keyed # hashes (e.g. InvokeAnthropic's context_management and usage iterations). @provider_data = options[:provider_data] || {} + # Streamed events the parser could not decode and dropped (see SkippedStreamEvent). + @skipped_stream_events = options[:skipped_stream_events] || 0 ensure_valid_role end diff --git a/lib/ruby_llm/protocols/converse/event_stream.rb b/lib/ruby_llm/protocols/converse/event_stream.rb index 4a9e5bc5c6..d6c371662d 100644 --- a/lib/ruby_llm/protocols/converse/event_stream.rb +++ b/lib/ruby_llm/protocols/converse/event_stream.rb @@ -14,6 +14,8 @@ class Converse # messages. What each endpoint puts INSIDE the payload differs, so event interpretation # stays with the including protocol. module EventStream + include SkippedStreamEvent::Reporting + ErrorResponse = Struct.new(:body, :status) # Bedrock stream exception type => HTTP status, so ErrorMiddleware raises the same error @@ -113,19 +115,19 @@ def event_type_header(message) header = headers[':event-type'] || headers[':exception-type'] value = header.respond_to?(:value) ? header.value : header value.is_a?(String) && !value.empty? ? value : nil - rescue StandardError - nil + rescue StandardError => e + skip_stream_event(:event_type_header, message.payload, error: e) end def exception_type_header(message) header = message.headers[':exception-type'] value = header.respond_to?(:value) ? header.value : header value.is_a?(String) && !value.empty? ? value : nil - rescue StandardError - nil + rescue StandardError => e + skip_stream_event(:exception_type_header, message.payload, error: e) end - def decode_event_payload(payload) + def decode_event_payload(payload, event_type: nil) outer = JSON.parse(payload) if outer['bytes'].is_a?(String) @@ -134,8 +136,7 @@ def decode_event_payload(payload) outer end rescue JSON::ParserError => e - RubyLLM.logger.debug { "Failed to decode Bedrock stream event payload: #{e.message}" } - nil + skip_stream_event(:event_payload, payload, event_type: event_type, error: e) end def raise_event_stream_exception(exception_type, message) diff --git a/lib/ruby_llm/protocols/converse/streaming.rb b/lib/ruby_llm/protocols/converse/streaming.rb index 83868832d7..0ad8260c3d 100644 --- a/lib/ruby_llm/protocols/converse/streaming.rb +++ b/lib/ruby_llm/protocols/converse/streaming.rb @@ -19,6 +19,7 @@ def stream_url def stream_response(payload, additional_headers = {}, &block) accumulator = StreamAccumulator.new + @stream_accumulator = accumulator decoder = event_stream_decoder thinking_state = {} @@ -45,10 +46,11 @@ def parse_stream_chunk(decoder, raw_chunk, accumulator, thinking_state) def decode_events(decoder, raw_chunk) decode_event_messages(decoder, raw_chunk).filter_map do |message| - event = decode_event_payload(message.payload.read) + type = event_type_header(message) + event = decode_event_payload(message.payload.read, event_type: type) next unless event - event = nest_event_under_type(event, message) + event = nest_event_under_type(event, type) RubyLLM.logger.debug { "Bedrock stream event keys: #{event.keys}" } if RubyLLM.config.log_stream_debug event end @@ -62,8 +64,7 @@ def decode_events(decoder, raw_chunk) # payload under its header type here. A payload already nested under its type (the # shape older specs fabricate) or a message with no type header passes through # unchanged. - def nest_event_under_type(event, message) - type = event_type_header(message) + def nest_event_under_type(event, type) return event unless type return event if event.key?(type) @@ -73,19 +74,20 @@ def nest_event_under_type(event, message) def build_chunk(event, thinking_state = {}) raise_stream_error(event) if stream_error_event?(event) - track_thinking_delta(event, thinking_state) + delta = normalized_delta(event) + track_thinking_delta(event, delta, thinking_state) metadata_usage, usage = event_usage(event) Chunk.new( role: :assistant, model_id: event['modelId'] || @model&.id, - content: extract_content_delta(event), + content: delta['text'], thinking: Thinking.build( - text: extract_thinking_delta(event), - signature: extract_thinking_signature(event), + text: extract_thinking_delta(delta), + signature: extract_thinking_signature(event, delta), blocks: extract_finalized_thinking_blocks(event, thinking_state) ), - tool_calls: extract_tool_calls(event), + tool_calls: extract_tool_calls(event, delta), input_tokens: extract_input_tokens(metadata_usage, usage), output_tokens: extract_output_tokens(metadata_usage, usage), cached_tokens: extract_cached_tokens(metadata_usage, usage), @@ -103,8 +105,8 @@ def build_chunk(event, thinking_state = {}) # block — shaped exactly like the blocks Converse::Chat.parse_thinking captures from # the non-streaming response — on that index's contentBlockStop # (see extract_finalized_thinking_blocks). - def track_thinking_delta(event, thinking_state) - index, reasoning_content = thinking_delta_source(event) + def track_thinking_delta(event, delta, thinking_state) + index, reasoning_content = thinking_delta_source(event, delta) return unless index && reasoning_content.is_a?(Hash) state = (thinking_state[index] ||= { text: +'', signature: nil, redacted: nil }) @@ -123,12 +125,12 @@ def apply_reasoning_content(state, reasoning_content) state[:redacted] = state[:redacted].to_s + redacted if redacted end - def thinking_delta_source(event) + def thinking_delta_source(event, delta) start_index = event.dig('contentBlockStart', 'contentBlockIndex') return [start_index, event.dig('contentBlockStart', 'start', 'reasoningContent')] if start_index delta_index = event.dig('contentBlockDelta', 'contentBlockIndex') - [delta_index, normalized_delta(event)['reasoningContent']] + [delta_index, delta['reasoningContent']] end # A thinking block finalizes on its own content_block_stop. As a safety net, messageStop @@ -231,19 +233,15 @@ def raise_stream_error(event) raise_event_stream_exception(key, event[key]['message']) end - def extract_content_delta(event) - normalized_delta(event)['text'] - end - - def extract_thinking_delta(event) - reasoning_content = normalized_delta(event)['reasoningContent'] || {} + def extract_thinking_delta(delta) + reasoning_content = delta['reasoningContent'] || {} reasoning_text = reasoning_content['reasoningText'] || {} reasoning_text['text'] || reasoning_content['text'] end - def extract_thinking_signature(event) - signature = extract_signature_from_delta(event) + def extract_thinking_signature(event, delta) + signature = extract_signature_from_delta(delta) return signature if signature signature = extract_signature_from_start(event) @@ -252,8 +250,8 @@ def extract_thinking_signature(event) nil end - def extract_signature_from_delta(event) - reasoning_content = normalized_delta(event)['reasoningContent'] || {} + def extract_signature_from_delta(delta) + reasoning_content = delta['reasoningContent'] || {} reasoning_text = reasoning_content['reasoningText'] || {} reasoning_text['signature'] || reasoning_content['signature'] end @@ -269,9 +267,9 @@ def extract_signature_from_start(event) nil end - def extract_tool_calls(event) + def extract_tool_calls(event, delta) return extract_tool_call_start(event) if tool_call_start_event?(event) - return extract_tool_call_delta(event) if tool_call_delta_event?(event) + return extract_tool_call_delta(delta) if tool_call_delta_event?(event) nil end @@ -298,13 +296,15 @@ def extract_tool_call_start(event) } end - def extract_tool_call_delta(event) - input = normalized_delta(event).dig('toolUse', 'input') + def extract_tool_call_delta(delta) + input = delta.dig('toolUse', 'input') return nil unless input { nil => ToolCall.new(id: nil, name: nil, arguments: input) } end + # Decoded once per event: a delta sent as a JSON string that fails to parse is reported as + # one skip, not once per extractor. def normalized_delta(event) delta = event.dig('contentBlockDelta', 'delta') || event['delta'] || {} return delta if delta.is_a?(Hash) @@ -314,7 +314,8 @@ def normalized_delta(event) else {} end - rescue JSON::ParserError + rescue JSON::ParserError => e + skip_stream_event(:content_block_delta, delta, event_type: 'contentBlockDelta', error: e) {} end end diff --git a/lib/ruby_llm/protocols/invoke_anthropic/streaming.rb b/lib/ruby_llm/protocols/invoke_anthropic/streaming.rb index 19d81f2531..d5f7504d4a 100644 --- a/lib/ruby_llm/protocols/invoke_anthropic/streaming.rb +++ b/lib/ruby_llm/protocols/invoke_anthropic/streaming.rb @@ -31,6 +31,7 @@ def stream_url def stream_response(payload, additional_headers = {}, &block) accumulator = StreamAccumulator.new + @stream_accumulator = accumulator decoder = event_stream_decoder thinking_state = {} block_state = {} @@ -65,12 +66,15 @@ def decode_invoke_event(message) payload = message.payload.read exception_type = exception_type_header(message) if exception_type - parsed = decode_event_payload(payload) || {} + parsed = decode_event_payload(payload, event_type: exception_type) || {} raise_event_stream_exception(exception_type, parsed['message'] || parsed['Message']) end data = decode_event_payload(payload) - return nil unless data.is_a?(Hash) + unless data.is_a?(Hash) + skip_stream_event(:event_payload, payload) unless data.nil? + return nil + end RubyLLM.logger.debug { "Bedrock invoke stream event: #{data['type']}" } if RubyLLM.config.log_stream_debug raise_anthropic_stream_error(data) if data['type'] == 'error' diff --git a/lib/ruby_llm/skipped_stream_event.rb b/lib/ruby_llm/skipped_stream_event.rb new file mode 100644 index 0000000000..b2d087def5 --- /dev/null +++ b/lib/ruby_llm/skipped_stream_event.rb @@ -0,0 +1,46 @@ +# frozen_string_literal: true + +module RubyLLM + SkippedStreamEvent = Struct.new(:provider, :model, :site, :event_type, :byte_size, :error_class, + keyword_init: true) + + # A streamed event a parser could not decode and dropped. Passed to config.on_stream_parse_error; + # its fields are documented in docs/_core_features/streaming.md and must stay stable. + class SkippedStreamEvent + SITES = %i[sse_data event_payload event_type_header exception_type_header content_block_delta].freeze + + def self.report(event, config) + RubyLLM.logger.warn("RubyLLM skipped a stream event it could not parse: #{event.log_fields}") + config.on_stream_parse_error&.call(event) + rescue StandardError => e + RubyLLM.logger.warn("RubyLLM on_stream_parse_error hook raised #{e.class}: #{e.message}") + end + + def log_fields + to_h.map { |key, value| "#{key}=#{value.nil? ? '-' : value}" }.join(' ') + end + + # Mixed into the streaming parsers. A skip never logs the payload itself, only its size. + module Reporting + private + + def skip_stream_event(site, payload, event_type: nil, error: nil) + event = SkippedStreamEvent.new( + provider: @provider&.slug, + model: @model&.id, + site: site, + event_type: event_type, + byte_size: skipped_payload_bytesize(payload), + error_class: error&.class&.name + ).freeze + @stream_accumulator&.skip_event + SkippedStreamEvent.report(event, @config || RubyLLM.config) + nil + end + + def skipped_payload_bytesize(payload) + payload.respond_to?(:bytesize) ? payload.bytesize : payload&.size + end + end + end +end diff --git a/lib/ruby_llm/stream_accumulator.rb b/lib/ruby_llm/stream_accumulator.rb index 57d10a3adb..a940cea94e 100644 --- a/lib/ruby_llm/stream_accumulator.rb +++ b/lib/ruby_llm/stream_accumulator.rb @@ -6,7 +6,7 @@ module RubyLLM # Assembles streaming responses from LLMs into complete messages. class StreamAccumulator - attr_reader :content, :model_id, :tool_calls + attr_reader :content, :model_id, :tool_calls, :skipped_events def initialize @content = +'' @@ -26,6 +26,7 @@ def initialize @pending_think_tag = +'' @latest_tool_call_id = nil @tool_call_ids_by_index = {} + @skipped_events = 0 end def add(chunk) @@ -41,6 +42,10 @@ def add(chunk) RubyLLM.logger.debug { inspect } if RubyLLM.config.log_stream_debug end + def skip_event + @skipped_events += 1 + end + def to_message(response) Message.new( role: :assistant, @@ -62,6 +67,7 @@ def to_message(response) model_id: model_id, tool_calls: tool_calls_from_stream, provider_data: @provider_data, + skipped_stream_events: @skipped_events, raw: response ) end diff --git a/lib/ruby_llm/streaming.rb b/lib/ruby_llm/streaming.rb index 45d8539a05..8f6e23b6f0 100644 --- a/lib/ruby_llm/streaming.rb +++ b/lib/ruby_llm/streaming.rb @@ -7,10 +7,13 @@ module RubyLLM # Handles streaming responses from AI providers. module Streaming + include SkippedStreamEvent::Reporting + module_function def stream_response(payload, additional_headers = {}, &block) accumulator = StreamAccumulator.new + @stream_accumulator = accumulator on_data = handle_stream do |chunk| accumulator.add chunk block.call chunk @@ -96,18 +99,23 @@ def handle_sse(chunk, parser, env) when :error handle_error_event(data, env) else - yield handle_data(data, env) unless data == '[DONE]' + yield handle_data(data, env, type) unless data == '[DONE]' end end end - def handle_data(data, env) + def handle_data(data, env, type = nil) parsed = JSON.parse(data) - return parsed unless parsed.is_a?(Hash) && parsed.key?('error') + return skip_sse_data(data, type) unless parsed.is_a?(Hash) + return parsed unless parsed.key?('error') raise_stream_error(data, parsed, env) rescue JSON::ParserError => e - RubyLLM.logger.debug { "Failed to parse data chunk: #{e.message}" } + skip_sse_data(data, type, e) + end + + def skip_sse_data(data, type, error = nil) + skip_stream_event(:sse_data, data, event_type: type.to_s.empty? ? nil : type, error: error) end def handle_error_event(data, env) diff --git a/spec/ruby_llm/configuration_spec.rb b/spec/ruby_llm/configuration_spec.rb index 2bc492cf1d..09665b4dfa 100644 --- a/spec/ruby_llm/configuration_spec.rb +++ b/spec/ruby_llm/configuration_spec.rb @@ -17,12 +17,14 @@ expect(config.tool_concurrency).to be(false) expect(config.deprecation_behavior).to eq(:warn) expect(config.faraday_adapter).to eq(:net_http) + expect(config.on_stream_parse_error).to be_nil end it 'exposes a discoverable options API' do expect(described_class.options).to include( :request_timeout, :tool_concurrency, + :on_stream_parse_error, :default_model, :default_speech_model, :model_registry_file, diff --git a/spec/ruby_llm/message_spec.rb b/spec/ruby_llm/message_spec.rb index 2c5416b4b5..5821a7e205 100644 --- a/spec/ruby_llm/message_spec.rb +++ b/spec/ruby_llm/message_spec.rb @@ -85,6 +85,18 @@ end end + describe '#skipped_stream_events' do + it 'defaults to zero' do + expect(described_class.new(role: :assistant, content: 'Hello').skipped_stream_events).to eq(0) + end + + it 'keeps the count it was built with' do + message = described_class.new(role: :assistant, content: 'Hello', skipped_stream_events: 2) + + expect(message.skipped_stream_events).to eq(2) + end + end + describe '#to_h' do it 'includes finish_reason when present' do message = described_class.new(role: :assistant, content: 'Hello', finish_reason: 'length') diff --git a/spec/ruby_llm/protocols/converse/streaming_spec.rb b/spec/ruby_llm/protocols/converse/streaming_spec.rb index 1fdda8d711..803949ba2e 100644 --- a/spec/ruby_llm/protocols/converse/streaming_spec.rb +++ b/spec/ruby_llm/protocols/converse/streaming_spec.rb @@ -366,7 +366,8 @@ def stream_to_message(*typed_payloads) payload: StringIO.new(JSON.generate({ 'message' => 'Too many requests' })) ) - event = streaming.send(:nest_event_under_type, { 'message' => 'Too many requests' }, message) + type = streaming.send(:event_type_header, message) + event = streaming.send(:nest_event_under_type, { 'message' => 'Too many requests' }, type) expect(event).to eq({ 'throttlingException' => { 'message' => 'Too many requests' } }) expect(streaming.send(:stream_error_event?, event)).to be(true) @@ -379,7 +380,126 @@ def stream_to_message(*typed_payloads) ) nested = { 'messageStop' => { 'stopReason' => 'end_turn' } } - expect(streaming.send(:nest_event_under_type, nested, message)).to equal(nested) + type = streaming.send(:event_type_header, message) + + expect(streaming.send(:nest_event_under_type, nested, type)).to equal(nested) + end + end + + describe 'skipped stream events' do + require 'aws-eventstream' + + let(:config) do + RubyLLM::Configuration.new.tap do |c| + c.bedrock_region = 'us-west-2' + c.bedrock_api_key = 'static-key' + c.bedrock_secret_key = 'static-secret' + end + end + let(:model) do + RubyLLM::Model::Info.new(id: 'anthropic.claude-test', name: 'claude-test', provider: 'bedrock', + created_at: nil, context_window: 200_000, max_output_tokens: 8192, + modalities: { input: [], output: [] }, capabilities: [], pricing: {}, metadata: {}) + end + let(:protocol) { RubyLLM::Protocols::Converse.new(RubyLLM::Providers::Bedrock.new(config), model) } + let(:skipped) { [] } + + before do + allow(RubyLLM.logger).to receive(:warn) + config.on_stream_parse_error = ->(event) { skipped << event } + end + + def wire_message(type, payload) + Aws::EventStream::Message.new( + headers: { + ':event-type' => Aws::EventStream::HeaderValue.new(value: type, type: 'string'), + ':message-type' => Aws::EventStream::HeaderValue.new(value: 'event', type: 'string') + }, + payload: StringIO.new(payload.is_a?(String) ? payload : JSON.generate(payload)) + ) + end + + def stream(*messages) + encoder = Aws::EventStream::Encoder.new + stub_request(:post, %r{/model/anthropic\.claude-test/converse-stream\z}) + .to_return(status: 200, body: messages.map { |message| encoder.encode(message) }.join, + headers: { 'Content-Type' => 'application/vnd.amazon.eventstream' }) + protocol.complete([RubyLLM::Message.new(role: :user, content: 'hi')], tools: {}, temperature: nil) { |_| nil } + end + + def tool_turn(*middle) + stream( + wire_message('contentBlockStart', { 'contentBlockIndex' => 0, + 'start' => { 'toolUse' => { 'toolUseId' => 'call_1', + 'name' => 'weather' } } }), + *middle, + wire_message('contentBlockDelta', { 'contentBlockIndex' => 0, + 'delta' => { 'toolUse' => { 'input' => '{"city":"SF"}' } } }), + wire_message('contentBlockStop', { 'contentBlockIndex' => 0 }), + wire_message('messageStop', { 'stopReason' => 'tool_use' }) + ) + end + + it 'reports an undecodable event payload, counts it, and still assembles the tool call' do + bad = '{"contentBlockIndex":0,"delta":{"toolUse":{"input":"{\"secret' + message = tool_turn(wire_message('contentBlockDelta', bad)) + + expect(message.tool_calls['call_1'].arguments).to eq('city' => 'SF') + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: 'anthropic.claude-test', site: :event_payload, + event_type: 'contentBlockDelta', byte_size: bad.bytesize, error_class: 'JSON::ParserError' }] + ) + expect(RubyLLM.logger).to have_received(:warn).with( + a_string_including('provider=bedrock model=anthropic.claude-test site=event_payload ' \ + "event_type=contentBlockDelta byte_size=#{bad.bytesize} error_class=JSON::ParserError") + ) + expect(RubyLLM.logger).not_to have_received(:warn).with(a_string_including('secret')) + end + + it 'reports a string delta that is not valid JSON exactly once' do + message = tool_turn(wire_message('contentBlockDelta', { 'contentBlockIndex' => 0, 'delta' => '{"text":' })) + + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: 'anthropic.claude-test', site: :content_block_delta, + event_type: 'contentBlockDelta', byte_size: 8, error_class: 'JSON::ParserError' }] + ) + end + + it 'records zero skipped events for a clean stream' do + message = tool_turn + + expect(message.skipped_stream_events).to eq(0) + expect(skipped).to be_empty + end + + it 'keeps streaming when the hook raises' do + config.on_stream_parse_error = ->(_event) { raise 'metrics down' } + + message = tool_turn(wire_message('contentBlockDelta', '{')) + + expect(message.tool_calls['call_1'].arguments).to eq('city' => 'SF') + expect(message.skipped_stream_events).to eq(1) + expect(RubyLLM.logger).to have_received(:warn) + .with('RubyLLM on_stream_parse_error hook raised RuntimeError: metrics down') + end + + it 'counts skips without a hook' do + config.on_stream_parse_error = nil + + expect(tool_turn(wire_message('contentBlockDelta', '{')).skipped_stream_events).to eq(1) + end + + it 'reports an event whose headers cannot be read' do + message = instance_double(Aws::EventStream::Message, payload: StringIO.new('{"a":1}')) + allow(message).to receive(:headers).and_raise(NoMethodError) + + expect(protocol.send(:event_type_header, message)).to be_nil + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: 'anthropic.claude-test', site: :event_type_header, + event_type: nil, byte_size: 7, error_class: 'NoMethodError' }] + ) end end end diff --git a/spec/ruby_llm/protocols/invoke_anthropic_spec.rb b/spec/ruby_llm/protocols/invoke_anthropic_spec.rb index 2076a4064c..aae377b351 100644 --- a/spec/ruby_llm/protocols/invoke_anthropic_spec.rb +++ b/spec/ruby_llm/protocols/invoke_anthropic_spec.rb @@ -558,6 +558,93 @@ def stream(messages = [user('hi')]) expect { stream }.to raise_error(RubyLLM::OverloadedError, /Overloaded/) end + + describe 'skipped events' do + before { allow(RubyLLM.logger).to receive(:warn) } + + def record_skips + [].tap { |skipped| config.on_stream_parse_error = ->(event) { skipped << event } } + end + + def raw_event(payload) + Aws::EventStream::Message.new( + headers: { ':event-type' => Aws::EventStream::HeaderValue.new(value: 'chunk', type: 'string') }, + payload: StringIO.new(payload) + ) + end + + def text_turn(*middle) + stub_stream(wire( + event('type' => 'content_block_start', 'index' => 0, + 'content_block' => { 'type' => 'text', 'text' => '' }), + event('type' => 'content_block_delta', 'index' => 0, + 'delta' => { 'type' => 'text_delta', 'text' => 'Hel' }), + *middle, + event('type' => 'content_block_delta', 'index' => 0, + 'delta' => { 'type' => 'text_delta', 'text' => 'lo' }), + event('type' => 'content_block_stop', 'index' => 0), + event('type' => 'message_delta', 'delta' => { 'stop_reason' => 'end_turn' }) + )) + stream.first + end + + it 'reports an undecodable outer payload and counts it on the message' do + skipped = record_skips + bad = '{"bytes":"secret' + message = text_turn(raw_event(bad)) + + expect(message.content).to eq('Hello') + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: model_id, site: :event_payload, event_type: nil, + byte_size: bad.bytesize, error_class: 'JSON::ParserError' }] + ) + expect(RubyLLM.logger).not_to have_received(:warn).with(a_string_including('secret')) + end + + it 'reports base64 bytes that do not decode to JSON' do + skipped = record_skips + bad = JSON.generate('bytes' => Base64.strict_encode64('{"type":"content_block_delta",')) + message = text_turn(raw_event(bad)) + + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map(&:site)).to eq([:event_payload]) + expect(skipped.first.byte_size).to eq(bad.bytesize) + end + + it 'reports an event that decodes to something other than a JSON object' do + skipped = record_skips + payload = JSON.generate('bytes' => Base64.strict_encode64('[1]')) + message = text_turn(raw_event(payload)) + + expect(message.content).to eq('Hello') + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: model_id, site: :event_payload, event_type: nil, + byte_size: payload.bytesize, error_class: nil }] + ) + end + + it 'records zero skipped events for a clean stream' do + skipped = record_skips + message = text_turn + + expect(message.skipped_stream_events).to eq(0) + expect(skipped).to be_empty + end + + it 'reports an event whose exception header cannot be read' do + skipped = record_skips + message = instance_double(Aws::EventStream::Message, payload: StringIO.new('{}')) + allow(message).to receive(:headers).and_raise(NoMethodError) + + expect(protocol.send(:exception_type_header, message)).to be_nil + expect(skipped.map(&:to_h)).to eq( + [{ provider: 'bedrock', model: model_id, site: :exception_type_header, event_type: nil, + byte_size: 2, error_class: 'NoMethodError' }] + ) + end + end end describe 'Converse replay of InvokeModel-captured thinking' do diff --git a/spec/ruby_llm/skipped_stream_event_spec.rb b/spec/ruby_llm/skipped_stream_event_spec.rb new file mode 100644 index 0000000000..db3f2fcd0b --- /dev/null +++ b/spec/ruby_llm/skipped_stream_event_spec.rb @@ -0,0 +1,102 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe RubyLLM::SkippedStreamEvent do + let(:config) { RubyLLM::Configuration.new } + let(:event) do + described_class.new(provider: 'bedrock', model: 'claude-test', site: :event_payload, + event_type: 'contentBlockDelta', byte_size: 42, error_class: 'JSON::ParserError') + end + + before { allow(RubyLLM.logger).to receive(:warn) } + + describe '.report' do + it 'logs every field at warn level' do + described_class.report(event, config) + + expect(RubyLLM.logger).to have_received(:warn).with( + 'RubyLLM skipped a stream event it could not parse: provider=bedrock model=claude-test ' \ + 'site=event_payload event_type=contentBlockDelta byte_size=42 error_class=JSON::ParserError' + ) + end + + it 'logs unknown fields as a dash' do + described_class.report(described_class.new(site: :sse_data, byte_size: 4), config) + + expect(RubyLLM.logger).to have_received(:warn).with( + a_string_including('provider=- model=- site=sse_data event_type=- byte_size=4 error_class=-') + ) + end + + it 'calls the configured hook with the event' do + received = [] + config.on_stream_parse_error = ->(skipped) { received << skipped } + + described_class.report(event, config) + + expect(received).to eq([event]) + end + + it 'only logs when no hook is configured' do + expect(config.on_stream_parse_error).to be_nil + expect { described_class.report(event, config) }.not_to raise_error + expect(RubyLLM.logger).to have_received(:warn).once + end + + it 'rescues and logs an exception the hook raises' do + config.on_stream_parse_error = ->(_skipped) { raise ArgumentError, 'metrics down' } + + expect { described_class.report(event, config) }.not_to raise_error + expect(RubyLLM.logger).to have_received(:warn) + .with('RubyLLM on_stream_parse_error hook raised ArgumentError: metrics down') + end + end + + describe described_class::Reporting do + let(:accumulator) { RubyLLM::StreamAccumulator.new } + let(:received) { [] } + let(:reporter) do + Object.new.tap do |object| + object.extend(described_class) + object.instance_variable_set(:@config, config) + object.instance_variable_set(:@stream_accumulator, accumulator) + object.instance_variable_set(:@provider, instance_double(RubyLLM::Provider, slug: 'bedrock')) + object.instance_variable_set(:@model, instance_double(RubyLLM::Model::Info, id: 'claude-test')) + end + end + + before { config.on_stream_parse_error = ->(skipped) { received << skipped } } + + it 'builds a frozen event from the protocol, counts it and reports it without the payload' do + result = reporter.send(:skip_stream_event, :event_payload, 'secret payload', + event_type: 'chunk', error: JSON::ParserError.new('secret payload')) + + expect(result).to be_nil + expect(received.first.to_h).to eq(provider: 'bedrock', model: 'claude-test', site: :event_payload, + event_type: 'chunk', byte_size: 14, error_class: 'JSON::ParserError') + expect(received.first).to be_frozen + expect(accumulator.skipped_events).to eq(1) + expect(RubyLLM.logger).not_to have_received(:warn).with(a_string_including('secret')) + end + + it 'measures IO payloads by size and counts bytes, not characters' do + reporter.send(:skip_stream_event, :event_type_header, StringIO.new('{"é":1}')) + + expect(received.first.byte_size).to eq(8) + expect(received.first.error_class).to be_nil + end + + it 'falls back to the global config outside a protocol' do + bare = Object.new.tap { |object| object.extend(described_class) } + global = [] + RubyLLM.config.on_stream_parse_error = ->(skipped) { global << skipped } + + bare.send(:skip_stream_event, :sse_data, 'x') + + expect(global.map(&:site)).to eq([:sse_data]) + ensure + RubyLLM.config.on_stream_parse_error = nil + end + end +end diff --git a/spec/ruby_llm/stream_accumulator_spec.rb b/spec/ruby_llm/stream_accumulator_spec.rb index 307bdf5cc6..899686ab66 100644 --- a/spec/ruby_llm/stream_accumulator_spec.rb +++ b/spec/ruby_llm/stream_accumulator_spec.rb @@ -3,6 +3,17 @@ require 'spec_helper' RSpec.describe RubyLLM::StreamAccumulator do + describe '#skip_event' do + it 'records skipped events on the final message' do + accumulator = described_class.new + expect(accumulator.to_message(nil).skipped_stream_events).to eq(0) + + 2.times { accumulator.skip_event } + + expect(accumulator.to_message(nil).skipped_stream_events).to eq(2) + end + end + describe '#add' do it 'handles tool call deltas that omit arguments' do accumulator = described_class.new diff --git a/spec/ruby_llm/streaming_spec.rb b/spec/ruby_llm/streaming_spec.rb index 2ab9c2cad8..e8ca6ff3ff 100644 --- a/spec/ruby_llm/streaming_spec.rb +++ b/spec/ruby_llm/streaming_spec.rb @@ -63,4 +63,102 @@ expect(response.status).to eq(429) expect(response.body).to eq(parsed_error) end + + describe 'skipped SSE events' do + let(:skipped) { [] } + + before do + allow(RubyLLM.logger).to receive(:warn) + RubyLLM.config.on_stream_parse_error = ->(event) { skipped << event } + end + + after { RubyLLM.config.on_stream_parse_error = nil } + + def feed(sse) + chunks = [] + handler = test_obj.send(:handle_stream) { |chunk| chunks << chunk } + handler.call(sse, 0, env) + chunks + end + + it 'reports data that is not valid JSON with its SSE event type, and keeps streaming' do + chunks = feed("event: content_block_delta\ndata: {\"x\":\"secret\n\ndata: {\"x\":\"ok\"}\n\n") + + expect(chunks).to eq(['chunk:ok']) + expect(skipped.map(&:to_h)).to eq( + [{ provider: nil, model: nil, site: :sse_data, event_type: 'content_block_delta', byte_size: 12, + error_class: 'JSON::ParserError' }] + ) + expect(RubyLLM.logger).to have_received(:warn) + .with(a_string_including('site=sse_data event_type=content_block_delta byte_size=12')) + expect(RubyLLM.logger).not_to have_received(:warn).with(a_string_including('secret')) + end + + it 'reports data that decodes to something other than a JSON object' do + expect(feed("data: true\n\n")).to eq([]) + + expect(skipped.map(&:to_h)).to eq( + [{ provider: nil, model: nil, site: :sse_data, event_type: nil, byte_size: 4, error_class: nil }] + ) + end + + it 'does not report the [DONE] sentinel' do + feed("data: [DONE]\n\n") + + expect(skipped).to be_empty + expect(RubyLLM.logger).not_to have_received(:warn) + end + + it 'still logs a skip when no hook is configured' do + RubyLLM.config.on_stream_parse_error = nil + + expect { feed("data: {\n\n") }.not_to raise_error + expect(RubyLLM.logger).to have_received(:warn).with(a_string_including('site=sse_data')) + end + end + + describe 'skipped event count on a streamed message' do + let(:config) { RubyLLM::Configuration.new.tap { |c| c.openai_api_key = 'test' } } + let(:protocol) do + model = RubyLLM::Model::Info.new(id: 'gpt-test', name: 'gpt-test', provider: 'openai', created_at: nil, + context_window: 128_000, max_output_tokens: 4096, + modalities: { input: [], output: [] }, capabilities: [], pricing: {}, + metadata: {}) + RubyLLM::Protocols::ChatCompletions.new(RubyLLM::Providers::OpenAI.new(config), model) + end + + before { allow(RubyLLM.logger).to receive(:warn) } + + def record_skips + [].tap { |skipped| config.on_stream_parse_error = ->(event) { skipped << event } } + end + + def stream(*data_lines) + body = data_lines.map { |line| "data: #{line}\n\n" }.join + stub_request(:post, 'https://api.openai.com/v1/chat/completions') + .to_return(status: 200, body: body, headers: { 'Content-Type' => 'text/event-stream' }) + protocol.complete([RubyLLM::Message.new(role: :user, content: 'hi')], tools: {}, temperature: nil) { |_| nil } + end + + it 'records how many events were skipped and reports provider and model' do + skipped = record_skips + message = stream('{"model":"gpt-test","choices":[{"delta":{"content":"Hel"}}]}', + '{"choices":[{"delta":{"tool_calls":[{"index":0,"id":"call_1"', + '{"choices":[{"delta":{"content":"lo"}}]}', + '[DONE]') + + expect(message.content).to eq('Hello') + expect(message.skipped_stream_events).to eq(1) + expect(skipped.map { |event| [event.provider, event.model, event.site] }) + .to eq([['openai', 'gpt-test', :sse_data]]) + end + + it 'records zero when every event decodes' do + skipped = record_skips + message = stream('{"model":"gpt-test","choices":[{"delta":{"content":"Hi"}}]}', '[DONE]') + + expect(message.skipped_stream_events).to eq(0) + expect(skipped).to be_empty + end + end end