Skip to content

staging: vc04_services: bcm2835-codec: Signal every buffer returned during a flush - #7713

Open
rahmounirabii wants to merge 1 commit into
raspberrypi:rpi-6.18.yfrom
rahmounirabii:bcm2835-codec-flush-last-buffer
Open

rahmounirabii wants to merge 1 commit into
raspberrypi:rpi-6.18.yfrom
rahmounirabii:bcm2835-codec-flush-last-buffer

Conversation

@rahmounirabii

Copy link
Copy Markdown

buffers_with_vpu is decremented in buffer_work_cb() before the buffer callback runs. With the port disabled, ip_buffer_cb() and op_buffer_cb() only signalled frame_cmplt if the counter was still non-zero, so the callback of the last returned buffer never signalled. When bcm2835_codec_flush_buffers() saw one buffer still with the VPU, it then slept the whole COMPLETE_TIMEOUT (2 s) under dev_mutex and logged "Timeout waiting for buffers to be returned - 0 outstanding", stalling QBUF/DQBUF of every other decoder instance meanwhile (a video playing in another instance freezes for ~2 s).

Signal frame_cmplt for every buffer returned while the port is disabled, including the EOS and error paths that are also counted in buffers_with_vpu, and make the flush wait loop while buffers remain with the VPU, each wait still bounded by COMPLETE_TIMEOUT.

Tested on a Pi 4B (6.18.50+rpt-rpi-v8): no occurrence of the timeout since 22 Oct, over ~45 h of signage playback plus an 8 h stress run (~4,700 decoder reuse cycles while another video plays). Details and reproduction in #7712.

Fixes #7712

@sapireli

Copy link
Copy Markdown

Independent confirmation from a different board, and evidence that the lost/early completion is more than a 2 s stall: on a decoder it leads to a use-after-free and a hung kernel.

Setup: Pi Zero W (armv6), 6.18.50+rpt-rpi-v6 (1:6.18.50-1+rpt1), two concurrent H.264 decodes with GStreamer 1.26 (v4l2h264dec ! … ! kmssink on overlay planes).

What happens. GStreamer's stateful decoder stops only the CAPTURE queue while OUTPUT keeps streaming on start-up renegotiation, on a resolution change and on an EOS drain (gstv4l2videodec.c: the draining path and negotiate()). In that case:

  1. bcm2835_codec_stop_streaming() disables the MMAL output port and calls bcm2835_codec_flush_buffers().
  2. buffer_work_cb() decrements buffers_with_vpu before calling op_buffer_cb(), and op_buffer_cb() only completes frame_cmplt while the counter is still non-zero — so with several buffers outstanding the flush wakes on the first returned buffer, not the last.
  3. Because the input port is still enabled, stop_streaming() then re-enables the output port ("keep the MMAL output port enabled for resolution changed events").
  4. The buffers the VPU returns after that see port->enabled and are resubmitted with vchiq_mmal_submit_buffer() instead of going back to videobuf2.

Kernel log (netconsole), in order:

WARNING: … at drivers/media/common/videobuf2/videobuf2-core.c:2222 __vb2_queue_cancel
 … v4l2_m2m_streamoff → vb2_core_streamoff → __vb2_queue_cancel
videobuf2_common: driver bug: stop_streaming operation is leaving buffer 1 in active state
videobuf2_common: driver bug: stop_streaming operation is leaving buffer 2 in active state
…
Workqueue: events buffer_work_cb [bcm2835_mmal_vchiq]
PC is at vc_sm_cma_import_dmabuf_internal+0x1b0/0x454 [vc_sm_cma]
 vc_sm_cma_import_dmabuf ← vchiq_mmal_submit_buffer ← buffer_work_cb

On other runs the first fault was buffer_from_host+0x6c (NULL write) from buffer_work_cb, or port_disable/release_all_event_contexts from bcm2835_codec_release; every time it cascaded into a hang that needed a power cycle. Stopping and restarting the two pipelines hung the board within 2–8 restarts, and it also happened on its own within a minute of start-up (the start-up renegotiation). Pipelines ending in fakesink instead of kmssink did not reproduce it in 20 stops, presumably because of timing. The single-buffer case also appeared once as Timeout waiting for buffers to be returned - 0 outstanding.

Our fix (minimal), which stopped it completely: signal only when the last buffer is back, at the three existing sites:

-	if (!port->enabled && atomic_read(&port->buffers_with_vpu))
+	if (!port->enabled && !atomic_read(&port->buffers_with_vpu))
 		complete(&ctx->frame_cmplt);

(likewise the if (atomic_read(&port->buffers_with_vpu)) in the empty-buffer branch of op_buffer_cb()). Built as an out-of-tree module against the exact Raspberry Pi kernel package and loaded in place of the stock module: repeated restarts, start-up renegotiations and orderly stops ran with no warnings at all, and decoding at full rate has run cleanly since.

This PR reaches the same end differently — completing for every returned buffer (including the EOS/error early returns, which our patch does not cover) and looping in bcm2835_codec_flush_buffers() until buffers_with_vpu is zero — which also prevents the early wake-up that leads to the resubmission above, and is the more thorough of the two. I haven't run this PR's exact patch on the Zero W yet, but the analysis matches; it would be worth noting in the commit message that the early completion can lead to buffers being resubmitted after stop_streaming() returns, not only to the 2 s stall.

🤖 Generated with Claude Code

…uring a flush

buffers_with_vpu is decremented in buffer_work_cb() before the buffer
callback runs. With the port disabled, ip_buffer_cb() and op_buffer_cb()
only signalled frame_cmplt if the counter was still non-zero, so the
callback of the last returned buffer never signalled. When
bcm2835_codec_flush_buffers() saw one buffer still with the VPU, it then
slept the whole COMPLETE_TIMEOUT (2 s) under dev_mutex and logged
"Timeout waiting for buffers to be returned - 0 outstanding", stalling
QBUF/DQBUF of every other decoder instance meanwhile (a video playing
in another instance freezes for ~2 s).

With several buffers outstanding, the same condition made the single
wait in bcm2835_codec_flush_buffers() return on the first returned
buffer rather than the last. For a decoder, stopping the CAPTURE queue
while the input port is still enabled then re-enables the output port,
and op_buffer_cb() resubmits the empty buffers the VPU returns after
that with vchiq_mmal_submit_buffer() instead of handing them back to
videobuf2. Eliran Sapir reported this on a Pi Zero W running two
concurrent H.264 decodes: videobuf2 warns that stop_streaming leaves
buffers in active state, followed by faults in buffer_work_cb and a
hung board.

Signal frame_cmplt for every buffer returned while the port is disabled,
including the EOS and error paths that are also counted in
buffers_with_vpu, and make the flush wait loop while buffers remain with
the VPU, each wait still bounded by COMPLETE_TIMEOUT. The flush now
returns once no buffer is left with the VPU, or after a COMPLETE_TIMEOUT
expires with one still outstanding.

Link: raspberrypi#7712
Link: raspberrypi#7713 (comment)
Signed-off-by: Rabii Rahmouni <rahmounirabii.me@gmail.com>
@rahmounirabii
rahmounirabii force-pushed the bcm2835-codec-flush-last-buffer branch from 3479796 to a8f2f90 Compare October 29, 2026 10:33
@rahmounirabii

Copy link
Copy Markdown
Author

Thank you for the independent report and the detailed log.

I checked the paths you describe against the driver source: with several buffers outstanding, the conditional complete() makes the single wait in bcm2835_codec_flush_buffers() return on the first returned buffer; bcm2835_codec_stop_streaming() on the CAPTURE queue of a decoder then re-enables the output port while the input port is enabled; and op_buffer_cb() resubmits empty buffers returned while the port is enabled with vchiq_mmal_submit_buffer(). I have updated the commit message to describe this consequence and to credit your report, as you suggested. The code is unchanged.

To be precise about what I observed myself: on the Pi 4B I only saw the 2 s stall described in #7712; I have not reproduced the resubmission or the crash there. With this patch each wait is still bounded by COMPLETE_TIMEOUT, so if the VPU does not return a buffer within that time, the flush still gives up and logs the timeout, as before.

If you get a chance to run this exact patch on the Zero W, the result would be very useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants