Repository navigation
mod_http3: fix a session allocator race, a stuck graceful drain and a blocked-stream spin - #947
Conversation
Worker threads allocate from stream subpools while the event thread allocates from the session pool. All of them use one allocator, which had no mutex, so concurrent allocations corrupted the heap and crashed the child under load. Set a mutex on the allocator, as mod_http2 does.
A pending handshake counts as an MPM connection. The event thread advanced pending handshakes only when not draining, so a handshake in flight at a graceful restart did not finish or time out. The old child then never reached zero connections and kept the port.
nghttp3_conn_resume_stream puts a blocked stream back in the write queue. The module blocked the stream only on the first refusal, so when the peer kept its window shut, nghttp3 offered the stream again and again and the event thread used 100% CPU.
c5a83b0 to
4a44dc9
Compare
|
@jfclere looks like different problems addressed (analysed it only with Claude, not with my eyes): #993 and #947 overlap but don't duplicate each other. #993 works around the symptoms in the shutdown code and in the tests; #947 fixes what most likely causes them. They merge cleanly together, and nothing in #947 has to change. #993 (merged to trunk): h3_server.c: the port-acquire thread is detached, and shutdown polls a flag instead of #993 also brought one new bug into trunk. If port_acquire_thread_fn can't create its memory pool, it returns without clearing port_acquire_running, so h3_c1_child_stopping waits forever.
I would take your 20-restart test_h3_014_04_stop_cleanly to xl32/mod_http3 and if you want, can add it to this pr also. |
…ake its pool Else h3_c1_child_stopping waits for ever.
A live session at stop also tears down its control streams, which is where the shutdown crashes were.
|
merged: Now I am rolling backing back the fix for #993, retesting and PR the rollback. |
Three small mod_http3 stability fixes. Each is its own commit with its own changes-entries file.
1. Lock the session allocator: workers and the event thread share it
Each QUIC connection gets an APR allocator owned by its session pool. Worker threads allocate from stream subpools of that pool while the event thread allocates from the session itself, and the allocator has no mutex. Under concurrent requests on one connection this corrupts the heap and crashes the child. The fix gives the allocator a mutex (
apr_allocator_mutex_set), as mod_http2 does for its per-connection allocators. The mutex is only taken when a pool needs a new block, not on each allocation.2. Advance pending handshakes while draining
If a handshake was still pending when the child began to drain (graceful restart or stop), the event loop no longer advanced it, so it never completed or timed out. The old child kept its connection count and held the UDP port, so the graceful restart never finished. A pending handshake counts as an MPM connection, so
progress_pending_handshakes()now also runs while draining.3. Block a write-blocked stream again each time nghttp3 offers it
When the QUIC send buffer of a stream is full,
flush_nghttp3()callednghttp3_conn_block_stream()only the first time (guarded bywrite_blocked). Oncenghttp3_conn_resume_stream()puts the stream back in the schedule, nghttp3 offers it again, so while a client keeps its flow-control window shut the event thread spun on it. The stream is now blocked again every time it is offered. Thewrite_blockedbookkeeping is unchanged.Tested: built httpd trunk with
--enable-http3=sharedon EL9 (OpenSSL 4.0.3 with QUIC, nghttp3 1.19), no new warnings. HTTP/3 GET returns 200 with the right body in 8 of 8 start/request/stop cycles.