Repository navigation
mod_http3: run the QUIC transport on ngtcp2 - #951
Conversation
A host with "h3" in Protocols now serves HTTP/3 with the certificate mod_ssl already resolved for it (SSLCertificateFile and mod_md alike), the way mod_http2 rides mod_ssl. The certificate and key are loaded in the ap_ssl_add_cert_files hook, during startup while httpd still runs privileged, so a key readable only by root loads for QUIC as it does for mod_ssl instead of failing in the unprivileged child. H3CertificatePath and H3CertificateKeyPath are removed; drop them from existing configurations. Each virtual host on a shared port now presents its own certificate over HTTP/3, chosen by the client SNI; an unmatched name gets the listener default. Names come from ServerName, ServerAlias and the wildcard ServerAlias names that httpd keeps in wild_names, and every entry is matched with ap_strcasecmp_match, the matcher httpd uses for its own virtual hosts. A cert_cb on the listener context applies the matched host certificate, key and chain to the connection: SSL_set_SSL_CTX does not switch the certificate of a QUIC connection. Landing it exposed that the module ran with AP_MODULE_FLAG_NONE, so a host without H3 directives shared the main server configuration; per-host H3AltSvc and H3AltSvcMaxAge were silently ignored. Set ALWAYS_MERGE.
httpd prepends virtual hosts to the server list, so the first host with an HTTP/3 context in that list is the last one defined. The listener took that host's settings and certificate, and a client whose SNI matched no host got that certificate, where mod_ssl serves the first virtual host defined for the address. Take the last match in both h3_post_config and find_h3_server instead.
Each child bound the UDP port itself after httpd dropped root, so a port below 1024 needed CAP_NET_BIND_SERVICE on the httpd binary. On Linux the parent now binds the socket in post_config, while it still runs as root, and keeps it in the process pool, so it stays bound across graceful restarts. The children inherit it. One child at a time serves it: the child that binds an abstract Unix socket name owns the port, and the kernel releases the name when that child exits. Other platforms keep the old per-child bind.
With ListenCoresBucketsRatio, httpd splits its TCP listeners into buckets. The parent now binds one SO_REUSEPORT UDP socket per bucket, and each child serves the first bucket that no other child owns. So more than one child serves HTTP/3. The parent keeps the sockets while children start and stop, so the kernel hash does not change and each peer stays on the same socket. Without buckets, there is one socket as before.
Some MPMs count their listener buckets in the run step, after post_config, so ap_num_listen_buckets was still 1 there. The module now uses the same count as httpd: online cores / ratio, capped at the MPM daemon limit, because a bucket with no child loses its peers.
Worker threads allocate from stream subpools while the event thread allocates from the session pool. All of them use one allocator, which had no mutex, so concurrent allocations corrupted the heap and crashed the child under load. Set a mutex on the allocator, as mod_http2 does.
A pending handshake counts as an MPM connection. The event thread advanced pending handshakes only when not draining, so a handshake in flight at a graceful restart did not finish or time out. The old child then never reached zero connections and kept the port.
nghttp3_conn_resume_stream puts a blocked stream back in the write queue. The module blocked the stream only on the first refusal, so when the peer kept its window shut, nghttp3 offered the stream again and again and the event thread used 100% CPU.
OpenSSL's own QUIC server drops 0-RTT packets, so 0-RTT cannot work on it. This change runs the QUIC transport on ngtcp2 and keeps OpenSSL only for the TLS 1.3 handshake, through its QUIC TLS API. This is the same split that nginx uses. The h3q_ API does not change. Its implementation now reads the UDP socket, maps connection IDs to connections, answers Retry and Version Negotiation, and runs the ngtcp2 timers on a monotonic clock. The BIO filter that recovered peer addresses from the OpenSSL listener is removed. The engine does not copy stream data: it reports each acknowledgement through stream_acked, and nghttp3 releases the buffer when the peer acknowledges it. Connection close follows RFC 9000 section 10. A local close keeps its CONNECTION_CLOSE packet and resends it, at most once per PTO, for three PTOs. An idle timeout closes silently. The transport idle timer runs 2 seconds past H3IdleTimeout, so the module's clean close comes first. H3EarlyData (default off) turns on 0-RTT. With it on, a returning client sends its first request with the handshake and gets the response one round trip sooner. Only safe methods run before the handshake completes; other methods wait for it, because 0-RTT data can be replayed (RFC 8470). The TLS layer sets SSL_OP_NO_ANTI_REPLAY, because OpenSSL's own anti-replay is built for TLS over TCP and refuses QUIC resumption with early data. ngtcp2 >= 1.22.1 with its OpenSSL crypto helper (ngtcp2_crypto_ossl) is a new build dependency: --with-ngtcp2 for configure, NGTCP2_INCLUDE_DIR and NGTCP2_LIBRARIES for CMake.
The upload windows were fixed at 1 MiB per connection and 256 KiB per stream, so an upload on a long, fast path stalled on flow control. Turn on ngtcp2 window autotuning: when a client fills a window within a few round trips, ngtcp2 doubles it. H3MaxWindow (default 6 MiB, the ngtcp2 example default) caps the growth, per connection and per stream. The windows still start at 1 MiB and 256 KiB.
The engine sends the packets of one flush to one peer with one sendmsg and UDP_SEGMENT. Reads use UDP_GRO, and the engine splits a joined read into datagrams. If the kernel or the device refuses GSO, the engine turns it off and sends each packet alone. A stream read now sends the new flow-control window at once. Before, the update waited for the next packet from the peer. A blocked peer sends no packet, and with GRO an upload could stop.
… buckets On Linux the [::] socket now reads the address each packet came to (IP_PKTINFO, IPV6_PKTINFO) and sends the reply from it. Before, the kernel picked the source, and a host with more than one address could reply from another one, which a client drops. ECN: the ECN mark of each packet goes to ngtcp2, which reports it in ACK_ECN frames, and the mark ngtcp2 asks for goes on each sent packet. The parent makes the key for Retry and stateless reset tokens and keeps it across graceful restarts, so a token from one child is good in all. With more than one SO_REUSEPORT socket the server sends disable_active_migration.
The QUIC connection keeps its connection IDs in 16 slots of its own, not in an APR pool of 8 KB or more. It makes the CONNECTION_CLOSE buffer only when it closes. The session and stream allocators keep at most MaxMemFree, as in mod_http2. cid_add does not add a CID that another connection has. Before, the map could keep a key that pointed into the pool of a freed connection.
|
Have you seen my investigation on the topic: https://github.com/jfclere/httpd-asf/tree/ngtcp2 |
@jfclere Thank you, I looked at it now. We go the same way: ngtcp2 with ngtcp2_crypto_ossl, OpenSSL keeps TLS 1.3, and the internal h3q_ API does not change. You also give back the stream credit when a stream closes. I missed this point, but will add it.
|
… idle sessions nghttp3 checks each PRIORITY_UPDATE against its own client stream limit, set once from H3MaxConcurrentStreams. ngtcp2 gives the peer one stream back each time a stream closes, but nghttp3 did not know. After 100 requests on one connection, a PRIORITY_UPDATE for a new stream gave H3_ID_ERROR, and the server stopped reading the client control stream. The engine now records each MAX_STREAMS raise (extend_max_remote_streams_bidi), and the session gives the new limit to nghttp3 before it reads the streams. From Jean-Frederic Clere's ngtcp2 branch. The event loop skipped a session only when it had no streams. Every session keeps its three control streams, so the skip never ran, and each wakeup served every connection. Now a session is skipped unless the engine received a packet for it, ran its timers or raised its stream limit, a worker has output or finished, or its last pass ran out of budget. Each session is still served once a second, for its idle and stream timeouts. test_015: 105 requests with PRIORITY_UPDATE on one connection, and a request after 1.5 s idle. Tested on EL10 with openssl-quic 4.0.3, ngtcp2 1.26.0 and nghttp3 1.19.0: test_015_01 fails without the change (nghttp3 error -607 on stream 2) and passes with it. 20 parallel 1 MiB downloads take the same time as before.
@jfclere thanks!
|
|
The ListenCoresBucketsRatio won't work correctly on UPD sockets: the origin of the datagram might change as there is no connection. That is why ngnix uses Ebpf for example. |
|
if you want claude explanation, it is more verbose, the problem is:
|
…T.md The certificate now comes from SSLCertificateFile and SSLCertificateKeyFile on a host with h3 in Protocols.
…uite H3CertificatePath and H3CertificateKeyPath are gone: a host serves HTTP/3 with the certificate mod_ssl has for it once h3 is in Protocols. The https vhosts already carry SSLEngine and the certificate, so the suite only adds the protocol. test_010 checked the removed directives; it now checks h3 on a host without SSLEngine (no host serves HTTP/3) and an unusable certificate, which now fails startup instead of each child.
With the UDP socket bound in the parent, a port held by another process now logs a warning that test_012_06 did not ignore, so its error log check failed at teardown. test_012_07 already ignores the same line.
…ckets On UDP, SO_REUSEPORT hashes each datagram on its 4-tuple. When a client's address changes (migration or NAT rebinding), its packets can reach another child, which does not know the connection ID and drops them. disable_active_migration stops active migration only. With ListenCoresBucketsRatio, byte 0 of each server connection ID now holds the bucket number. The parent attaches an SK_REUSEPORT eBPF program to the socket group, while it is still root. The program reads byte 0 of the destination connection ID and sends the packet to the socket of that bucket, so a client keeps its child when its address changes. nginx does the same with quic_bpf. The program is 24 hand-written BPF instructions, loaded with bpf() calls: no clang or libbpf. It needs Linux 4.19 or later and CAP_BPF or CAP_SYS_ADMIN. If the load fails (old kernel, no capability, SELinux, seccomp), a warning gives the reason, the kernel hash stays, and clients get disable_active_migration, as before. Tests can run so without root. test_016: the server connection IDs of one connection share byte 0, and a connection survives six changes of the client port. Tested on EL10 with openssl-quic 4.0.3, ngtcp2 1.26.0 and nghttp3 1.19.0, with bpf() allowed (routing on: 5 passed) and with Docker's default seccomp (routing off: 3 passed, 2 skipped).
|
@jfclere I already had this part but didn't wanted to overload you with big patch set. |
|
Cool thanks for the work. |
|
Oops I caused the conflict I think, you have a look. |
h3_server.c: keep the includes from both sides. The port acquirer also clears port_acquire_running when it cannot create its pool, else h3_c1_child_stopping waits for ever.
Merged trunk, conflict resolved :-) |
|
@xl32 Can you please have a look to the conflicts I would like to retest and merge before or during the week-end. |
|
@xl32 I have noted 2 problems: 1 can be xfailed for the moment |
|
I have done a PR against your repo for the --with-http3-quic=ngtcp2|openssl, do you want to take it, or should we do that later. |
I would skip plain openssl, this overcomplicates code, please give me 5 minutes to fix ngtcp2 part :-) |
|
@jfclere please run the checks |
@jfclere Claude found test error on 1: Why only sometimes: when the response headers and the end-of-stream arrive in the same packet, aioquic fails before it reports the headers, so the test sees status=None. When they arrive separately, the headers are already recorded and the test passes. How often they arrive together depends on timing, which is why it fails more often on jfclere's machine than in CI. Fix: in test/pyhttpd/h3client.py, the client now skips aioquic's content-length check for HEAD requests. This overrides _check_content_length, a private aioquic method, which is acceptable in test code. I pushed it to the stress branch and the run is going now. Reviewing it. |
|
@jfclere ASan failed on unrelated problem, please check |
Depends on #947, #948 and #949. This branch is stacked on #949, which contains #948. The first three commits are #947's; the five after them are this PR.
Why
modules/http3/README_FIRST.mdsays:#949 removes the single-process limit. This PR removes the other one.
With OpenSSL's QUIC server, OpenSSL owns the transport and mod_http3 works around it:
OPENSSL_memdupinh3q_addr.c), then pumpsSSL_handle_eventsin budgeted loops until the queue drains.sendmsg.ngtcp2 comes from the same project as nghttp2 and nghttp3, which httpd already uses. With it, OpenSSL still does TLS 1.3 through its QUIC TLS API, and the module owns the socket and the packet path.
Measured
httpd trunk with
--enable-http3=shared, OpenSSL 4.0, nghttp3 1.19, clienth2load --h3, medians of 3 runs. Left column is trunk with #947–#949. Client and server shared one 16-vCPU VM over loopback, so the ratios matter more than the absolute numbers.Per connection and per packet, ngtcp2 is 1.8× to 11.7× faster and uses 2.4× to 8× less server CPU per request. With few connections and many parallel requests the transport is not the bottleneck, and the two are even. On that host OpenSSL QUIC also failed some of the large downloads; ngtcp2 completed all of them.
What changes
H3EarlyData(0-RTT). The internalh3q_API is unchanged. The per-datagram BIO copy is gone, and stream data is not copied: nghttp3 releases each buffer when the peer acknowledges it. Connection close follows RFC 9000 §10.H3EarlyData(default off) lets a resumed client send its first request as 0-RTT data, one round trip sooner. Only safe methods run before the handshake completes (RFC 8470).H3MaxWindow(default 6 MiB), so uploads on long, fast paths are no longer held to 1 MiB.[::], replies come from the address the client sent to (IP_PKTINFO/IPV6_PKTINFO). ECN marks are passed through. The parent makes the Retry and stateless-reset token key, so tokens are valid in every child and across graceful restarts. With more than one bucket the server sendsdisable_active_migration.MaxMemFree, as mod_http2 does. A client cannot insert an Initial DCID that belongs to another connection.What this opens
Because the module now creates its own connection IDs, it can put the bucket number in them. That is what the eBPF idea from #935 ("eBPF linux kernel filters to allow multi process on the UDP socket") needs. A follow-up PR adds an
SK_REUSEPORTeBPF program that routes each packet to its bucket's socket by connection ID, so a client can change address and keep its connection across processes. Until then, the server sendsdisable_active_migrationwhen it uses more than one bucket.Dependency
This adds ngtcp2 ≥ 1.22.1 with its OpenSSL crypto helper:
--with-ngtcp2for configure,NGTCP2_INCLUDE_DIR/NGTCP2_LIBRARIESfor CMake. Without it mod_http3 is not built, the same as a missing nghttp3 today. ngtcp2 is MIT-licensed C with no dependencies beyond OpenSSL, and curl uses it for HTTP/3. EPEL 9 ships it asngtcp2-crypto-ossl-devel1.22.1.Tested on EL9 with nghttp3 1.19: OpenSSL 4.0 with ngtcp2 1.26, and the stock EL9 OpenSSL 3.5.8 with EPEL's ngtcp2 1.22.1, which gives the same numbers. No new warnings.