[climagabriel]

Adds server_name_resolver_by_lua_block. During server {} selection, when the host name (TLS SNI, Host or :authority) matches no exact server_name, a Lua chunk can name the server {} block to select. The request's host is not changed.

The chunk runs before wildcard and regex matching, for every host that has no exact server_name, on every request (see the TLS exception below). It cannot yield.

Draft. Requires both commits of nginx/nginx#1972, which is not merged. The first adds a server_name_resolver callback to ngx_http_core_main_conf_t, called after the exact-name lookup misses and before wildcard names. The second sets fc->reusable = 0 on HTTP/2 stream connections. If that PR is rejected, this one is void. Without openresty/lua-resty-core#549 (lua-resty-core), ngx.get_phase() raises unknown phase: 524288 in the chunk (0x00080000, the new context bit); with it, it returns "server_name_resolver". The tests here do not call it.

lua_shared_dict aliases 10m;

server_name_resolver_by_lua_block {
    return ngx.shared.aliases:get(...)
}

server {
    listen 443 ssl;
    server_name tenant-1.example.com;

    # also selected for any host H where aliases[H] == "tenant-1.example.com",
    # so host names can be attached at run time without a reload
}

The directive:

• http context, one per configuration, block form only.
• The chunk receives the host name as ..., in lower case, and returns a server name. The answer is lower-cased and must be an exact server_name on the listening address of the connection.
• nil, any value that is not a string, an empty string, a Lua error, or an answer that is not such a name leaves the selection to wildcard names, regex names and the default server.
• The chunk is not run for a host that is an exact server_name.
• Over TLS it runs once for the SNI name. It runs again for each request on the connection, unless the SNI name selected a server and Host equals the SNI name. Tested here: the case where it does not run again. The other cases are tested at the callback level in HTTP: added a server name resolver hook for modules nginx/nginx#1972.

The implementation:

• config searches src/http/ngx_http_core_module.h for the text server_name_resolver and src/http/v2/ngx_http_v2.c for the text fc->reusable = 0;, and defines NGX_HTTP_LUA_HAVE_SERVER_NAME_RESOLVER when it finds both. On an nginx without either, the module builds, and the directive is rejected with an [emerg] message when the configuration is loaded.
• The chunk runs on the module's fake connection and fake request, in a new context, NGX_HTTP_LUA_CONTEXT_SERVER_NAME_RESOLVER (0x00080000). No existing API lists that context, so APIs guarded by ngx_http_lua_check_context() should fail with API disabled in the context of server_name_resolver_by_lua*. Tested: ngx.sleep is refused; ngx.log and ngx.shared.DICT:get work.
• The chunk has its own runner, modelled on ngx_http_lua_log_by_chunk(). ngx_http_lua_set_by_chunk() and the other shared runners are not changed.

c->reusable:

• The handler saves c->reusable, calls ngx_reusable_connection(c, 0) before ngx_http_lua_create_fake_connection(), which calls ngx_get_connection(), and calls ngx_reusable_connection(c, saved) when it returns, before the chunk runs. ngx_get_connection() then cannot reuse the connection being served.
• By code reading, the case that needs this is HTTP/3: a QUIC connection is reusable during its handshake, and the chunk runs inside that handshake's SNI callback. Observed: without the two calls, a QUIC handshake on a worker that is out of connections makes the worker exit on SIGSEGV in 12 of 14 runs; with them, in none (Testing). The mechanism is inferred from the log line worker_connections are not enough, reusing connections before the exit and from the calls removing the crash.
• On HTTP/2 the connection passed to the callback (c) is the stream's ngx_connection_t, a struct copy of the client connection. With the second nginx commit the copy has reusable = 0, and both calls do nothing. The client connection stays in the reusable queue while the handler runs. Not tested: HTTP/2 with the worker out of connections at that moment. By code reading nginx then sends GOAWAY on the client connection, because the stream is already counted in h2c->processing.
• Without the second nginx commit the two calls crash HTTP/2 (Testing). By code reading: after an incomplete frame the client connection is reusable and the copy has reusable = 1. The first call unlinks through the copied queue links, which removes the client connection from the queue while its flag stays set. The second inserts the copy, and the copy is freed with the HTTP/2 connection's pool while linked.
• The handler makes the two calls on every connection. Skipping them on a stream copy would remove the need for the second nginx commit. I did not do that: the only test I see for a copy is whether c lies inside ngx_cycle->connections.
• An earlier revision of this PR did not make the two calls. It passed the HTTP/2 test without the second nginx commit and had the HTTP/3 crash.
• No backtrace was taken for either SIGSEGV.

Other open points:

• There is no _file variant.
• No .t test covers the HTTP/2 or the HTTP/3 crash. Both clients are Python scripts, in the collapsed section below.
• APIs that are not guarded by context (ngx.var, ngx.ctx, ngx.req.*) are not tested on the fake request.
• t/199-server-name-resolver.t skips when the nginx binary contains the text requires nginx with the server_name_resolver hook, which is compiled in only when config did not find both nginx changes. If they reach an nginx release, a version check should replace that, and the two text searches in config.

Testing, on nginx 1.31.8 built by util/build.sh (this repository's CI configure flags, including the no-pool patch), OpenSSL 3.5.9. Pass means the assertions hold and no worker exits on a signal; nothing was run under valgrind or ASan.

• With both nginx commits: t/199-server-name-resolver.t passes (60 assertions). It covers an exact name (chunk not run), a mapped name, a mapped name that a wildcard also matches, an unmapped name under the wildcard, an unknown name, answers that are not an exact name or not a string, lower-casing, a shared dict lookup, a Lua error, ngx.sleep (refused), TLS with the SNI name mapped, and a duplicate directive. The position of regex names in the selection order is tested in HTTP: added a server name resolver hook for modules nginx/nginx#1972, not here.
• With both nginx commits, on the current code: t/001-set.t, t/086-init-by.t, t/124-init-worker.t, t/162-exit-worker.t, t/166-ssl-client-hello.t and t/062-count.t pass. I ran these 6 of the existing test files, not the whole suite.
• HTTP/2, cleartext, h2_split_headers.py below, worker_connections 32. Sequence: open an idle HTTP/1 connection; send a HEADERS frame with a mapped :authority in two writes, to a location that sleeps one second; meanwhile send a request on the HTTP/1 connection; after the HTTP/2 response, open 40 idle connections, and the worker logs worker_connections are not enough, reusing connections. With both nginx commits: the response comes from the mapped server and no worker exits on a signal (2 runs). With the first nginx commit only: the worker exits on SIGSEGV (2 of 2 runs). With the frame sent in one write: no crash either way. The HTTP/1 steps are in the sequence by code reading; I did not run it without them.
• HTTP/3, client aioquic, quic_draining_worker.py below, worker_connections 16, both nginx commits. Sequence: hold N half-sent HTTP/1 requests; send an HTTP/3 request for a mapped name; release the held requests; send a second HTTP/3 request. Without the two calls, crashes per runs by N: 8: 2/2; 9: 3/3; 10: 3/3; 11: 1/2; 12: 1/2; 13: 1/1; 14: 1/1. With the two calls: none in the same 14 runs; the first HTTP/3 request times out at the client while nginx logs worker_connections are not enough while handling frames, and the second returns 200 from the mapped server. With N of 0, 6 or 7 nothing crashes in one run each, with or without the calls.
• With the first nginx commit only: the module builds, t/199-server-name-resolver.t is skipped, and nginx -t rejects the directive with server_name_resolver_by_lua_block requires nginx with the server_name_resolver hook in ngx_http_core_main_conf_t and with the reusable flag cleared in HTTP/2 stream connections. An nginx with neither change takes the same code path; I ran that build before the message was extended.
• Not tested: the rest of the suite, HTTP/3 beyond the case above, HTTP/2 over TLS for the crash case, other nginx versions, a build without the no-pool patch.

HTTP/2 and HTTP/3 clients and their nginx configurations

python3 h2_split_headers.py sends the frame in two writes; python3 h2_split_headers.py whole sends it in one.

#!/usr/bin/env python3
"""Raw HTTP/2 client: HEADERS frame sent in two writes, with an :authority
the server_name resolver maps, then idle connections until the worker reuses
connections.  Usage: h2_split_headers.py [whole]"""

import socket
import struct
import sys
import time

PORT = 18357

STREAM_SLEEP = 1

# More than worker_connections (32): ngx_get_connection() drains the queue
# once fewer than 32 / 16 connections are free.
IDLE_CONNECTIONS = 40

DRAIN = b'worker_connections are not enough, reusing connections'


def _frame(kind, flags, stream, payload):
    return (struct.pack('>I', len(payload))[1:] + bytes([kind, flags])
            + struct.pack('>I', stream) + payload)


def _literal(name, value):
    """HPACK literal header field without indexing, new name, no Huffman."""
    name, value = name.encode(), value.encode()
    assert len(name) < 127 and len(value) < 127
    return b'\x00' + bytes([len(name)]) + name + bytes([len(value)]) + value


def _headers_frame():
    block = (_literal(':method', 'GET') + _literal(':scheme', 'http')
             + _literal(':path', '/?sleep=%d' % STREAM_SLEEP)
             + _literal(':authority', 'alias.test'))
    return _frame(0x1, 0x4 | 0x1, 1, block)


def _read_until(sock, marker, timeout):
    sock.settimeout(timeout)
    data = b''
    try:
        while marker not in data:
            chunk = sock.recv(65536)
            if not chunk:
                break
            data += chunk
    except OSError:
        pass
    return data








def _drive(split_headers):
    socks = []
    try:
        # an idle HTTP/1 connection, in the queue before the HTTP/2 one
        idle_http1 = socket.create_connection(('127.0.0.1', PORT))
        socks.append(idle_http1)
        time.sleep(0.2)

        h2 = socket.create_connection(('127.0.0.1', PORT))
        socks.append(h2)
        h2.sendall(b'PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n'
                   + _frame(0x4, 0, 0, b''))
        # SETTINGS processed, no stream: the connection goes idle
        time.sleep(0.3)

        headers = _headers_frame()
        if split_headers:
            h2.sendall(headers[:5])
            time.sleep(0.3)
            h2.sendall(headers[5:])
        else:
            h2.sendall(headers)

        # the HTTP/1 connection leaves and re-enters the queue while the
        # stream sleeps
        time.sleep(0.3)
        idle_http1.sendall(b'GET / HTTP/1.1\r\nHost: default.invalid\r\n\r\n')
        http1_reply = _read_until(idle_http1, b'server_name=', 2)

        h2_reply = _read_until(h2, b'server_name=', STREAM_SLEEP + 5)

        # the stream has ended; idle connections exhaust worker_connections
        time.sleep(0.3)
        for _ in range(IDLE_CONNECTIONS):
            try:
                socks.append(socket.create_connection(('127.0.0.1', PORT),
                                                      timeout=1))
            except OSError:
                break
        time.sleep(1)

        return http1_reply, h2_reply
    finally:
        for s in socks:
            try:
                s.close()
            except OSError:
                pass


if __name__ == '__main__':
    http1_reply, h2_reply = _drive(len(sys.argv) < 2)
    print('HTTP/1 answered by default.invalid:',
          b'server_name=default.invalid' in http1_reply)
    print('HTTP/2 answered by named.test:',
          b'server_name=named.test' in h2_reply)

daemon off;
user root;
master_process on;
worker_processes 1;

pid       /tmp/h2_split_headers.pid;
error_log /tmp/h2_split_headers.error info;

events {
    worker_connections 32;
}

http {
    access_log off;
    http2 on;

    lua_package_path '/path/to/lua-resty-core/lib/?.lua;/path/to/lua-resty-lrucache/lib/?.lua;;';

    client_body_temp_path /tmp/h2_split_headers_tmp;
    proxy_temp_path       /tmp/h2_split_headers_tmp;
    fastcgi_temp_path     /tmp/h2_split_headers_tmp;
    uwsgi_temp_path       /tmp/h2_split_headers_tmp;
    scgi_temp_path        /tmp/h2_split_headers_tmp;

    server_name_resolver_by_lua_block {
        local host = ...
        if host == "alias.test" then
            return "named.test"
        end
    }

    server {
        listen 127.0.0.1:18357 default_server;
        server_name default.invalid;
        return 200 "server_name=$server_name\n";
    }

    server {
        listen 127.0.0.1:18357;
        server_name named.test;

        location / {
            content_by_lua_block {
                ngx.sleep(tonumber(ngx.var.arg_sleep) or 0)
                ngx.say("server_name=", ngx.var.server_name)
            }
        }
    }
}

quic_draining_worker.py; it needs aioquic and takes the counts of held requests as arguments:

#!/usr/bin/env python3
"""A QUIC handshake on a worker that is almost out of connections, with a
server name resolver that maps the SNI name.

A new QUIC connection stays reusable until its handshake completes, and the
resolver takes a worker connection inside the SNI callback.  Near
worker_connections, taking a connection makes nginx reuse connections, and
with every other connection busy the handshaking one is the only candidate.

For each count of held half-sent HTTP/1 requests the script starts nginx,
holds the requests, sends one HTTP/3 request for the mapped name (it may
fail for lack of connections), releases the held requests, sends a second
HTTP/3 request, and reports whether a worker exited on a signal.

Usage:
    quic_draining_worker.py --nginx /path/to/nginx \\
        --conf quic_draining_worker.conf --workdir /tmp/quic-draining-worker \\
        --lua-resty-core /path/to/lua-resty-core \\
        --lua-resty-lrucache /path/to/lua-resty-lrucache \\
        --cert /path/to/http3.crt --key /path/to/http3.key [held ...]

Needs aioquic.
"""

import argparse
import asyncio
import os
import shutil
import signal
import socket
import ssl
import subprocess
import time

from aioquic.asyncio import connect
from aioquic.asyncio.protocol import QuicConnectionProtocol
from aioquic.h3.connection import H3Connection
from aioquic.h3.events import HeadersReceived
from aioquic.quic.configuration import QuicConfiguration

HTTP_PORT = 18882
QUIC_PORT = 18442
ALIAS = 'alias.test'


class _Proto(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.h3 = None
        self.headers = asyncio.get_event_loop().create_future()

    def quic_event_received(self, event):
        if self.h3 is None:
            self.h3 = H3Connection(self._quic)
        for ev in self.h3.handle_event(event):
            if isinstance(ev, HeadersReceived) and not self.headers.done():
                self.headers.set_result(ev.headers)


def get_http3(host):
    async def run():
        config = QuicConfiguration(is_client=True, alpn_protocols=['h3'])
        config.server_name = host
        config.verify_mode = ssl.CERT_NONE
        async with connect('127.0.0.1', QUIC_PORT, configuration=config,
                           create_protocol=_Proto) as proto:
            if proto.h3 is None:
                proto.h3 = H3Connection(proto._quic)
            sid = proto._quic.get_next_available_stream_id()
            proto.h3.send_headers(sid, [
                (b':method', b'GET'), (b':scheme', b'https'),
                (b':authority', host.encode()), (b':path', b'/'),
            ], end_stream=True)
            proto.transmit()
            return await asyncio.wait_for(proto.headers, 5)

    try:
        headers = asyncio.run(asyncio.wait_for(run(), 10))
    except Exception as e:
        return 'failed (%s)' % type(e).__name__
    hdrs = {k.decode().lower(): v.decode() for k, v in headers}
    return '%s x-served=%s' % (hdrs.get(':status'), hdrs.get('x-served'))


def hold_half_sent_requests(n):
    """n connections that sent part of a request: each holds a worker
    connection that is not reusable."""
    socks = []
    for _ in range(n):
        s = socket.create_connection(('127.0.0.1', HTTP_PORT), timeout=5)
        s.sendall(b'GET / HTTP/1.1\r\nHost: half')
        socks.append(s)
    time.sleep(0.2)
    return socks


def wait_port(port, timeout=5.0):
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            with socket.create_connection(('127.0.0.1', port), timeout=0.2):
                return True
        except OSError:
            time.sleep(0.05)
    return False


def render(args):
    conf = open(args.conf).read()
    conf = conf.replace('/path/to/lua-resty-core', args.lua_resty_core)
    conf = conf.replace('/path/to/lua-resty-lrucache', args.lua_resty_lrucache)
    conf = conf.replace('/path/to/http3.crt', args.cert)
    conf = conf.replace('/path/to/http3.key', args.key)
    conf = conf.replace('/path/to/workdir', args.workdir)
    path = os.path.join(args.workdir, 'nginx.conf')
    with open(path, 'w') as f:
        f.write(conf)
    return path


def run_one(args, conf, held):
    err = os.path.join(args.workdir, 'error.log')
    open(err, 'w').close()
    nginx = subprocess.Popen([args.nginx, '-c', conf],
                             stdout=subprocess.DEVNULL,
                             stderr=subprocess.DEVNULL)
    try:
        if not wait_port(HTTP_PORT):
            return 'held=%d: nginx did not start' % held

        socks = hold_half_sent_requests(held)
        try:
            first = get_http3(ALIAS)
        finally:
            for s in socks:
                s.close()
        time.sleep(0.3)
        second = get_http3(ALIAS)
        time.sleep(0.2)
    finally:
        nginx.send_signal(signal.SIGTERM)
        try:
            nginx.wait(timeout=10)
        except subprocess.TimeoutExpired:
            nginx.kill()
            nginx.wait()

    with open(err, errors='replace') as f:
        lines = f.readlines()
    signals = [l.split(': ', 1)[-1].strip() for l in lines
               if 'exited on signal' in l]
    reuse = sum('reusing connections' in l for l in lines)
    return ('held=%d: first=%s; second=%s; reusing-connections lines=%d; %s'
            % (held, first, second, reuse,
               ('WORKER DIED: ' + '; '.join(signals)) if signals
               else 'no worker exited on a signal'))


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--nginx', required=True)
    p.add_argument('--conf', required=True)
    p.add_argument('--workdir', required=True)
    p.add_argument('--lua-resty-core', required=True)
    p.add_argument('--lua-resty-lrucache', required=True)
    p.add_argument('--cert', required=True)
    p.add_argument('--key', required=True)
    p.add_argument('held', nargs='*', type=int, default=list(range(8, 13)))
    args = p.parse_args()

    shutil.rmtree(args.workdir, ignore_errors=True)
    os.makedirs(os.path.join(args.workdir, 'tmp'))
    conf = render(args)

    for held in args.held:
        print(run_one(args, conf, held), flush=True)


if __name__ == '__main__':
    main()

quic_draining_worker.conf:

# A QUIC handshake whose SNI name the server name resolver maps, on a worker
# that is almost out of connections.  The paths are placeholders;
# quic_draining_worker.py substitutes them.

daemon off;
master_process on;
worker_processes 1;
user root;

pid       /path/to/workdir/nginx.pid;
error_log /path/to/workdir/error.log info;

events {
    worker_connections 16;
}

http {
    access_log off;

    lua_package_path '/path/to/lua-resty-core/lib/?.lua;/path/to/lua-resty-lrucache/lib/?.lua;;';

    client_body_temp_path /path/to/workdir/tmp;
    proxy_temp_path       /path/to/workdir/tmp;
    fastcgi_temp_path     /path/to/workdir/tmp;
    uwsgi_temp_path       /path/to/workdir/tmp;
    scgi_temp_path        /path/to/workdir/tmp;

    ssl_certificate     /path/to/http3.crt;
    ssl_certificate_key /path/to/http3.key;

    server_name_resolver_by_lua_block {
        local host = ...
        if host == "alias.test" then
            return "named.test"
        end
    }

    # four listening sockets: plain HTTP, TLS, QUIC and one spare port

    server {
        listen 127.0.0.1:18882 default_server;
        listen 127.0.0.1:18442 ssl default_server;
        listen 127.0.0.1:18442 quic reuseport default_server;
        listen 127.0.0.1:18883;
        server_name default.invalid;
        add_header X-Served default always;
        return 200 "served-by=default\n";
    }

    server {
        listen 127.0.0.1:18882;
        listen 127.0.0.1:18442 ssl;
        listen 127.0.0.1:18442 quic;
        server_name named.test;
        add_header X-Served named always;
        return 200 "served-by=named\n";
    }
}

I hereby granted the copyright of the changes in this pull request
to the authors of this lua-nginx-module project.