Repository navigation
Feat/bwrap private netns - #37
Thomas A Caswell (tacaswell) wants to merge 36 commits into
Conversation
ce250d8 to
8823c86
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Blocking writes to a stalled logging subprocess can prevent pasta logs from being drained and stall the sandbox’s network.
3 open issues
What changed in this PR
This PR adds private-network controls to the bw* sandbox wrappers, separating agent services from host loopback services while retaining outbound access and explicit port forwarding.
Changes:
- Integrates private networking, port publication, host-port forwarding, and a shared-network escape hatch across all five wrappers.
- Adds network-log processing, a DNS relay, and sandboxed OpenCode 2 web serving with password authentication and pairing links.
- Adds tests, fake network backends, documentation, and CI integration.
| File | Description |
|---|---|
tests/test_serve_timeout.py |
Tests stalled pairing-request timeout handling. |
tests/test_netlog.py |
Tests log condensation, DNS parsing, and relay behavior. |
tests/fakes/pasta-bin/pasta |
Simulates pasta networking and flow logs. |
tests/fakes/bin/unshare |
Simulates namespace setup and failures. |
tests/fakes/bin/systemd-cat |
Captures journal output for testing. |
tests/fakes/bin/slirp4netns |
Simulates slirp setup and port forwarding. |
tests/fakes/bin/logger |
Captures syslog output for testing. |
tests/fakes/_nsproxy.py |
Provides namespace socket and forwarding helpers. |
tests/fake-net-run |
Runs wrappers with fake network backends. |
tests/data/pasta-debug.log |
Supplies a realistic log-parser fixture. |
README.md |
Documents network dependencies and common options. |
pixi.toml |
Adds actionlint and a unit-test task. |
pixi.lock |
Locks actionlint packages across supported platforms. |
lib/bwrap_netlog.py |
Implements flow-log condensation and DNS logging. |
docs/bw-wrappers.md |
Documents networking, logging, descriptor relay, and web serving. |
bin/bwopencode2 |
Adds network options, sandboxed web serving, and prek cache mounting. |
bin/bwopencode |
Integrates network controls and prek cache mounting. |
bin/bwcopilot |
Integrates networking, descriptor relay, and prek cache mounting. |
bin/bwcodex |
Integrates networking and descriptor-based configuration copies. |
bin/bwclaude |
Integrates networking and descriptor-based credential copies. |
AI_DISCLOSURE.md |
Updates model disclosure and its date. |
.pre-commit-config.yaml |
Runs actionlint through the pixi environment. |
.github/workflows/lint.yml |
Adds unit tests to CI. |
.codespell.ignore |
Allows the passt package name. |
Review effort: Balanced
| out.write((text + "\n").encode("utf-8", "replace")) | ||
| out.flush() |
| while True: | ||
| try: | ||
| hdr = self._recv_exact(conn, 2) | ||
| query = self._recv_exact(conn, struct.unpack("!H", hdr)[0]) | ||
| # DNS servers may close after one response. A fresh | ||
| # connection per query also avoids replaying a request | ||
| # after an ambiguous upstream failure. | ||
| with socket.create_connection(self.upstream, timeout=5) as up: | ||
| up.settimeout(10) | ||
| up.sendall(hdr + query) | ||
| rhdr = self._recv_exact(up, 2) | ||
| response = self._recv_exact(up, struct.unpack("!H", rhdr)[0]) | ||
| self.log(query, response, "tcp") | ||
| conn.sendall(rhdr + response) | ||
| except OSError: | ||
| break |
| esac | ||
| path="${here}/fakes/bin:${PATH}" | ||
| [[ "${backend}" != pasta ]] || path="${here}/fakes/pasta-bin:${path}" | ||
| exec bwrap --dev-bind / / --dev /dev --dev-bind /dev/null /dev/net/tun \ |
Runs opencode2 serve inside the sandbox on 127.0.0.1, so tools stay sandboxed, with one server per invocation on the first free port of a fixed range (24700-24709 by default) so one ssh_config block covers every session. The password is random for each session unless BWOPENCODE2_SERVE_PASSWORD is set, and reaches the server only as OPENCODE_PASSWORD. Ports are claimed with a host-side file so two launchers cannot pick the same one. --serve-ssh-config prints the matching ssh_config block. Assisted-by: opencode:claude-sonnet-5-5
The web UI opens on a page that asks for a pairing link or the password, and opencode pair points at the shared service, which the wrapper refuses. Inside the sandbox a small wrapper now starts the server, waits for it, asks it for a pairing code and prints the link, so the browser can sign in without typing the password. The password reaches curl on stdin, never on a command line. Use password still works after the link expires. Assisted-by: opencode:claude-sonnet-5-5
pasta, which will start bwrap in a private network namespace, closes every inherited descriptor above 2, so the --file and --ro-bind-data descriptors opened by the launcher would never reach bwrap. The launcher now only reserves descriptor numbers and records their source (a host file, or generated text carried in the environment); a fixed bash relay opens them at those numbers and execs bwrap. bwcodex no longer stages temporary copies of its config and daemon settings, and --dry-run shows real descriptor numbers instead of placeholders. Assisted-by: opencode:claude-opus-5-5
pasta (package passt) now starts each sandbox in its own network namespace. Outbound connections still leave from the host's address, but the host's loopback services are unreachable and the sandbox's listeners are invisible to the host. There is no fallback to the host network: without a working pasta the launchers exit 3. Crossing the boundary is a per-port, per-launch opt-in: - --publish PORT[:SANDBOXPORT] binds a per-user, per-session host address 127.H1.H2.S (--publish-addr overrides it, e.g. for OAuth); - --host-port PORT[:SANDBOXPORT] makes the host's 127.0.0.1:PORT reachable as host.internal, with a deny list (CUPS, Docker, X11, Tailscale, the OpenCode 2 service); - --net-log FILE records every flow. pasta maps the caller to uid 0 in its user namespace, so bwrap now runs the agent as the real uid in a nested one (--unshare-user --uid --gid); without it the agent would be root with a full capability set there. bwrap also restores the real hostname that pasta replaces. /etc/resolv.conf and /etc/hosts are generated, the context file gains a Network section, and BWRAP_NET_* describe the forwards. This breaks --serve, will fix later. Assisted-by: opencode:claude-opus-5-5
Document the pasta requirement, --publish / --publish-addr / --host-port / --net-log, the per-user publish addresses and what they do not guarantee, host.internal and the loopback-trust caveat, and the descriptor relay. Assisted-by: opencode:claude-opus-5-5
passt is in RHEL 9 but not RHEL 8. Without a working pasta the sandbox now shares the host network as before, with a warning, instead of refusing to start. --local-net chooses the host network explicitly and silences the warning. The forwarding options still need pasta: with --local-net they are an error, and without pasta they stop the launch (exit 3). BWRAP_NET_MODE tells the sandbox which mode it is in, and the context file describes both. Assisted-by: opencode:claude-opus-5-5
Use bwrap's own term for sharing the host network namespace. Assisted-by: opencode:claude-opus-5-5
Without a usable private network namespace the launchers now exit 3 naming the reason, instead of warning and sharing the host network. Only --share-net shares it. BWRAP_NET_BACKEND (pasta or none) replaces BWRAP_NET_MODE, ready for a second backend. Assisted-by: opencode:claude-opus-5-5
RHEL 8 has slirp4netns but no passt. When pasta is not usable, the launchers now use slirp4netns: util-linux unshare gives the launcher a user namespace that owns a new network namespace, bwrap runs the sandbox in it as the real user, and a background helper attaches slirp4netns (--disable-host-loopback, 192.168.254.0/24, built-in DNS) before releasing the sandbox through --block-fd. The launcher still execs in the foreground; slirp4netns exits with bwrap (--sync-fd). On a setup failure the helper kills the blocked sandbox before the tool starts. unshare, not bwrap --unshare-net, makes the namespaces: bwrap's devpts setup nests the sandbox in a second user namespace, from which slirp4netns would have no rights over the network namespace. --publish works through the slirp4netns API socket; --host-port and --net-log need pasta. --net-backend restricts the choice to one backend. Assisted-by: opencode:claude-opus-5-5
Document the selection order (pasta, slirp4netns, else stop unless --share-net), --net-backend, what each backend supports, and how the slirp4netns launch works. Assisted-by: opencode:claude-opus-5-5
--serve now adds its port to --publish: the server binds 0.0.0.0 inside the private namespace and is reached on the host at the user's publish address 127.H1.H2.S (normally .1), with both backends. With --share-net it binds 127.H1.H2.1 itself. The port is chosen free on that address, the printed ssh -L command and pairing links name it, and --serve-ssh-config forwards to it. net_publish_prefix exposes the per-user prefix. Assisted-by: opencode:claude-opus-5-5
The slirp4netns pipes and API socket were in a mktemp directory under XDG_RUNTIME_DIR or TMPDIR, whose parent was not checked: another user able to write it could swap the directory during the launch. They now go in a per-launch session directory under $XDG_RUNTIME_DIR/bwrap-net, else /tmp/bwrap-net-UID, used only when no other user can change its parent and it is the user's own 0700 directory. TMPDIR is never used. Both locations are blocked paths, so no sandbox of the user can mount them and reach another sandbox's API socket. Assisted-by: opencode:claude-opus-5-5
The slirp4netns helper kept polling after the launcher had gone (for example when unshare failed) and then sent SIGKILL to the saved pid, which by then could belong to another process. It now exits quietly as soon as its parent is no longer the launcher, and before killing it checks that the launcher is still its parent and that the sandbox pid is the launcher's child. Assisted-by: opencode:claude-opus-5-5
The publish address came from one ss snapshot, and the backend binds it only after exec, so two launches at the same moment could pick the same address and port. The choice is now made under flock in the runtime directory, and each launch leaves a claim (launcher pid and start time, host ports) that later launches respect; claims whose process is gone are removed. --serve port claims also record the start time, so a reused pid does not keep them alive. Assisted-by: opencode:claude-opus-5-5
The header still said that without pasta the sandbox shares the host network with a warning; the launch tries slirp4netns and otherwise stops unless --share-net is given. Assisted-by: opencode:claude-opus-5-5
pasta writes one log file, and the network log becomes an always-on journal record in the next commit; a file the user names is easier to edit, lose, or leave where the agent can write it. Assisted-by: opencode:claude-opus-5-5
…m log Every launch writes start and end records (tag bwrap-net, a session id also given to the sandbox as BWRAP_SESSION) with systemd-cat, else logger; with neither it warns and runs without the log. There is no option to turn it off. With pasta every flow is logged too: pasta runs with --debug and a FIFO as --log-file (huge --log-size, as its rotation fails on a FIFO) and as its stderr, and a supervisor outside the sandbox copies each line to the log, holding the FIFO open so pasta never blocks. The descriptor relay prints a marker and reopens the user's stderr, so pasta's setup lines go to the log, and if pasta fails before the marker its lines are shown on the terminal. Assisted-by: opencode:claude-opus-5-5
tests/fake-net-run runs a launcher against fake pasta, slirp4netns, unshare, systemd-cat and logger, where /dev/net/tun is missing (as in a development sandbox). The fakes really create or join the namespaces and forward TCP, so --publish, --serve and the network log can be tested end to end; failure modes are read from $FAKE_NET_DIR. Assisted-by: opencode:claude-opus-5-5
A --serve session could move to another session index when one of its ports was busy on 127.H1.H2.1, and the --serve-ssh-config alias then did not reach it. --serve now fixes the session's publish address to .1: its port is chosen free and unclaimed there, a clash with another --publish port stops the launch with the reason, and --publish-addr is refused with --serve. The fake pasta now handles comma-separated port lists and never starts the command when its setup failed. Assisted-by: opencode:claude-opus-5-5
pasta's -T takes no listening address ("Listening address not allowed
for -T"): it always listens on the namespace's loopback. So
--host-port PORT[:SANDBOXPORT] now passes -T SANDBOXPORT:PORT and the
agent reaches the host service at 127.0.0.1:SANDBOXPORT; host.internal,
127.0.42.1 and BWRAP_NET_HOST_ADDR are gone. That sandbox port is held
by pasta, so a --publish to it is refused, and the context file says
the agent cannot serve on it.
The fake pasta now rejects an address on -T/-U and listens on the
namespace's loopback, as the real one does; it had accepted the old
form, which is why the tests passed.
Assisted-by: opencode:claude-opus-5-5
pasta's debug log has about 34 lines per connection (state changes, timers, ACK bookkeeping) plus a doubled setup dump, and only addresses. lib/bwrap_netlog.py condense now turns it into one logfmt line when a connection opens and one when it closes (with its duration), a fail line for connections that never became active, and pasta's warnings and errors; 5000 lines of a real session became 280. To name destinations, bwrap_netlog.py dns runs as a DNS relay on 127.53.0.1:53 in the sandbox's network namespace, started by the descriptor relay before bwrap (it waits until the relay listens, at most 0.25 s) and dropping its capabilities after binding. It logs one dns line per lookup and forwards to pasta's DNS address, which resolv.conf lists second so lookups go on working if the relay is down. The condenser adds name= to connections from the answers. Both need Python 3.6+ (platform-python on RHEL 8); without it the log is unfiltered as before. Assisted-by: opencode:claude-opus-5-5
Unit tests (pixi run -e dev test) against part of a real pasta debug log with documentation addresses, and a DNS relay against a local fake upstream. The fake pasta now writes pasta's debug format for its forwards and answers DNS on --dns-forward, so the launchers' log can be checked end to end with tests/fake-net-run. Assisted-by: opencode:claude-opus-5-5
On a real session the first IPv6 connections were logged without a name: the client connected as soon as it had the answer, before the relay had written its dns line. The relay now logs first. CNAMEs go to cname= so answers= holds addresses only. Assisted-by: opencode:claude-opus-5-5
…rk log name= did not say what it was. Connection and dns lines now use domain= (the domain the relay looked up), which is also what an allow-list is written in. Assisted-by: opencode:claude-opus-5-5
| if event == "open": | ||
| parts.append("src=%s" % _hostport(*f["src"])) | ||
| parts.append("dst=%s" % _hostport(*f["dst"])) | ||
| if f["tdst"][0] != f["dst"][0] and f["tdst"][0] not in ("0.0.0.0", "::"): |
| exec bwrap --dev-bind / / --dev /dev --dev-bind /dev/null /dev/net/tun \ | ||
| --ro-bind /dev/null /usr/bin/passt \ | ||
| "${etc[@]}" \ | ||
| ${logsock[@]+"${logsock[@]}"} \ | ||
| --chdir "${PWD}" \ | ||
| -- env PATH="${path}" "$@" |
|
the slirp4nets stuff is all broken, still having the 🦜 work on it. |
|
Thomas A Caswell (@tacaswell) two things from running 1. The server password is in the environment of the agent's tool shells.
What I checked, so this is not a guess:
What I did not check: whether the Why it matters beyond the shell: blanking the variable only in tool shells would not be a control, because of the
Which of these do you want for 2. Who owns the published socket? On a real launch with pasta, Good news from the same run: the private namespace works. |
|
Thomas A Caswell (@tacaswell) a follow-up on the database guard under pasta. Copilot's review of #43 said, in a summary line with no detail, that pasta startup "releases the database session lock". I checked it on a real launch. What I found
So the guard works today, but only because the log helper happens to hold the lock. What I think follows, from the code and not tested
In either case a second host or OpenCode build could pass the guard while the session runs, and the database is on NFS. It affects Possible fixes, your call
Is there a reason the lock is not passed through the relay, which already carries the other descriptors? A test with |
On Fedora and RHEL pasta runs as pasta_t, which passt's policy lets only append to user_tmp_t FIFOs. pasta opens --log-file O_RDWR (to rotate it), so the open of the netlog FIFO is denied and pasta exits before the sandbox starts; the launcher returned to the prompt with nothing on the terminal. With --debug and no --log-file pasta writes the same lines to its stderr, and an inherited append-only descriptor is allowed. Open the launcher's stderr on the FIFO with >> instead of <>, holding a temporary read-write descriptor only until the open returns: the kernel replaces any descriptor the new domain may not use with /dev/null at the exec, so a read-write one must not survive into pasta. Drop --log-size, which only mattered for a log file. The stderr format has no header line and no level column, so the condenser's version and level records go away; the pasta version moves into the start record. Fixtures and the fake pasta use the stderr format. Verified with pasta 2026_10_02 on Fedora 45 enforcing. Assisted-by: opencode:claude-fable-5-1
The bash supervisor read the FIFO line by line and piped each line into the Python condenser; the condenser now opens the FIFO, watches the launcher pid, prints pasta's early lines if the start fails, bridges the user's stderr when it is a pipe, writes the end record and removes the session directory. One process instead of a subshell, a process substitution and a pipe, and no bash read loop with sub-second timeouts. The launcher waits for the condenser's "ready" on a pipe before it opens its stderr on the FIFO, so nothing pasta writes is lost. The bash supervisor stays as the fallback when Python is missing. Assisted-by: opencode:claude-fable-5-1
RHEL 8 ships bubblewrap 0.4.0, which cannot start in a user namespace that maps only uid 0: drop_privs() tests geteuid() == 0 instead of is_privileged (fixed in 0.4.1), calls setuid() on the real uid, which that namespace does not map, and dies with "unable to drop root uid: Invalid argument". unshare --map-root-user built exactly that namespace, so every slirp4netns launch failed. The launcher now unshares and writes the id maps itself (_NET_NS_IDENTITY, Python standard library), mapping the real uid and gid to themselves; RHEL 8's util-linux (2.32) has no --map-user. The backend needs python3 or platform-python instead of unshare. slirp4netns --enable-sandbox does setegid(0) after joining the namespace, so it is no longer passed; --enable-seccomp still is. An unprivileged process may write a single map line, so the namespace maps uid 0 or the real uid, not both. pasta also maps the caller to uid 0 alone: refuse it, with the reason, when bwrap is older than 0.4.1, instead of failing at exec. The fake slirp4netns execs ip(8) after joining the namespace, and execve clears capabilities under a non-root map; raise them into the ambient set, as the real one configures the interface in-process. Remove the fake unshare, which nothing runs any more. Assisted-by: opencode:claude-opus-5-5
slirp4netns has no flow log, so its sessions recorded only a start and an end. The DNS relay is not tied to pasta: it runs in the sandbox's network namespace and forwards to the backend's own resolver (192.168.254.3 here). With it, the lookups are the record of what a slirp4netns sandbox reached for. Under the identity map the relay cannot bind port 53 for itself: execve clears capabilities unless the process is uid 0 in the namespace, and the relay starts two execs after the unshare. The mapper (--dns) raises loopback and binds the relay's UDP and TCP sockets before it execs, on descriptors 88 and 89, and the relay adopts them (DnsRelay.adopt). Under pasta it still binds for itself. The netlog FIFO and its condenser now run whenever flows or lookups are logged (_NET_LOG_PIPE); the start record says dns=logged or dns=unavailable, and a failed start is labelled with the backend's name instead of always pasta. Fixes needed along the way: - the spawn relay entry grew from 6 to 8 words; _bwrap_fd_data_numbers must step the same, or it skips data entries and resolv.conf and hosts reach the sandbox empty. tests/test_fd_relay.py runs one list through both. - slirp4netns kept its pipes in the session directory and removed it, which now also holds the log's FIFO; it uses a slirp/ subdirectory when the log owns the directory. - when the condenser does not start, undo the relay's hand-over (its marker and the DNS relay's lines would print on the terminal) and remove the session directory, which nothing else would. Assisted-by: opencode:claude-opus-5-5
Nothing here ran pasta with bwrap 0.4.0: the RHEL 8 hosts tested have bwrap 0.4.0 and no pasta. The check refused pasta when bwrap was older than 0.4.1 on the strength of reading the source, so it goes, and _net_try_pasta is upstream's again. The docs say the combination is not tested. The comments and docs on the identity map now hold only what was seen or read in a named source. Gone: the util-linux version numbers, the comparison with rootless podman, what restoring --enable-sandbox would take, and the claim that nothing in the sandbox runs as uid 0 (the sandbox's own uid_map, 409598 0 1, maps it onto uid 0 of the namespace above it). Assisted-by: opencode:claude-opus-5-5
Describe the syslog line format, that only the start and end records name the user, and how to find one user's DNS lookups through their session IDs. State which sessions have no DNS or flow records. Assisted-by: opencode:claude-sonnet-5-5
--perms needs bwrap 0.5.0 and RHEL 8 ships 0.4.0. The daemon directory and settings file live in the sandbox's private tmpfs, so their modes guard nothing.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can release the database guard while OpenCode is running and allow stalled logging to freeze sandbox networking.
7 open issues
Use Bash to read file descriptors above 9 in descriptor-relay tests · New Keep draining logs when the logging sink stalls Forward piped stderr bytes immediately without decoding them · New Preserve the selected FAKE_NET_DIR for backend fakes Compare full endpoints when recording port translations Preserve custom fake-network settings after environment scrubbing Log failed complete TCP DNS queries
Review effort: Balanced
| """Copy PATH (a FIFO the relay opens as the sandbox's stderr when the | ||
| user's is a pipe) to our stderr until STOP is set and it is drained.""" | ||
| reader = FifoReader(path, stop.is_set) | ||
| for line in reader.lines(): | ||
| try: | ||
| sys.stderr.write(line + "\n") | ||
| sys.stderr.flush() | ||
| except (OSError, ValueError): | ||
| pass |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Blocking logger writes can stall FIFO draining and freeze sandbox networking, while line-buffered stderr forwarding hides prompts and alters the original byte stream.
6 open issues
Keep draining logs when the logging sink stalls Forward piped stderr bytes immediately without decoding them Preserve the selected FAKE_NET_DIR for backend fakes Compare full endpoints when recording port translations Preserve custom fake-network settings after environment scrubbing Log failed complete TCP DNS queries
1 resolved since last review
Review effort: Balanced
|
Found while reviewing #43 (Copilot raised it there). Not caused by this PR: it is on What. The per-tool state directories are checked with Reproduced with
With Where. Severity. Low for the fixed paths: the symlink has to be made on the host by the user, and the agent cannot replace a mount point from inside. Worth closing because the rule is meant to be "nothing that contains a blocked path is bound read-write". Suggested fix. In each launcher that binds the directory whole, use Assisted-by: opencode:claude-sonnet-5-5 |
Only the start and end records carried user=, so a dns, open, close or fail line could be attributed only by looking up its session. Every record now begins session=ID user=NAME, written by the condenser (--user) and by the fallback supervisor used without Python. The name is the account of the launcher process (id -un), not $USER, which the caller can set, and characters that could split a logfmt field are replaced. The start and end records use the same value. Assisted-by: opencode:claude-sonnet-5-5
There was a problem hiding this comment.
🔵 Needs a closer look
Blocking writes to a stalled logging subprocess can stop FIFO draining and block networking and DNS replies.
6 open issues
Keep draining logs when the logging sink stalls Forward piped stderr bytes immediately without decoding them Preserve the selected FAKE_NET_DIR for backend fakes Compare full endpoints when recording port translations Preserve custom fake-network settings after environment scrubbing Log failed complete TCP DNS queries
Review effort: Balanced
|
Pushed one commit to this branch: Until now only the
Assisted-by: opencode:claude-sonnet-5-5 |


Summary
This is a draft of starting to add network controls to the sandbox. Initially we took the position "the agents need the network, guess we have to allow it", but there are a number of steps we can take short of full network cut off.
Description
Taken together these will keep LSP or opencode2's servers from talking to each other from inside of the sandbox but allows for e.g. letting the agent run a dev server you poke from a browser.
Testing
Not much yet, have not tried running with this as my local wrappers in anger yet.
Checklist
pixi run -e dev lintlocally (or wait for CI) — see.github/workflows/lint.ymlrunansibledeployment)