Skip to content

Windows: boot the avocado-vm, run the engine client, provision - #404

Open
mobileoverlord wants to merge 8 commits into
jschneck/engine-apifrom
jschneck/windows-engine
Open

mobileoverlord wants to merge 8 commits into
jschneck/engine-apifrom
jschneck/windows-engine

Conversation

@mobileoverlord

Copy link
Copy Markdown
Contributor

Stacked on jschneck/engine-api.

What this does

  • VM on Windows: QEMU under WHPX, QMP/QGA/control chardevs on loopback TCP with port files next to where the sockets would be (utils/vm/endpoint.rs), Windows process control, the hibernation supervisor with a TCP docker tunnel, sparse var.btrfs on NTFS, 8 MiB main-thread stack.
  • Workspace share over SMB: Windows QEMU has no 9p, so the guest mounts %USERPROFILE%\Avocado over CIFS. This needs a one-time elevated setup (local account + share). Treat it as test-grade: the product answer for a share with no user setup is still open.
  • Built-in engine client on Windows: removes the !cfg!(windows) gate. Verified against the avocado-vm engine over TCP: piped stdin, half-close, exit codes, separate streams, exec, and an interactive TTY. Named pipes were not exercised.
  • 32k command-line limit: engine::command_for_argv writes a long docker argv to a temp file and runs avocado __docker @<file>.
  • cp: a leading / or \ is a local path on Windows.
  • Stone include paths are treated as container paths on Windows.
  • A signing-keys folder outside the shared workspace is skipped with a warning instead of failing the run with "invalid volume specification".
  • provision: an explicit --env AVOCADO_USB_PASSTHROUGH=... now wins over the platform default, so USB profiles can run with a device imported into the VM.

Testing

  • Windows 11: release build; cargo test --lib 2194 pass, 40 fail. The same 40 fail on the base branch: they shell out to sh/bash/Python or use Unix-only fixtures. No new failures.
  • End to end from Avocado Desktop on Windows: install, build, img provision for qemux86-64; install, build and emmc provision of an RZ/V2N SoM (2026/next) that booted to login.
  • Not run on Linux or macOS after the rebase. An earlier revision passed Linux fmt/clippy/tests; this one relies on CI.
  • cargo clippy -D warnings on Windows still reports older unused-code findings outside this change, plus engine/run.rs:275 unreachable expression.

Known issues found along the way (not fixed here)

  • The supervisor does not bring the docker tunnel back after a wake that was not a user docker call, so the idle-exempt lane fails until the next build.
  • The supervisor hibernates the VM after its idle timeout even with a USB/IP device attached.
  • ps --format '{{json .}}' prints Labels as a map and no CreatedAt; there is no events. The desktop works around both.

🤖 Generated with Claude Code

mobileoverlord and others added 8 commits November 14, 2026 18:08
`avocado vm start/status/shell/stop` now work on a Windows host with QEMU
on PATH. Unix behaviour is unchanged.

- QEMU serves the QMP, qga and control-plane chardevs on loopback TCP on
  Windows, with the ports recorded in qmp.port / qga.port / control.port
  in the VM state dir. tokio has no UnixStream there and QEMU's pipe
  chardev takes a single client. A new `Endpoint` type hides the
  difference; the QMP and qga clients are no longer cfg(unix). Any local
  process can connect to a loopback port, which is the accepted trade-off.
- `-accel whpx`, `.exe` lookup on PATH, detached spawn without a console.
- Windows QEMU is built without -fsdev, so the 9p workspace share is not
  declared or mounted there; `vm start` warns. A replacement is a
  separate change.
- The docker forward lands on a loopback TCP port (docker.port) because
  `ssh -L` cannot bind a Unix socket on Windows; DOCKER_HOST routing uses
  tcp://127.0.0.1:<port>.
- PID liveness and termination through the Win32 API (windows-sys, which
  tokio already pulls in).
- The Windows binary links with an 8 MiB main-thread stack. With the
  1 MiB default even `avocado --version` overflowed the stack.
- A detached QEMU or ssh forwarder no longer inherits the CLI's own
  stdout, so a caller reading `vm start` to EOF is not held open for the
  life of the VM.

Not covered yet on Windows: the folder share, the hibernation supervisor,
swtpm (the guest boots with systemd.tpm2_wait=false).

Tested on Windows 11 with QEMU 11.1: image 0.5.0 (qemux86-64) boots in
about 15 s, ssh and qga answer, the docker API answers through the
forward, and `vm stop` quits through QMP. `cargo test --lib` on Windows
fails the same 42 tests main fails there (bash and Unix-path tests) and
fixes vm::state::tests::pid_round_trip_and_alive. Not run on macOS or
Linux.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`avocado vm supervise` was Unix-only because it listens on Unix sockets
for docker. On Windows those endpoints are now loopback TCP ports,
recorded in the file of the same stem: docker.sock -> docker.port,
docker-stream.sock -> docker-stream.port. The ssh tunnel's local end is
a loopback port held in memory, and QMP is reached through the endpoint
the socket path stands for. Unix keeps its sockets; the listener and
tunnel code is shared behind a small transport seam.

`vm start` therefore spawns the supervisor on every platform, with the
same internal and infra SSH ports as on Unix, instead of binding QEMU's
hostfwd to the user-facing port on Windows.

There is no SIGTERM on Windows for the supervisor to catch, so `vm stop`
ends it as a process tree (taking its ssh tunnel with it) and relies on
QMP `quit`, which QEMU answers while paused.

Tested on Windows 11 with a 20 s idle timeout: ssh and the docker API
work through the proxy, the VM pauses when idle (QEMU at 0 CPU, `vm
status` reports hibernated), an ssh connection resumes it in about
100 ms, and `vm stop` leaves no supervisor, tunnel or port files behind.
`cargo test --lib` on Windows is unchanged from the previous commit.
Not run on macOS or Linux.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Windows builds of QEMU have no virtio-9p, so a Windows host had no
workspace share and nothing could be built. The guest kernel already has
a CIFS client built in, and the VM's user-mode network reaches the
host's own SMB server at 10.0.2.2, so the share needs no new server and
no change to the VM image.

- One share covers the whole workspace root, so no project is shared on
  its own. The default root on Windows is %USERPROFILE%\Avocado;
  AVOCADO_VM_WORKSPACE overrides it as elsewhere.
- Host setup is one-time and elevated (Windows asks once): a local
  account used only for this, Modify rights for it on the workspace
  folder, and a share open to that account alone
  (smb_share_setup.ps1). Its random password is stored DPAPI-encrypted
  for the current user in share.cred in the VM state dir.
- `vm start` (and the lazy mount before a container run) checks the
  share, sets it up if missing or pointing elsewhere, and mounts it at
  /run/workspace. The mount script goes to the guest over stdin
  (`SshTarget::exec_script`) because it carries the password, and a
  refused mount redoes the setup once.
- Host paths translate to `/`-separated guest paths, and the `\\?\`
  form `canonicalize` returns on Windows no longer defeats the
  under-the-workspace check.

Limits, measured on Windows 11: every file appears root-owned 0755, the
share is case-insensitive, FIFOs cannot be created, symlinks are
emulated (mfsymlinks). Through the user-mode network a 256 MiB file
writes at about 11 MB/s and reads at about 64 MB/s; small files cost 3
to 4 ms each.

Known gap: the CLI also bind-mounts files outside that folder (the
signing keys under %APPDATA%, helper scripts under %TEMP%), which do not
resolve in the VM. A build needs AVOCADO_SIGNING_KEYS_DIR pointed inside
the workspace until the root is widened or those mounts are staged.

Tested on Windows 11: a fresh `vm start` creates the account and share
and mounts it; a second start reuses them; `avocado install` and
`avocado build` complete with the project on the share. `cargo test
--lib` on Windows adds no failures over main. Not run on macOS or Linux.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`stone_include_paths` entries become paths inside the build container,
but they were classified and joined with the host's path rules. On a
Windows host a declared `/abs/shared` is not "absolute", so it came out
as `/opt/src//abs/shared`, and a nested relative entry was written with
a backslash. Both are now handled by POSIX rules: a leading `/` is
absolute, and the relative part is joined with `/`.

Fixes test_stone_include_paths_absolute_path_passes_through and
test_stone_include_paths_dedup_preserves_first_occurrence on Windows;
output on Unix hosts is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`vm start` grows var.btrfs to its target size (50 GiB by default) with
`set_len`, relying on the result being sparse. That is implicit on macOS
and Linux. NTFS allocates every byte unless the file is first marked
sparse, so on Windows the VM's disk cost its full logical size in real
disk space from the first start.

The file is now marked sparse (FSCTL_SET_SPARSE) before it is extended.
A var disk already extended by an earlier build keeps its allocation;
marking it sparse afterwards does not release it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The built-in Engine API client was held back on Windows until console
input and stream half-close had been exercised there. They have, against
an avocado-vm reached over tcp://127.0.0.1, so the gate is gone and a
Windows host no longer needs a docker client installed.

Removing the gate alone leaves one failure: a container's shell script
rides inline as `bash -c <script>`, the entrypoint alone is tens of
kilobytes, and the re-exec `avocado __docker <argv>` is still a process
command line. Windows refuses to start one past 32,767 characters.
`engine::command_for_argv` takes the whole argv instead of a tool name;
on Windows, past the limit, it writes the argv to a temporary file and
starts `avocado __docker @<file>`. The shim reads it back, and the
script then travels in the create/exec request body, which has no such
limit. The five spawn sites that carry a script use it. A real docker
CLI or another tool still gets its argv as given.

`docker cp` told a local path from CONTAINER:PATH with the host's notion
of absolute, so on Windows `/tmp/a:b` read as a container named `/tmp/a`.
A leading `/` or `\` is now local on every host.

Tested on Windows 11 with no docker client on PATH: `avocado build` of a
qemux86-64 project completes, including the runtime image step whose
command is over the limit; and through `avocado __docker` directly,
piped stdin to `run -i`, end of input ending the command (`wc -l`), exit
code passthrough, stdout/stderr separation, `exec -i`, `rm -f` and
`volume ls`. An interactive TTY was not exercised. Named pipes were not
exercised either; the avocado-vm is reached over TCP.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The default signing-keys folder sits under the roaming profile, outside
the workspace shared with the avocado-vm. Its drive-letter path went to
the engine as a bind source and every SDK run from a plain shell died on
"invalid volume specification". Warn and leave the mount out instead, and
name AVOCADO_SIGNING_KEYS_DIR as the way to supply keys.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The flag was forced to 0 on macOS and Windows after the caller's --env
values were read, so a profile that needs USB (the fastboot `emmc` ones
exit unless it is 1) could not be run even with a device imported into the
avocado-vm over USB/IP. Keep the default and let an explicit value win.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant