Repository navigation
Talk to the Docker daemon over the Engine API; add a container: block - #402
mobileoverlord wants to merge 6 commits into
Conversation
The CLI ran the `docker` command for every container operation, so a host whose daemon lives in an avocado-vm still needed the docker client installed. It now re-executes itself as a hidden `avocado __docker <argv>` that answers the subset of the docker CLI the code uses, over the Engine API with bollard. Call sites only change `Command::new(tool)` to `engine::command(tool)`, so their argv builders, stdio handling, exit codes and process-group signalling stay as they were. The connection follows DOCKER_HOST, then the current docker context, then the default socket. The docker command still runs for `container dev`, `--runs-on` remotes, Windows, non-docker container tools and ssh/TLS DOCKER_HOST; AVOCADO_DOCKER_CLI=1 forces it everywhere.
`sdk` and provision profiles take `container:` with privileged, network_mode, volumes, devices, extra_hosts, cap_add and environment, spelled as Docker Compose spells them. It becomes docker flags after `container_args`, deduplicated so the block's setting replaces the list's (docker treats a second --network as a second network, not an override). `container_args` stays for anything without a key. A target's block replaces the base one, and the block is part of the SDK stamp.
- A docker context naming an ssh or tcp endpoint keeps the docker CLI, as DOCKER_HOST already did; its TLS settings are not read here. - Images spelled docker.io/... find the Hub login `docker login` files under its v1 index URL, instead of pulling anonymously. - SIGHUP and SIGQUIT are forwarded as well as SIGINT and SIGTERM, and with a tty too, so a closed session no longer leaves the container running and a terminated tty run puts the terminal back. - Any 404 from create pulls when --pull=missing, which covers an image present only for another platform.
Running the whole container_args list through the provision dedup split `--network host`, `--pid host --ipc host` and repeated `--group-add` apart. The block now takes out of container_args only the network, privileged and mounts on the same path, in either spelling, and leaves every other entry as written.
# Conflicts: # CHANGELOG.md
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical issues can route operations to the wrong Docker daemon and leave provisioning privileged despite an explicit opt-out.
7 open issues
Emit an explicit false flag to override SDK privileged mode · New Fail when a selected Docker context cannot be loaded · New Preserve structured container values as atomic arguments · New Replace grouped and compact container flags correctly · New Honor DOCKER_TLS when selecting the Docker client · New Synchronize wait registration before starting auto-removed containers · New Preserve bare environment names when host variables are unset · New
What changed in this PR
This PR adds a built-in Docker Engine API client so supported Avocado operations no longer require the host Docker CLI. It also introduces structured container settings for SDK and provisioning configurations.
Changes:
- Routes container operations through a hidden re-execution entry point, retaining CLI fallbacks.
- Adds
container:settings, override handling, schema validation, and SDK stamp hashing. - Adds Engine request/parsing tests and documents the new behavior.
| File | Description |
|---|---|
tests/schema_rules.rs |
Tests acceptance and validation of container blocks. |
src/utils/volume.rs |
Routes volume management through the Engine wrapper. |
src/utils/stamps.rs |
Includes container settings in SDK input hashes. |
src/utils/nfs_server.rs |
Routes NFS container operations through the wrapper. |
src/utils/mod.rs |
Exposes the Engine module. |
src/utils/host_copy.rs |
Uses the wrapper for container-to-host copies. |
src/utils/ext_source_reader.rs |
Routes volume inspection through the wrapper. |
src/utils/engine/run.rs |
Implements container execution, I/O, and request conversion. |
src/utils/engine/mod.rs |
Adds routing, connections, parsing, and Engine operations. |
src/utils/engine/cp.rs |
Implements copies using archive endpoints. |
src/utils/container.rs |
Integrates Engine wrappers into container execution. |
src/utils/config.rs |
Adds structured settings and argument merging. |
src/main.rs |
Handles the hidden Engine client entry point. |
src/commands/sdk/run.rs |
Includes structured settings in SDK run arguments. |
src/commands/sdk/package.rs |
Routes packaging copies and temporary containers through the wrapper. |
src/commands/save.rs |
Uses the wrapper for archive export. |
src/commands/runtime/sign.rs |
Routes signing container operations through the wrapper. |
src/commands/runtime/provision.rs |
Routes provisioning helpers through the wrapper. |
src/commands/prune.rs |
Routes inspection and cleanup through the wrapper. |
src/commands/load.rs |
Uses the wrapper for volume imports and cleanup. |
src/commands/hitl/server.rs |
Routes server inspection, logs, and cleanup through the wrapper. |
src/commands/fetch.rs |
Uses the wrapper for image pulls. |
src/commands/ext/package.rs |
Routes extension packaging helpers through the wrapper. |
src/commands/ext/image.rs |
Routes image export helpers through the wrapper. |
src/commands/ext/checkout.rs |
Uses the wrapper for checkout copies. |
src/commands/connect/upload.rs |
Routes upload streaming containers through the wrapper. |
schemas/avocado-config.json |
Defines container blocks for SDK and provision profiles. |
CHANGELOG.md |
Documents Engine integration and structured settings. |
Cargo.toml |
Adds Bollard and archive-stream bridging support. |
Cargo.lock |
Locks the new Engine client dependencies. |
Review effort: Balanced
- A selected docker context that cannot be loaded is an error, as with the docker CLI, instead of falling back to the default socket. - DOCKER_TLS, like DOCKER_TLS_VERIFY, keeps the docker CLI. - A bare `-e NAME` the host does not have stays bare, so the daemon unsets it rather than keeping an image default. - `privileged: false` becomes `--privileged=false`, which overrides an earlier `--privileged` in the provision merge. - Block values with spaces are quoted, so the container_args splitter keeps them whole. - The block's overrides are found in grouped entries and in the compact `-v/src:/dst` spelling; the rest of a grouped entry is kept. - Rust 1.100 reads awaiting `pending()` on an uninhabited type as diverging; await a unit one so the code after it is not unreachable.
| /// outside double quotes, so a value with a space is quoted to stay whole. | ||
| // ponytail: that splitter has no escape for `"` itself, so a value holding | ||
| // one is still split; carry block values as argv end to end if that matters. | ||
| fn atom(value: &str) -> String { |
There was a problem hiding this comment.
error: Literal double quotes in structured environment values are still silently removed. For example, sdk.container.environment: { PAYLOAD: '{"name":"x"}' } passes through atom() and then SdkContainer::parse_container_arg() at container.rs:2738, producing PAYLOAD={name:x}. This corrupts valid JSON even without whitespace. Preserve literal quotes through structured-value serialization and final argv parsing, and add a regression test that checks the final environment value.
| .as_ref() | ||
| pub fn get_provision_profile_container_args(&self, profile_name: &str) -> Option<Vec<String>> { | ||
| let profile = self.get_provision_profile(profile_name)?; | ||
| ContainerConfig::with_args(&profile.container_args, &profile.container) |
There was a problem hiding this comment.
error: The profile block is applied only to the profile's own arguments before merge_provision_container_args() combines them with the SDK arguments. An SDK container.volumes: ["/tmp:/data:ro"] and profile container.volumes: ["/tmp:/data:rw"] therefore retain both mounts, and container creation fails with Duplicate mount point: /data. Likewise, SDK container_args: ["--network bridge"] plus profile container.network_mode: host retains two networks and the shim rejects the run. Apply the profile block to the combined inherited SDK and profile arguments before applying CLI overrides, with regression tests for both combinations.
| // not using the daemon's auto-remove if it is ever seen. | ||
| let removed = a | ||
| .has("--rm") | ||
| .then(|| tokio::spawn(wait(docker.clone(), id.clone(), "removed"))); |
There was a problem hiding this comment.
error: Non-blocking follow-up: the wait-registration race is reproducible. With a proxy delaying only forwarding of POST /containers/<id>/wait by one second, docker run --rm ubuntu:24.04 true exits 0, while the same command through avocado __docker exits 125 with No such container. Ordinary occurrence frequency is unmeasured. Docker's ContainerWait implementation explicitly waits for the server's response headers before returning; it does not merely spawn the wait request. Identify a bounded repair that preserves exit status and cleanup when the shim is killed; simply awaiting the complete wait before starting would deadlock, and moving removal into the shim needs to preserve forced-termination cleanup.


Why
On macOS and Windows the Docker daemon lives in the avocado-vm, but the CLI still ran the host's
dockercommand for every container operation, so users had to install the docker client anyway.What
Engine API instead of the docker command. The CLI re-executes itself as a hidden
avocado __docker <docker argv>, which answers the subset of the docker CLI this codebase uses (run/create/exec/start,ps,inspect,logs,stop/kill/rm,volume,pull,cp) over the Engine API with bollard. Call sites only changeCommand::new(tool)toengine::command(tool)/engine::std_command(tool), so argv builders, stdio, exit codes, process groups and the existing fake-tool unit tests are unchanged.DOCKER_HOST, then the current docker context, then the default socket. The avocado-vm route already setsDOCKER_HOST=unix://…/docker.sock.avocado container dev,--runs-onremotes, Windows (console input and named-pipe half-close not exercised yet), a--container-toolother thandocker, and ssh/TLS endpoints (env or context).AVOCADO_DOCKER_CLI=1forces the docker command everywhere.docker runflag fails the step with a message namingAVOCADO_DOCKER_CLI=1, rather than being dropped silently.inspectof a missing object prints the daemon's error rather than[];cp SRC/.into an existing directory is refused; pull progress is one line.Structured
container:block.sdkand provision profiles takecontainer:withprivileged,network_mode,volumes,devices,extra_hosts,cap_addandenvironment(Docker Compose spellings).container_argsstays as the fallback for anything without a key. The block replaces the matching network, privileged and same-path mounts incontainer_args; everything else is left as written. A target's block replaces the base one, and the block is part of the SDK stamp. Separate commit, so it can be split out if preferred.Testing
cargo fmt --check,cargo clippy --all-targets -D warnings, fullcargo test(5070 passed).init,sdk install,install,build(plain and TUI),fetch,provision,save/clean/load,prune --dry-run,sdk run(plain, piped stdin, interactive under a pty). Zero docker command calls.init(qemuarm64),sdk install,install,build(plain and TUI),provision,sdk run(piped and interactive),save/clean/load. Zero docker command calls.-itrun/exec under a pty, SIGINT/SIGTERM/SIGHUP forwarding (container traps, exit code, nothing left behind), and thecontainer:block's effect insdk run.