Repository navigation
feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine - #4773
Open
matthewgrossman wants to merge 9 commits into
Open
matthewgrossman wants to merge 9 commits into
matthewgrossman wants to merge 9 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4773.docs.buildwithfern.com/openshell |
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
3 times, most recently
from
October 21, 2026 20:41
e06a0eb to
a23e140
Compare
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
2 times, most recently
from
October 22, 2026 04:51
be4af48 to
63f4141
Compare
matthewgrossman
changed the base branch from
test/3954-land-testkit/matthewgrossman
to
test/3954-testkit-ci-lane/matthewgrossman
October 22, 2026 04:51
5 tasks done
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 05:03
63f4141 to
df44e02
Compare
matthewgrossman
changed the title
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 16:01
df44e02 to
4d19336
Compare
matthewgrossman
changed the base branch from
test/3954-testkit-ci-lane/matthewgrossman
to
test/3954-land-testkit/matthewgrossman
October 22, 2026 16:01
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 16:37
4d19336 to
6c9a275
Compare
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 16:56
6c9a275 to
1da9c3b
Compare
matthewgrossman
marked this pull request as ready for review
October 22, 2026 17:10
matthewgrossman
requested review from
derekwaynecarr,
mrunalp and
sjenning
as code owners
October 22, 2026 17:10
Base automatically changed from
test/3954-land-testkit/matthewgrossman
to
main
October 22, 2026 17:24
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 17:24
1da9c3b to
4781a50
Compare
2 tasks done
drew
reviewed
Oct 22, 2026
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
3 times, most recently
from
October 23, 2026 04:00
8e7e5d4 to
2b879b0
Compare
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
2 times, most recently
from
October 23, 2026 05:29
c4f944f to
be103d6
Compare
6 tasks done
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 23, 2026 22:44
be103d6 to
f244394
Compare
|
Label |
This was referenced Oct 26, 2026
matthewgrossman
changed the title
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 26, 2026 20:12
f244394 to
6106cd0
Compare
Add a library API that boots an environment, runs commands in it with timeouts, copies files in and out, collects diagnostics and tears it down, so a caller can drive a machine without writing an Ansible playbook per suite. SSH and exec timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from leaving QEMU running. Add the `openshell-test-runner` flake package, which runs tests/runner with the toolchains and VM tools it needs. The Session API and test-runner package are adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…like the root `cargo --locked` only checks a lockfile against its own manifest. A workspace outside the root that builds root crates through path dependencies, such as the testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two lockfiles can resolve a shared crate to different versions without any check failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link against the AWS-LC that Nix provides. check_cargo_lockfile_alignment.py fails when an aligned workspace locks a registry crate that the root lockfile also contains at a version the root lockfile does not have. With --fix it re-locks the workspace starting from the root lockfile. tests/testkit is the first aligned workspace. The script and its tests come from #4629 (a37ebe9). Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add the `sandbox` crate to the conformance workspace with one testkit test that creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone. It is deliberately small: it exists so the runner can run a real test in a tmachine environment on CI before the existing suites move to testkit. The workspace's other crate, the CLI conformance suite, keeps running as it does today. The conformance workspace joins the lockfile alignment check, so its lockfile is realigned with the root lockfile. The test is adapted from the `sandbox-lifecycle` suite in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
`openshell-test-runner` builds a family's packages (a workspace under
tests/suites/) on the host as a nextest archive for the guest's musl target,
boots one tmachine environment by name with the named installer through the
Session API, runs nextest in it and brings back the JUnit report and log. No
per-suite Ansible playbook is involved:
nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful \
--installer deb --family conformance --package sandbox
The runner points the tests at the system under test with environment
variables: OPENSHELL_GATEWAY (the gateway the installer registered, `--gateway`,
default `openshell`) and OPENSHELL_BIN (the installed CLI). It waits for the
gateway with `testkit-ready` before the tests start. Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test.
A run passes if nextest succeeds and its JUnit report shows at least one test
and no failures. `--filter` narrows the tests with a nextest filterset, and
`--keep` leaves the VM up for debugging. Results from an earlier run are
removed first, and the guest's diagnostics are collected after a failure.
Which tests run in which environments is left to the caller for now; there are
no plans, environment registry or per-test requirements, and a broken test is
`#[ignore]`d.
Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and clippy for the new `sandbox` suite crate, wherever tests/testkit already runs them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they run through `mise run e2e:testkit` against a local Docker gateway and, on CI, through the test runner rather than the unit-test job. The lockfile task and the Cargo lockfiles job also run the alignment check, and its own tests run with `mise run test`. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…workflow Add a reusable workflow with an explicit matrix: each row names a tmachine environment and installer, a family and package, and the runner label. Each row runs `nix run .#openshell-test-runner` once. The jobs use the same integration inputs as the existing tmachine lanes, cache the tmachine disks keyed on the Ansible tree, upload the JUnit report and logs whether or not the run passes, and an aggregator job reports the overall result. The only row runs the conformance sandbox suite on amd64 Ubuntu with rootful Docker and the Debian package, which CI can provision today. Branch E2E calls the workflow when the test:e2e label is set. The job is not in the Core E2E result's needs yet, so it starts out advisory. The existing conformance, feature and driver lanes are unchanged. Modeled on the E2E Suites workflow in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Describe how the runner runs a family in a tmachine environment, locally and on CI, in TESTING.md, add a README for the sandbox suite, point the testkit README at the runner, and add rows for tests/suites, tests/tmachine and tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
- Replace tmachine's global QEMU PID registry with until_signal, which drops the work on SIGTERM/SIGINT/SIGHUP so kill_on_drop stops QEMU. This also lets Ctrl-C end a --keep run. - Decide a run's status from nextest's exit code (with --no-tests fail) instead of parsing JUnit; print the end of the nextest log on failure. - Move gateway diagnostics from tmachine's Session into the runner, drop Session::run_playbooks, and make Session own its VM directly. - Read TMACHINE_SSH_PORT once and reject invalid values; drop the racy port pre-check. - Build the runner with Nix so tmachine's firmware paths are compiled in, and set ANSIBLE_CONFIG only in the wrapper. - Copy the bundle directory in directly, fold lib.rs into main.rs, and make the gateway name required. - Drop the semver hint from the lockfile alignment check and trim duplicated docs. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Restore the check that the SSH forward port is free before QEMU starts. QEMU reports a taken port only by exiting, so a layer build would otherwise run its playbooks against the VM that already holds the port and cache the result. Report nextest's no-tests exit code as "no test ran" instead of a bare exit status. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 26, 2026 23:42
6106cd0 to
d6f93fa
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes it possible to run a testkit test on top of tmachine. A small runner builds a test family's tests for the guest, boots one tmachine environment with OpenShell installed, runs the tests inside it with nextest, and reports the result. On CI, an advisory lane runs one smoke test on Ubuntu with rootful Docker.
nix run .#openshell-test-runner -- \ --environment ubuntu-docker-rootful --installer deb \ --family conformance --package sandboxMost of the runner and the tmachine
SessionAPI come from @elezar's draft #4629.Stack
The stack is linear; each PR targets the one before it.
SessionAPI, a runner for one family in one tmachine environment, a smoke suite, and an advisory CI lane with an explicit matrix.OPENSHELL_TEST_*variables, and a suite fixture reads them. The Keycloak provider-refresh test is the example.openshell-conformance, move the remainingfeaturesanddriverssuites, then bring back plans (which suites run where) once features show what they need.Deferred from this PR
testkit-spec. They are kept ontest/3954-deferred-test-plans/matthewgrossman(be103d6, the previous head of this PR) and come back after features. Until then, CI lists what runs where as an explicit matrix ine2e-suites.yml.e2e-suiteslane is advisory and not part of the Core E2E result.Dropped from #4629, not planned:
requires, capabilities and waivers. A test that fails in an environment fails the lane.#[ignore = "<issue link>"]takes a broken test out everywhere.OPENSHELL_GATEWAYandOPENSHELL_BIN.Out of scope for the whole stack: version compatibility guarantees between the conformance suite and gateways, and running tests from other repositories.
Related Issue
Closes #4934
Part of #3954
Changes
SessionAPI; SSH, scp, and exec timeouts;down()never panics;until_signaldrops a binary's work on SIGTERM, SIGINT or SIGHUP so QEMU stops with it; the SSH port isTMACHINE_SSH_PORTand is checked before QEMU starts. Anopenshell-test-runnerflake package that builds the runner with Nix. Environment setup is unchanged.tests/runner(openshell-test-runner): picks the tmachine environment and installer by name, builds the family's packages (a workspace undertests/suites/) as a nextest archive for the guest's musl target, waits for the gateway withtestkit-ready, runs nextest in the guest with--no-tests fail, and writesjunit.xml,nextest.logand, after a failure, the gateway's journal to--results. The result comes from nextest's exit code: a run passes if nextest succeeds, fails if a test failed (the end of the nextest log, which names the failed tests, is printed), and is an error otherwise, including when no test ran.--filternarrows with a nextest filterset, and--keepkeeps the VM for debugging until Enter or Ctrl-C.tests/suites/conformance/sandbox: the smoke suite (create a sandbox, exececho, delete it).mise run e2e:testkitruns it against a local Docker gateway..github/workflows/e2e-suites.yml(reusable), with an explicitincludematrix of one row, and ane2e-suitesjob in Branch E2E (labeltest:e2e). JUnit and logs are uploaded ase2e-results-<family>-<package>-<environment>-<installer>. It is advisory: not part of the Core E2E result.TESTING.mdand the testkit and suite READMEs describe the runner;AGENTS.mdandCONTRIBUTING.mdlisttests/suites/.Testing
clippy -D warningsfor the runner, tmachine and the conformance suite; the runner and tmachine unit tests; the lockfile alignment check and its tests.nix build .#openshell-test-runnersucceeds..deb): the smoke test passed with the Nix-built runner, and Ctrl-C at the--keepprompt exited at once with 130 and stopped QEMU. With port 2222 already taken, the run stopped before booting with a clear port error.Checklist