test(suites): port the conformance suites onto testkit and run conformance-integration on the runner by matthewgrossman · Pull Request #4774 · NVIDIA/OpenShell

matthewgrossman · 2026-10-20T23:38:28Z

Summary

Ports the universal conformance suites (CLI, lifecycle, file transfer, persistent workspace, policy advisor) onto testkit and runs them with openshell-test-runner on Docker, Podman (rootful and rootless) and k3s (single and HA). Then it switches the required conformance-integration job to the runner and deletes the old tmachine conformance and policy-advisor testsuites.

The lifecycle, persistent-workspace, file-transfer and policy-advisor ports start from @elezar's draft #4629.

Stack

Each PR targets the one below it.

feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773: the test runner, tmachine Session API and a smoke suite running on CI.
test(features): run feature tests against environment services, starting with Keycloak #4891: the Keycloak features suite, as the pattern for a features suite.
test(suites): port the conformance suites onto testkit and run conformance-integration on the runner #4774 (this PR): the conformance suites, run in every environment, and conformance-integration on the runner.

Later: ship openshell-conformance as one program and delete the old CLI suite; migrate the features and drivers suites; bring back test plans.

Related Issue

Part of #3954

Changes

tests/suites/conformance/sandbox: test files lifecycle, cli, persistent_workspace, file_transfer and policy_advisor next to smoke. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start.
cx.track and cx.adopt in testkit, for sandboxes a test creates through the CLI.
e2e-suites.yml: a row for each conformance environment (ubuntu-docker-rootful/deb, fedora-podman-rootful/rpm, fedora-podman-rootless/rpm, ubuntu-k3s/k3s, ubuntu-k3s/k3s-ha), each running --family conformance --package sandbox. A gateway field passes --gateway (openshell, or tmachine for k3s). A family input runs the rows of one family. The disk cache is keyed per environment, and k3s skips it, as integration-runner.yml does; its history doesn't say why, so that is still to find out.
conformance-integration in Branch E2E, Release Dev and Release Qualification now calls e2e-suites.yml with family: conformance. The job ID is unchanged. The Branch E2E e2e-suites job runs the features family, so conformance doesn't run twice.
The tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive are deleted. The integration-runner and integration-test defaults no longer name conformance.
The k3s installers wait for the Agent Sandbox controller to reconcile (k3s-controller-ready.yaml, moved out of the old conformance playbook). Tests don't wait for it.
Git is installed in every tmachine test environment (test-tools.yaml), for the .gitignore filtering test.
TESTING.md says what a conformance test may assert; CI.md and the watch-github-actions skill point at the runner's e2e-results-* artifacts.

Old test to new test

Old	New	Notes
`smoke` (scenario: status, create, get Ready, list, exec, delete, list empty)	`cli.rs::cli_create_exec_list_delete`	Same CLI steps. `smoke.rs::create_exec_delete` is the SDK variant (in #4773).
`sandbox_lifecycle` stop_start_preserves_workspace	`persistent_workspace.rs::persistent_workspace_survives_stop_start` (SDK) and `cli.rs::cli_stop_start_delete` (CLI)	Also `lifecycle.rs::restart_stopped_sandbox`.
`sandbox_lifecycle` stopped_can_be_deleted	`lifecycle.rs::stopped_sandbox_can_be_deleted`	Also at the end of `cli_stop_start_delete`.
`file_transfer::round_trip`	`file_transfer.rs::upload_download_round_trip`
`file_transfer::git_filtering`	`file_transfer.rs::upload_respects_gitignore`	Needs `git` where the tests run.
`file_transfer::path_safety`	`file_transfer.rs::download_stays_inside_workspace`
`policy_local`	`policy_advisor.rs::policy_local_accepts_proposals`
`mechanistic_proposal`	`policy_advisor.rs::denied_address_produces_draft_rule`
`new_hostname_proposal`	`policy_advisor.rs::denied_hostname_produces_draft_rule`

New with no old counterpart (from #4629): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

Not in this PR

The old CLI suite (tests/suites/conformance/cli, crates/openshell-conformance, e2e/support/conformance.sh) stays: the host lanes and mise run e2e:* still use it.
The features and drivers suites (oci-image, Podman userns, e2e-podman).
arm64: needs an arm64 KVM runner and arm64 build inputs.

Testing

Checks appropriate to the affected code and behavior pass
- Locally: fmt (runner, tmachine, testkit), clippy -D warnings, runner and tmachine tests, lockfile alignment, license headers, and Nix evaluation of the runner, tmachine config and test archives.
- On CI: Branch E2E run 40402056078 on 270c660f0. All five conformance-integration rows passed in tmachine VMs (Docker/deb, Podman rootful and rootless/rpm, k3s, HA k3s), as did the Keycloak row and both Select E2E suites jobs. The one failure was the Kubernetes E2E Rust smoke job (a timeout in openshell-e2e::credential_gating, unrelated to these changes; it failed on a different test in test(features): run feature tests against environment services, starting with Keycloak #4891's run). That job was re-run on its own and passed, so the run is green on attempt 2.
Unit tests added/updated
E2E tests added/updated

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (TESTING.md, CI.md)

copy-pr-bot · 2026-10-20T23:38:32Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-20T23:39:04Z

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

copy-pr-bot · 2026-10-21T20:37:55Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

matthewgrossman · 2026-10-26T21:10:03Z

/ok to test 270c660

github-actions · 2026-10-26T21:10:27Z

Label test:e2e applied for 270c660. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Add cx.track and cx.adopt, so a test that creates a sandbox through the CLI still cleans it up after a failure: track the name before the command, adopt it afterwards for an SDK handle. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ent-workspace suites to testkit Add the lifecycle, cli, persistent_workspace and file_transfer test files to the sandbox crate. The lifecycle, persistent-workspace and file-transfer tests are adapted from #4629. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start, and the one that can't yet (MXC) isn't conformant until it can. The policy-advisor tests are ported in the next commit but one. The OCI image, Keycloak and Podman userns suites are ported separately, in the features and drivers families. The old conformance/cli crate stays until the host lanes move off it. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The k3s installer now waits until the Agent Sandbox controller reconciles a gated probe Sandbox, so a test that starts right after boot doesn't race it. The check moves out of the old conformance playbook into tests/tmachine/ansible/playbooks/k3s-controller-ready.yaml, which the installer and the old playbook both include. Tests themselves don't wait for the controller. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Port the old suite's policy.local, denied-address and denied-hostname scenarios as the policy-advisor suite, so they survive the deletion of the old CLI suite. Adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add a row for each conformance environment (Docker, Podman rootful and rootless, k3s and HA k3s) next to the Docker and Keycloak rows. Each row names the gateway its installer registers (`openshell`, or `tmachine` for k3s) and the runner passes it with --gateway. A `family` input runs the rows of one family, so a caller can run the conformance rows alone. The disk cache is keyed per environment, and k3s skips it as integration-runner.yml does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Switch the required conformance-integration job in Branch E2E, Release Dev and Release Qualification from the tmachine conformance testsuite to the E2E Suites workflow, selecting the conformance family. The job ID is unchanged. Delete the tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive: the sandbox package covers the same tests, including the ported policy-advisor ones. The Branch E2E E2E Suites job now runs the features family only, so conformance does not run twice. Point the integration runner defaults and the docs at what remains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-10-27T04:53:48Z

/ok to test ff09b71

matthewgrossman added this pull request to stack #4776 October 20, 2026 23:38

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch 2 times, most recently from 6d64703 to e3d9f01 Compare October 21, 2026 20:40

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from e3d9f01 to 5e2cbc5 Compare October 21, 2026 20:49

This was referenced Oct 21, 2026

test(testkit): add the testkit library for end-to-end suites #4724

Merged

feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773

Open

feat(runner): add the test runner, plans and tmachine Session API #4775

Closed

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 5e2cbc5 to cda205f Compare October 21, 2026 23:45

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from cda205f to 7c75fbf Compare October 22, 2026 04:51

matthewgrossman removed this pull request from stack #4776 October 22, 2026 04:51

matthewgrossman added this pull request to stack #4824 October 22, 2026 04:51

matthewgrossman mentioned this pull request Oct 22, 2026

test(sandbox): run a testkit suite in Branch E2E #4822

Closed

5 tasks done

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 7c75fbf to bb39723 Compare October 22, 2026 05:29

matthewgrossman changed the title ~~test(suites): move the conformance, oci-image, provider-refresh and podman suites to testkit~~ test(suites): port the old suites onto testkit and run them from the plan Oct 22, 2026

matthewgrossman removed this pull request from stack #4824 October 22, 2026 05:29

matthewgrossman added this pull request to stack #4825 October 22, 2026 05:30

matthewgrossman removed this pull request from stack #4825 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from bb39723 to 1cb1dc8 Compare October 22, 2026 16:01

matthewgrossman added this pull request to stack #4836 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 1cb1dc8 to fbcd389 Compare October 22, 2026 16:37

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 33b8dc1 to db85da9 Compare October 23, 2026 03:37

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from db85da9 to 6f5c621 Compare October 23, 2026 04:00

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 6f5c621 to b9e6bfe Compare October 23, 2026 04:46

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from b9e6bfe to 919f767 Compare October 23, 2026 05:30

matthewgrossman mentioned this pull request Oct 26, 2026

test(features): run feature tests against environment services, starting with Keycloak #4891

Open

6 tasks done

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 919f767 to 270c660 Compare October 26, 2026 21:08

matthewgrossman changed the title ~~test(suites): port the universal conformance suites onto testkit and run them from the plan~~ test(suites): port the conformance suites onto testkit and run conformance-integration on the runner Oct 26, 2026

matthewgrossman removed this pull request from stack #4836 October 26, 2026 21:09

matthewgrossman changed the base branch from test/3954-first-testkit-lane/matthewgrossman to test/3954-keycloak-features-example/matthewgrossman October 26, 2026 21:09

matthewgrossman added this pull request to stack #4938 October 26, 2026 21:09

matthewgrossman added the test:e2e Requires end-to-end coverage label Oct 26, 2026

matthewgrossman and others added 8 commits October 26, 2026 16:48

test(tmachine): install Git in every test environment

6a0b65e

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

docs(testing): say what a conformance test may assert

9dee1c7

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 270c660 to ff09b71 Compare October 27, 2026 04:53

Navigation Menu

Repository navigation

test(suites): port the conformance suites onto testkit and run conformance-integration on the runner - #4774

test(suites): port the conformance suites onto testkit and run conformance-integration on the runner#4774
matthewgrossman wants to merge 8 commits into
test/3954-keycloak-features-example/matthewgrossmanfrom
test/3954-migrate-suites-to-testkit/matthewgrossman

matthewgrossman commented Oct 20, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

matthewgrossman commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

matthewgrossman commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

matthewgrossman commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Related Issue

Changes

Old test to new test

Not in this PR

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

matthewgrossman commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

matthewgrossman commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewgrossman commented Oct 20, 2026 •

edited

Loading