test(integration): add an envtest integration framework and a trace test by jcogilvie · Pull Request #482 · crossplane/cli

jcogilvie · 2026-10-30T19:33:44Z

Description of your changes

Summary

Adds an integration-test framework: CLI commands run against an envtest API server, a real kube-apiserver and etcd with no controllers. It is made up of:

a Nix app, integration-test, with pinned envtest binaries and Crossplane CRDs;
a CI job that runs it;
an importable fixtures package, internal/testutils/integration;
a first test, which traces a claim through its XR to its composed resources with crossplane resource trace.

Why

Unit tests stub the API server, so they can't catch problems that only show up against a real one: CRDs generated from XRDs, server-side apply field ownership, owner references, and claim and XR binding. #436 needs this to test project simulate. Here the framework is split out on its own, with one small test, so it can be reviewed apart from simulate.

What changed

internal/testutils/integration (new). Its main pieces:
- Start(t, Setup, timeout) *Cluster starts an envtest cluster. It serves Crossplane's CRDs at the version go.mod pins, the test's own CRDs, and the XR and claim CRDs generated from the test's XRDs.
- Start also sets up the test's manifests the way Crossplane's controllers would leave them: controller owner references, resourceRefs and composite/claim labels; composed resources server-side applied under Crossplane's field manager; CompositionRevisions; XRD ownership of the generated CRDs. It can also provision a user with only given RBAC rules.
- Start skips the test when the envtest binaries or Crossplane's CRDs are missing.
- These fixtures build only with the integration tag.
- Untagged: CRDsForXRD, PinnedCrossplaneCRDsDir, and ReapOrphanedEnvtest. The last one kills kube-apiserver and etcd processes that a crashed earlier run left behind; call it from TestMain. It kills only orphans (parent PID 1) run from an envtest binary dir whose command line has a --data-dir/--cert-dir under envtest's k8s_test_framework_ temp dir, so a system server is spared.
cmd/crossplane/trace: TestTraceIntegration sets up a v1 XRD with a claim, the claim's XR, and two composed resources (one ready, one failing to sync). It then runs trace nopclaim test-claim against the cluster and checks the tree the default printer draws.
Nix:
- nix/envtest-assets.nix pins envtest 1.30.3 for linux and darwin, amd64 and arm64.
- nix/crossplane-crds.nix pins Crossplane's cluster/crds at v2.4.1, matching crossplane/apis/v2 in go.mod.
- The integration-test app runs go test -tags integration with both of these on every package containing a test file with //go:build integration (found with go list; it fails if it finds none), and writes coverage-integration.txt. Integration tests are any test files with //go:build integration, so unit tests in those packages also run there. It keeps the host PATH (Nix inputs first), so tests that run functions can reach the host's Docker; nothing here does, but feat(project): add alpha project simulate on the ported crossplane-diff engine #436's simulate tests need it.
- The test app and check export CROSSPLANE_CRDS_DIR for the untagged CRD helper tests.
CI: a new integration-tests job runs the app. It caches Go modules and the build cache (outside the Nix sandbox nothing else does), and uploads coverage under the codecov flag integration.
Lint: golangci-lint also builds with the integration tag.
gitignore: /coverage-integration.txt.
go.mod: adds github.com/shirou/gopsutil/v4 (used by the reaper). Tidying also pulled in its indirect dependencies and raised go.yaml.in/yaml/v3 from v3.0.4 to v3.0.5, which gopsutil requires. The vendor hash was regenerated with nix run .#tidy.

How to run the integration tests

Locally:

./nix.sh run .#integration-test                 # all integration-tagged tests
./nix.sh run .#integration-test -- -run TestTraceIntegration -v

Arguments after -- go to go test. Nix provides envtest's binaries and Crossplane's CRDs. To use your own, set KUBEBUILDER_ASSETS or CROSSPLANE_CRDS_DIR. Without them, go test -tags integration ./... skips the cluster tests.

In CI, the integration-tests job in .github/workflows/ci.yml runs nix run .#integration-test on every PR and push.

To add a test: give its file //go:build integration (the app runs every package with such a file), call integration.Start with a Setup, and call integration.ReapOrphanedEnvtest from the package's TestMain.

Testing

./nix.sh run .#integration-test passes. TestTraceIntegration runs (it doesn't skip) and passes.
./nix.sh run .#test, nix build .#checks.<system>.test and ./nix.sh run .#lint pass.
Without CROSSPLANE_CRDS_DIR (plain go test ./..., IDE runs), the CRD helper tests that need Crossplane's CRDs skip, as Start does; nix run .#test and the test check set it, so they run there.
TestSelectServers covers the reaper's selection, including a system etcd in an envtest dir and a k8s_test_framework_ dir outside the temp dir.
The four envtest tarball hashes match the release assets (nix store prefetch-file).

Split out of feat(project): add alpha project simulate on the ported crossplane-diff engine #436 at @adamwg's request. feat(project): add alpha project simulate on the ported crossplane-diff engine #436 will rebase onto this: its simulate tests will import internal/testutils/integration and keep their simulate-specific harness and testdata.
The trace test here doesn't use Setup's UnestablishedXRDs, FieldManagerApplies, DeleteAfterSetup or RBACRules, Cluster.Config, Cluster.Client, Cluster.Context, Cluster.UserConfig, FieldManagerApply, CompositeReconcilerFieldManager, ReadResourcesFromFile, or AwaitAdmissionDenial. feat(project): add alpha project simulate on the ported crossplane-diff engine #436's simulate tests use them.
Sibling split-outs from feat(project): add alpha project simulate on the ported crossplane-diff engine #436: fix(render): stale-backend errors, CROSSPLANE_DOCKER_NETWORK for every engine #477, refactor(project): share one xpkg client constructor; clamp build concurrency to 1 #478, refactor(simulate): move resource simulate's planning and diff server lifecycle into internal/simulate #479, feat(simulate)!: exit 3 on changes and 1 on errored plans in resource simulate #480.

I have:

Read and followed Crossplane's contribution process.
Run ./nix.sh flake check to ensure this PR is ready for review. (./nix.sh run .#test, ./nix.sh run .#lint and ./nix.sh run .#integration-test pass.)
Added or updated unit tests.
~~Linked a PR or a docs tracking issue to document this change.~~ (no docs change)
~~Added backport release-x.y labels to auto-backport this PR.~~

Need help with this checklist? See the cheat sheet.

🤖 Generated with Claude Code

Add internal/testutils/integration, an importable package of fixtures for tests that run CLI commands against an envtest API server: Start sets up a cluster serving Crossplane's CRDs and those generated from a test's XRDs, with its manifests as Crossplane would have left them; CRDsForXRD and PinnedCrossplaneCRDsDir generate and locate CRDs; ReapOrphanedEnvtest kills envtest servers an earlier run left behind. The cluster fixtures build only with the integration tag. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

Pin envtest binaries and Crossplane's CRDs in Nix, add an integration-test app that runs the integration-tagged tests against them, and run it in a new integration-tests CI job with Go module caching. The unit tests get CROSSPLANE_CRDS_DIR for the CRD helper tests, and lint covers the integration tag. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

coderabbitai · 2026-10-30T19:47:52Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: crossplane/cli
Branch: main
File: .coderabbit.yaml
Commit: ba029f2
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: 4b091802-f3fb-4ba1-8ad3-69648c0e242c

Commits

Reviewing files that changed from the base of the PR and between ba029f2 and 3c9fd47.

Files ignored due to path filters: 13

.gitignore is excluded by none and included by none
cmd/crossplane/trace/testdata/integration/composed-failing.yaml is excluded by !**/testdata/** and included by **/*.yaml
cmd/crossplane/trace/testdata/integration/composed-ready.yaml is excluded by !**/testdata/** and included by **/*.yaml
cmd/crossplane/trace/testdata/integration/crds/nopresource.yaml is excluded by !**/testdata/** and included by **/*.yaml
cmd/crossplane/trace/testdata/integration/setup.yaml is excluded by !**/testdata/** and included by **/*.yaml
cmd/crossplane/trace/testdata/integration/xr.yaml is excluded by !**/testdata/** and included by **/*.yaml
go.mod is excluded by none and included by none
go.sum is excluded by !**/*.sum and included by none
nix/apps.nix is excluded by none and included by none
nix/checks.nix is excluded by none and included by none
nix/crossplane-crds.nix is excluded by none and included by none
nix/envtest-assets.nix is excluded by none and included by none
nix/vendor-hashes.nix is excluded by none and included by none

Files selected for processing: 11

.github/workflows/ci.yml
.golangci.yml
cmd/crossplane/trace/trace_integration_test.go
flake.nix
internal/testutils/integration/cluster.go
internal/testutils/integration/crds.go
internal/testutils/integration/crds_test.go
internal/testutils/integration/doc.go
internal/testutils/integration/fixtures.go
internal/testutils/integration/reaper.go
internal/testutils/integration/reaper_test.go

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

adamwg

A few questions/comments inline. I haven't reviewed fixtures.go in detail yet, but overall this looks good for me and the tests pass on my machine.

adamwg · 2026-10-30T21:28:38Z

+# integration tests and the CRD helpers' unit tests. Pinned to the
+# github.com/crossplane/crossplane/apis/v2 version in go.mod; bump rev and
+# hash together when that changes. Tests find them through CROSSPLANE_CRDS_DIR.


Can we get renovate to update this for us? One option would be to have nix run .#tidy update it (renovate runs that after every go mod bump).

Done in tidy as you asked.

adamwg · 2026-10-30T21:32:01Z

+# envtest build, so callers can skip wiring them.
+{ pkgs }:
+let
+  version = "1.30.3";


Any reason not to match the k8s version from our go.mod (currently 1.35.3)? 1.30 has been EOL for over a year.

envtest only has 1.35.0 so far, so using that.

adamwg · 2026-10-30T21:38:46Z

I noticed nixpkgs has a package for setup-envtest, which is the controller-tools utility for downloading the envtest binaries and setting relevant envvars. Seems like using it rather than constructing the hashes and URLs ourselves would be more robust.

This is already pretty tidy, so I don't feel strongly about using setup-envtest instead. But, I think it's worth a look to see if it can make it easier to keep this up-to-date.

I think we can achieve the best of both worlds by making tidy fill in the hashes for us. That way, the tests never need network access (since setup-envtest downloads lazily) and the binaries still live in the nix store.

Integration tests only build with the integration tag, so reaching Start without envtest's binaries or Crossplane's CRDs means a broken environment. Fail loudly instead of skipping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

…eout Start's per-call timeout duplicated go test's -timeout, which already bounds t.Context(). Drop the parameter and use the test's context. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

Match the k8s.io/api v0.35 minor that go.mod pins. 1.35.0 is the only 1.35 envtest release controller-tools publishes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

After go mod tidy, tidy sets nix/crossplane-crds.nix's rev to go.mod's crossplane/crossplane/apis/v2 version and refreshes its hash the way it refreshes the vendor hash, so renovate's go.mod bumps carry it along. It fails rather than guess when the module is replaced or at a pseudo-version. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

tidy now sets nix/envtest-assets.nix to the newest envtest release for go.mod's k8s.io/api minor, taking each platform's sha512 from controller-tools' envtest-releases.yaml, the index setup-envtest verifies downloads against. The fetchurl hashes move from sha256 to sha512 to match. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

jcogilvie and others added 3 commits October 30, 2026 15:20

test(trace): trace a claim's tree against envtest

8bc2996

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>

jcogilvie mentioned this pull request Oct 30, 2026

refactor(xpkg): fold Resolver into a resolving xpkg Client #483

Merged

3 of 5 tasks

jcogilvie marked this pull request as ready for review October 30, 2026 19:47

jcogilvie requested review from a team and tampakrap as code owners October 30, 2026 19:47

jcogilvie requested review from adamwg and removed request for a team October 30, 2026 19:47

adamwg reviewed Oct 30, 2026

View reviewed changes

jcogilvie and others added 5 commits October 31, 2026 14:39

Navigation Menu

Repository navigation

test(integration): add an envtest integration framework and a trace test - #482

test(integration): add an envtest integration framework and a trace test#482
jcogilvie wants to merge 8 commits into
crossplane:mainfrom
jcogilvie:jco-integration-framework

jcogilvie commented Oct 30, 2026

Uh oh!

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading

Review limit reached

Uh oh!

adamwg left a comment

Uh oh!

Uh oh!

Uh oh!

adamwg Oct 30, 2026

Uh oh!

jcogilvie Oct 31, 2026

Uh oh!

adamwg Oct 30, 2026

Uh oh!

jcogilvie Oct 31, 2026

Uh oh!

adamwg Oct 30, 2026

Uh oh!

jcogilvie Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jcogilvie commented Oct 30, 2026

Description of your changes

Summary

Why

What changed

How to run the integration tests

Testing

Related

Uh oh!

coderabbitai Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

adamwg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adamwg Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jcogilvie Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

adamwg Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jcogilvie Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

adamwg Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jcogilvie Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading