[WIP] Add the testing property of pipelines to bundles by tharunbhuk-db · Pull Request #7300 · databricks/cli

tharunbhuk-db · 2026-11-02T19:45:55Z

🥞 Stacked PR

stack/pipelines-test-pr1-command [Files changed]
- stack/pipelines-test-pr2-output [Files changed]
  - stack/pipelines-test-pr3-junit [Files changed]
    - stack/pipelines-test-pr4-bundles [Files changed] ← this PR

Design doc: https://docs.google.com/document/d/1atW-JG_hQu-G-YX-m1M7OodNZdH7LMc9H7CBlQAv5gw/edit

Changes

Add resources.pipelines.<key>.testing, with the test sources and test-only dependencies that databricks pipelines test runs with.
The property is experimental. It needs DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true. While the variable is off, bundle validate rejects testing and names the variable, and bundles deploy exactly as before: pipelines are read with the SDK, and the existing path rules are unchanged.
Relative paths in testing are translated like libraries. bundle validate catches common mistakes in test dependencies, such as package names, URLs and a requirements file without -r.
Local wheels in test dependencies are uploaded like other libraries, and the same-name check covers them, so two wheels with one file name cannot overwrite each other.
bundle plan shows changes to testing made outside the bundle, and bundle deploy undoes them.
With the variable on, a local project folder after -e, such as -e ./project[test], maps to its deployed folder instead of failing.

Why

A bundle needs to declare where a pipeline's tests and their dependencies live. The CLI uses its own types and raw calls for testing, because the Go SDK does not have the field yet. Depends on API support for testing on Create and Edit. Keep as draft until then.

Tests

Unit tests for path translation, the dependency checks and the read-back. The translation and read-back tests also cover the variable off, where the old behavior stays.
Acceptance tests in acceptance/bundle/resources/pipelines/testing* (testing-disabled covers the variable unset), acceptance/bundle/environments/editable-folder and acceptance/pipelines/test. The live test also installs a local test dependency.

This pull request and its description were written by Isaac.

github-actions · 2026-11-02T19:59:03Z

Commit: e038a13a6 | Workflow run

Regression test report

Selected 16 changed tests (limit=20, 0 not selected)

22 tests: regression=22 unreleased=0 coverage=0 failing=0

test	branch	main (`aed6702`)	latest (v1.23.0)
TestAccept/bundle/environments/editable-folder/DMS=	✅	❌	❌
TestAccept/bundle/environments/editable-folder/DMS=true	✅	❌	❌
TestAccept/bundle/refschema/DMS=	✅	❌	❌
TestAccept/bundle/refschema/DMS=true	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-dependencies/DMS=	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-dependencies/DMS=true	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-disabled/DMS=	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-disabled/DMS=true	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing/DMS=	✅	❌	❌
TestAccept/bundle/resources/pipelines/testing/DMS=true	✅	❌	❌
TestAccept/pipelines/databricks-cli-help	✅	❌	❌
TestAccept/pipelines/test/arguments	✅	❌	❌
TestAccept/pipelines/test/immutable	✅	❌	❌
TestAccept/pipelines/test/live	✅	❌	❌
TestAccept/pipelines/test/many-cases	✅	❌	❌
TestAccept/pipelines/test/pipeline-id	✅	❌	❌
TestAccept/pipelines/test/results	✅	❌	❌
TestAccept/pipelines/test/run-control	✅	❌	❌
TestAccept/pipelines/test/selection	✅	❌	❌
TestAccept/pipelines/test/submit-error	✅	❌	❌

TestAccept/bundle/environments/editable-folder/DMS=: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1341
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1082
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:659
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\n{\n  \"job\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project\"\n  ],\n  \"pipeline\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project\"\n  ],\n  \"testing\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]\"\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: unknown field: testing\n  at resources.pipelines.my_pipeline\n  in databricks.yml:27:7\n\nError: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory\n\n{\n  \"job\": [\n    \"-e project\"\n  ],\n  \"pipeline\": [\n    \"-e project\"\n  ],\n  \"testing\": null\n}\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,13 +2,19 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.my_pipeline
        	            	+  in databricks.yml:27:7
        	            	+
        	            	+Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        	            	+
        	            	 {
        	            	   "job": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
        	            	+    "-e project"
        	            	   ],
        	            	   "pipeline": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
        	            	+    "-e project"
        	            	   ],
        	            	-  "testing": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]"
        	            	-  ]
        	            	+  "testing": null
        	            	 }
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/environments/editable-folder/DMS=
        	Messages:   	bundle/environments/editable-folder/output.txt vs /tmp/TestAcceptbundleenvironmentseditable-folderDMS=475759570/002/output.txt