[chenyuem-db]

Changes

Expose the existing UI “Run file” capability through pipelines run --refresh and --full-refresh, so users can refresh the tables defined in a source file without listing their names.

databricks pipelines deploy
databricks pipelines run my_pipeline --refresh ./src/pipeline.py
databricks pipelines run my_pipeline --full-refresh ./src/pipeline.py

Local files resolve through bundle sync configuration, including notebook names and exclusions. Absolute /Workspace/ paths are also accepted. The CLI passes these paths to the existing API through the shared bundle runner and reports the selected source paths on completion.

Each flag accepts table names or files; mixing both within one flag is rejected before restarting an update. Existing local files take precedence over bare names; backticks select ambiguous table identifiers. Unresolved immutable deployment paths require an absolute workspace path.

Why

Implements file-based selective refresh from the SDP Turbo CLI PRD. Local changes use the existing deploy-then-run workflow. Displaying the resolved table-name list is deferred to a follow-up; this PR reports source paths.

Tests

• Resolver and shared-runner tests cover table names (including long qualified names), file paths, notebook conversion, sync exclusions, deployment modes, and incompatible selectors.
• Existing acceptance tests check the serialized API request and completion output and rejection before restart. The fake API retains selectors per update; it does not execute pipeline code.
• Passed go test ./cmd/pipelines ./bundle/run ./libs/testserver, 117 pipeline/bundle-run acceptance subtests, and ./task fmt, ./task checks, ./task lint, using the unchanged dependencies from main. After restoring the original --no-wait behavior, re-ran go test ./cmd/pipelines and the no-wait and refresh-flags acceptance cases; all passed.

No live-workspace refresh was performed.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 9d78b59

Run: 41359451873

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	281	16	5:19
✅​	aws-windows-latest-4core-16gb	283	14	4:20
✅​	azure linux-2core-8gb	280	16	5:25
✅​	azure-windows-latest-4core-16gb	282	14	5:24
✅​	gcp linux-2core-8gb	281	16	5:19
✅​	gcp-windows-latest-4core-16gb	283	14	5:13

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:16	azure-windows-latest-4core-16gb	TestAccept
5:06	gcp-windows-latest-4core-16gb	TestAccept
4:15	aws-windows-latest-4core-16gb	TestAccept
4:01	azure linux-2core-8gb	TestAccept
3:50	gcp linux-2core-8gb	TestAccept
3:46	aws linux-2core-8gb	TestAccept

[github-actions]

Commit: 9d78b5988 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test	branch	main (3aa5c96)	latest (v1.23.0)
TestAccept/pipelines/run/refresh-flags	✅	❌	❌

TestAccept/pipelines/run/refresh-flags: branch ✅ | main (3aa5c96) ❌ | latest (v1.23.0) ❌

main (3aa5c96):

acceptance_test.go:1341: Diff:
        --- pipelines/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptpipelinesrunrefresh-flags3264895559/002/output.txt
        @@ -129,15 +129,15 @@
         Update for pipeline completed successfully.
         
         Pipeline configurations for this update:
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py]
        +• Refreshed [pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
           "path": "/api/2.0/pipelines/[UUID]/updates",
           "body": {
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "pipeline_file.py"
             ]
           }
         }
        @@ -155,15 +149,15 @@
         Update for pipeline completed successfully.
         
         Pipeline configurations for this update:
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py]
        +• Full refreshed [pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
           "path": "/api/2.0/pipelines/[UUID]/updates",
           "body": {
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "pipeline_file.py"
             ]
           }
         }
        @@ -174,37 +162,15 @@
         
         If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
         
        -Error: invalid --refresh: source file "./missing.sql" not found; use a table name or an existing source file
        +Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
         
        ->>> musterr [CLI] pipelines run --refresh pipeline_file.py --full-refresh-all --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Update ID: [UUID]
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Update for pipeline completed successfully.
         
        -Error: cannot use --refresh or --full-refresh together with --full-refresh-all, these flags are mutually exclusive
        +Pipeline configurations for this update:
        +• Refreshed [./missing.sql]
         
        ->>> musterr [CLI] pipelines run --full-refresh ./missing.sql --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Unexpected success
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: invalid --full-refresh: source file "./missing.sql" not found; use a table name or an existing source file
        -
        ->>> test ! -e [OUTPUT_DIR]/out.requests.txt
        -
        -=== Rejecting mixed selectors before restarting the pipeline
        ->>> musterr [CLI] pipelines run --refresh table1,pipeline_file.py --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot mix table names and source files in --refresh; use only table names or only source files
        -
        ->>> musterr [CLI] pipelines run --full-refresh table2,pipeline_file.py --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot mix table names and source files in --full-refresh; use only table names or only source files
        -
        ->>> test ! -e [OUTPUT_DIR]/out.requests.txt
        +Exit code: 1
        
FAIL

latest (v1.23.0): same output as main (3aa5c96)