Repository navigation
Conversation
## Changes
Add support for installing libraries on all-purpose clusters. Libraries
are declared as a field of the cluster resource, not a separate
resource:
```yaml
resources:
clusters:
my_cluster:
cluster_name: my-cluster
spark_version: 15.4.x-scala2.12
node_type_id: i3.xlarge
num_workers: 1
libraries:
- pypi:
package: requests
- whl: ./dist/*.whl
```
A cluster's libraries are not part of the cluster create/edit spec, so
they are managed through the dedicated Libraries API (`install`,
`uninstall`, `cluster-status`) as part of the cluster lifecycle.
Installed libraries are read back via `cluster-status`, so library
changes participate in normal drift detection. Local `.whl` artifacts
are built, uploaded, and rewritten to their workspace path before
install.
Install/uninstall cases:
- **Create** — libraries install on the freshly started cluster; no
restart is needed (a new cluster has no active sessions).
- **Add or change a library** — the desired set is installed and any
removed libraries uninstalled. A running cluster is restarted so the
change takes effect; a stopped cluster applies it on its next start.
- **Remove all libraries** — the libraries are uninstalled, restarting a
running cluster.
## Why
Resolves databricks#1860.
## Tests
Acceptance tests covering create / add / remove, local wheel
build-and-upload, repo-hosted wheels, and the stopped-cluster (no
restart) case. They run locally against the fake server and additionally
against a real workspace (`Cloud = true`).
Also manually verified on a real Azure workspace: a job with a
`python_wheel_task` and no task-level library successfully ran a wheel
installed only via the cluster's `libraries:` field, confirming a
cluster-installed library is actually usable by workloads.
---------
Co-authored-by: Isaac <no-reply@databricks.com>
## Changes - Change `air get pool` and `air list pools` commands to `air pools get` and `air pools list` to be resource-first. - Update help, user-facing references, command-shape tests, and acceptance tests. ## Why - Group pool operations under their plural resource command, consistent with other Databricks CLI resources. ## Tests - Relevant AIR unit tests. - AIR `help`, `pools`, and `config-help` acceptance tests. - Stale command-reference scan. _This PR was written with Codex._ Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
## Changes - Assert AIR workload accelerator fields independently instead of comparing the full SDK compute struct. ## Why - SDK unmarshalling populates `ComputeSpec.ForceSendFields`, causing the whole-struct assertion to fail even when the submitted payload is correct. ## Tests - `./task fmt` - `./task checks` - `./task lint` - `./task test` - `go test ./experimental/air/... -count=1` _This PR was written with Codex._ Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
## Changes - Persist task run IDs and MLflow enrichment for cached terminal AIR runs. - Reuse complete enrichment on warm-cache `air list --all-status` calls and refresh legacy cache entries that lack task run IDs. ## Why Previously, terminal rows were cached before MLflow enrichment. The first `air list --all-status` call displayed the MLflow run, but later cache hits showed an empty value and attempted `get-output` with the parent run ID instead of the task run ID. ## Tests - `go test ./experimental/air/cmd -run "Test(List|Index|NewList|IsTerminal)" -count=1` - `go test -race ./experimental/air/cmd -run "^TestIndexStrategyCachesMLflowEnrichment$" -count=1` - `./task checks` - Full Go lint: 0 issues. The Python lint phase could not download Ruff because PyPI DNS was unavailable. _This PR was written with Codex._ Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
) ## Changes - Show the configured retry policy when `experimental air run --watch` starts. - Emit each observed retry once in text and JSON output, detecting `WAITING_FOR_RETRY` early and preserving attempt-number detection across Bricklens and MLflow. ## Why Retry backoff was silent until Jobs created the next attempt, making active workloads appear hung after a failed attempt. ## Tests - `./task checks` - `./task lint` (all Go linters passed with zero issues; the Python-only Ruff step could not download Ruff because PyPI DNS was unavailable, and this PR changes no Python files) - Focused retry tests passed normally and under the race detector - Bricklens-to-MLflow fallback test passed 20 consecutive runs - `./task build` - Full test suite deferred to GitHub CI <img width="780" height="668" alt="Screenshot 2026-09-21 at 3 33 53 PM" src="https://github.com/user-attachments/assets/f750ea6e-3f9f-4188-8148-41c0081d8311" /> _This PR was written with Codex._ --------- Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
…server shutdown (databricks#6387) ## Changes Adds `--keep-detached-processes` to `databricks ssh connect` and `databricks ssh setup` (dedicated clusters, default off), which lets processes detached from the SSH session (`tmux`, `setsid`, `nohup`) outlive the tunnel: - **Teardown** (`ssh-server-bootstrap.py`): when the flag is set, terminate only the server's own process group - the server and its `sshd` children - and then hold the job run open while any detached process is still alive. Unset, the notebook sweeps its children exactly as before. - **Server** (`internal/server/`): the idle-timeout path returns instead of calling `os.Exit(0)`, and before returning it looks for detached processes. With the flag unset it logs a WARN naming them (readable via `/logs`), and it emits a new `SshTunnelTeardownEvent`. - **Plumbing**: flag → notebook base parameter → server flag → `metadata.json`, so a reconnect that asks for the mode against a server started without it gets a new server. Same shape as `--usage-policy-id`. The flag is also carried in the persisted `ProxyCommand`, like `--max-clients` and `--server-timeout`, so a host configured through `ssh setup` can ask for it - that ProxyCommand is the invocation that submits the run, and the mode is fixed at submission. - Narrowed the startup sweep, which matched any command line containing `databricks_cli` and would otherwise let a new session destroy the previous session's detached work. Implements option A from DECO-28187. ## Why The tunnel currently destroys work users deliberately detached: on server exit the bootstrap notebook runs `pkill -P` on everything it parents, and since it sets `PR_SET_CHILD_SUBREAPER` it parents every orphan in the session - which is exactly what `tmux`, `setsid` and `nohup` produce. Scoping that sweep to the server's process group is not enough on its own, and this is the part measured on real compute: survivors do live through teardown, but the notebook is the WSFS-registered process and WSFS authorizes an I/O by walking the live process tree for a registered ancestor. Once the notebook returns, survivors reparent to PID 1 and every `/Workspace` and `/Volumes` call fails with `EPERM` - a visible failure traded for an invisible one. So the narrowed teardown and the held-open run ship together behind one flag. **Why a boolean, bounded by `--server-timeout`.** The hold ends when the last detached process exits; the only other bound is the run's own lifetime, which `--server-timeout` already sets (24h by default) and Jobs already enforces. An earlier revision of this PR took a duration instead (`--keep-detached-for`), which was a second budget for the same run measured from a different origin: the run timeout runs from run start, while the linger clock only started once the server exited. `--keep-detached-for=24h` under the default `--server-timeout` therefore validated but could never deliver 24h, and because `Validate` compared the two in isolation, the ceiling it accepted silently depended on how long the session had already run. Presence is the condition the feature actually wants, and one knob for "how long this run may live" is enough. Dropped with the duration: the negative-value check, the ceiling check, the seconds-based notebook widget, and duration matching on reconnect. What the flag spends is time on a cluster: a `RUNNING` run suppresses autotermination (measured). That is why it is off by default, and why it is rejected for serverless, where the container teardown takes survivors regardless. Because the hold now has no deadline, the notebook's "holding the run open" line is throttled to one every five minutes while the set of survivors is unchanged, instead of one per 15s poll for as long as the work lives. The teardown warning is useful on its own: it turns "my work vanished with no explanation" into an actionable message even for users who never set the flag. `SshTunnelTeardownEvent` is a new event rather than fields on `SshTunnelEvent`, so teardown rows can't be counted as connections by existing `is_success` queries. It needs a matching lumberjack proto in universe (follow-up). Only the request is reported, not how long the hold lasted: the notebook outlives every Go process in the session, so no CLI process can observe when the hold ends - the run's own duration carries that. ## Tests - Unit tests for the procfs-based detection (fixture tree + one case pinned against real `/proc`), the teardown warning across flag/compute combinations, and the teardown telemetry asserted end-to-end through the upload path. - Extended existing tests for flag validation, `ToProxyCommand`, the submit-run base parameters, `keepDetachedMatches`, the connect event field, and the `ssh setup` host config. - Acceptance: the three `acceptance/ssh` connect goldens carry the new base parameter, and `acceptance/ssh/setup` pins the flag in the ProxyCommand `ssh setup` writes. - `./task lint`, `./task fmt`, `./task ws`, `go vet`, ruff format/check on the bootstrap, and the full `experimental/ssh` + `libs/telemetry` unit suites pass. **Live verification.** The feature was verified end to end on two dedicated dogfood clusters ([results](https://databricks.atlassian.net/browse/DECO-28187?focusedCommentId=9532479), [findings](https://databricks.atlassian.net/browse/DECO-28187?focusedCommentId=9532512)): detached work survives the tunnel, the run is held open, WSFS stays intact for the whole hold, `tmux attach` works on the next connection, and with the flag unset teardown is indistinguishable from the old build. Two defects came out of those runs and are fixed here: - **The hold never released early.** `detached_descendants()` asked `ps` for this process's children, but `ps` is itself one of them and `subprocess.run` leaves it in the notebook's process group, so it passed the "outside the server's group" filter on every poll. The survivor list was never empty, so the run - and the cluster - was held for the full duration even with zero detached work. It now reads `/proc` directly, mirroring `detachedDescendants` in `internal/server/descendants.go`, which counted correctly for exactly this reason. - **A server handover marked the previous run `FAILED`.** A new session's bootstrap terminates a server already running on the cluster, which a reconnect asking for a different mode reaches on a normal path; the previous notebook saw `-15` and raised. A SIGTERM exit is now a handover rather than a failure. That verification ran while the flag was still a duration. Everything it exercised - group-scoped teardown, the run held open, WSFS intact for the whole hold, `tmux attach` on reconnect, teardown unchanged with the flag unset - is untouched by the rename; what changed is how the hold ends (the work exiting, or the run timing out, instead of a preset deadline). Not re-verified on live compute after the `/proc` and handover fixes, or after the rename. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Once `resources.json` lands on the workspace (serial `tf+1`), any leftover terraform state files cannot affect engine selection because they carry a lower serial. The terraform backup/delete steps are cleanup, not commit. Simplify: `pushDirectState` only uploads `resources.json`. Reuse `BackupRemoteTerraformState` (already fail-open) for the remote terraform cleanup, and make the local renames fail-open too. Remove `rollbackRemoteMigration` — there is nothing to roll back. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Extends the invariant tests with a field-removal check: after the baseline no-drift deploy, fields tagged `# CAN_REMOVE` are stripped from the config, and the test asserts the removal is applied as an update that leaves no drift. This guards the class of bug where a field the config stops declaring is dropped from the update request instead of being cleared (databricks#6343) — which either fails the deploy or never converges. Every resource config with a cleanly-removable scalar is tagged (`comment` on the UC securables, `description` on jobs/models/cluster policies/apps, etc.). Fields whose removal still misbehaves are tagged `# CANNOT_REMOVE` with the reason inline: - `catalogs.custom_max_retention_hours` — not force-sent, drifts back every plan - `genie_spaces.description` — not force-sent, perpetual drift - `postgres_projects.display_name` — added to the update_mask but dropped from the body, backend 400s Misbehaving fields extracted from databricks#6452 This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…tabricks#6789) Clearing a genie space's `description` or a secret's `comment` by removing it from config left the old value in place — the field is `omitempty`, so it was dropped from the update request and every subsequent plan re-proposed the change. Force-send both on update (the same fix databricks#6343 applied to the UC `comment`), and teach the testserver's genie/secret update handlers to honor an explicit empty value as a clear rather than treating it as "not provided". Verified against a real AWS workspace: `UpdateSpace` accepts `{"description": ""}` and `UpdateSecret` accepts `{"comment": ""}` as clears, and the `no_drift` invariant's removal cycle (added in databricks#6773) converges for both. This flips `genie_spaces.description` and `secrets.comment` from `CANNOT_REMOVE` to `CAN_REMOVE`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…nd (databricks#6787) A direct-engine `bundle deploy` recreates a UC-tracing MLflow experiment on every deploy when the experiment's `trace_location` was bound out-of-band (e.g. by an enable-UC-tracing job the bundle config never declares). `trace_location` is `recreate_on_changes` (`spec:immutable`) with no `backend_defaults` entry, so a config-absent, remote-populated value plans a delete + create — dropping the UC trace location and stopping trace writes. This is the experiment analog of the secret-scope / external-volume recreate bug (fixed via `backend_defaults` in databricks#4834). The fix adds a `backend_defaults` entry for `trace_location`, so a config-absent, remote-populated value is left in place instead of forcing an immutable-field recreate. This mirrors the existing `artifact_location` suppression. Covered by the dresources classifier tests. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Reorder the steps in `ProcessBundleRet`. The single pre-build `shouldReadState` block is split in two: the state **pull** (which determines the engine) stays up front, while opening the resolved state moves to after Build. **Old:** pull-state → open-state → FastValidate → Validate → Build → deploy **New:** pull-state → FastValidate → Validate → Build → open-state → deploy where: - **pull-state** = `PullResourcesState` — determine the engine, set `MigratingToDirect`, print the deploy notice, and validate `--select`. Needs only the pulled engine descriptor, so it stays before FastValidate (unchanged). - **open-state** = open the direct state, DMS fetch/open, deployment-history enforcement, `InitIDs`, and the `--plan` file load. Needs the opened/resolved state, so it moves to after Build. The reorder is safe because the two sides are independent: - validate/build do not read the deployment state, and - open-state does not depend on validate/build output. (`Validate` may issue workspace API calls — that's why it is separate from `FastValidate` — but it does not need the state, so running it before open-state is fine.) The auto-migration still runs post-deploy, so behavior is unchanged. ## Why Prep for databricks#6749 — move the terraform→direct migration before deploy: that migration converts the bundle config into the direct state and must run **after Build**, because library and `${artifacts.*}` references (e.g. `whl: ./dist/*.whl`) are only resolved during Build. Running migration before Build would bake unexpanded globs into the migrated state instead of resolved remote paths. Landing this reorder on its own keeps databricks#6749's diff to the actual behavior change. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…6767) ## Why Raw-`$CLI` acceptance tests that document, against the real API, what the Postgres project `Update` endpoint actually accepts — no `bundle deploy` involved. These pin the behavior the direct engine's mask logic has to work around. ## Changes Two tests under `postgres_projects/factcheck/`: - **`update_mask_star`**: create accepts a minimal body (`display_name` + `pg_version`; the server materializes the rest), but that same body with `update_mask=*` is rejected field by field (`Field '...' is in update_mask but not provided`). Which field comes first isn't contractual, so a `Repl` normalizes it. - **`update_mask_spec`**: `update_mask=spec` applies the body but, unlike `*`, does **not** require every field — a body that omits `history_retention_duration` succeeds and the omitted scalar is preserved, not reset. (So `spec` is a lenient patch, not a full replace.) Both inherit `Cloud = true`, so they also run against a real workspace; verified on AWS. To make the local run match cloud, `validateUpdateMask` in the testserver now handles the wildcard: `*` is accepted, then every leaf of the resource's update-mask path list must be present in the body (a oneof group satisfied by any member), returning the first missing. Previously the fake rejected `*` as an unknown path.
Prep toward replacing all hand-written `RemapState` with an automatic copier. `ResourceSecretScopeAcls.RemapState` is `return remote`, and its remote and state types are identical, so the adapter's default path already returns the remote unchanged. Removing the method is behavior-preserving (verified: `secret_scopes` acceptance passes) and independent of the copier work. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
databricks#6792) Removing `custom_max_retention_hours` from a catalog or schema left the old value in place — the field is `omitempty`, so it was dropped from the update request and every later plan re-proposed the change. Force-send it on catalog and schema updates. Verified against a real AWS workspace with raw API calls: `{"custom_max_retention_hours": 0}` clears it (create with 240h → PATCH 0 → GET returns 0) for both catalogs and schemas, whereas an empty PATCH is otherwise rejected with "Nothing to update". **Catalogs** are direct-only (no terraform converter — `validate_direct_only_resources` rejects them in terraform mode), so there is no cross-engine payload to diverge from. Flips `catalog_optional_fields.custom_max_retention_hours` to `CAN_REMOVE`. **Schemas** run through terraform, but the terraform `databricks_schema` resource cannot set this field at all (it silently drops it), so a schema config that uses it would drift under the `migrate` invariant. Handled by: - carrying the field on `schema.yml` (for the direct field-removal check) and excluding `schema.yml` from `migrate`, and - adding a terraform-compatible `schema_tf_compat.yml` that keeps `migrate` coverage for schemas. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Adds a GitHub Actions workflow (actions/labeler v5) that labels PRs from the paths they touch, driven by a glob→label map in .github/labeler.yml. New label mappings are added by editing that file. Runs on pull_request_target so it can label fork PRs, with minimal permissions (contents: read, pull-requests: write) and scoped to main. It never checks out or runs PR code, and sync-labels is off so it only adds labels, never removing manually-applied ones. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## What did you change, and why? - Move AIR commands and acceptance tests from their experimental locations into `cmd/air` and `acceptance/air`. - Register `databricks air` as a top-level command and update help, ownership, and release notes. - Keep a hidden `databricks experimental air` command that returns an error directing users to `databricks air`. - Note that AIR commands remain under active development. - This PR does not otherwise change AIR APIs, configuration formats, or workload behavior. ## How do you know it works? - Ran `go test ./cmd/air ./cmd/experimental -count=1`. - Ran all AIR and root-help acceptance tests. - Ran repository checks and Go lint with zero issues. _This PR was written with Codex._ --------- Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
## Changes 1. Remove the maintainer-approval workflow 2. Consolidate `.github/OWNERS` & `.github/OWNERTEAMS` into `.github/CODEOWNERS` 3. To model "CLI maintainers can approve everything", include that team at the start of every entry 4. Linter to make sure 3. is always true ## Why `maintainer-approval` check run has shown to be unreliable (requiring rebase despite getting sufficient approvals) ## Tests GitHub does not have a CODEOWNERS playground :-(
…into DoRead (databricks#6797) Prep toward replacing hand-written `RemapState` with the automatic copier. `genie_spaces`' `RemapState` only excluded `SerializedSpace` from `ForceSendFields`. That exclusion moves into `responseToGenieSpaceConfig` — the builder every remote-producing path (`DoRead`/`DoCreate`/`DoUpdate`) already uses — so the remote is identical at the source and `RemapState` is an identity no-op. Removed; the adapter returns the remote unchanged. No golden changes; acceptance passes. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…icks#6804) ## Changes Close conversion gaps for fields already supported by the DAB schema. - Map `compute.pool_id` to `provisioned_capacity_id`. - Preserve `mlflow_artifact_location` in the generated AI Runtime task. - Set `retry_on_timeout` when `max_retries > 0`. ## Why Converted AIR workloads should retain settings that the DAB schema already supports instead of silently dropping them. ## Tests - `go test ./cmd/air` - `./task checks` - Manually converted, validated, deployed, and ran a one-A10 workload using a provisioned GPU pool. The deployed job preserved the pool ID and completed successfully. A native `air run` control using the same configuration also succeeded. - Manually deployed a workload with a custom MLflow artifact location in a managed Volume. Verified the experiment artifact root, run artifact URI, and uploaded artifact contents. _This PR was written with Codex._ Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
## Changes Skip the expensive dirty-working-tree check for AIR plain-tar and commit-pinned snapshots. Branch snapshots still check for local changes, and provenance sidecars continue to request dirty-state metadata when enabled. ## Why Dirty state cannot change plain-tar packaging (which includes working-tree contents) or commit-pinned packaging (which intentionally archives a specific commit). In the launch benchmark, avoiding the unnecessary `git status` reduced this client step from about 1.34 s to 0.61 s, saving about 0.73 s; the raw status scan took about 0.58 s. ## Tests - `go test ./cmd/air -count=1` - Full unit suite (11,141 tests) - `./task fmt`, `./task checks`, and `./task lint` - `GOFLAGS=-buildvcs=false ./task build` - `./task test-acc` was attempted, but this host has jq 1.6 and the suite requires jq >= 1.7, so it stopped at prerequisite validation. _This PR was written with Codex._
## Changes - Default `databricks air` workloads to environment version 6 when no version is configured. - Document the default in config help and update unit and acceptance coverage. ## Why Environment version 6 is now the preferred AIR runtime. All current public examples explicitly set an environment version, so this does not affect existing examples. ## Tests - `go test ./cmd/air` - `go test ./acceptance -run '^TestAccept/air/(config-help|run-submit)$' -timeout=60m` - `./task checks` _This PR was written by Codex._ --------- Co-authored-by: Caroline Chen <324939130+caroline-db@users.noreply.github.com>
…lt (databricks#6790) An immutable (`recreate_on_changes`) field the bundle config never set but the backend populated — `old` and `new` nil, `remote` set — is now left in place instead of triggering delete + create, reported as `immutable_backend_value`. This generalizes the `trace_location` fix ([databricks#6787](databricks#6787)) to **every** optional immutable field: the user isn't managing the field, and recreating to drop a backend-populated immutable value is destructive. The check is shape-based (config-absent + backend-present), so it only fires when the user omitted the field; a genuine change to an immutable field still recreates. ### Fields that already behaved this way (explicit `backend_defaults`, now removed — implicit) These had a hand-written `backend_defaults` entry that duplicated the new implicit rule; removed here: | Field | Prior link | |---|---| | `secret_scopes.scope_backend_type` | [databricks#4834](databricks#4834); note this field is also `missing_in_remote`, so the entry was already inert | | `volumes.storage_location` | [databricks#5145](databricks#5145) (external volumes) | | `experiments.artifact_location` | initial direct-engine field classification | | `experiments.trace_location` | [databricks#6787](databricks#6787) | | `registered_models.storage_location` | initial direct-engine field classification | | `pipelines.storage` | initial direct-engine field classification | | `model_serving_endpoints.route_optimized` | was `values:[false]`-restricted; now unconditional | | `vector_search_indexes.index_subtype` | initial direct-engine field classification | `backend_defaults` entries on **non**-recreate fields are kept (e.g. `pipelines.clusters[*].node_type_id`, `model_serving_endpoints.config.traffic_config`, `registered_models.owner/full_name/metastore_id`). ### Fields newly covered by the implicit default Previously these would recreate on a config-absent, backend-populated value; now left in place. (Impact is real only for the subset the backend actually populates when omitted — flagged **⚠**.) - **catalogs**: `storage_root` ⚠, `connection_name`, `provider_name`, `share_name` - **schemas**: `storage_root` ⚠ - **dashboards**: `parent_path` ⚠ - **model_serving_endpoints**: `config.auto_capture_config.catalog_name` ⚠, `.schema_name` ⚠, `.table_name_prefix` ⚠, `description` - **quality_monitors**: `assets_dir` ⚠ - **volumes**: `volume_type` ⚠ - **vector_search_indexes**: `index_type`, `primary_key`, `endpoint_name`, `delta_sync_index_spec`, `direct_access_index_spec` (non-ignored leaves) - **vector_search_endpoints**: `endpoint_type` ⚠ - **external_locations**: `credential_name`, `encryption_details`, `file_event_queue` - **database_instances**: `parent_instance_ref` (non-ignored leaves) - **instance_pools**: `disk_spec`, `node_type_flexibility`, `preloaded_spark_versions`, `preloaded_docker_images` - **pipelines**: `gateway_definition.*`, `ingestion_definition.*` - **model_provider_services**: `config.provider_type` - **postgres_roles**: `parent`, `role_id` - **secrets**: `catalog_name`, `name`, `schema_name` (identity fields; inert in practice) ### Whole-struct (field-less) recreate resources These declare the entire resource immutable (`recreate_on_changes: - reason: immutable`). Any config-absent, backend-populated field is now left in place rather than recreating the whole resource: `database_catalogs`, `internal_immutable_snapshots`, `job_runs`, `postgres_synced_tables`, `synced_database_tables`, `postgres_catalogs`. (`job_runs`' re-run-on-result-state logic is driven by `OverrideChangeDesc`, which runs after classification and is unaffected.) ### Not affected - Recreate fields already covered by `ignore_remote_changes` (already skipped when config-absent): `postgres_branches.*`, `postgres_endpoints.endpoint_type`, `postgres_projects.pg_version`, `postgres_roles.{postgres_role,auth_method,identity_type}`, `postgres_catalogs.postgres_database`, `experiments.trace_location.uc_trace_location.effective_table_prefix`, `vector_search_indexes.direct_access_index_spec.schema_json`, `database_instances.parent_instance_ref.{uid,lsn,effective_lsn}`. - `missing_in_remote` recreate fields (never echoed, so the shape can't occur): `secret_scopes.{scope_backend_type,backend_azure_keyvault,initial_manage_principal}`. ### Testing `TestRecreateBackendDefault` covers the new classifier path directly on real adapters; plan goldens that show the skip are updated from `backend_default` to `immutable_backend_value`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…atabricks#6796) Prep toward replacing hand-written `RemapState` with the automatic copier. `secret_scopes`' `RemapState` renamed `name`→`scope`, `backend_type`→`scope_backend_type`, `keyvault_metadata`→`backend_azure_keyvault`. This introduces a `SecretScopeRemote` type, populated in `DoRead`, whose field names already match the state — so `RemapState` becomes a plain field copy the copier can later take over. `name` is retained on the remote because `${resources.secret_scopes.<key>.name}` references resolve against the remote type. Plan `remote_state` display now shows the mapped field names (`scope`/`scope_backend_type`); goldens regenerated. Verified: `secret_scopes` acceptance passes. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…abricks#6795) Prep toward replacing all hand-written `RemapState` with an automatic copier. `DoRead` for these three already computes and stores `remote.Lifecycle` (3-way for clusters/warehouses: running → started, terminated → stopped, otherwise unset). `RemapState` recomputed the same `started` value (2-way) — duplicating `DoRead` and disagreeing for transient states. This copies `remote.Lifecycle` through instead, so `RemapState` is a plain field copy the copier can later take over. Behavior is unchanged for running/terminated; transient states now follow `DoRead`'s mapping (unset), which is the authoritative one. Verified: apps/clusters/sql_warehouses acceptance passes. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…s#6727) ## Stack - databricks#6727 — generate the PyDABs `.rst` doc pages from the generated modules (base) ← **this PR** - databricks#6729 — publish the built HTML to `gh-pages` + make the docs build strict ## Why The PyDABs Python reference docs (https://databricks.github.io/cli/python/) went stale. Codegen now emits **31** resource modules under `python/databricks/bundles/*/`, but the Sphinx `.rst` pages were hand-written and listed only 5 (`core`, `jobs`, `pipelines`, `schemas`, `volumes`). ## What Generate the per-resource `.rst` pages and the `index.rst` toctree inside `pydabs-codegen`: - New `python/codegen/codegen/generated_docs.py` — `write_docs()` scans the generated `databricks/bundles/*/` package dirs (decoupled from `RESOURCE_NAMESPACE`, robust to future renames), writes one page per resource, and regenerates the `index.rst` toctree. Called from `main.py` after the existing generation. - Stale generated pages are removed on each run so a dropped resource loses its page. - `core.rst` stays hand-written. The 4 previously hand-written resource pages (`jobs`, `pipelines`, `schemas`, `volumes`) regenerate **byte-identical** — no diff. `index.rst`'s prose header is preserved verbatim; only its toctree grows. - Acronym titles (`SQL Warehouses`, `MCP Services`) handled via a small override map. - The doc structure lives in `doc_page.rst.tmpl` and `doc_index.rst.tmpl` (loaded via `string.Template`, like the existing `*.py.tmpl` codegen templates) so it can be reviewed separately from the generation code. - `python/Taskfile.yml`: added the generated doc outputs to the task's `generates:` list. Also fold in a related fix surfaced in review: some upstream API descriptions aren't valid reStructuredText and break the Sphinx build (e.g. `sql.SpotInstancePolicy`, whose comment is a hard-wrapped ASCII grid table docutils rejects as malformed). - New burn-down override list `jsonschema_patch.override_descriptions` rewrites such descriptions into valid RST until the proto comment is fixed upstream (see `sqlgateway/scheduler/api/proto/endpoint_common.proto`). The override raises if it becomes a no-op, flagging entries to remove once upstream lands. - The strict `-W` docs gate that surfaces any *new* such breakage (at PR/SDK-bump time) lands in databricks#6729. CI's `generate-check` + `git diff --exit-code` now enforces the documented resource list can never drift from the generated modules. ## Testing - `./task pydabs-codegen` — regenerates code + 31 `.rst` pages + `index.rst`; the 4 existing pages show no diff; idempotent on re-run. - `./task pydabs-docs` — Sphinx build succeeds and now emits all 32 `databricks.bundles.*.html` pages; the `sql_warehouses` page renders a proper table instead of the malformed one. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…#6729) ## Stack - databricks#6727 — generate the PyDABs `.rst` doc pages from the generated modules (base) - databricks#6729 — publish the built HTML to `gh-pages` + make the docs build strict ← **this PR** ## Changes Two related pieces of the PyDABs docs pipeline: **1. Auto-publish the docs to `gh-pages` on release.** A new `update-pydabs-docs` GitHub Actions workflow rebuilds the PyDABs Python reference docs and publishes the HTML to the `gh-pages` branch on every release tag (and on demand via `workflow_dispatch`), replacing the manual copy that had left https://databricks.github.io/cli/python/ stale (5 pages vs. the 32 the build now emits). Modeled on `update-schema-docs.yml`: - Trigger: `push` tags `v[0-9]+.[0-9]+.[0-9]+*` + `workflow_dispatch`. - Permissions: `contents: write`, `id-token: write` (setup-jfrog GOPROXY exchange). - Checkout `main`, set up JFrog / Go / uv, then `git worktree add` `origin/gh-pages`. - Build with `go tool -modfile=tools/task/go.mod task pydabs-docs`. - Refresh only the `gh-pages` `python/` directory wholesale from `python/docs/_output/`, so a dropped resource loses its page; top-level `.nojekyll`, `index.html`, and `experimental/` are untouched. - Commit as `github-actions[bot]` and push directly to `gh-pages`, no-op when there is no diff. **2. Make the docs build strict (`-W`).** Add `-W` to the `pydabs-docs` sphinx-build command so warnings — e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump — fail the build instead of passing silently. This makes both the existing `python_docs` CI job (every PR, including SDK bumps) and the new publish workflow strict, surfacing docs breakage at bump time. Fixes the one pre-existing warning this exposes: the `Resources` docstring referenced `add_job` with `:meth:`, but `add_job` is a generated method Sphinx cannot resolve — changed to an inline literal. Stacked on top of databricks#6727, so the strict build already includes its generated pages and the override for malformed upstream descriptions. ## Tests From repo root, `./task pydabs-codegen` then `./task pydabs-docs` both exit 0 under `-W` (the `add_job` fix + databricks#6727's override clear the only warnings). YAML validated locally. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Prep toward replacing hand-written `RemapState` with the automatic copier. `dashboards`' `RemapState` forced `EmbedCredentials` into `ForceSendFields` and excluded output-only fields from it. But `DoRead` and the shared create/update builder (`responseToState`) already force `EmbedCredentials`, and the output-only fields are never present in the API response's `ForceSendFields`, so the exclusion was a no-op. `RemapState` was identity-equivalent — removed. No golden changes; acceptance passes. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Add an invariant target that deploys on terraform, then deploys on the direct default (which auto-migrates the state), then asserts no drift — across every resource config. Mirrors `invariant/migrate` but exercises the deploy-triggered auto path rather than the explicit `bundle deployment migrate` command. ## Why Prep for databricks#6749 (move terraform→direct state migration before deploy): landing the auto-migration drift coverage first, off main, so that PR stays focused on the behavior change. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…cesState (databricks#6808) ## Changes The user-agent `engine` dimension used to be set inside `PullResourcesState`. Move it to the callers (`bundle deploy`/`plan` in process.go and the two `bundle generate` resource paths), so the engine tag lives next to where the resolved engine is known. Follow-on cleanups from that move: - `PullResourcesState` no longer mutates the context, so it now returns only `*StateDesc` instead of `(context.Context, *StateDesc)`. - `generate dashboard` and `generate genie-space` had byte-identical state-loading blocks (initialize, resolve engine, pull state, tag engine, load). Extracted into one `loadStateForGenerate` helper so the tag is set in a single place. ## Why Prep for databricks#6749 (move terraform→direct state migration before deploy): that PR needs the engine tag set from the resolved (post-migration) engine, so consolidating the tagging off main keeps its diff focused on the behavior change. ## Tests Behavior-preserving. `acceptance/bundle/user_agent` asserts the `engine/` tag on every recorded request for both engines; it plus the `generate` and `state` suites and the generate unit tests pass with no golden changes. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Add a `PyDABs` label mapping to the PR labeler config (`.github/labeler.yml`). It reuses the existing `PyDABs` label and applies to PRs touching: - `python/**` — the PyDABs `databricks-bundles` package - `bundle/config/mutator/python/**` — the Go-side PyDABs mutator - `acceptance/bundle/python/**` and `acceptance/bundle/templates/pydabs/**` — PyDABs-specific acceptance tests ## Why PyDABs changes were only covered by the broad `DABs` label. A dedicated `PyDABs` label makes them easier to find and route. The Go mutator also matches `bundle/**`, so those PRs get both `DABs` and `PyDABs` — labeler applies every matching label independently. ## Tests Labeler config only, no code change. Verified the file parses as YAML and passes the whitespace linter. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…utside of bundle root (databricks#7122) ## Changes Fix `bundle validate` warning that `sync.include` and `sync.exclude` patterns do not match any files when `sync.paths` includes a directory outside the bundle root. ## Why Fixes databricks#7115 ## Tests Added an acceptance test <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. -->
…#7121) ## Changes Replace `github.com/google/uuid` with the [`uuid`](https://pkg.go.dev/uuid) package added to the standard library in Go 1.27. `github.com/google/uuid` remains an indirect dependency. `uuid.New` behaves the same: both read 16 bytes from `crypto/rand` and set the v4 version and variant bits (the CLI never used google/uuid's `SetRand` or `EnableRandPool`). Text and JSON encoding are identical, so existing deployment state and lock files decode unchanged. `Parse` is stricter: it rejects 38-character forms not wrapped in `{…}` (google/uuid ignores the first and last character) and a non-lowercase `urn:uuid:` prefix. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up to databricks#7105. The comments on databricks#6989 and databricks#7109 were 34-44 KB, close to the 60 KB cap, mostly because a new test fails the same way on main and on the latest release and the same diff was printed twice. - A failing run whose output equals an earlier column's (ignoring the random temp dir) now says "same output as <column>". - When the comment is over the cap, whole failure details are dropped (with a count) instead of cutting one in the middle and leaving its markup open. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…atabricks#7124) Follow-up to databricks#7105. The tests are selected from changes under `acceptance/`, but the job set up the whole build environment and compiled the test binary (~200s) just to find nothing to compare. A lightweight `changes` job on a hosted runner now checks the diff first and skips the job; it also deletes the comment of an earlier commit, which the skipped job can no longer do. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…tabricks#7125) Follow-up to databricks#7105. The `go build` of main's CLI (about 60s on CI) does not depend on the branch run, so start it before the branch phase and wait for it afterwards. Also fixes `git archive` being waited on before `tar` had drained the pipe. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes
Adds `features` as a bundle resource type, so a Feature Engineering
feature can be declared in `databricks.yml` and deployed, updated and
destroyed like any other resource.
`UpdateFeature` accepts `description` and nothing else, so the adapter
sends a static `update_mask=description`; every other field is excluded
at plan level, with `full_name`, `source` and `function` marked
`recreate_on_changes` from the spec's `IMMUTABLE` annotations. Literal
catalog and schema names in `full_name` are rewritten into
`${resources.…}` references so a feature deploys after the catalog and
schema it lives in.
## Why
Features are currently created through the Python client or the API.
Onboarding Features to direct engine is part of the proper DAB support
effort for the Feature Engineering API.
## Tests
- `go test ./bundle/... ./libs/... ./internal/... ./cmd/...
./experimental/ssh/... .` — passed
- `go test ./acceptance` — passed
- `task generate-check` — no drift
- `task checks`, `task fmt` — passed
Acceptance tests added:
- `bundle/resources/features/basic` — full lifecycle, request trace
- `bundle/resources/features/lifecycle` — cloud-capable, provisions its
own table
- `bundle/resources/features/remote-delete` — out-of-band delete drift
- `bundle/deployment/bind/feature` — bind and unbind
- `bundle/invariant/configs/feature.yml.tmpl` — invariant coverage
- `libs/testserver/feature.go` — fake create/get/update/delete
- `cmd/experimental/workspace_open_test.go` — `features` URL and
resource-type lists
Integration test is skipped: The cloud test environments are production
workspaces, and the feature-engineering service there has not yet
received a build compatible with the create payload this resource sends,
so features/lifecycle fails on create for reasons unrelated to this
change. Running that same acceptance test against a staging workspace
already serving a compatible build passes both DMS variants with no diff
against the golden, including the two bundle plan assertions that
confirm no field reads back as drift.
---------
Co-authored-by: Isaac <no-reply@databricks.com>
…vice calls (databricks#7129) ## Changes Retry idempotent calls that were not retried before: deploy lock stat/read/delete, state file reads, remote reads and waits after create/update in the direct engine, and the deployment-service get, list, update and delete calls. Reads also retry `unexpected EOF` transport errors. ## Why A cloud integration run (https://github.com/databricks-eng/eng-dev-ecosystem/actions/runs/39339632441) failed 8 scenarios on the first attempt with transient 500/504s. A failed lock release left a stale lock, and a `clusters/get` `unexpected EOF` failed a deploy. Non-idempotent creates (vector search POST, `CreateDeployment`, `CreateVersion`) are intentionally not retried. ## Tests Unit tests for `IsTransientRead` and the retrying filer. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes - Add `app-appkit` to `databricks bundle init` by calling the existing `databricks apps init` initializer. The production template remains in `databricks/appkit`. - Support non-interactive initialization with `--config-file` and `project_name`, and update help and changelog. ## Why AppKit apps use bundles for deployment. Listing AppKit in `bundle init` lets users discover and scaffold one alongside the other bundle templates. ## Tests - Existing `apps init` acceptance tests cover AppKit initializer options. The new `bundle init app-appkit` acceptance test covers non-interactive config-file handling and validation, deploy, and destroy of the generated bundle using a small local test fixture. - `./task checks` and `GOTOOLCHAIN=go1.26.9 ./task lint-go` passed. PR CI passed across Linux, macOS, and Windows, including integration tests. - An AppKit 0.76.1 scaffold initialized with closed stdin passed strict validation. On dogfood, the app reached `ACTIVE` and its deployment reached `SUCCEEDED`; the temporary app was deleted. *Authored with Codex.*
## Changes One resolver for a resource given as `key` or `type.key`, replacing three copies: `ResolveSelect`, and `FindResourceByConfigKey` used by `bundle deployment bind`/`unbind` and `apps import`. `bind` and `unbind` now also accept `type.key`. ## Why Preparation for `lifecycle.depends_on`, which needs the same resolution: databricks#7120 (and databricks#7132 on top of it). This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Add `python/docs/.gitattributes` marking `index.rst` and the per-resource `databricks.bundles.*.rst` pages as `linguist-generated`. `databricks.bundles.core.rst` is hand-written and stays visible. ## Why pydabs-codegen has written these pages since databricks#6727 (`python/codegen/codegen/generated_docs.py`), but the gitattributes from databricks#6394 predate it and only cover the Python modules and test data. So new-resource PRs like databricks#7094 show the doc pages as regular diffs. `python/databricks_tests/core/public_api.txt` is left unmarked on purpose: it's the golden snapshot of the hand-written `core` API, and reviewers are meant to see changes to it. ## Tests `git check-attr linguist-generated` resolves to `true` for `index.rst` and `databricks.bundles.jobs.rst`, and to `false` for `databricks.bundles.core.rst`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…emplate to >=0.0.3,<0.1 (databricks#7102) ## Changes Changes the `databricks-lakeflow-integrations` dependency in the `lakeflow-integrations` template from `==0.0.3` to `>=0.0.3,<0.1`. ## Why With an exact pin, the template needs a CLI update every time a new SDK version ships on PyPI. The range still picks up new patch releases, and `<0.1` keeps out potentially breaking minor releases. ## Tests Acceptance test `bundle/templates/lakeflow-integrations` passes. Follow-up to databricks#6992. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Changes `databricks auth login` now creates the `--timeout` context before `u2m.NewPersistentAuth` instead of after it, so the deadline also bounds the browser login step. This matches what the discovery login (`login.databricks.com`) path already does. ## Why `PersistentAuth` keeps the context it is constructed with, and `Challenge()` takes no context. Since databricks#2076 changed `persistentAuth.Challenge(ctx)` to `persistentAuth.Challenge()`, the timeout context has been created after `NewPersistentAuth`, so it never reached the challenge. As a result, `--timeout` (default 1h) had no effect on the browser step: if the OAuth redirect never arrives, the CLI waits for it indefinitely. The timeout only applied to the steps after the token was issued (workspace selection, cluster picker). Timeout description clearly states that it should cover the browser step. `--timeout duration Timeout for completing login challenge in the browser (default 1h0m0s)` ## Tests - New acceptance test `acceptance/cmd/auth/login/timeout`: it uses a fake browser (`echo_browser.py`) that never completes the redirect, with `--timeout 1s`, and expects `Error: authorize: context deadline exceeded`. Without the fix, this test hangs until the Go test timeout. - Existing `cmd/auth` unit tests and `cmd/auth/login` acceptance tests pass. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This PR updates the Go SDK dependency and generates APIs from that release's OpenAPI specification. --------- Co-authored-by: databricks-ci-ghec-2[bot] <184307802+databricks-ci-ghec-2[bot]@users.noreply.github.com> Co-authored-by: Jan Rose <jan.rose@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
## Changes Add `DATABRICKS_BUNDLE_STRICT_PROFILE_MATCH`, read as a boolean. When it is true and the target has a host or workspace ID, a strict loader runs before the config file, so the profile name always comes from the config file, even if `DATABRICKS_TOKEN` is set. The host and workspace ID must match exactly one workspace profile, and a profile without `workspace_id` only matches targets without one. A target with a workspace ID but no host must name a profile with that ID. A target with neither resolves as without strict mode. The loader does the same lookup as `ResolveProfileFromHost`, but fails when there is no single match, a different profile is selected, or the file is missing. ## Why With env auth the CLI skips host-based profile matching, so the token goes to whatever host the target sets. A caller that passes its own token and a file of trusted profiles needs the target checked against them first, including on a custom host shared by several workspaces. ## Tests Unit tests in `loader_test.go` list the loader cases side by side. Acceptance tests under `acceptance/bundle/strict_profile_match/` cover the same rules end to end, including `--profile`, `workspace.profile`, `default_profile` and the behavior without the variable. Accepted cases print the auth a bundle script receives, which shows the selected profile and host. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…tabricks#7148) ## Changes - Add `./task bump-sdk-post`: the mechanical rest of an SDK bump once `cli.json` is refreshed (`generate-check`, then `test-update`). - Run it as the last step of `generate-clijson-post`, so `generate-clijson` and the SDK bumper both get it. Its error is ignored there (see Why); run the task directly to see failures. - Rework the `bump-sdk` skill around the bumper: what its `automation/go-sdk-vX.Y.Z` PRs contain and how to finish one (merge `main`, `bump-sdk-post`, hand fixes, push to the bot branch). Also documents a local environment issue that corrupts acceptance goldens: a `uv.toml` that makes an internal PyPI proxy the default index breaks the tests' offline wheel builds, so run acceptance with `UV_DEFAULT_INDEX=https://pypi.org/simple`. ## Why The SDK bumper opens CLI PRs that stop after `cli.json` and `generate-clijson-post`, so they stay red until someone regenerates the rest: databricks#7138 failed lint and every test job on stale command stubs. Apart from the hand fixes (`exhaustruct` wiring, a `job_runs.yml` entry, the changelog PR link), everything that finishing databricks#7138 took is now this one task. The bumper runs `./task generate-clijson-post` from the CLI checkout and aborts before opening a PR on any error from it. `bump-sdk-post` fails whenever a bump needs hand fixes, because a compile break fails `generate-refschema` and `test-update`. Without `ignore_error`, those bumps would get no PR at all instead of a red one. The call is the last command rather than a `deps:` entry because deps run first, and the cleanup's `.gitattributes` restore would then undo cligen's output. ## Tests - Ran the task's two halves on a fresh checkout of databricks#7138's bot commit merged with `main` (`test-update` with `UV_DEFAULT_INDEX=https://pypi.org/simple`). The result matches databricks#7138's fix commit except for the four hand-fixed files. - Checked with the pinned go-task (v3.50) that `ignore_error` on a `task:` call swallows a failure two task levels down; `./task --dry generate-clijson-post` shows the expected order. Not run end to end through `generate-clijson` or the bumper, which need genkit output. - `./task checks`, and `go test -count=1 ./testmask` in `tools/` (it reads the Taskfile). This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
The test ran real npm about 60 times and timed out on slow runners. The CLI delegates manifest handling to the package manager, so cases that only exercised npm's own behavior (invalid/null/BOM manifests, workspaces, install/typegen lifecycle) are dropped. Package manager detection moves to its own `apps/validate-package-manager` test, using the shims. This pull request and its description were written by Isaac. The test has been very flaky on windows, causing retries there: last 7 days: 42 flaky, 40 passed of 82 jobs --------- Co-authored-by: Isaac <no-reply@databricks.com>
…abricks#7149) Adds `DATABRICKS_BUNDLE_RETRY_DISABLE_WARNING`; set in root `acceptance/test.toml`, overridden in the 6 tests that assert the warning. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…ectory (databricks#7153) ## Changes `air run` now asks SCIM Me for only `userName` (`attributes=userName`) when it resolves the launch directory. ## Why The launch directory needs only the user name, but the full SCIM Me response also resolves groups and entitlements. On a test workspace the trimmed call took 50ms at the median versus 94ms for the full one (40 calls each), and `air run --dry-run` dropped from 150ms to 101ms at the median (25 runs per build, alternating). ## Tests - New unit test asserts the Me request sends `attributes=userName` and the launch directory is unchanged. - `go test ./cmd/air/` and `go test ./acceptance -run TestAccept/air` pass; `./task fmt`, `./task checks`, and `./task lint` are clean. - `air run --dry-run` against a real workspace sends `GET /api/2.0/preview/scim/v2/Me?attributes=userName` and resolves the same launch directory as before. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…#7154) ## Changes Align CLI containers.name with backend service naming convention. ## Why May cause unexpected failures for users. ## Tests Unit tests --------- Co-authored-by: Vincent Chen <v-chen_data@example.com>
…tabricks#7137) Bumps [certifi](https://github.com/certifi/python-certifi) from 2025.1.31 to 2026.10.19. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/certifi/python-certifi/commit/662df63d7906df66f1f07c0751a3dfe9da14d914"><code>662df63</code></a> Restore unaffected emSign roots (<a href="https://redirect.github.com/certifi/python-certifi/issues/439">#439</a>) (<a href="https://redirect.github.com/certifi/python-certifi/issues/440">#440</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/5cb238caeeb6c30ec1a7309ad3e16a5aa7b10ce8"><code>5cb238c</code></a> Remove emsign (<a href="https://redirect.github.com/certifi/python-certifi/issues/437">#437</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/69494a764d0a08f90838d53d4b0fefd3e84045d6"><code>69494a7</code></a> Bump actions/download-artifact from 8.0.1 to 8.0.2 (<a href="https://redirect.github.com/certifi/python-certifi/issues/436">#436</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/029c7b0913d79e2efb73a216fc5c5e13ef958536"><code>029c7b0</code></a> Bump actions/upload-artifact from 7.0.1 to 7.0.2 (<a href="https://redirect.github.com/certifi/python-certifi/issues/435">#435</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/9d0a8f1f3a0d6b2e38d5773db7afe1a37e4527b6"><code>9d0a8f1</code></a> Add support for Python 3.15 (<a href="https://redirect.github.com/certifi/python-certifi/issues/432">#432</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/db472abf0a37067cb4488f036f4c19adf6b6ec3c"><code>db472ab</code></a> Bump pypa/gh-action-pypi-publish from 1.14.1 to 1.14.2 (<a href="https://redirect.github.com/certifi/python-certifi/issues/431">#431</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/f4bc676bc101fe2235846e37044e8c693d6cbaf4"><code>f4bc676</code></a> 2026.07.22 (<a href="https://redirect.github.com/certifi/python-certifi/issues/428">#428</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/4c91f9c5f9b4676d40d2930025ba04d80dd536f6"><code>4c91f9c</code></a> Bump actions/setup-python from 6.3.0 to 7.0.0 (<a href="https://redirect.github.com/certifi/python-certifi/issues/427">#427</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/01a66c5cf32eadb8f03a0504c24cb44f6d581248"><code>01a66c5</code></a> Bump actions/checkout from 7.0.0 to 7.0.1 (<a href="https://redirect.github.com/certifi/python-certifi/issues/426">#426</a>)</li> <li><a href="https://github.com/certifi/python-certifi/commit/eb355f41cb71f71012ecf1a4d27a57d0ee2b1337"><code>eb355f4</code></a> Bump pypa/gh-action-pypi-publish from 1.14.0 to 1.14.1 (<a href="https://redirect.github.com/certifi/python-certifi/issues/425">#425</a>)</li> <li>Additional commits viewable in <a href="https://github.com/certifi/python-certifi/compare/2025.01.31...2026.10.19">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/databricks/cli/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… in /tools/task (databricks#7133) Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.5 to 1.18.7. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/klauspost/compress/releases">github.com/klauspost/compress's releases</a>.</em></p> <blockquote> <h2>v1.18.7</h2> <ul> <li>Fix s2.NewDict OOB read.</li> </ul> <p>Security release without other changes.</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/klauspost/compress/compare/v1.18.6...v1.18.7">https://github.com/klauspost/compress/compare/v1.18.6...v1.18.7</a></p> <h2>v1.18.6</h2> <h2>What's Changed</h2> <ul> <li>s2: Fix amd64 stack frame corruption by <a href="https://github.com/klauspost"><code>@klauspost</code></a> in <a href="https://redirect.github.com/klauspost/compress/pull/1145">klauspost/compress#1145</a></li> <li>gzhttp: Canonicalize ETag header by <a href="https://github.com/justinmayhew"><code>@justinmayhew</code></a> in <a href="https://redirect.github.com/klauspost/compress/pull/1139">klauspost/compress#1139</a></li> <li>perf: pool hash tables in Go encode paths to reduce allocations by <a href="https://github.com/huynhanx03"><code>@huynhanx03</code></a> in <a href="https://redirect.github.com/klauspost/compress/pull/1143">klauspost/compress#1143</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/justinmayhew"><code>@justinmayhew</code></a> made their first contribution in <a href="https://redirect.github.com/klauspost/compress/pull/1139">klauspost/compress#1139</a></li> <li><a href="https://github.com/huynhanx03"><code>@huynhanx03</code></a> made their first contribution in <a href="https://redirect.github.com/klauspost/compress/pull/1143">klauspost/compress#1143</a></li> <li><a href="https://github.com/thaJeztah"><code>@thaJeztah</code></a> made their first contribution in <a href="https://redirect.github.com/klauspost/compress/pull/1144">klauspost/compress#1144</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/klauspost/compress/compare/v1.18.5...v1.18.6">https://github.com/klauspost/compress/compare/v1.18.5...v1.18.6</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/klauspost/compress/commit/8668e357e776d5152ed62f33c17f21b8690664fa"><code>8668e35</code></a> Merge commit from fork</li> <li><a href="https://github.com/klauspost/compress/commit/ac2f5e87a3089bd58c974ca53cef71c7aa8bf877"><code>ac2f5e8</code></a> docs: use unix line-endings for markdown files (<a href="https://redirect.github.com/klauspost/compress/issues/1144">#1144</a>)</li> <li><a href="https://github.com/klauspost/compress/commit/620d7b5cfbf89fbfdf69884f8a561a2981c25b11"><code>620d7b5</code></a> s2: Fix amd64 stack frame corruption (<a href="https://redirect.github.com/klauspost/compress/issues/1145">#1145</a>)</li> <li><a href="https://github.com/klauspost/compress/commit/1b63f2fece784da6334068c671e3f94366bf9d68"><code>1b63f2f</code></a> build(deps): bump the github-actions group with 2 updates (<a href="https://redirect.github.com/klauspost/compress/issues/1141">#1141</a>)</li> <li><a href="https://github.com/klauspost/compress/commit/3d86b893733380c370eba876f5beaaf63aeeea94"><code>3d86b89</code></a> s2: pool hash tables in Go encode paths to reduce allocations (<a href="https://redirect.github.com/klauspost/compress/issues/1143">#1143</a>)</li> <li><a href="https://github.com/klauspost/compress/commit/15967def6890f5494829394274686436976c4936"><code>15967de</code></a> gzhttp: Canonicalize ETag header (<a href="https://redirect.github.com/klauspost/compress/issues/1139">#1139</a>)</li> <li>See full diff in <a href="https://github.com/klauspost/compress/compare/v1.18.5...v1.18.7">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/databricks/cli/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ks#7147) ## Changes Unhide the `lakeflow-integrations` template so it appears in the `databricks bundle init` interactive selector and `--help` output. ## Why The template was hidden, so users could only find it by knowing its name. ## Tests Updated `TestTemplateHelpDescriptions`, `TestTemplateOptions`, and the `bundle/help/bundle-init` acceptance output. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…atabricks#7141) ## Changes `ssh connect` now creates each release folder (`<version>/databricks_cli_<version>_linux_<arch>/`) with `mkdirs` before uploading the archive to `import-file`. `mkdirs` succeeds when the folder already exists. `CreateParentDirectories` stays as the fallback if the upload still gets a 404. ## Why `import-file` reports a missing parent folder with a 404 only after the whole archive has been sent. On the first upload of every new version, the filer then created the folder and sent the archive again. Every archive (~17 MB) was therefore uploaded twice, and half of the `import-file` calls showed up as failures. The upload stays on `import-file`. Moving it to `/workspace/import` was considered, but `format=AUTO` imports a zip as a new folder that can't be overwritten. Overwriting is needed for dev builds and for concurrent connects. It also places an archive with a single top-level file next to that folder instead of inside it. ## Tests - `ssh/connect-serverless-feature-disabled` now records the `import-file` requests: one per architecture. Without the change it records four. - New `TestUploadReleasesSendsEachArchiveOnce`: a fresh release upload, and a dev re-upload into existing folders, both send one `import-file` request per architecture. Without the change the fresh upload records the reported pattern: `amd64 404`, `amd64 200`, `arm64 404`, `arm64 200`. - Against a real AWS workspace with the v1.21.0 release archives: the old upload made `import-file 404 → mkdirs 200 → import-file 200`. The new one makes `mkdirs 200 → import-file 200` per architecture, with the same extracted layout. - `./task lint-go` and the ssh unit tests pass locally. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…bricks#7157) ## Changes When migrating terraform state, a job in the state that references a resource not yet deployed (e.g. a new alert used by a new `alert_task`) failed with `cannot resolve field ... not found in TF state`. Such fields are now recorded as unset, so the first direct plan applies them. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.22.0 ### CLI * Fix `databricks aitools install` overwriting newer raw skills with an older pinned release. CLI 1.22.0 and newer now install the latest published skills and prevent automatic downgrades. ([databricks#7069](databricks#7069)) * Fix `auth login --timeout` not applying to the browser login, which waited for the OAuth redirect indefinitely. ([databricks#7103](databricks#7103)) * Remove the 50-workspace limit in `auth login` and support searching workspaces by name or ID. ([databricks#7071](databricks#7071)) * `databricks ssh connect` now fails right away with the workspace's reason when port forwarding to serverless compute is disabled, instead of waiting out the server start timeout. ([databricks#7057](databricks#7057)) ### AI Runtime * `air run` now spends less time before submitting: it fetches only the user name from SCIM. ([databricks#7153](databricks#7153)) * Align multi-image container name validation with backend services: allow hyphens and up to 32 characters, and reject uppercase letters and underscores. ([databricks#7154](databricks#7154)) ### Bundles * The `compute_id` deprecation warning for a target now shows the full path (`targets.<name>.compute_id`). ([databricks#7048](databricks#7048)) * Add `DATABRICKS_BUNDLE_STRICT_PROFILE_MATCH`, which requires a target's host and workspace ID to match exactly one profile in the config file before authenticating. Targets without a host or workspace ID are not affected. ([databricks#7109](databricks#7109)) * Zero-value overrides of `presets`, `lifecycle.prevent_destroy` and `deployment` settings in includes now replace non-zero base values, and `bundle validate -o json` shows explicit zero presets. ([databricks#7117](databricks#7117)) * `bundle deployment bind` and `unbind` now accept a resource as `type.key`, e.g. `jobs.foo`. ([databricks#7128](databricks#7128)) * Retry more transient server errors during `bundle deploy` and `bundle destroy`, including when releasing the deploy lock and reading state. ([databricks#7129](databricks#7129)) * Fix migration from Terraform failing when a deployed resource references a resource that was never deployed. ([databricks#7157](databricks#7157)) * Add `app-appkit` to bundle templates to simplify starting a Databricks Apps project. ([databricks#7004](databricks#7004), [databricks#7007](databricks#7007)) * Add support for the `features` resource type. ([databricks#6952](databricks#6952)) * Fix `bundle validate` warning that `sync.include` and `sync.exclude` patterns do not match any files when `sync.paths` includes a directory outside the bundle root. ([databricks#7122](databricks#7122)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([databricks#7083](databricks#7083)) * Bump Go to 1.27.2. Release binaries now require macOS 13 Ventura or later. ([databricks#7052](databricks#7052)) * Update the Go SDK to [v0.189.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.189.0). ([databricks#7138](databricks#7138)) * Bump Go toolchain to 1.27.3. ([databricks#7116](databricks#7116))
## Changes Pin CLI 1.22.0+ to the current latest skills release, `0.2.32`. Adjust existing tests to support the pin. ## Why After [databricks#7069](databricks#7069), `aitools list` and plugin installs fail when the unauthenticated GitHub latest-release lookup fails. The [VS Code CLI bump](databricks/databricks-vscode#2269) reproduces `resolve latest skills release: HTTP 403 fetching latest skills release` on Linux and Windows. This remote manifest change mitigates the failure for released CLI 1.22.0 once its compatibility cache refreshes (one-hour TTL). Keep 1.22.0 pinned when restoring latest tracking for a fixed CLI release. ## Tests `go test ./libs/clicompat/... ./libs/aitools/installer/...` _This PR was written by Codex._ --------- Signed-off-by: simonfaltum <simon.faltum@databricks.com>
## Release v1.22.1 ### CLI * Fix GitHub API rate limit errors in `aitools list` and plugin installs by pinning skills to `0.2.32`. ([databricks#7159](databricks#7159))
Preserve upstream ancestry and use the unmodified release tree. Assisted-by: Replit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Synchronize the fork with upstream Databricks CLI v1.22.1. The resulting source tree is exactly the upstream release, with no custom edits. Preserve both histories without rewriting main.
Why
Separate the mechanical upstream upgrade from the small sync extension review. The large generated and upstream diff belongs here; the companion PR restores the sync extensions. Do not release or update consumers from this intermediate branch. Merge the companion before publishing a fork release.
Tests
Git tree identity against upstream v1.22.1 verified. The companion tree matches the previously built fork source exactly. No authenticated workspace calls were run.
~ written by ⠕ Replit