You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: verify GPU driver installer SHA-256 before execution - #3249
Custom GPU driver installers are the most privileged downloads in our CI: they run as root/Administrator, and on Linux install_gpu_driver.sh loads kernel modules directly on the runner host via nsenter. Until now their integrity rested on TLS alone, which the eMudhra/emSign CA compromise (Ubuntu LP #2171507, Python advisory) showed to be fragile on unpatched runners.
This PR pins a SHA-256 digest for each installer in ci/test-matrix.yml and verifies it before the existing driver is touched or the installer runs, so a tampered download fails even if TLS is compromised. The incident prompted the change, but the check is meant to be permanent. Rows with DRIVER: 'latest' or 'earliest' use the runner's pre-installed driver and are unaffected.
us.download.nvidia.com publishes no checksum files, so the digests were computed on a patched Ubuntu 24.04 machine (ca-certificates 20260816ubuntu0.24.04.1). CI confirmed the 610.43.02 digest in run #40585595821. The 580.65.06 and 596.36 rows run only in nightly; their digests are pending confirmation by a nightly dispatch on pull-request/3249.
Updating a driver version
Changing a custom DRIVER in ci/test-matrix.yml now also means updating its DRIVER_SHA256, and whoever makes the change is responsible for that value:
Compute the digest of the exact installer the script downloads (curl -fsSL <installer-url> | sha256sum) on a machine with an up-to-date trust store. Windows data-center and desktop installers have different digests.
Say in the PR how the digest was obtained. Reviewers should recompute it.
Treat a digest mismatch in CI as possible tampering. Never copy the actual digest from a job log into the matrix.
The same steps are documented in the ci/test-matrix.yml header comment.
Changes
ci/tools/install_gpu_driver.sh: require DRIVER_SHA256, preserve it across sudo -E and nsenter, and verify installer.run with sha256sum --check --strict before unloading kernel modules, purging nvidia-* packages, or running the installer. The nsenter failure message no longer blames container options for every host-side failure.
ci/tools/install_gpu_driver.ps1: require DRIVER_SHA256 up front and verify the .exe with Get-FileHash before launching it.
ci/test-matrix.yml: add DRIVER_SHA256 to the custom-driver rows (610.43.02, 580.65.06, 596.36) and document the update procedure in the header comment.
.github/workflows/test-wheel-{linux,windows}.yml: pass DRIVER_SHA256 to the installer, and run the installer from a checkout of ci/tools at the workflow revision (github.sha). Nightly runs check out test sources at an older wheel-build commit, so without this they would run an installer script that ignores the digest.
ci/tools/tests/test_test_matrix_driver_hashes.py: check that every custom-driver row has a well-formed DRIVER_SHA256 and that rows sharing an installer agree.
Checklist
New or existing tests cover these changes. (Digests confirmed by the CI run linked above; nightly-only digests pending. Matrix format guard in ci/tools/tests/test_test_matrix_driver_hashes.py.)
The documentation is up to date with these changes.
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.
GPU driver installers on Linux and Windows now verify downloaded installers against their expected SHA-256 hashes before installation. On Linux, verification happens before existing driver packages or services are changed.
CI workflows provide expected hashes and use installer scripts from the workflow revision for custom driver versions. Pinned driver configurations include hashes, and automated checks flag missing, invalid, or inconsistent values.
Linux container setup errors now advise checking the required privilege and host process namespace options.
CI wheel-test workflows pass expected SHA-256 hashes to the Linux and Windows driver installers. The scripts verify downloaded installers before installation. The CI matrix includes hashes for custom driver versions, and a test validates matrix hash entries.
Changes: Driver hash verification
Layer / File(s)
Summary
Matrix hashes and workflow wiring ci/test-matrix.yml, .github/workflows/test-wheel-linux.yml, .github/workflows/test-wheel-windows.yml
Custom driver rows now specify installer hashes. The wheel-test workflows check out installer scripts from the workflow revision and pass each row’s DRIVER_SHA256 value to the installer.
Both installer scripts require and check the expected digest. The Linux script verifies the downloaded file before stopping services, unloading modules, or purging packages, then reuses that file for installation.
The test checks hash omission for preinstalled drivers, hash format and consistency for custom drivers, and the presence of at least one custom-driver entry.
Possibly related PRs
NVIDIA/cuda-python#3176: Updates the wheel-test workflows and test matrix that this change extends.
Suggested labels:ci-workflow-health
Priority: Low
Change: Feature
Merge Risk: Low · up to 06707
A malformed driver matrix entry can cause custom-driver CI jobs to fail; the localized validation fix should be applied before merge.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Reviewing files that changed from the base of the PR and between f9212dd and 08cba07.
Files selected for processing: 6
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml
ci/test-matrix.yml
ci/tools/install_gpu_driver.ps1
ci/tools/install_gpu_driver.sh
ci/tools/tests/test_test_matrix_driver_hashes.py
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
Reviewing files that changed from the base of the PR and between 08cba07 and 2e775e1.
Files selected for processing: 2
ci/tools/install_gpu_driver.sh
ci/tools/tests/test_test_matrix_driver_hashes.py
Files skipped from review as they are similar to previous changes: 1
ci/tools/install_gpu_driver.sh
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
Reviewing files that changed from the base of the PR and between 2e775e1 and 4d2aa83.
Files selected for processing: 2
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
Reviewing files that changed from the base of the PR and between 5151c9a and 06707f2.
Files selected for processing: 3
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml
ci/tools/tests/test_test_matrix_driver_hashes.py
Files skipped from review as they are similar to previous changes: 2
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
Supported by static analysis
Script executed:
git diff --unified=80 20a4aeeebd2d3226955d3caa142bd19b4d828ced 06707f203d1c8e7890f0d8f922bfb7cc9e825c95 -- ci/tools/tests/test_test_matrix_driver_hashes.py .github/workflows/test-wheel-linux.yml .github/workflows/test-wheel-windows.yml ci/tools/install_gpu_driver.sh ci/tools/install_gpu_driver.ps1 ci/test-matrix.yml
sed -n '1,120p' ci/tools/tests/test_test_matrix_driver_hashes.py
sed -n '140,175p' .github/workflows/test-wheel-linux.yml
sed -n '115,150p' .github/workflows/test-wheel-windows.yml
rg -n 'DRIVER|param|Argument|Validate|mandatory|Require' ci/tools/install_gpu_driver.sh ci/tools/install_gpu_driver.ps1
Repository: NVIDIA/cuda-python
Length of output: 41814
Script executed:
nl -ba ci/test-matrix.yml | sed -n '1,45p'printf'%s\n''--- hash test ---'
nl -ba ci/tools/tests/test_test_matrix_driver_hashes.py | sed -n '15,40p'printf'%s\n''--- Linux workflow custom path ---'
nl -ba .github/workflows/test-wheel-linux.yml | sed -n '145,180p'printf'%s\n''--- Windows workflow custom path ---'
nl -ba .github/workflows/test-wheel-windows.yml | sed -n '120,155p'printf'%s\n''--- Linux installer entrypoint ---'
nl -ba ci/tools/install_gpu_driver.sh | sed -n '30,48p'printf'%s\n''--- Windows installer entrypoint ---'
nl -ba ci/tools/install_gpu_driver.ps1 | sed -n '1,45p'printf'%s\n''--- Windows installer verification and execution ---'
nl -ba ci/tools/install_gpu_driver.ps1 | sed -n '65,95p'
Repository: NVIDIA/cuda-python
Length of output: 12459
important: Reject empty or non-string DRIVER values.
An empty DRIVER with a valid digest passes this test and both workflows select the custom-driver install step. Both installers then reject the empty value. Scalar non-string values also bypass the preinstalled-driver branch, although the installer behavior is not type-based after the workflow passes the value through an environment variable.
The matrix contract permits only latest, earliest, or a version string. Add assert isinstance(driver, str) and driver before incrementing custom_rows.
Suggested fix
if driver in PREINSTALLED_DRIVERS:
assert "DRIVER_SHA256" not in row, f"{platform}.{section_name}[{index}] sets DRIVER_SHA256"
continue
+ assert isinstance(driver, str) and driver, f"{platform}.{section_name}[{index}] has invalid DRIVER"
custom_rows += 1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI/CDCI/CD infrastructureto-be-backportedTrigger the bot to raise a backport PR upon merge
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
juenglin
added
CI/CD
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
rwgk
left a comment
Description
Refs #3195
Custom GPU driver installers are the most privileged downloads in our CI: they run as
root/Administrator, and on Linuxinstall_gpu_driver.shloads kernel modules directly on the runner host viansenter. Until now their integrity rested on TLS alone, which the eMudhra/emSign CA compromise (Ubuntu LP #2171507, Python advisory) showed to be fragile on unpatched runners.This PR pins a SHA-256 digest for each installer in
ci/test-matrix.ymland verifies it before the existing driver is touched or the installer runs, so a tampered download fails even if TLS is compromised. The incident prompted the change, but the check is meant to be permanent. Rows withDRIVER: 'latest'or'earliest'use the runner's pre-installed driver and are unaffected.us.download.nvidia.compublishes no checksum files, so the digests were computed on a patched Ubuntu 24.04 machine (ca-certificates 20260816ubuntu0.24.04.1). CI confirmed the610.43.02digest in run #40585595821. The580.65.06and596.36rows run only in nightly; their digests are pending confirmation by a nightly dispatch onpull-request/3249.Updating a driver version
Changing a custom
DRIVERinci/test-matrix.ymlnow also means updating itsDRIVER_SHA256, and whoever makes the change is responsible for that value:curl -fsSL <installer-url> | sha256sum) on a machine with an up-to-date trust store. Windows data-center and desktop installers have different digests.The same steps are documented in the
ci/test-matrix.ymlheader comment.Changes
ci/tools/install_gpu_driver.sh: requireDRIVER_SHA256, preserve it acrosssudo -Eandnsenter, and verifyinstaller.runwithsha256sum --check --strictbefore unloading kernel modules, purgingnvidia-*packages, or running the installer. Thensenterfailure message no longer blames container options for every host-side failure.ci/tools/install_gpu_driver.ps1: requireDRIVER_SHA256up front and verify the.exewithGet-FileHashbefore launching it.ci/test-matrix.yml: addDRIVER_SHA256to the custom-driver rows (610.43.02,580.65.06,596.36) and document the update procedure in the header comment..github/workflows/test-wheel-{linux,windows}.yml: passDRIVER_SHA256to the installer, and run the installer from a checkout ofci/toolsat the workflow revision (github.sha). Nightly runs check out test sources at an older wheel-build commit, so without this they would run an installer script that ignores the digest.ci/tools/tests/test_test_matrix_driver_hashes.py: check that every custom-driver row has a well-formedDRIVER_SHA256and that rows sharing an installer agree.Checklist
ci/tools/tests/test_test_matrix_driver_hashes.py.)