Check and backfill notarization of darwin tarballs

doc/jenkins-jobs.md

@@ -15,3 +15,46 @@ https://github.com/nodejs/email/blob/HEAD/iojs.org/aliases.json.
 
 This job needs to be updated each time a we roll over to a new Current
 release.
+
+## notarize-darwin-tarballs (ci-release, to be created)
+
+Nightly job on ci-release that checks the darwin tarballs of the supported
+release lines for an Apple notarization ticket and submits the ones that lack
+it, see [nodejs/node#67140](https://github.com/nodejs/node/issues/67140). It
+has to run on a macOS release node (full Xcode and the `NODE_RELEASE_PROFILE`
+keychain profile), so it cannot live on the test CI.
+
+Freestyle job restricted to a `macos*-release-*` label, checking out
+`nodejs/build`, with the `SSH_RELEASE_*` credential binding from
+`iojs+release` and `NOTARIZATION_TEAM_ID` set to the Node.js team id,
+running the same keychain unlock `iojs+release` uses (the notary credentials
+are read from the keychain profile, so `NOTARIZATION_PASSWORD` is not
+needed):
+
+```shell
+security unlock-keychain -p "$SSH_RELEASE_PASSWORD"
+tools/macos-tarball-notarization.sh --submit --all-supported --latest 3
+```
+
+Each check downloads a tarball, so only the three most recent releases per
+line are covered nightly; older ones are a manual run with a larger
+`--latest`. Supported lines come from the Release schedule, so the job needs
+no editing on roll-over. Submissions are serial: each waits for Apple's
+verdict (bounded to 30 minutes by `notarytool --timeout`) and then up to
+eight minutes for the ticket to become visible, so 18 binaries can take over
+11 hours if Apple is slow on every one. Disable concurrent builds and give the
+first run a 16 hour timeout, or run it by hand in two or three batches with
+explicit versions. A rerun skips every binary that already has a ticket and
+resubmits the rest, so a run that hits the timeout is simply rerun. One hour
+is enough once everything is ticketed. A rejected submission prints
+Apple's notary log in the build output.
+
+The ticket check queries Apple's ticket service by code directory hash, the
+same lookup macOS performs; a failed lookup is reported as such and never
+counted as a missing ticket. Exit 1 means a completed check found a published tarball whose
+`bin/node` has no ticket, or an invalid signature: in the nightly `--submit` run that
+should not happen and points at a tarball axis of `iojs+release` that
+stopped notarizing. Exit 3 means a check or a submission could not complete:
+download, extraction, a rejected or timed-out notarytool submission, or a
+binary not signed by the expected team; read the build output for which.
+Notify `release-validation-alert` on both, as for `validate-downloads`.

doc/release-overview.md

@@ -159,5 +159,40 @@ flowchart TD
     r2Staging ~~~ website
 ```
 
+## macOS code signing and notarization
+
+The macOS release VMs are built from
+[`orka/templates/macos-release.pkr.hcl`](../orka/templates/macos-release.pkr.hcl),
+which installs the Developer ID certificate and a `notarytool` keychain
+profile (`NODE_RELEASE_PROFILE`) on every release image, pkg and tarball nodes
+alike. In `nodejs/node`, `make pkg` and `make binary` codesign `bin/node` and
+call `tools/osx-notarize.sh` on their output; the pkg gets its ticket stapled,
+the tarball binary cannot and relies on Gatekeeper fetching the ticket by
+code directory hash, so its contents and checksums are unaffected.
+
+The script only runs when `NOTARIZATION_ID`, `NOTARIZATION_PASSWORD` and
+`NOTARIZATION_TEAM_ID` are set. `iojs+release` injects them from a
+conditional step per node type; the tarball steps need them as well as the
+pkg step, otherwise tarballs ship signed but not notarized
+([nodejs/node#67140](https://github.com/nodejs/node/issues/67140)). A
+successful tarball axis logs `Verification was successful.`; `Skipping
+notarization` means the variables were not injected.
+
+Tarball binaries often receive a ticket anyway, days after the release,
+when a third party notarizes an app that bundles them; the window right after
+a release and any version nobody bundles stay uncovered, which is what the
+build step closes. To check a published release from any Mac (the script asks
+Apple's ticket service by code directory hash; `codesign -R=notarized` alone
+only reads the local cache), or to backfill tickets for already published
+tarballs from a machine with the keychain profile:
+
+```shell
+tools/macos-tarball-notarization.sh v26.13.0
+tools/macos-tarball-notarization.sh --submit --all-supported --latest 10
+```
+
+The scheduled `notarize-darwin-tarballs` job in [jenkins-jobs.md](./jenkins-jobs.md)
+runs the second form nightly.
+
 [nodejs.org]: https://nodejs.org
 [release process]: https://github.com/nodejs/node/blob/main/doc/contributing/releases.md

orka/templates/README.md

@@ -57,6 +57,8 @@ We need the private key for node-www for the release images, as well as the appl
 
     Xcode Command-line tools are not enough to perform a full notarization cycle, full Xcode must be fully installed on the release images.
 
+    The release image is used by both the `.pkg` and the tarball nodes of `iojs+release`, and both notarize their output, so every release image needs full Xcode and the `NODE_RELEASE_PROFILE` notarytool profile that this template stores in the System keychain.
+
     * Login to https://developer.apple.com using the apple-operations@openjsf.org account
     * Download Xcode: https://developer.apple.com/download/more/ - find non-beta version, open Developer Tools in browser, Networking tab, start download (then cancel), in Networking tab "Copy as cURL" (available in Chrome & FF)
         * On OSX 15 we currently install 16.4