From c9e0277ace2aa17f1ab892121bbd2aaef59238b8 Mon Sep 17 00:00:00 2001 From: Ulises Gascon Date: Fri, 30 Oct 2026 16:30:12 +0100 Subject: [PATCH] tools,doc: check and backfill notarization of darwin tarballs Add tools/macos-tarball-notarization.sh. Given release versions, or --all-supported for the lines currently in their support window, it downloads the darwin tarballs, verifies the bin/node signature and asks Apple's Gatekeeper ticket service, by code directory hash, whether a notarization ticket exists. That is the lookup macOS itself performs; `codesign --verify -R=notarized` only reads the local ticket cache and reports a notarized binary as unticketed on a machine that never fetched its ticket. A failed lookup is reported as such, never as a missing ticket. With --submit it zips and submits the binaries that lack one through the NODE_RELEASE_PROFILE keychain profile, requires Apple's Accepted verdict, and waits for the ticket. --latest N limits the run to the N most recent releases per line, since every check downloads a tarball. Notarizing a published binary does not modify it, so checksums stay valid; this is how already shipped tarballs get backfilled. Document in release-overview.md how macOS signing and notarization are wired, that tarball binaries usually receive a ticket only days after a release when third parties notarize apps that bundle them, that the NOTARIZATION_* variables in iojs+release must be exported on the tarball axes too, and how to check and backfill a release. Describe in jenkins-jobs.md the scheduled ci-release job that runs the script nightly, which both backfills and detects a tarball axis that stopped notarizing. Note in the orka README that pkg and tarball nodes share the release image. Refs: https://github.com/nodejs/node/issues/67140 Refs: https://github.com/nodejs/node/pull/67143 Refs: https://github.com/nodejs/build/issues/3538 Refs: https://github.com/nodejs/build/issues/4545 Assisted-by: Claude Assisted-by: Codex Signed-off-by: Ulises Gascon --- doc/jenkins-jobs.md | 43 +++ doc/release-overview.md | 35 +++ orka/templates/README.md | 2 + tools/macos-tarball-notarization.sh | 391 ++++++++++++++++++++++++++++ 4 files changed, 471 insertions(+) create mode 100755 tools/macos-tarball-notarization.sh diff --git a/doc/jenkins-jobs.md b/doc/jenkins-jobs.md index 00244199c..64e028951 100644 --- a/doc/jenkins-jobs.md +++ b/doc/jenkins-jobs.md @@ -15,3 +15,46 @@ https://github.com/nodejs/email/blob/HEAD/iojs.org/aliases.json. This job needs to be updated each time a we roll over to a new Current release. + +## notarize-darwin-tarballs (ci-release, to be created) + +Nightly job on ci-release that checks the darwin tarballs of the supported +release lines for an Apple notarization ticket and submits the ones that lack +it, see [nodejs/node#67140](https://github.com/nodejs/node/issues/67140). It +has to run on a macOS release node (full Xcode and the `NODE_RELEASE_PROFILE` +keychain profile), so it cannot live on the test CI. + +Freestyle job restricted to a `macos*-release-*` label, checking out +`nodejs/build`, with the `SSH_RELEASE_*` credential binding from +`iojs+release` and `NOTARIZATION_TEAM_ID` set to the Node.js team id, +running the same keychain unlock `iojs+release` uses (the notary credentials +are read from the keychain profile, so `NOTARIZATION_PASSWORD` is not +needed): + +```shell +security unlock-keychain -p "$SSH_RELEASE_PASSWORD" +tools/macos-tarball-notarization.sh --submit --all-supported --latest 3 +``` + +Each check downloads a tarball, so only the three most recent releases per +line are covered nightly; older ones are a manual run with a larger +`--latest`. Supported lines come from the Release schedule, so the job needs +no editing on roll-over. Submissions are serial: each waits for Apple's +verdict (bounded to 30 minutes by `notarytool --timeout`) and then up to +eight minutes for the ticket to become visible, so 18 binaries can take over +11 hours if Apple is slow on every one. Disable concurrent builds and give the +first run a 16 hour timeout, or run it by hand in two or three batches with +explicit versions. A rerun skips every binary that already has a ticket and +resubmits the rest, so a run that hits the timeout is simply rerun. One hour +is enough once everything is ticketed. A rejected submission prints +Apple's notary log in the build output. + +The ticket check queries Apple's ticket service by code directory hash, the +same lookup macOS performs; a failed lookup is reported as such and never +counted as a missing ticket. Exit 1 means a completed check found a published tarball whose +`bin/node` has no ticket, or an invalid signature: in the nightly `--submit` run that +should not happen and points at a tarball axis of `iojs+release` that +stopped notarizing. Exit 3 means a check or a submission could not complete: +download, extraction, a rejected or timed-out notarytool submission, or a +binary not signed by the expected team; read the build output for which. +Notify `release-validation-alert` on both, as for `validate-downloads`. diff --git a/doc/release-overview.md b/doc/release-overview.md index 57c2c430b..e9da8bddf 100644 --- a/doc/release-overview.md +++ b/doc/release-overview.md @@ -159,5 +159,40 @@ flowchart TD r2Staging ~~~ website ``` +## macOS code signing and notarization + +The macOS release VMs are built from +[`orka/templates/macos-release.pkr.hcl`](../orka/templates/macos-release.pkr.hcl), +which installs the Developer ID certificate and a `notarytool` keychain +profile (`NODE_RELEASE_PROFILE`) on every release image, pkg and tarball nodes +alike. In `nodejs/node`, `make pkg` and `make binary` codesign `bin/node` and +call `tools/osx-notarize.sh` on their output; the pkg gets its ticket stapled, +the tarball binary cannot and relies on Gatekeeper fetching the ticket by +code directory hash, so its contents and checksums are unaffected. + +The script only runs when `NOTARIZATION_ID`, `NOTARIZATION_PASSWORD` and +`NOTARIZATION_TEAM_ID` are set. `iojs+release` injects them from a +conditional step per node type; the tarball steps need them as well as the +pkg step, otherwise tarballs ship signed but not notarized +([nodejs/node#67140](https://github.com/nodejs/node/issues/67140)). A +successful tarball axis logs `Verification was successful.`; `Skipping +notarization` means the variables were not injected. + +Tarball binaries often receive a ticket anyway, days after the release, +when a third party notarizes an app that bundles them; the window right after +a release and any version nobody bundles stay uncovered, which is what the +build step closes. To check a published release from any Mac (the script asks +Apple's ticket service by code directory hash; `codesign -R=notarized` alone +only reads the local cache), or to backfill tickets for already published +tarballs from a machine with the keychain profile: + +```shell +tools/macos-tarball-notarization.sh v26.13.0 +tools/macos-tarball-notarization.sh --submit --all-supported --latest 10 +``` + +The scheduled `notarize-darwin-tarballs` job in [jenkins-jobs.md](./jenkins-jobs.md) +runs the second form nightly. + [nodejs.org]: https://nodejs.org [release process]: https://github.com/nodejs/node/blob/main/doc/contributing/releases.md diff --git a/orka/templates/README.md b/orka/templates/README.md index 975e11620..a6351ed66 100644 --- a/orka/templates/README.md +++ b/orka/templates/README.md @@ -57,6 +57,8 @@ We need the private key for node-www for the release images, as well as the appl Xcode Command-line tools are not enough to perform a full notarization cycle, full Xcode must be fully installed on the release images. + The release image is used by both the `.pkg` and the tarball nodes of `iojs+release`, and both notarize their output, so every release image needs full Xcode and the `NODE_RELEASE_PROFILE` notarytool profile that this template stores in the System keychain. + * Login to https://developer.apple.com using the apple-operations@openjsf.org account * Download Xcode: https://developer.apple.com/download/more/ - find non-beta version, open Developer Tools in browser, Networking tab, start download (then cancel), in Networking tab "Copy as cURL" (available in Chrome & FF) * On OSX 15 we currently install 16.4 diff --git a/tools/macos-tarball-notarization.sh b/tools/macos-tarball-notarization.sh new file mode 100755 index 000000000..7a6636bc7 --- /dev/null +++ b/tools/macos-tarball-notarization.sh @@ -0,0 +1,391 @@ +#!/usr/bin/env bash + +# Check, and optionally backfill, Apple notarization tickets for the bin/node +# binaries shipped in the Node.js darwin release tarballs. +# +# Background: the release tarballs are codesigned but, until nodejs/node +# started notarizing them in the tarball build, nothing submitted their +# bin/node to the notary service. Only the .pkg was notarized, and its +# bin/node is a different build (different install prefix), so its ticket +# does not cover the tarball binary. Gatekeeper looks tickets up by code +# directory hash. See https://github.com/nodejs/node/issues/67140 for the +# full analysis, plus https://github.com/nodejs/build/issues/3538 and +# https://github.com/nodejs/build/issues/4545. +# +# Notarizing an already published binary does not modify it: Apple records +# its code directory hash and Gatekeeper fetches the ticket online. The +# published SHASUMS256.txt stays valid. +# +# Usage: +# tools/macos-tarball-notarization.sh [--submit] [--arch x64|arm64|all] ... +# tools/macos-tarball-notarization.sh [--submit] --all-supported [--latest N] +# +# --submit submit every binary that has no ticket to the Apple +# notary service. Requires a Mac with full Xcode and a +# notarytool keychain profile (default NODE_RELEASE_PROFILE, +# as provisioned on the release images by +# orka/templates/macos-release.pkr.hcl), and +# NOTARIZATION_TEAM_ID set to the team the binaries must +# be signed by. Without --submit the script only reports. +# --arch which tarball architectures to check for the versions +# given on the command line (default all). +# --all-supported instead of explicit versions, take the releases of every +# release line that is currently supported according to +# the Release WG schedule, and for each one only the +# darwin tarballs it actually published. Releases only; +# nightlies are never included. +# --latest N with --all-supported, only the N most recent releases +# of each line. 0 (the default) means all of them. Every +# check downloads one tarball per architecture, so the +# scheduled job uses a small N and a deeper backfill is a +# one-off run with a larger one. +# +# Environment: +# NOTARIZATION_PROFILE keychain profile name (default NODE_RELEASE_PROFILE) +# NOTARIZATION_TEAM_ID required with --submit; binaries whose signature +# is not from this team are reported and skipped +# NODE_DIST_URL, NODE_SCHEDULE_URL +# alternative sources for check mode only, for +# example the nightly tree; refused with --submit +# +# The ticket check queries Apple's Gatekeeper ticket service by code directory +# hash, the same lookup macOS performs, so it needs network access and does +# not depend on what this machine has cached. +# +# Exit status: 0 when every checked binary has a ticket, 1 when at least one +# binary lacks a ticket after a completed check, 2 on usage errors, 3 when an +# infrastructure problem (download, extraction, ticket lookup, notary +# service, signature from an unexpected team) prevented a check from +# completing. + +set -euo pipefail + +default_dist="https://nodejs.org/dist" +default_schedule="https://raw.githubusercontent.com/nodejs/Release/HEAD/schedule.json" + +submit=0 +arch=all +all_supported=0 +latest=0 +versions=() +dist="${NODE_DIST_URL:-$default_dist}" +schedule="${NODE_SCHEDULE_URL:-$default_schedule}" +profile="${NOTARIZATION_PROFILE:-NODE_RELEASE_PROFILE}" +team="${NOTARIZATION_TEAM_ID:-}" + +usage_error() { echo "error: $*" >&2; exit 2; } + +while [ $# -gt 0 ]; do + case "$1" in + --submit) submit=1 ;; + --all-supported) all_supported=1 ;; + --arch) [ $# -ge 2 ] || usage_error "--arch needs a value"; shift; arch="$1" ;; + --arch=*) arch="${1#--arch=}" ;; + --latest) [ $# -ge 2 ] || usage_error "--latest needs a value"; shift; latest="$1" ;; + --latest=*) latest="${1#--latest=}" ;; + -h|--help) sed -n '3,55p' "$0"; exit 0 ;; + v*) versions+=("$1") ;; + *) usage_error "unknown argument $1" ;; + esac + shift +done + +if [ "$all_supported" -eq 0 ] && [ "${#versions[@]}" -eq 0 ]; then + usage_error "give at least one version, for example v26.13.0, or --all-supported" +fi +if [ "$all_supported" -eq 1 ] && [ "${#versions[@]}" -gt 0 ]; then + usage_error "--all-supported cannot be combined with explicit versions" +fi +case "$latest" in + ''|*[!0-9]*) usage_error "--latest needs a non-negative number (0 means all)" ;; +esac +if [ "$latest" -gt 0 ] && [ "$all_supported" -eq 0 ]; then + usage_error "--latest only applies with --all-supported" +fi +case "$arch" in + all) arches="x64 arm64" ;; + x64|arm64) arches="$arch" ;; + *) usage_error "--arch must be x64, arm64 or all" ;; +esac +# Versions become URL and tar member paths. Check mode accepts nightly +# strings such as v28.0.0-nightly20261030abcdef1234; a credentialed run only +# accepts canonical release versions. +for v in "${versions[@]+"${versions[@]}"}"; do + case "$v" in + *[!A-Za-z0-9.-]*) usage_error "invalid version string $v" ;; + esac + if [ "$submit" -eq 1 ]; then + echo "$v" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$' || usage_error "--submit only accepts release versions, got $v" + fi +done +if [ "$submit" -eq 1 ]; then + # A credentialed run must only ever submit what nodejs.org publishes. + [ "$dist" = "$default_dist" ] || usage_error "--submit does not accept NODE_DIST_URL" + [ "$schedule" = "$default_schedule" ] || usage_error "--submit does not accept NODE_SCHEDULE_URL" + [ -n "$team" ] || usage_error "--submit needs NOTARIZATION_TEAM_ID" +fi + +if [ "$(uname -s)" != "Darwin" ]; then + echo "error: this script needs macOS (codesign, ditto, notarytool)" >&2 + exit 2 +fi + +# Temporary workspace, removed on any exit. Same shape as the +# tools/dep_updaters scripts in nodejs/node. Signals exit with their +# conventional status so Jenkins sees an abort, not a success. +workdir=$(mktemp -d 2> /dev/null || mktemp -d -t 'tmp') +cleanup () { + EXIT_CODE=$? + [ -d "$workdir" ] && rm -rf "$workdir" || true + exit $EXIT_CODE +} +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +# Ask Apple's ticket service directly, keyed by the binary's code directory +# hash. This is the lookup syspolicyd performs and it is the only reliable +# check for a bare binary: `codesign --verify -R=notarized` only consults the +# local ticket cache and answers "no" for a notarized binary this machine has +# never fetched a ticket for, and spctl rejects command line tools outright. +# Returns 0 ticket present, 1 no ticket, 2 lookup failed (network, format). +ticket_api="https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup" +has_ticket() { + local cdhash reply + # Exactly one 40-hex CDHash: the tarball binaries are thin. A universal + # binary prints one per slice and would need every slice ticketed, so it + # is refused here rather than half-checked. + cdhash=$(codesign -dvvv "$1" 2>&1 | sed -n 's/^CDHash=//p') + if ! echo "$cdhash" | grep -Eqx '[0-9a-f]{40}' || [ "$(echo "$cdhash" | wc -l)" -ne 1 ]; then + echo " could not read a single CDHash from codesign" + return 2 + fi + reply=$(curl -fsS -X POST "$ticket_api" -H 'Content-Type: application/json' \ + -d "{\"records\":[{\"recordName\":\"2/2/$cdhash\"}]}" 2>/dev/null) || return 2 + # Structural check: one record, for the hash we asked about. Anything else + # is a lookup failure, never "present". The reply travels in the + # environment because stdin carries the program. + TICKET_REPLY="$reply" TICKET_WANT="2/2/$cdhash" python3 -I - <<'PY' +import json, os, sys +want = os.environ["TICKET_WANT"] +try: + records = json.loads(os.environ["TICKET_REPLY"])["records"] +except Exception: + sys.exit(2) +if len(records) != 1 or records[0].get("recordName") != want: + sys.exit(2) +r = records[0] +if r.get("serverErrorCode") == "NOT_FOUND": + sys.exit(1) +if "serverErrorCode" not in r and r.get("recordType") == "DeveloperIDTicket": + sys.exit(0) +sys.exit(2) +PY +} + +signed_by_team() { + codesign -dvv "$1" 2>&1 | grep -Fqx "TeamIdentifier=$team" +} + +# Tickets can take a few minutes to become visible after notarytool reports +# Accepted, so poll with backoff (about 8 minutes in total). +# Returns 0 once the ticket is visible, 1 if it never appeared, 2 if the +# last lookup itself failed (so an Apple outage is not reported as delay). +wait_for_ticket() { + local delay=15 attempt=1 rc=0 + while :; do + rc=0 + has_ticket "$1" || rc=$? + [ "$rc" -eq 0 ] && return 0 + if [ "$attempt" -ge 6 ]; then + return "$rc" + fi + if [ "$rc" -eq 2 ]; then + echo " ticket lookup failed, retrying in ${delay}s (attempt $attempt)" + else + echo " ticket not visible yet, retrying in ${delay}s (attempt $attempt)" + fi + sleep "$delay" + attempt=$((attempt + 1)) + delay=$((delay * 2)) + done +} + +# Zip, submit, remove the zip. Each step guarded so a failure is reported +# where it happens instead of cascading into notarytool with a bad archive. +# notarytool exits 0 whenever the submission completes, including when Apple +# rejects it, so the verdict has to be read from its output: only +# "status: Accepted" counts. On any other verdict the notary log is printed, +# it names the signing or entitlement problem Apple found. +submit_binary() { + local binary="$1" zip="$1.zip" out id + # ditto keeps the code signature intact and is the archiver Apple documents + # for notarization uploads. + if ! ditto -c -k --keepParent "$binary" "$zip"; then + echo " ditto failed to create $zip" + return 1 + fi + out=$(xcrun notarytool submit --keychain-profile "$profile" --wait --timeout 30m "$zip" 2>&1) || { + rm -f "$zip" + echo "$out" | sed 's/^/ | /' + echo " notarytool submission failed or timed out" + return 1 + } + rm -f "$zip" + echo "$out" | sed 's/^/ | /' + if ! echo "$out" | grep -Eq '^ *status: Accepted$'; then + id=$(echo "$out" | sed -n 's/^ *id: //p' | head -1) + echo " notarization NOT accepted by Apple" + if [ -n "$id" ]; then + xcrun notarytool log "$id" --keychain-profile "$profile" 2>&1 | sed 's/^/ | /' || true + fi + return 1 + fi +} + +# Build the list of " " pairs to check into a file, so that a +# failed fetch or resolver is a hard error instead of an empty loop. For +# explicit versions use the requested architectures. For --all-supported read +# the release schedule for the lines whose support window includes today, +# then the dist index for the releases of those lines and the darwin tarballs +# each one actually published (v28+ has no darwin-x64, for example). +resolve_targets() { + if [ "$all_supported" -eq 0 ]; then + for v in "${versions[@]}"; do + for a in $arches; do echo "$v $a"; done + done + return + fi + curl -fsSL "$schedule" -o "$workdir/schedule.json" || return 1 + curl -fsSL "$dist/index.json" -o "$workdir/index.json" || return 1 + python3 -I - "$workdir/schedule.json" "$workdir/index.json" "$latest" <<'PY' +import json, re, sys, datetime +schedule = json.load(open(sys.argv[1])) +index = json.load(open(sys.argv[2])) +latest = int(sys.argv[3]) +release = re.compile(r"^v\d+\.\d+\.\d+$") + +def day(value, line, field): + try: + return datetime.date.fromisoformat(value) + except (TypeError, ValueError): + sys.exit(f"schedule.json: line {line} has no valid {field} date") + +supported = set() +today = datetime.date.today() +for line, info in schedule.items(): + if day(info.get("start"), line, "start") <= today < day(info.get("end"), line, "end"): + supported.add(line) +if not supported: + sys.exit("schedule.json: no release line is in its support window today") + +def key(entry): + return tuple(int(x) for x in entry["version"][1:].split(".")) + +entries = [e for e in index if release.match(e["version"])] +entries.sort(key=key, reverse=True) # do not rely on the index being newest-first +seen = {} +for entry in entries: + major = "v" + str(key(entry)[0]) + if major not in supported: + continue + seen[major] = seen.get(major, 0) + 1 + if latest and seen[major] > latest: + continue + for a in ("x64", "arm64"): + if f"osx-{a}-tar" in entry["files"]: + print(entry["version"], a) +PY +} + +missing=0 # checks completed, binary has no ticket +failed=0 # checks that could not complete (download, extract, notary, team) +checked=0 + +# Check one " " tarball. +check_one() { + local version="$1" a="$2" + local name="node-$version-darwin-$a" + local tarball="$name.tar.gz" + local binary="$workdir/$name/bin/node" + + echo "==> $name" + if ! curl -fsSL -o "$workdir/$tarball" "$dist/$version/$tarball"; then + echo " download failed: $dist/$version/$tarball" + failed=$((failed + 1)) + return + fi + if ! tar -xzf "$workdir/$tarball" -C "$workdir" "$name/bin/node"; then + echo " extract failed: $tarball" + failed=$((failed + 1)) + return + fi + + if ! codesign --verify --strict "$binary" 2>/dev/null; then + echo " codesign: INVALID signature" + missing=$((missing + 1)) + return + fi + + local rc=0 + has_ticket "$binary" || rc=$? # bare call would trip errexit + case "$rc" in + 0) echo " notarization: ticket present"; return ;; + 2) echo " notarization: ticket lookup failed"; failed=$((failed + 1)); return ;; + esac + + if [ "$submit" -eq 0 ]; then + echo " notarization: NO ticket (run with --submit to backfill)" + missing=$((missing + 1)) + return + fi + + if ! signed_by_team "$binary"; then + echo " signature is not from team $team, refusing to submit" + failed=$((failed + 1)) + return + fi + + echo " notarization: no ticket, submitting with profile $profile" + if ! submit_binary "$binary"; then + failed=$((failed + 1)) + return + fi + local wrc=0 + wait_for_ticket "$binary" || wrc=$? + case "$wrc" in + 0) echo " notarization: ticket present after submission" ;; + 2) echo " notarization: accepted by Apple, ticket lookup failing after submission" + failed=$((failed + 1)) ;; + *) echo " notarization: accepted by Apple, ticket still not visible after retries" + failed=$((failed + 1)) ;; + esac +} + +if ! resolve_targets > "$workdir/targets"; then + echo "error: could not resolve the tarballs to check" >&2 + exit 3 +fi + +while read -r version a; do + [ -n "$version" ] || continue + checked=$((checked + 1)) + check_one "$version" "$a" + # Free the disk between binaries; a full --all-supported run is many GB. + rm -rf "$workdir/node-$version-darwin-$a" "$workdir/node-$version-darwin-$a.tar.gz" || true +done < "$workdir/targets" + +if [ "$checked" -eq 0 ]; then + echo "error: no tarballs matched; check the schedule and index" >&2 + exit 2 +fi +echo "checked $checked, without ticket $missing, could not check $failed" +if [ "$failed" -gt 0 ]; then + echo "infrastructure problems prevented some checks; see above" >&2 + exit 3 +fi +if [ "$missing" -gt 0 ]; then + echo "published binaries without a notarization ticket; see above" >&2 + exit 1 +fi +echo "all checked binaries carry a notarization ticket"