Skip to content

chore: authenticate the artifacts API call in the PilotWrapper workflow - #8879

Merged
aldbr merged 1 commit into
DIRACGrid:integrationfrom
fstagni:ci/pilotwrapper-artifact-token
Nov 10, 2026
Merged

aldbr merged 1 commit into
DIRACGrid:integrationfrom
fstagni:ci/pilotwrapper-artifact-token

Conversation

@fstagni

@fstagni fstagni commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

CI fix: authenticate the artifacts API listing call

The run pilot wrapper test step lists the Pilot artifacts with an unauthenticated curl to the GitHub API — rate limited to 60 requests/hour per IP, and runner IPs are shared, so it intermittently returns an error body without the artifacts key:

jq: error (at <stdin>:1): Cannot iterate over null (null)
##[error]Process completed with exit code 5.

Seen on PR #8870 (PilotWrapper (3.14.6, devel) failed while its master twin passed in the same run, and the same leg had passed 40 minutes earlier).

The artifact download a few lines below already authenticates with ${{ secrets.GITHUB_TOKEN }}: this adds the same header to the listing call. One line, CI only.

Evidence

Merge Danger

Door: two-way — adds an auth header to an API call; at worst the same flakiness as today.

Blast Radius: CI only.

Comment on lines 47 to 79
- name: run pilot wrapper test
run: |
export INVALID_UTF8_VAR=$'\xff'
cp tests/Integration/WorkloadManagementSystem/Test_GenerateAndExecutePilotWrapper.py .
eval "$(conda shell.bash hook)" && conda activate python_${{ matrix.python }}
# use github APIs to get the artifacts URLS from https://github.com/DIRACGrid/Pilot/, for those named Pilot_${{ matrix.pilot_branch }}
url=$(curl -s https://api.github.com/repos/DIRACGrid/Pilot/actions/artifacts | jq -r '.artifacts[] | select(.name == "Pilot_${{ matrix.pilot_branch }}") | .archive_download_url')
# authenticate: unauthenticated API calls are rate limited per IP, which
# makes this step flaky on shared runner IPs
url=$(curl -s -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" https://api.github.com/repos/DIRACGrid/Pilot/actions/artifacts | jq -r '.artifacts[] | select(.name == "Pilot_${{ matrix.pilot_branch }}") | .archive_download_url')
echo $url

# download and unzip the url above
curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
-H "X-GitHub-Api-Version: 2022-11-28" \
$url --output Pilot_${{ matrix.pilot_branch }}.zip

file_type=$(file --mime-type -b Pilot_${{ matrix.pilot_branch }}.zip)

if [ "$file_type" != "application/zip" ]; then
echo "The downloaded file is not a ZIP file. File type: $file_type"
exit 1
fi


mkdir -p ${{ matrix.pilot_branch }}/pilot
cp Pilot_${{ matrix.pilot_branch }}.zip ${{ matrix.pilot_branch }}/pilot
cd ${{ matrix.pilot_branch }}/pilot
unzip Pilot_${{ matrix.pilot_branch }}.zip
cd ../..

python Test_GenerateAndExecutePilotWrapper.py file://${{ github.workspace }}/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py file://${{ github.workspace }}/${{ matrix.pilot_branch }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using gh?

- name: run pilot wrapper test
    env: 
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |
        export INVALID_UTF8_VAR=$'\xff'
        cp tests/Integration/WorkloadManagementSystem/Test_GenerateAndExecutePilotWrapper.py .
        eval "$(conda shell.bash hook)" && conda activate python_${{ matrix.python }}
        gh run download -R DIRACGrid/Pilot -n Pilot_${{ matrix.pilot_branch }} -D ${{ matrix.pilot_branch }}/pilot
        python Test_GenerateAndExecutePilotWrapper.py file://${{ github.workspace }}/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py file://${{ github.workspace }}/${{ matrix.pilot_branch }}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented on this branch (and folded into the rel-v8r0/rel-v9r0 backports): the whole listing+download+unzip block is replaced by gh run download -R DIRACGrid/Pilot -n Pilot_${{ matrix.pilot_branch }} -D ... with GH_TOKEN in the step env — authenticated, no rate limits, and it extracts the artifact directly, so the zip handling and the file-type check go away. Verified locally that gh run download -n locates the named artifact even when the most recent runs of the Pilot repo don't produce it (the artifacts come from the Pilot Nightly).

@fstagni
fstagni force-pushed the ci/pilotwrapper-artifact-token branch from 69259b0 to 14ff047 Compare November 9, 2026 16:17
Replaces the unauthenticated artifacts listing (rate limited per IP,
which intermittently flaked this job with 'jq: Cannot iterate over
null') and the manual zip download/extraction with 'gh run download',
authenticated with the workflow token. Suggested by aldbr in review.
@fstagni
fstagni force-pushed the ci/pilotwrapper-artifact-token branch from 14ff047 to f32dbe0 Compare November 9, 2026 16:31
fstagni added a commit to fstagni/Pilot that referenced this pull request Nov 9, 2026
The four calls listing the DIRAC releases are unauthenticated and
therefore rate limited per IP; on shared runner IPs they intermittently
return an error body, leaving the DIRAC release version empty and
failing the job. Authenticate them with the workflow token (same
treatment as DIRACGrid/DIRAC#8879 for the Pilot artifacts listing).
fstagni added a commit to fstagni/Pilot that referenced this pull request Nov 9, 2026
The four calls listing the DIRAC releases were unauthenticated and
therefore rate limited per IP; on shared runner IPs they intermittently
return an error body, leaving the DIRAC release version empty and
failing the job. Use 'gh api' with the workflow token instead (same
treatment as DIRACGrid/DIRAC#8879 for the Pilot artifacts listing).
@aldbr
aldbr merged commit 366ad67 into DIRACGrid:integration Nov 10, 2026
24 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweep:ignore Prevent sweeping from being ran for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants