Repository navigation
chore: authenticate the artifacts API call in the PilotWrapper workflow - #8879
Merged
aldbr merged 1 commit intoNov 10, 2026
Merged
Conversation
This was referenced Nov 9, 2026
aldbr
reviewed
Nov 9, 2026
Comment on lines
47
to
79
| - name: run pilot wrapper test | ||
| run: | | ||
| export INVALID_UTF8_VAR=$'\xff' | ||
| cp tests/Integration/WorkloadManagementSystem/Test_GenerateAndExecutePilotWrapper.py . | ||
| eval "$(conda shell.bash hook)" && conda activate python_${{ matrix.python }} | ||
| # use github APIs to get the artifacts URLS from https://github.com/DIRACGrid/Pilot/, for those named Pilot_${{ matrix.pilot_branch }} | ||
| url=$(curl -s https://api.github.com/repos/DIRACGrid/Pilot/actions/artifacts | jq -r '.artifacts[] | select(.name == "Pilot_${{ matrix.pilot_branch }}") | .archive_download_url') | ||
| # authenticate: unauthenticated API calls are rate limited per IP, which | ||
| # makes this step flaky on shared runner IPs | ||
| url=$(curl -s -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" https://api.github.com/repos/DIRACGrid/Pilot/actions/artifacts | jq -r '.artifacts[] | select(.name == "Pilot_${{ matrix.pilot_branch }}") | .archive_download_url') | ||
| echo $url | ||
|
|
||
| # download and unzip the url above | ||
| curl -L \ | ||
| -H "Accept: application/vnd.github+json" \ | ||
| -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \ | ||
| -H "X-GitHub-Api-Version: 2022-11-28" \ | ||
| $url --output Pilot_${{ matrix.pilot_branch }}.zip | ||
|
|
||
| file_type=$(file --mime-type -b Pilot_${{ matrix.pilot_branch }}.zip) | ||
|
|
||
| if [ "$file_type" != "application/zip" ]; then | ||
| echo "The downloaded file is not a ZIP file. File type: $file_type" | ||
| exit 1 | ||
| fi | ||
|
|
||
|
|
||
| mkdir -p ${{ matrix.pilot_branch }}/pilot | ||
| cp Pilot_${{ matrix.pilot_branch }}.zip ${{ matrix.pilot_branch }}/pilot | ||
| cd ${{ matrix.pilot_branch }}/pilot | ||
| unzip Pilot_${{ matrix.pilot_branch }}.zip | ||
| cd ../.. | ||
|
|
||
| python Test_GenerateAndExecutePilotWrapper.py file://${{ github.workspace }}/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py file://${{ github.workspace }}/${{ matrix.pilot_branch }} |
Contributor
There was a problem hiding this comment.
What about using gh?
- name: run pilot wrapper test
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
export INVALID_UTF8_VAR=$'\xff'
cp tests/Integration/WorkloadManagementSystem/Test_GenerateAndExecutePilotWrapper.py .
eval "$(conda shell.bash hook)" && conda activate python_${{ matrix.python }}
gh run download -R DIRACGrid/Pilot -n Pilot_${{ matrix.pilot_branch }} -D ${{ matrix.pilot_branch }}/pilot
python Test_GenerateAndExecutePilotWrapper.py file://${{ github.workspace }}/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py file://${{ github.workspace }}/${{ matrix.pilot_branch }}
Contributor
Author
There was a problem hiding this comment.
Implemented on this branch (and folded into the rel-v8r0/rel-v9r0 backports): the whole listing+download+unzip block is replaced by gh run download -R DIRACGrid/Pilot -n Pilot_${{ matrix.pilot_branch }} -D ... with GH_TOKEN in the step env — authenticated, no rate limits, and it extracts the artifact directly, so the zip handling and the file-type check go away. Verified locally that gh run download -n locates the named artifact even when the most recent runs of the Pilot repo don't produce it (the artifacts come from the Pilot Nightly).
fstagni
force-pushed
the
ci/pilotwrapper-artifact-token
branch
from
November 9, 2026 16:17
69259b0 to
14ff047
Compare
Replaces the unauthenticated artifacts listing (rate limited per IP, which intermittently flaked this job with 'jq: Cannot iterate over null') and the manual zip download/extraction with 'gh run download', authenticated with the workflow token. Suggested by aldbr in review.
fstagni
force-pushed
the
ci/pilotwrapper-artifact-token
branch
from
November 9, 2026 16:31
14ff047 to
f32dbe0
Compare
fstagni
added a commit
to fstagni/Pilot
that referenced
this pull request
Nov 9, 2026
The four calls listing the DIRAC releases are unauthenticated and therefore rate limited per IP; on shared runner IPs they intermittently return an error body, leaving the DIRAC release version empty and failing the job. Authenticate them with the workflow token (same treatment as DIRACGrid/DIRAC#8879 for the Pilot artifacts listing).
fstagni
added a commit
to fstagni/Pilot
that referenced
this pull request
Nov 9, 2026
The four calls listing the DIRAC releases were unauthenticated and therefore rate limited per IP; on shared runner IPs they intermittently return an error body, leaving the DIRAC release version empty and failing the job. Use 'gh api' with the workflow token instead (same treatment as DIRACGrid/DIRAC#8879 for the Pilot artifacts listing).
aldbr
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI fix: authenticate the artifacts API listing call
The
run pilot wrapper teststep lists the Pilot artifacts with an unauthenticatedcurlto the GitHub API — rate limited to 60 requests/hour per IP, and runner IPs are shared, so it intermittently returns an error body without theartifactskey:Seen on PR #8870 (
PilotWrapper (3.14.6, devel)failed while itsmastertwin passed in the same run, and the same leg had passed 40 minutes earlier).The artifact download a few lines below already authenticates with
${{ secrets.GITHUB_TOKEN }}: this adds the same header to the listing call. One line, CI only.Evidence
129226434925(run 42636648792, PR CPU power from the HEPiX HS23 table (phase 1: record and report) #8870): jq null-iteration error immediately after the listingcurl.masterleg passed — classic per-IP rate-limit flakiness, not content.Merge Danger
Door: two-way — adds an auth header to an API call; at worst the same flakiness as today.
Blast Radius: CI only.