Repository navigation
Staged builds for pull requests and releases - #216
Draft
EzraBrooks wants to merge 6 commits into
Draft
EzraBrooks wants to merge 6 commits into
EzraBrooks wants to merge 6 commits into
Conversation
The PR build of each distribution and platform is no longer one job that needs re-running after six hours: a plan job lists vinca-gha's stages (--json), pr-stages.yaml runs them in fixed stage slots, one matrix job per batch of recipes, and the packages pass between the stages as artifacts, read as a local channel. A collect job saves them as the PR's build cache and runs the ABI check. pr-stages.yaml is generated by tools/pr_stages.py (rs check checks it). vinca is pinned to the --json change on top of the previous pin.
…requests and releases staged-build.yaml (tools/staged_build.py) has 64 stage slots shared by all distributions and platforms instead of 64 per distribution and platform. Each job reads its recipes from the schedule, downloads only the packages of the jobs it needs (vinca-gha's needs), and fails when one of them failed, so the other distributions and platforms go on. Jobs that succeeded in an earlier attempt skip their build when later stages are re-run. release.yaml runs the same workflow with uploads, by hand while the build branches still build the releases.
download-artifact's pattern only sees a run's first 1,000 artifacts: the stage jobs and collect download the artifacts they need by name (tools/download_artifacts.cjs), and the build log is an artifact only after a failure. A failed recipe now holds back only the recipes that need it (vinca-gha's requires): the job builds the rest with --continue-on-failure=skip-dependents, and status/<job>.json records each recipe's outcome for the later jobs. Releases upload the packages of the recipes that built.
…b isn't built either rattler-build's skip-dependents skips the outputs that need a failed one to build, not the ones that only run with it, e.g. ros-<distro>-<name> of ros2-<name>.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull requests, and by hand for now releases, build in vinca-gha's dependency stages, one job per batch of recipes. Today each distribution and platform gets one six-hour job that needs re-running. Releases use the same workflow and only add the uploads, so the build branches can retire once a release run has uploaded correctly.
How (
staged-build.yaml, generated bytools/staged_build.py workflow;rs checkverifies it is up to date):vinca-gha --json(vinca-gha --json: write the batched stages instead of a workflow vinca#201). Each job carriesneeds, the earlier jobs building its requirements.stage-job.yaml: downloads only the artifacts of the jobs it needs (directly and transitively, plus the restored cache, with onedownload-artifactbrace pattern; no REST API calls, so no rate limit and fork PRs work), indexes them as a local channel (ROBOSTACK_LOCAL_CHANNELinrattler_build()), builds withrs build-ci, and uploadspkgs-<distro>-<platform>-<job>with astatus/<job>file.!failure(), so other distributions and platforms keep building.rs upload: prefix.dev trusted publishing, plus the distribution's anaconda token viasecrets[...]).noarch/, as the PR cache, then runs the ABI check and thers ghacheck (while the build branches remain).release.yaml:workflow_dispatchwith optional distributions and platforms, calling the same workflow withupload: true.Notes
EzraBrooks/vinca@0a04e39: vinca-gha --json: write the batched stages instead of a workflow vinca#201 cherry-picked onto the current pin. Repin after it merges..github/workflows/stage-job.yaml; today it allows the build branches'build.yaml.staged-…), so open PRs start from an empty cache once.tools/ci_build.py(the timeout wrapper) is no longer used by PR builds.