Skip to content

Staged builds for pull requests and releases - #216

Draft
EzraBrooks wants to merge 6 commits into
RoboStack:mainfrom
EzraBrooks:pr-staged-builds
Draft

EzraBrooks wants to merge 6 commits into
RoboStack:mainfrom
EzraBrooks:pr-staged-builds

Conversation

@EzraBrooks

@EzraBrooks EzraBrooks commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Pull requests, and by hand for now releases, build in vinca-gha's dependency stages, one job per batch of recipes. Today each distribution and platform gets one six-hour job that needs re-running. Releases use the same workflow and only add the uploads, so the build branches can retire once a release run has uploaded correctly.

How (staged-build.yaml, generated by tools/staged_build.py workflow; rs check verifies it is up to date):

  • plan (one job per distribution × platform): generates the recipes once (artifact), restores the PR cache (PRs only), leaves out the recipes whose every variant is cached, and lists the jobs with vinca-gha --json (vinca-gha --json: write the batched stages instead of a workflow vinca#201). Each job carries needs, the earlier jobs building its requirements.
  • schedule: puts every plan into one shared set of 64 stage slots. Slot k runs stage k of every distribution and platform as one matrix, capped at 256 jobs by merging the smallest batches. It also writes the schedule each job reads its recipes from. A run is about 20 plan jobs + schedule + 64 slots + 20 collect jobs, no longer 20 × 66.
  • stage_k → stage-job.yaml: downloads only the artifacts of the jobs it needs (directly and transitively, plus the restored cache, with one download-artifact brace pattern; no REST API calls, so no rate limit and fork PRs work), indexes them as a local channel (ROBOSTACK_LOCAL_CHANNEL in rattler_build()), builds with rs build-ci, and uploads pkgs-<distro>-<platform>-<job> with a status/<job> file.
    • A job whose requirements failed fails at once. Slots don't use !failure(), so other distributions and platforms keep building.
    • "Re-run failed jobs" re-runs every later slot; jobs that already succeeded find their own artifact and skip the build.
    • In release mode the job also uploads to the channels (rs upload: prefix.dev trusted publishing, plus the distribution's anaconda token via secrets[...]).
  • collect (PRs only, per distribution × platform): saves the packages, now including noarch/, as the PR cache, then runs the ABI check and the rs gha check (while the build branches remain).
  • release.yaml: workflow_dispatch with optional distributions and platforms, calling the same workflow with upload: true.

Notes

  • vinca is pinned to EzraBrooks/vinca@0a04e39: vinca-gha --json: write the batched stages instead of a workflow vinca#201 cherry-picked onto the current pin. Repin after it merges.
  • Before a release run, prefix.dev's trusted publishing (Repository Access) must allow .github/workflows/stage-job.yaml; today it allows the build branches' build.yaml.
  • Slots are a fixed ceiling (the workflow can't change once a run starts). Lyrical's full linux-64 build has 53 stages; stages past slot 63 fold into one job.
  • Downloading only the needed artifacts saves less than hoped: a lyrical job needs 115 of the 135 earlier jobs on average, because almost everything depends on the core packages.
  • The PR cache key changes (staged-…), so open PRs start from an empty cache once.
  • tools/ci_build.py (the timeout wrapper) is no longer used by PR builds.

The PR build of each distribution and platform is no longer one job that needs re-running after six hours: a plan job lists vinca-gha's stages (--json), pr-stages.yaml runs them in fixed stage slots, one matrix job per batch of recipes, and the packages pass between the stages as artifacts, read as a local channel. A collect job saves them as the PR's build cache and runs the ABI check.

pr-stages.yaml is generated by tools/pr_stages.py (rs check checks it). vinca is pinned to the --json change on top of the previous pin.
…requests and releases

staged-build.yaml (tools/staged_build.py) has 64 stage slots shared by all distributions and platforms instead of 64 per distribution and platform. Each job reads its recipes from the schedule, downloads only the packages of the jobs it needs (vinca-gha's needs), and fails when one of them failed, so the other distributions and platforms go on. Jobs that succeeded in an earlier attempt skip their build when later stages are re-run.

release.yaml runs the same workflow with uploads, by hand while the build branches still build the releases.
@EzraBrooks EzraBrooks changed the title Build pull requests in the build branches' stages Staged builds for pull requests and releases Nov 13, 2026
download-artifact's pattern only sees a run's first 1,000 artifacts: the stage jobs
and collect download the artifacts they need by name (tools/download_artifacts.cjs),
and the build log is an artifact only after a failure.

A failed recipe now holds back only the recipes that need it (vinca-gha's requires):
the job builds the rest with --continue-on-failure=skip-dependents, and
status/<job>.json records each recipe's outcome for the later jobs. Releases upload
the packages of the recipes that built.
…b isn't built either

rattler-build's skip-dependents skips the outputs that need a failed one to build, not
the ones that only run with it, e.g. ros-<distro>-<name> of ros2-<name>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant