Skip to content

fix: honor boot grace before scale-down identity lookup - #5583

Merged
guicaulada merged 2 commits into
mainfrom
gc/scale-down-registration-grace
Nov 10, 2026
Merged

guicaulada merged 2 commits into
mainfrom
gc/scale-down-registration-grace

Conversation

@guicaulada

Copy link
Copy Markdown
Contributor

Scale-down can emit UNVERIFIABLE_RUNNER for a healthy instance that is still registering. On v8.1.0 we observed an instance launch at 19:46:33 UTC, an exact-name lookup miss at 19:46:55, and successful JIT registration at 19:47:04. Identity verification currently runs before the boot-grace check, so an expected startup delay is logged as an error and consumes GitHub requests.

Defer identity lookup for runners without a valid GitHub runner ID until the compute provider's boot grace expires. Runners with a valid ID continue through normal evaluation, and an exact-name miss after the grace period still retains the instance conservatively.

Regression coverage exercises registration arriving during the grace period and an unresolved identity after grace expiry, asserting that the early sweep makes no GitHub client/lookup calls and does not mark or terminate the instance.

@guicaulada
guicaulada requested a review from a team as a code owner November 6, 2026 19:59
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified in the change to defer runner identity lookup during boot grace.

No open issues 🎉

What changed in this PR

This PR makes scale-down honor the compute provider’s boot grace before looking up runners without a valid GitHub runner ID, avoiding premature errors and unnecessary GitHub requests during registration.

Changes:

  • Defer identity lookup during boot grace while preserving normal evaluation for runners with valid IDs.
  • Add regression coverage for registration during grace and conservative retention after grace expires.
  • Disable boot grace in MiniStack smoke tests to preserve immediate-cleanup assertions.
File Description
tests/​ministack/​run-smoke.sh Sets boot grace to zero for immediate cleanup.
tests/​ministack/​README.md Explains smoke-test timing settings and separate boot-grace coverage.
lambdas/​functions/​control-plane/​src/​scale-runners/​scale-down.ts Skips identity lookup for booting runners without valid GitHub IDs.
lambdas/​functions/​control-plane/​src/​scale-runners/​scale-down.test.ts Tests deferred lookup, subsequent registration, and retention after grace expiry.

Review effort: Balanced

@guicaulada
guicaulada merged commit e7aa391 into main Nov 10, 2026
16 checks passed
@guicaulada
guicaulada deleted the gc/scale-down-registration-grace branch November 10, 2026 15:11
guicaulada pushed a commit that referenced this pull request Nov 12, 2026
🤖 I have created a release *beep* *boop*
---


##
[8.1.1](v8.1.0...v8.1.1)
(2026-11-12)


### Bug Fixes

* honor boot grace before scale-down identity lookup
([#5583](#5583))
([e7aa391](e7aa391))
* preserve sensitive bootstrap boundaries in v2 provider selection
([#5588](#5588))
([1381802](1381802))
* **runners:** Skip runner file ownership change when user_name is unset
([#5574](#5574))
([2cf37e5](2cf37e5))
* **termination-watcher:** look up the runner by exact name instead of
paginating all runners
([#5573](#5573))
([f6fc4e6](f6fc4e6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants