Skip to content

fix(termination-watcher): look up the runner by exact name instead of paginating all runners - #5573

Merged
guicaulada merged 3 commits into
github-aws-runners:mainfrom
ronniecoleman-hadrian:fix/termination-watcher-exact-name-lookup
Nov 12, 2026
Merged

guicaulada merged 3 commits into
github-aws-runners:mainfrom
ronniecoleman-hadrian:fix/termination-watcher-exact-name-lookup

Conversation

@ronniecoleman-hadrian

Copy link
Copy Markdown
Contributor

Description

findRunnerByInstanceId() in the termination watcher iterated every page of the owner's runner list (per_page: 100) and matched name.includes(instanceId). On ephemeral fleets the registration is usually already gone when the shutting-down event arrives, so the common outcome was a full scan that found nothing, at a cost of one request per 100 registered runners, every termination, charged to the installation's shared primary rate limit. Details and measurements in #5572.

This PR resolves the runner by its exact name instead. The runner modules tag every instance with ghr:runner_name_prefix (modules/runners/main.tf), and runners register as <prefix><instance-id>, so the watcher can call the list-runners API with its name filter and get the single matching runner (or none) in one request, independent of fleet size.

  • New findRunnerByName() used when the tag is present; findRunnerByInstanceId() is kept as the fallback for instances without the tag (logged at debug).
  • Exact match is enforced on the response (r.name === runnerName) rather than trusting the filter alone.
  • Delete / busy-runner (422) retry / not-found handling is unchanged.
  • modules/termination-watcher/README.md: short paragraph on how the lookup works and its cost, and a pointer to registration-janitor as the reconciliation path.

No Terraform changes; no change to inputs or behaviour for callers.

Test Plan

  • Added 4 tests to deregister.test.ts covering the exact-name path: org runner found and deleted with { org, name, per_page: 1 } and no paginate.iterator call; repo runner likewise; not-found makes exactly one request and no scan; a non-exact name in the response is ignored. Existing 16 deregister tests (fixtures without the prefix tag) now exercise the fallback path unchanged.
  • yarn nx run-many --target=lint,test --projects=termination-watcher: lint clean, 9 test files / all tests passing (deregister.test.ts 20 tests).
  • yarn nx run @aws-github-runner/termination-watcher:build: builds.
  • Verified the name query parameter is typed on actions/list-self-hosted-runners-for-org and -for-repo in the pinned @octokit/openapi-types.
  • Not deployed to a live environment from this branch; the behaviour it replaces is well characterised in termination-watcher: per-event runner lookup paginates every runner; dominates App rate limit on ephemeral fleets #5572.

Related Issues

Fixes #5572 (item 1). Items 2 and 3 in that issue (default for enable_runner_deregistration in multi-runner, metric emission from a request hook) are left for follow-up discussion.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The deregistration lookup can use an overridden tag prefix that differs from the JIT registration prefix, causing it to miss runners and skip deregistration.

1 open issue
What changed in this PR

This PR reduces termination-watcher API usage by replacing full runner-list scans with exact-name lookups when the instance has a runner-prefix tag, addressing item 1 of #5572.

Changes:

  • Adds name-filtered lookups for organization and repository runners, retaining the scan fallback when the prefix tag is absent.
  • Adds four tests covering successful lookups, missing runners, and non-exact responses.
  • Documents lookup costs and the registration janitor’s reconciliation role.
File Description
modules/​termination-watcher/​README.md Documents lookup behavior, API costs, and reconciliation.
lambdas/​functions/​termination-watcher/​src/​deregister.ts Adds tag-derived exact-name lookup with a legacy scan fallback.
lambdas/​functions/​termination-watcher/​src/​deregister.test.ts Tests exact-name lookup and response validation.

Review effort: Balanced

@Brend-Smits Brend-Smits left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this fix! However, CoPilot is right on this one, can you take another look at this please?

@guicaulada guicaulada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we apply the bounded lookup to handleDeregisterRetry() as well? It still calls findRunnerByInstanceId() and scans the entire runner list. We confirmed production retry failures are 403 rate-limit errors from the same exhausted installation used by the initial watcher. Carrying the resolved runner ID (or a verified exact name) in DeregisterRetryMessage would let retries avoid another org-wide scan, while retaining compatibility for older queued messages. A regression test covering the SQS retry path would help ensure both paths benefit from this fix.

Brend-Smits
Brend-Smits previously approved these changes Nov 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The watcher trusts an instance-writable runner ID tag, allowing a job with instance credentials to trigger deletion of another runner’s registration using the watcher’s GitHub App privileges.

2 open issues
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the changes reducing termination-watcher API usage.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

… paginating all runners

On every instance termination the watcher resolved the runner's GitHub
registration by iterating every page of the owner's runner list and
matching `name.includes(instanceId)`. The cost of that lookup grows with
the number of registered runners (one request per 100), and on an
ephemeral fleet the registration is normally already gone when the
termination event arrives, so the common outcome is a full scan that
finds nothing.

The runner modules tag every instance with `ghr:runner_name_prefix`, and
runners are registered as `<prefix><instance-id>`. Use that to call the
list-runners API with its `name` filter, which returns the single
matching runner (or none) in one request. Instances without the tag keep
the previous paginated scan as a fallback.

The behaviour is otherwise unchanged: a found runner is deleted, a busy
runner (422) is retried through SQS, and a missing runner is logged.
@guicaulada
guicaulada force-pushed the fix/termination-watcher-exact-name-lookup branch from 7c36cff to 14ebbb3 Compare November 12, 2026 17:07
@guicaulada

guicaulada commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Had to re-sign all commits, since the commit by @ronniecoleman-hadrian wasn't signed.

@guicaulada
guicaulada merged commit f6fc4e6 into github-aws-runners:main Nov 12, 2026
58 of 59 checks passed
guicaulada pushed a commit that referenced this pull request Nov 12, 2026
🤖 I have created a release *beep* *boop*
---


##
[8.1.1](v8.1.0...v8.1.1)
(2026-11-12)


### Bug Fixes

* honor boot grace before scale-down identity lookup
([#5583](#5583))
([e7aa391](e7aa391))
* preserve sensitive bootstrap boundaries in v2 provider selection
([#5588](#5588))
([1381802](1381802))
* **runners:** Skip runner file ownership change when user_name is unset
([#5574](#5574))
([2cf37e5](2cf37e5))
* **termination-watcher:** look up the runner by exact name instead of
paginating all runners
([#5573](#5573))
([f6fc4e6](f6fc4e6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

termination-watcher: per-event runner lookup paginates every runner; dominates App rate limit on ephemeral fleets

4 participants