Skip to content

fix: treat VolumeLimitExceeded as a retryable scale-up error - #5598

Open
tehlers320 wants to merge 2 commits into
github-aws-runners:mainfrom
tehlers320:fix/retry-volume-limit-exceeded
Open

tehlers320 wants to merge 2 commits into
github-aws-runners:mainfrom
tehlers320:fix/retry-volume-limit-exceeded

Conversation

@tehlers320

Copy link
Copy Markdown
Contributor

Description

Add VolumeLimitExceeded to the default scale_errors list.

When an account reaches its EBS volume quota (for example the gp3 storage limit during a burst of runners), RunInstances fails with VolumeLimitExceeded. The error is not in the default retry set, so the scale-up Lambda treats it as non-retryable and the SQS message is deleted instead of requeued. The queued job is then lost. VolumeLimitExceeded is the same class of transient account-limit error as InsufficientInstanceCapacity and ResourceLimitExceeded, which are already retried.

The default list is duplicated, so the value is added in each place:

  • root variables.tf
  • modules/multi-runner/variables.tf (top-level and per-runner schema)
  • modules/runners/variables.tf
  • modules/runner-config/variables.compute-provider.tf
  • modules/compute-providers/aws/ec2/variables.tf

Anyone who sets scale_errors explicitly is unaffected, since that list replaces the default.

Test Plan

  • terraform validate on the root, multi-runner, runners, runner-config and compute-providers/aws/ec2 (Terraform 1.9.8); terraform fmt -check clean.
  • terraform test in modules/multi-runner: 21 passed.
  • Observed in a load test: with the account at its gp3 quota, RunInstances returned VolumeLimitExceeded and the affected jobs were dropped rather than retried.
  • Not done: no deployment of this change. README default tables are left to the docs bot.

Related Issues

None found. I searched open and closed issues and PRs for VolumeLimitExceeded before opening this. #5578 and #5304 also touch scale_errors handling but are not about this value.

When an account hits its EBS volume quota (for example the gp3 storage
limit during a burst of runners), RunInstances fails with
VolumeLimitExceeded. The error is not in the default scale_errors set, so the
scale-up Lambda classifies it as non-retryable and the SQS message is deleted
instead of requeued, which permanently loses the job.

VolumeLimitExceeded is the same class of transient account-limit error as
InsufficientInstanceCapacity and ResourceLimitExceeded, which are already
retried. Add it to every place the default list is duplicated: the root module,
multi-runner (top-level and per-runner schema), runners, runner-config and the
EC2 compute provider.

Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
@tehlers320
tehlers320 requested review from a team as code owners November 17, 2026 22:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant