Repository navigation
fix: treat VolumeLimitExceeded as a retryable scale-up error - #5598
Open
tehlers320 wants to merge 2 commits into
Open
tehlers320 wants to merge 2 commits into
tehlers320 wants to merge 2 commits into
Conversation
When an account hits its EBS volume quota (for example the gp3 storage limit during a burst of runners), RunInstances fails with VolumeLimitExceeded. The error is not in the default scale_errors set, so the scale-up Lambda classifies it as non-retryable and the SQS message is deleted instead of requeued, which permanently loses the job. VolumeLimitExceeded is the same class of transient account-limit error as InsufficientInstanceCapacity and ResourceLimitExceeded, which are already retried. Add it to every place the default list is duplicated: the root module, multi-runner (top-level and per-runner schema), runners, runner-config and the EC2 compute provider. Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add
VolumeLimitExceededto the defaultscale_errorslist.When an account reaches its EBS volume quota (for example the gp3 storage limit during a burst of runners),
RunInstancesfails withVolumeLimitExceeded. The error is not in the default retry set, so the scale-up Lambda treats it as non-retryable and the SQS message is deleted instead of requeued. The queued job is then lost.VolumeLimitExceededis the same class of transient account-limit error asInsufficientInstanceCapacityandResourceLimitExceeded, which are already retried.The default list is duplicated, so the value is added in each place:
variables.tfmodules/multi-runner/variables.tf(top-level and per-runner schema)modules/runners/variables.tfmodules/runner-config/variables.compute-provider.tfmodules/compute-providers/aws/ec2/variables.tfAnyone who sets
scale_errorsexplicitly is unaffected, since that list replaces the default.Test Plan
terraform validateon the root,multi-runner,runners,runner-configandcompute-providers/aws/ec2(Terraform 1.9.8);terraform fmt -checkclean.terraform testinmodules/multi-runner: 21 passed.RunInstancesreturnedVolumeLimitExceededand the affected jobs were dropped rather than retried.Related Issues
None found. I searched open and closed issues and PRs for
VolumeLimitExceededbefore opening this. #5578 and #5304 also touchscale_errorshandling but are not about this value.