Repository navigation
feat(storage): DynamoDB runner config storage for multi-runner v1 (builds on #5279, #5458, #5266) - #5597
Open
tehlers320 wants to merge 11 commits into
Open
feat(storage): DynamoDB runner config storage for multi-runner v1 (builds on #5279, #5458, #5266)#5597tehlers320 wants to merge 11 commits into
tehlers320 wants to merge 11 commits into
Conversation
(cherry picked from commit 8b2fef4) Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit fc9cf6d) Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit 409206e) Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit 9e04ba1) Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
Prerequisite contract for the DynamoDB runner config store ported from github-aws-runners#5279/github-aws-runners#5458 (see review on github-aws-runners#5458: core lacked RunnerConfigRecord.accessScope). Optional, so the SSM provider is unaffected. Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ders Compose the DynamoDB runner config, runner group cache and GitHub App credentials stores when RUNNER_CONFIG_STORAGE_PROVIDER=aws_dynamodb, and make RunnerConfigStorage.consumer optional (EC2 runners consume their own config in the instance bootstrap, the control plane never calls it). Builds on github-aws-runners#5279 / github-aws-runners#5458 (edersonbrilhante) and github-aws-runners#5266 (wadherv). Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tehlers320
marked this pull request as ready for review
November 17, 2026 21:29
Adapted to v8.1.1 (changes by the fork maintainer): - keep upstream's storage_provider capability hooks in orchestration-providers as they are - merge the DynamoDB capabilities into the existing storage_provider variables of modules/webhook, compute-providers/aws/ec2 and runner-config instead of adding second variables - extend global_config_storage_provider with an optional aws.dynamodb block and forward the selected type and per-entry capabilities from multi-runner through runner-config - fix the ec2 module test fixtures for the for_each launch template and merged variable (cherry picked from commit 4148f80) Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
…ig store The DynamoDB runner config store keys one-time registration/JIT config on the compute identity so that IAM can restrict each runner to its own record. Add an optional getRunnerConfigAccessScope hook to StartRunnerConfigOptions and have the EC2 provider supply the instance ARN from EC2_INSTANCE_ARN_PREFIX (set by the DynamoDB wiring). SSM deployments never set it and keep writing the same record. Based on the accessScope handling in github-aws-runners#5279. Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Consolidates the open DynamoDB runner-config work onto
v8.1.1so that the experimental v2 config (global_config_storage_provider) can use DynamoDB end to end, for deployments hitting SSM API rate limits at scale (see #5243).The existing drafts overlap and all conflict with
main, so this takes the parts that apply tov8.1.1and keeps their original authors:feat(storage): add DynamoDB provider(stores, per-capability factories, tests)-x, original author keptmodules/storage-providers/aws/dynamodbTerraform module (tables, IAM, KMS), dependency / Terraform 1.5 fix-x, original author keptfix(storage): add KMS permissions for DynamoDB consumers(review by @guicaulada)-x, original author keptfeat(terraform): wire shared DynamoDB storage(multi-runner composition, webhook module capabilities, EC2 bootstrap scripts for Linux / macOS / Windows)-x, original author kept, adapted tov8.1.1(see below)Each commit carries a DCO
Signed-off-byfrom the fork owner as committer; the author of every cherry-picked commit is unchanged.What the adaptation changes on top of #5279
v8.1.1already contains the capability hooks inorchestration-providers(storage_provider.scale_up/scale_down/pool/job_retry), so those modules are left as they are.storage_providervariables ofmodules/webhook,compute-providers/aws/ec2andrunner-configinstead of adding second variables.global_config_storage_providergains an optionalaws.dynamodbblock. Setting it selectsaws_dynamodb; omitting it keeps SSM, so existing deployments are unchanged.multi-runnerforwards the selected type and per-entry capabilities throughrunner-configto the webhook, control-plane and EC2 modules.createStorageProviders()/createCommonStorage()select the DynamoDB stores,RunnerConfigRecord.accessScopeis added to the core contract (the prerequisite @guicaulada pointed out on feat(storage): add AWS DynamoDB storage provider #5458),RunnerConfigStorage.consumerbecomes optional (EC2 runners consume their own config in the instance bootstrap), and the EC2 provider supplies the instance ARN as the access scope fromEC2_INSTANCE_ARN_PREFIX.Review feedback already covered by this design
dynamodb:LeadingKeyswith${ec2:SourceInstanceARN}(asked for by @Brend-Smits on feat(storage): add shared DynamoDB runner storage #5279 and by @guicaulada / Copilot on feat: add optional DynamoDB runner config storage #5266).DeleteItemthat requiresexpires_at > nowand returns the old item, so an expired-but-not-yet-deleted record is never used.Deliberately not included
scale-downchanges. They conflict with the reworked scale-down in 8.x (deadlines, pagination, boot grace), and the orphan-retry bug @guicaulada reported on feat(storage): add shared DynamoDB runner storage #5279 is unresolved there.multi_runner_configpath stays on SSM:storage_provider_typeis only DynamoDB when the v2 config selects it, which fits the plan to deprecate multi-runner v1.Test Plan
terraform validate: root,multi-runner,webhook,runner-config,compute-providers/aws/ec2,storage-providers/aws/dynamodb(Terraform 1.9.8, freshinit -backend=false);terraform fmt -checkclean.terraform test:storage-providers/aws/dynamodb7/7,compute-providers/aws/ec211/11,multi-runner21/21.vitest:libs/storage-providers226 passed,functions/control-plane493 passed (includes two new tests for the access scope),libs/compute-providers/aws/ec2384 passed.tsc --noEmitfor storage-providers, control-plane, webhook and termination-watcher;eslintclean on the touched packages.terraform applyor any deployment, and no test of the Windows / macOS bootstrap scripts on real instances. README tables are left to the docs bot.Related Issues
DeleteParameterrate limit)