Skip to content

feat(storage): DynamoDB runner config storage for multi-runner v1 (builds on #5279, #5458, #5266) - #5597

Open
tehlers320 wants to merge 11 commits into
github-aws-runners:mainfrom
tehlers320:feat/dynamodb-runner-config-storage-v2
Open

tehlers320 wants to merge 11 commits into
github-aws-runners:mainfrom
tehlers320:feat/dynamodb-runner-config-storage-v2

Conversation

@tehlers320

@tehlers320 tehlers320 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Consolidates the open DynamoDB runner-config work onto v8.1.1 so that the experimental v2 config (global_config_storage_provider) can use DynamoDB end to end, for deployments hitting SSM API rate limits at scale (see #5243).

The existing drafts overlap and all conflict with main, so this takes the parts that apply to v8.1.1 and keeps their original authors:

Source What was taken Credit
#5279 (edersonbrilhante) feat(storage): add DynamoDB provider (stores, per-capability factories, tests) cherry-picked with -x, original author kept
#5458 (edersonbrilhante) final DynamoDB stores and the modules/storage-providers/aws/dynamodb Terraform module (tables, IAM, KMS), dependency / Terraform 1.5 fix cherry-picked with -x, original author kept
#5279 fix(storage): add KMS permissions for DynamoDB consumers (review by @guicaulada) cherry-picked with -x, original author kept
#5279 feat(terraform): wire shared DynamoDB storage (multi-runner composition, webhook module capabilities, EC2 bootstrap scripts for Linux / macOS / Windows) cherry-picked with -x, original author kept, adapted to v8.1.1 (see below)
#5266 (wadherv) the idea and the review findings (instance-scoped access, TTL expiry on read, integer validation). Its per-label-table design for the stable v1 path is not carried over no code taken

Each commit carries a DCO Signed-off-by from the fork owner as committer; the author of every cherry-picked commit is unchanged.

What the adaptation changes on top of #5279

  • v8.1.1 already contains the capability hooks in orchestration-providers (storage_provider.scale_up/scale_down/pool/job_retry), so those modules are left as they are.
  • DynamoDB capabilities are merged into the existing storage_provider variables of modules/webhook, compute-providers/aws/ec2 and runner-config instead of adding second variables.
  • global_config_storage_provider gains an optional aws.dynamodb block. Setting it selects aws_dynamodb; omitting it keeps SSM, so existing deployments are unchanged.
  • multi-runner forwards the selected type and per-entry capabilities through runner-config to the webhook, control-plane and EC2 modules.
  • Lambda side: createStorageProviders() / createCommonStorage() select the DynamoDB stores, RunnerConfigRecord.accessScope is added to the core contract (the prerequisite @guicaulada pointed out on feat(storage): add AWS DynamoDB storage provider #5458), RunnerConfigStorage.consumer becomes optional (EC2 runners consume their own config in the instance bootstrap), and the EC2 provider supplies the instance ARN as the access scope from EC2_INSTANCE_ARN_PREFIX.

Review feedback already covered by this design

Deliberately not included

Test Plan

  • terraform validate: root, multi-runner, webhook, runner-config, compute-providers/aws/ec2, storage-providers/aws/dynamodb (Terraform 1.9.8, fresh init -backend=false); terraform fmt -check clean.
  • terraform test: storage-providers/aws/dynamodb 7/7, compute-providers/aws/ec2 11/11, multi-runner 21/21.
  • vitest: libs/storage-providers 226 passed, functions/control-plane 493 passed (includes two new tests for the access scope), libs/compute-providers/aws/ec2 384 passed.
  • tsc --noEmit for storage-providers, control-plane, webhook and termination-watcher; eslint clean on the touched packages.
  • Not done: terraform apply or any deployment, and no test of the Windows / macOS bootstrap scripts on real instances. README tables are left to the docs bot.

Related Issues

edersonbrilhante and others added 8 commits November 17, 2026 21:25
(cherry picked from commit 8b2fef4)
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit fc9cf6d)
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit 409206e)
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
(cherry picked from commit 9e04ba1)
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
Prerequisite contract for the DynamoDB runner config store ported from
github-aws-runners#5279/github-aws-runners#5458 (see review on github-aws-runners#5458: core lacked RunnerConfigRecord.accessScope).
Optional, so the SSM provider is unaffected.

Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ders

Compose the DynamoDB runner config, runner group cache and GitHub App
credentials stores when RUNNER_CONFIG_STORAGE_PROVIDER=aws_dynamodb, and make
RunnerConfigStorage.consumer optional (EC2 runners consume their own config in
the instance bootstrap, the control plane never calls it).

Builds on github-aws-runners#5279 / github-aws-runners#5458 (edersonbrilhante) and github-aws-runners#5266 (wadherv).

Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tehlers320
tehlers320 marked this pull request as ready for review November 17, 2026 21:29
@tehlers320
tehlers320 requested a review from a team as a code owner November 17, 2026 21:29
edersonbrilhante and others added 2 commits November 17, 2026 21:45
Adapted to v8.1.1 (changes by the fork maintainer):
- keep upstream's storage_provider capability hooks in orchestration-providers as they are
- merge the DynamoDB capabilities into the existing storage_provider variables of modules/webhook,
  compute-providers/aws/ec2 and runner-config instead of adding second variables
- extend global_config_storage_provider with an optional aws.dynamodb block and forward the
  selected type and per-entry capabilities from multi-runner through runner-config
- fix the ec2 module test fixtures for the for_each launch template and merged variable

(cherry picked from commit 4148f80)
Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
…ig store

The DynamoDB runner config store keys one-time registration/JIT config on the
compute identity so that IAM can restrict each runner to its own record. Add an
optional getRunnerConfigAccessScope hook to StartRunnerConfigOptions and have the
EC2 provider supply the instance ARN from EC2_INSTANCE_ARN_PREFIX (set by the
DynamoDB wiring). SSM deployments never set it and keep writing the same record.

Based on the accessScope handling in github-aws-runners#5279.

Signed-off-by: tehlers320 <16598119+tehlers320@users.noreply.github.com>
@tehlers320
tehlers320 requested a review from a team as a code owner November 17, 2026 21:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants