Skip to content

feat(deploy): run the e2e suite against dev and stage after every deploy - #1757

Merged
dawsontoth merged 13 commits into
stagefrom
claude/e2e-post-deploy-gate
Nov 13, 2026
Merged

dawsontoth merged 13 commits into
stagefrom
claude/e2e-post-deploy-gate

Conversation

@dawsontoth

@dawsontoth dawsontoth commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

⊙ Problem

Nothing tells a deployer that a deploy broke Studio. The harness's trusted lane has been red on every dev run since 2026-10-28, and stage fails the same 9 tests, but no CI job runs the e2e specs, so it went unreported for two weeks.

💡 Solution

  • Each dev and stage deploy runs the suite against the build it just deployed, from the same commit, and goes red if it fails. Nothing rolls back; prod is not wired yet.
  • The gate waits until the public origin and every replicated peer serve the new version, then fails on any failure or skip. It uploads no traces or screenshots (the repo is public) and masks values the specs generate.
  • The 9 failures: the clusters and help-card specs did not mock GET /OrganizationRegion/, which 401s without a session and signs the fixture out. Both now answer unmocked calls with a 404, as account-menu does.

For the human reviewer

  1. e2e-stage must exist before merge (stage job), or every stage deploy goes red. The signup round-trip also needs stage's Mailosaur domain in ALLOWLIST_EMAIL_DOMAINS; the stage job can exclude @roundtrip instead.
  2. Skips fail the gate (judge), so a missing secret cannot read green.
  3. The deploy node is trusted on its exit code, not probed (docs): its hostname is inside a secret.
  4. A red suite holds the environment's deploy queue until it finishes (~4 minutes, capped at 50).
  5. To reach dev, cherry-pick: merging stage into dev conflicts with the pricing epic.

🔧 Changes

✅ Verification

  • The 9 failures reproduce from each branch's own specs against dev_611d7e7 and v2.215.0, every trace showing the 401; with the fix, 80/80 against stage and 13/13 against dev.
  • The readiness check passed live on stage's two nodes, refused a stale version, and failed at once on an empty peer list.
  • Unit tests and workflow pins for both jobs; disabling each of 21 guards turns a test red. Full suite green on every commit (700 files).
  • Two cross-model rounds after the rebase onto stage @ 7e1dad9c8; four nits are open as review deferrals. The last commit was not re-reviewed.
More evidence
  • GitHub-runner dry runs before the rebase: first (stage green, dev red on skew, wrong version refused), second (verdict plus 8 per-test annotations).
  • A test runs the deploy step's own sed over real deploy_component output from dev, stage and prod; today's output still yields stage-2.
  • Rebase conflicts kept stage's changes: Harper 4.x path retired (fix(deploy): let Renovate update @fastify/static and retire the Harper 4.x path #2039), per-run mail cleanup, harper called by path.
  • Not verified before merge: the real post-deploy path on a push to dev or stage.

Related PRs: #2054 overlaps (both edit .github/deploying.md), #2030 independent (merged; its mail.ts change is in the base)
Complexity: medium

Deferred findings

  • 9ec1dc7d14031007 nit maintainability — e2e/tests/mail.ts — maskedLinks masks every href, including non-secret links — NOT FILED
  • 22b088c6b7f977f6 nit maintainability — .github/actions/studio-e2e/postDeploy.ts — unconsumed headline export, comments repeating deploying.md, extensionless imports — NOT FILED
  • dc4c6173665a6947 nit reliability — .github/workflows/deploy-dev.yaml — 45-minute job limit could cut off the Report step — NOT FILED
  • fa4ba0778e60a4a8 nit reliability — .github/actions/studio-e2e/postDeploy.ts — version check reads only the entry chunk; no real-build smoke test — NOT FILED

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=2; full=2 @ 6d6c428

Review-Attention: study ~20m (sensitive: deploy-dev.yaml, deploy-stage.yaml; decisions: do-less-alternative, e2e-holds-deploy-group, red-after-deploy-no-rollback, skip-fails-gate, deploy-node-trusted-by-exit) @ 6d6c428

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a post-deploy end-to-end (E2E) testing workflow using Playwright. It adds a new studio-e2e composite action that waits for the newly deployed version to go live before running tests, implements a postDeploy.ts script to handle deployment observation and report generation, and integrates secret masking in CI logs. Additionally, unit tests and documentation are updated to support this flow. Feedback suggests normalizing the baseUrl in observeDeployment to prevent malformed URLs caused by trailing slashes.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 84.86% 21885 / 25787
🔵 Statements 84.67% 23223 / 27427
🔵 Functions 80.7% 6029 / 7470
🔵 Branches 80.19% 18751 / 23382
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
.github/actions/studio-e2e/postDeploy.ts 77.2% 69.18% 84.9% 77.4% 84, 86, 307, 398-400, 433-490, 494-499
Generated in workflow #2572 for commit 6d6c428 by the Vitest Coverage Report Action

@dawsontoth
dawsontoth marked this pull request as ready for review September 25, 2026 15:54
@dawsontoth
dawsontoth requested a review from a team as a code owner September 25, 2026 15:54

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Reviewed with Codex

Comment thread e2e/README.md Resolved

@cb1kenobi cb1kenobi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new CI masking happens too late on the malformed-email failure path, so a verification URL and token can be printed in this public repository’s logs. Mask every candidate link before validating it or including it in an error.

—
Reviewed 4af06f4

Comment thread e2e/tests/mail.ts Resolved

@cb1kenobi cb1kenobi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No confirmed blocking defects remain in the supplied diff. Existing discussions cover the prior readiness and masking concerns, and the changed lines address them.

—
Reviewed b358323

@dawsontoth

Copy link
Copy Markdown
Contributor Author

I'm going to hold on this one for several days while we work on landing the billing plane. After that, I'll get stuff really set up in Dev and think about the other environments too.

@cb1kenobi cb1kenobi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No confirmed blocking defects remain. Existing discussions cover the prior readiness and masking concerns, and this commit addresses them.

—
Reviewed b358323

dawsontoth and others added 9 commits November 12, 2026 11:34
The dev deploy workflow gains an `e2e` job that waits until
dev.studio.harperfabric.com serves the version the deploy just built,
runs e2e/ from that same commit, and fails the deploy run on any
failure or skip. It runs on its own runner with only the `e2e-dev`
environment's test-account secrets, and uploads no traces or
screenshots because the repo is public.

Testing a deployment with its own commit's specs is what the trusted
lane got wrong on 2026-09-25: it ran stage's specs against dev, which
deploys the diverged dev branch, and all 36 failures were features dev
does not have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups on the post-deploy gate:

- mask the round-trip's generated address, password and verification
  token with ::add-mask:: before a failure could print them
- redact the parsed report before formatting it, so escaping or clipping
  cannot turn a secret into a form that survives; also redact
  credential-bearing URL parameters and the org and Mailosaur ids
- bound Playwright with a 25-minute global timeout inside a 40-minute
  job, so a slow red run still writes its report
- refuse a non-numeric version timeout instead of waiting forever
- report tests cut off by --max-failures or --global-timeout as not run
- name every version-looking token when the deploy never lands

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 45 minutes outlasts the 10-minute build wait plus Playwright's
  25-minute global timeout with margin for a cold install, so a run
  that hits the global timeout still writes its summary
- the verdict names tests that never ran even when others failed, so a
  --max-failures stop no longer understates how much of the suite ran

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub keeps 10 error annotations per step. On the dry run, Playwright's
github reporter spent them on every retry of the first failing tests and
the verdict annotation was dropped. The report step now emits the
verdict first, then one annotation per failed test (not per retry), at
most 8, each linked to its spec file and line. The github reporter is
removed again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-3 review nits:

- stop credential-parameter redaction at `:`, so redacting a finished
  annotation cannot swallow its `::` separator when a test title
  carries `?token=`
- mask the URL-encoded form of a generated value too (the throwaway
  address prints as `local%40server` inside URLs)
- build the Playwright arguments as a bash array
- escape the project cell in the summary table like the others

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oad balancer

Review feedback: three matching responses through a load balancer do
not prove every node updated; with two equally routed nodes and one
stale, three requests can all land on the fresh one.

deploy_component already reports each peer it replicated to (stage:
stage-2.studio.harperfabric.com, prod: studio-2.harperfabric.com, dev:
none), and every node serves Studio under its own hostname. The deploy
step now passes those peers on as a `nodes` output, and the e2e gate
probes the public origin and each peer directly: a round passes only
when every one serves the new version, still three rounds in a row.
Nothing about the deploy itself changes; an unparseable report only
leaves the list empty.

The README also says outright that the trusted lane still runs stage's
specs against dev until the harness tests the commit each site serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-4 review follow-ups:

- a credential parameter value may contain `:` again, so
  `?token=user:session` is redacted whole; a value starting with `[`
  is skipped instead, which keeps a second pass from eating the `::`
  after an already-redacted value in an annotation
- the "no verification link" error lists the links it saw without their
  query strings, so a code or password in some other link cannot reach
  the public log
- a test runs the deploy step's exact sed program over the verbatim
  deploy_component output of dev, stage and prod
- the docs name the peers in full and state the assumption the probe
  rests on: each peer's Harper node name is its public hostname

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review feedback: when no verification link matched, the error printed
the email's links before any of them was masked, so a changed template
could publish its verification URL and token in the public log.
Stripping query strings did not close that, since a token can sit in
the path. Every collected link and its token is now masked first; in
CI the error then prints them masked, and locally in full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage added a set-password branch to the signup round-trip after this
branch was written: a second generated password and a reset-password
email. Both email helpers now collect their links through one function
that masks every link and token before anything can print them, and the
new password is masked when it is generated.

Also moves studio-e2e to the pnpm/action-setup and setup-node pins the
rest of the repo uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dawsontoth
dawsontoth force-pushed the claude/e2e-post-deploy-gate branch from b358323 to a58e900 Compare November 12, 2026 16:38
dawsontoth and others added 4 commits November 12, 2026 11:51
Since the custom-regions work, the clusters overview fetches
GET /OrganizationRegion/. Neither spec mocked it, so the request reached
the real backend with no session, came back 401, and the app signed the
fixture user out: all seven clusters tests and both help-card tests ended
on the sign-in page. The trusted lane has failed these same 9 on every dev
run since 2026-10-28, and stage (v2.215.0) fails them too.

Both specs now answer any unmocked fetch or XHR with a 404, as
account-menu already does. The region query reads a 404 as "no custom
regions", and the next new endpoint cannot sign the fixture out either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same post-deploy job dev has, wired into deploy-stage.yaml: it waits
until stage and its replicated peer serve the version the deploy just
built (the semantic-release tag on a release push), runs e2e/ from that
commit, and fails the deploy run on red. Stage is the steadier signal
while dev integrates unfinished epics, so the two together show whether
the front end holds as changes move forward.

Its secrets come from a new e2e-stage environment restricted to the
stage branch, which must exist before this merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups:

- the job summary clipped long messages at a fixed length, which could
  cut a link in half; GitHub masks a link whole, so the cut-off prefix
  of a link that carries its token in the path would print. Messages
  are now clipped at a word boundary.
- an empty peer list on a multi-node environment means deploy_component
  changed its output, not that there are no peers. A new expected-peers
  input fails the gate in that case; stage sets it to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…proves

Closing-round review follow-ups:

- stage raised the signup round-trip to 840 s an attempt, retried once
  in CI, so it can need 28 minutes; the 25-minute global timeout would
  report a test still inside its own budget as not run. The suite cap is
  now 30 minutes and both e2e jobs get 50.
- the deploy node is trusted on deploy_component's exit code, not probed
  (its hostname is part of a secret); the docs now say so, including
  that a restart deploy can leave it briefly down while the peers pass.
- types nothing outside postDeploy.ts imports are no longer exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dawsontoth dawsontoth changed the title feat(deploy): run the e2e suite against dev after every dev deploy feat(deploy): run the e2e suite against dev and stage after every deploy Nov 12, 2026
@dawsontoth
dawsontoth marked this pull request as ready for review November 13, 2026 20:58
@dawsontoth
dawsontoth merged commit b3978f7 into stage Nov 13, 2026
6 checks passed
@dawsontoth
dawsontoth deleted the claude/e2e-post-deploy-gate branch November 13, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants