Skip to content

feat(deploy): run the e2e suite against dev after every dev deploy - #2107

Merged
dawsontoth merged 17 commits into
devfrom
claude/e2e-post-deploy-gate-dev
Nov 16, 2026
Merged

dawsontoth merged 17 commits into
devfrom
claude/e2e-post-deploy-gate-dev

Conversation

@dawsontoth

@dawsontoth dawsontoth commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📝 Summary

Brings the post-deploy e2e gate from #1757 and its follow-up #2103, both merged into stage, onto dev: every dev deploy now runs the e2e suite against the build it just deployed and fails the workflow when the suite fails. It is the same 17 commits, cherry-picked with -x, because merging stage into dev conflicts with the pricing epic.

🔧 Changes

The changed files match stage's exactly; #1757 and #2103 describe each one.

✅ Verification

🤖 Generated with Claude Code

Related PRs: #1757 overlaps (the same commits, on stage), #2103 overlaps (the same commits, on stage), #2054 overlaps (also edits deploying.md), 2 others independent
Complexity: easy

Accepted behavior

Behavior the owner ruled intended, so there is nothing to file.

5 minor or nit

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=4; full=3 @ feae247

Review-Attention: study ~20m (sensitive: deploy-dev.yaml, deploy-stage.yaml; decisions: do-less-alternative, gate-lives-with-code-under-test, e2e-in-deploy-run, skip-fails-gate, peer-discovery-by-sed, skip-is-failure, red-without-rollback, dev-gate-on-integration-branch) @ feae247

dawsontoth and others added 17 commits November 13, 2026 16:01
The dev deploy workflow gains an `e2e` job that waits until
dev.studio.harperfabric.com serves the version the deploy just built,
runs e2e/ from that same commit, and fails the deploy run on any
failure or skip. It runs on its own runner with only the `e2e-dev`
environment's test-account secrets, and uploads no traces or
screenshots because the repo is public.

Testing a deployment with its own commit's specs is what the trusted
lane got wrong on 2026-09-25: it ran stage's specs against dev, which
deploys the diverged dev branch, and all 36 failures were features dev
does not have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 81f07f1)
Review follow-ups on the post-deploy gate:

- mask the round-trip's generated address, password and verification
  token with ::add-mask:: before a failure could print them
- redact the parsed report before formatting it, so escaping or clipping
  cannot turn a secret into a form that survives; also redact
  credential-bearing URL parameters and the org and Mailosaur ids
- bound Playwright with a 25-minute global timeout inside a 40-minute
  job, so a slow red run still writes its report
- refuse a non-numeric version timeout instead of waiting forever
- report tests cut off by --max-failures or --global-timeout as not run
- name every version-looking token when the deploy never lands

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e01bd8d)
- 45 minutes outlasts the 10-minute build wait plus Playwright's
  25-minute global timeout with margin for a cold install, so a run
  that hits the global timeout still writes its summary
- the verdict names tests that never ran even when others failed, so a
  --max-failures stop no longer understates how much of the suite ran

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit add4a6c)
GitHub keeps 10 error annotations per step. On the dry run, Playwright's
github reporter spent them on every retry of the first failing tests and
the verdict annotation was dropped. The report step now emits the
verdict first, then one annotation per failed test (not per retry), at
most 8, each linked to its spec file and line. The github reporter is
removed again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5083c1b)
Round-3 review nits:

- stop credential-parameter redaction at `:`, so redacting a finished
  annotation cannot swallow its `::` separator when a test title
  carries `?token=`
- mask the URL-encoded form of a generated value too (the throwaway
  address prints as `local%40server` inside URLs)
- build the Playwright arguments as a bash array
- escape the project cell in the summary table like the others

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1532801)
…oad balancer

Review feedback: three matching responses through a load balancer do
not prove every node updated; with two equally routed nodes and one
stale, three requests can all land on the fresh one.

deploy_component already reports each peer it replicated to (stage:
stage-2.studio.harperfabric.com, prod: studio-2.harperfabric.com, dev:
none), and every node serves Studio under its own hostname. The deploy
step now passes those peers on as a `nodes` output, and the e2e gate
probes the public origin and each peer directly: a round passes only
when every one serves the new version, still three rounds in a row.
Nothing about the deploy itself changes; an unparseable report only
leaves the list empty.

The README also says outright that the trusted lane still runs stage's
specs against dev until the harness tests the commit each site serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 3a8f10e)
Round-4 review follow-ups:

- a credential parameter value may contain `:` again, so
  `?token=user:session` is redacted whole; a value starting with `[`
  is skipped instead, which keeps a second pass from eating the `::`
  after an already-redacted value in an annotation
- the "no verification link" error lists the links it saw without their
  query strings, so a code or password in some other link cannot reach
  the public log
- a test runs the deploy step's exact sed program over the verbatim
  deploy_component output of dev, stage and prod
- the docs name the peers in full and state the assumption the probe
  rests on: each peer's Harper node name is its public hostname

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 91b6097)
Review feedback: when no verification link matched, the error printed
the email's links before any of them was masked, so a changed template
could publish its verification URL and token in the public log.
Stripping query strings did not close that, since a token can sit in
the path. Every collected link and its token is now masked first; in
CI the error then prints them masked, and locally in full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 23528e3)
Stage added a set-password branch to the signup round-trip after this
branch was written: a second generated password and a reset-password
email. Both email helpers now collect their links through one function
that masks every link and token before anything can print them, and the
new password is masked when it is generated.

Also moves studio-e2e to the pnpm/action-setup and setup-node pins the
rest of the repo uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a1745a4)
Since the custom-regions work, the clusters overview fetches
GET /OrganizationRegion/. Neither spec mocked it, so the request reached
the real backend with no session, came back 401, and the app signed the
fixture user out: all seven clusters tests and both help-card tests ended
on the sign-in page. The trusted lane has failed these same 9 on every dev
run since 2026-10-28, and stage (v2.215.0) fails them too.

Both specs now answer any unmocked fetch or XHR with a 404, as
account-menu already does. The region query reads a 404 as "no custom
regions", and the next new endpoint cannot sign the fixture out either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 360ad84)
The same post-deploy job dev has, wired into deploy-stage.yaml: it waits
until stage and its replicated peer serve the version the deploy just
built (the semantic-release tag on a release push), runs e2e/ from that
commit, and fails the deploy run on red. Stage is the steadier signal
while dev integrates unfinished epics, so the two together show whether
the front end holds as changes move forward.

Its secrets come from a new e2e-stage environment restricted to the
stage branch, which must exist before this merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 88f309d)
Review follow-ups:

- the job summary clipped long messages at a fixed length, which could
  cut a link in half; GitHub masks a link whole, so the cut-off prefix
  of a link that carries its token in the path would print. Messages
  are now clipped at a word boundary.
- an empty peer list on a multi-node environment means deploy_component
  changed its output, not that there are no peers. A new expected-peers
  input fails the gate in that case; stage sets it to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f5869fb)
…proves

Closing-round review follow-ups:

- stage raised the signup round-trip to 840 s an attempt, retried once
  in CI, so it can need 28 minutes; the 25-minute global timeout would
  report a test still inside its own budget as not run. The suite cap is
  now 30 minutes and both e2e jobs get 50.
- the deploy node is trusted on deploy_component's exit code, not probed
  (its hostname is part of a secret); the docs now say so, including
  that a restart deploy can leave it briefly down while the peers pass.
- types nothing outside postDeploy.ts imports are no longer exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b3978f7)
The first stage gate run failed two specs on setup, not on Studio, and
both messages pointed the wrong way. Signup's 403 blamed the email-domain
allowlist, which was already right: the refusal was stage's reCAPTCHA
check ("Verification failed"), which local and dev skip. The org-users
failure called a missing organization unlikely, and that was the cause.

The signup message now names the check from central-manager's title,
and for reCAPTCHA says whether the page sent a token. The org-users spec
recognises Studio's "Create an organization" page. The deploy docs list
the three things a target's central-manager needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 831b135)
A manual Deploy to Dev or Stage from any other branch still deploys, but
the e2e environment admits only its own branch, so the job failed on the
environment and turned a working deploy red. It now runs only on the
branch its workflow deploys from, and a workflow test pins that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b76de05)
…llowlist

Review follow-ups. The signup classifier now reads the same shapes
Studio's isCaptchaVerificationError does (a plain string, error,
message, title, detail) and checks reCAPTCHA before Forbidden, so a
mixed envelope is not sent to the allowlist. E2E_ALLOW_SIGNUP_403_SKIP
now skips only a Forbidden refusal, so it cannot hide reCAPTCHA or an
authz regression. The org-users check waits briefly for the new-org page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7762750)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0d37876)
@github-actions

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 84.97% 21912 / 25785
🔵 Statements 84.81% 23370 / 27555
🔵 Functions 80.65% 6068 / 7523
🔵 Branches 80.32% 19112 / 23792
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
.github/actions/studio-e2e/postDeploy.ts 77.2% 69.18% 84.9% 77.4% 84, 86, 307, 398-400, 433-490, 494-499
Generated in workflow #2592 for commit feae247 by the Vitest Coverage Report Action

@dawsontoth
dawsontoth marked this pull request as ready for review November 16, 2026 15:02
@dawsontoth
dawsontoth requested a review from a team as a code owner November 16, 2026 15:02
@dawsontoth
dawsontoth merged commit 9ac888c into dev Nov 16, 2026
4 checks passed
@dawsontoth
dawsontoth deleted the claude/e2e-post-deploy-gate-dev branch November 16, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant