Repository navigation
feat(deploy): run the e2e suite against dev after every dev deploy - #2107
Merged
Merged
Conversation
The dev deploy workflow gains an `e2e` job that waits until dev.studio.harperfabric.com serves the version the deploy just built, runs e2e/ from that same commit, and fails the deploy run on any failure or skip. It runs on its own runner with only the `e2e-dev` environment's test-account secrets, and uploads no traces or screenshots because the repo is public. Testing a deployment with its own commit's specs is what the trusted lane got wrong on 2026-09-25: it ran stage's specs against dev, which deploys the diverged dev branch, and all 36 failures were features dev does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 81f07f1)
Review follow-ups on the post-deploy gate: - mask the round-trip's generated address, password and verification token with ::add-mask:: before a failure could print them - redact the parsed report before formatting it, so escaping or clipping cannot turn a secret into a form that survives; also redact credential-bearing URL parameters and the org and Mailosaur ids - bound Playwright with a 25-minute global timeout inside a 40-minute job, so a slow red run still writes its report - refuse a non-numeric version timeout instead of waiting forever - report tests cut off by --max-failures or --global-timeout as not run - name every version-looking token when the deploy never lands Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e01bd8d)
- 45 minutes outlasts the 10-minute build wait plus Playwright's 25-minute global timeout with margin for a cold install, so a run that hits the global timeout still writes its summary - the verdict names tests that never ran even when others failed, so a --max-failures stop no longer understates how much of the suite ran Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit add4a6c)
GitHub keeps 10 error annotations per step. On the dry run, Playwright's github reporter spent them on every retry of the first failing tests and the verdict annotation was dropped. The report step now emits the verdict first, then one annotation per failed test (not per retry), at most 8, each linked to its spec file and line. The github reporter is removed again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5083c1b)
Round-3 review nits: - stop credential-parameter redaction at `:`, so redacting a finished annotation cannot swallow its `::` separator when a test title carries `?token=` - mask the URL-encoded form of a generated value too (the throwaway address prints as `local%40server` inside URLs) - build the Playwright arguments as a bash array - escape the project cell in the summary table like the others Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1532801)
…oad balancer Review feedback: three matching responses through a load balancer do not prove every node updated; with two equally routed nodes and one stale, three requests can all land on the fresh one. deploy_component already reports each peer it replicated to (stage: stage-2.studio.harperfabric.com, prod: studio-2.harperfabric.com, dev: none), and every node serves Studio under its own hostname. The deploy step now passes those peers on as a `nodes` output, and the e2e gate probes the public origin and each peer directly: a round passes only when every one serves the new version, still three rounds in a row. Nothing about the deploy itself changes; an unparseable report only leaves the list empty. The README also says outright that the trusted lane still runs stage's specs against dev until the harness tests the commit each site serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 3a8f10e)
Round-4 review follow-ups: - a credential parameter value may contain `:` again, so `?token=user:session` is redacted whole; a value starting with `[` is skipped instead, which keeps a second pass from eating the `::` after an already-redacted value in an annotation - the "no verification link" error lists the links it saw without their query strings, so a code or password in some other link cannot reach the public log - a test runs the deploy step's exact sed program over the verbatim deploy_component output of dev, stage and prod - the docs name the peers in full and state the assumption the probe rests on: each peer's Harper node name is its public hostname Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 91b6097)
Review feedback: when no verification link matched, the error printed the email's links before any of them was masked, so a changed template could publish its verification URL and token in the public log. Stripping query strings did not close that, since a token can sit in the path. Every collected link and its token is now masked first; in CI the error then prints them masked, and locally in full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 23528e3)
Stage added a set-password branch to the signup round-trip after this branch was written: a second generated password and a reset-password email. Both email helpers now collect their links through one function that masks every link and token before anything can print them, and the new password is masked when it is generated. Also moves studio-e2e to the pnpm/action-setup and setup-node pins the rest of the repo uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a1745a4)
Since the custom-regions work, the clusters overview fetches GET /OrganizationRegion/. Neither spec mocked it, so the request reached the real backend with no session, came back 401, and the app signed the fixture user out: all seven clusters tests and both help-card tests ended on the sign-in page. The trusted lane has failed these same 9 on every dev run since 2026-10-28, and stage (v2.215.0) fails them too. Both specs now answer any unmocked fetch or XHR with a 404, as account-menu already does. The region query reads a 404 as "no custom regions", and the next new endpoint cannot sign the fixture out either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 360ad84)
The same post-deploy job dev has, wired into deploy-stage.yaml: it waits until stage and its replicated peer serve the version the deploy just built (the semantic-release tag on a release push), runs e2e/ from that commit, and fails the deploy run on red. Stage is the steadier signal while dev integrates unfinished epics, so the two together show whether the front end holds as changes move forward. Its secrets come from a new e2e-stage environment restricted to the stage branch, which must exist before this merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 88f309d)
Review follow-ups: - the job summary clipped long messages at a fixed length, which could cut a link in half; GitHub masks a link whole, so the cut-off prefix of a link that carries its token in the path would print. Messages are now clipped at a word boundary. - an empty peer list on a multi-node environment means deploy_component changed its output, not that there are no peers. A new expected-peers input fails the gate in that case; stage sets it to 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f5869fb)
…proves Closing-round review follow-ups: - stage raised the signup round-trip to 840 s an attempt, retried once in CI, so it can need 28 minutes; the 25-minute global timeout would report a test still inside its own budget as not run. The suite cap is now 30 minutes and both e2e jobs get 50. - the deploy node is trusted on deploy_component's exit code, not probed (its hostname is part of a secret); the docs now say so, including that a restart deploy can leave it briefly down while the peers pass. - types nothing outside postDeploy.ts imports are no longer exported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b3978f7)
The first stage gate run failed two specs on setup, not on Studio, and
both messages pointed the wrong way. Signup's 403 blamed the email-domain
allowlist, which was already right: the refusal was stage's reCAPTCHA
check ("Verification failed"), which local and dev skip. The org-users
failure called a missing organization unlikely, and that was the cause.
The signup message now names the check from central-manager's title,
and for reCAPTCHA says whether the page sent a token. The org-users spec
recognises Studio's "Create an organization" page. The deploy docs list
the three things a target's central-manager needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 831b135)
A manual Deploy to Dev or Stage from any other branch still deploys, but the e2e environment admits only its own branch, so the job failed on the environment and turned a working deploy red. It now runs only on the branch its workflow deploys from, and a workflow test pins that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b76de05)
…llowlist Review follow-ups. The signup classifier now reads the same shapes Studio's isCaptchaVerificationError does (a plain string, error, message, title, detail) and checks reCAPTCHA before Forbidden, so a mixed envelope is not sent to the allowlist. E2E_ALLOW_SIGNUP_403_SKIP now skips only a Forbidden refusal, so it cannot hide reCAPTCHA or an authz regression. The org-users check waits briefly for the new-org page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7762750)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0d37876)
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary
Brings the post-deploy e2e gate from #1757 and its follow-up #2103, both merged into
stage, ontodev: everydevdeploy now runs the e2e suite against the build it just deployed and fails the workflow when the suite fails. It is the same 17 commits, cherry-picked with-x, because mergingstageintodevconflicts with the pricing epic.🔧 Changes
The changed files match
stage's exactly; #1757 and #2103 describe each one.deploy-dev.yamlgains thee2ejob: it runs after the deploy, ondevonly, with thee2e-devenvironment's secrets, againsthttps://dev.studio.harperfabric.com.deploy-stage.yamlgains the stage job so the file matchesstage. That workflow only runs on pushes tostage, so ondevit changes nothing.studio-deployreports the deployed version and peers; the newstudio-e2eaction andpostDeploy.tswait for the version and report the results.maskInCi.ts,mail.ts), and the signup and org-users failures name the setup step that is missing.deploying.mdande2e/README.mddocument the gate and what a target's central-manager needs.✅ Verification
origin/dev@611d7e7cbwithout conflicts, andgit diff origin/stageis empty for every file they change.tsc -b, the e2e project'stsc --noEmitandunitproject, oxlint and dprint. The gate's own tests:postDeployE2e.test.ts,workflowPrivilege.test.ts,signupRefusal.unit.spec.ts.stagepassed after fix(e2e): say which check refused a stage run, and skip e2e on other branches' manual runs #2103 merged: run 43631299561, 109 passed.e2e-devhas existed with its secrets since September, and dev's central-manager skips reCAPTCHA. Not verified before merge: the dev job itself, which first runs on the deploy that merging this triggers.🤖 Generated with Claude Code
Related PRs: #1757 overlaps (the same commits, on
stage), #2103 overlaps (the same commits, onstage), #2054 overlaps (also editsdeploying.md), 2 others independentComplexity: easy
Accepted behavior
Behavior the owner ruled intended, so there is nothing to file.
5 minor or nit
e21b9431f8fc3f63nit reliability —.github/deploying.md— restart deploy: unprobed deploy node may be mid-restart while peers pass — accepted: Dawson Toth (PR owner) ruled it intended in the Claude Code session on 2026-11-16 for feat(deploy): run the e2e suite against dev after every dev deploy #2107acd572fe721713eenit reliability —.github/deploying.md— dev pushes go red until e2e-dev environment, secrets and Mailosaur allowlist exist — accepted: Dawson Toth (PR owner) ruled it intended in the Claude Code session on 2026-11-16 for feat(deploy): run the e2e suite against dev after every dev deploy #2107f89fec0a0d9776denit reliability —.github/workflows/deploy-dev.yaml— dev leaves expected-peers unset — accepted: Dawson Toth (PR owner) ruled it intended in the Claude Code session on 2026-11-16 for feat(deploy): run the e2e suite against dev after every dev deploy #21073f48ead57401da02nit test-coverage —e2e/tests/cluster-form.anon.spec.ts— pricing-epic specs fail the dev gate if UI copy changes before their mocks — accepted: Dawson Toth (PR owner) ruled it intended in the Claude Code session on 2026-11-16 for feat(deploy): run the e2e suite against dev after every dev deploy #2107fdcad2c75616eb32nit maintainability —e2e/tests/mail.ts— new relative imports omit file extensions — accepted: Dawson Toth (PR owner) ruled it intended in the Claude Code session on 2026-11-16 for feat(deploy): run the e2e suite against dev after every dev deploy #2107Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=4; full=3 @ feae247
Review-Attention: study ~20m (sensitive: deploy-dev.yaml, deploy-stage.yaml; decisions: do-less-alternative, gate-lives-with-code-under-test, e2e-in-deploy-run, skip-fails-gate, peer-discovery-by-sed, skip-is-failure, red-without-rollback, dev-gate-on-integration-branch) @ feae247