Repository navigation
feat(deploy): run the e2e suite against dev and stage after every deploy - #1757
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a post-deploy end-to-end (E2E) testing workflow using Playwright. It adds a new studio-e2e composite action that waits for the newly deployed version to go live before running tests, implements a postDeploy.ts script to handle deployment observation and report generation, and integrates secret masking in CI logs. Additionally, unit tests and documentation are updated to support this flow. Feedback suggests normalizing the baseUrl in observeDeployment to prevent malformed URLs caused by trailing slashes.
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
cb1kenobi
left a comment
There was a problem hiding this comment.
The new CI masking happens too late on the malformed-email failure path, so a verification URL and token can be printed in this public repository’s logs. Mask every candidate link before validating it or including it in an error.
—
Reviewed 4af06f4
|
I'm going to hold on this one for several days while we work on landing the billing plane. After that, I'll get stuff really set up in Dev and think about the other environments too. |
The dev deploy workflow gains an `e2e` job that waits until dev.studio.harperfabric.com serves the version the deploy just built, runs e2e/ from that same commit, and fails the deploy run on any failure or skip. It runs on its own runner with only the `e2e-dev` environment's test-account secrets, and uploads no traces or screenshots because the repo is public. Testing a deployment with its own commit's specs is what the trusted lane got wrong on 2026-09-25: it ran stage's specs against dev, which deploys the diverged dev branch, and all 36 failures were features dev does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups on the post-deploy gate: - mask the round-trip's generated address, password and verification token with ::add-mask:: before a failure could print them - redact the parsed report before formatting it, so escaping or clipping cannot turn a secret into a form that survives; also redact credential-bearing URL parameters and the org and Mailosaur ids - bound Playwright with a 25-minute global timeout inside a 40-minute job, so a slow red run still writes its report - refuse a non-numeric version timeout instead of waiting forever - report tests cut off by --max-failures or --global-timeout as not run - name every version-looking token when the deploy never lands Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 45 minutes outlasts the 10-minute build wait plus Playwright's 25-minute global timeout with margin for a cold install, so a run that hits the global timeout still writes its summary - the verdict names tests that never ran even when others failed, so a --max-failures stop no longer understates how much of the suite ran Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub keeps 10 error annotations per step. On the dry run, Playwright's github reporter spent them on every retry of the first failing tests and the verdict annotation was dropped. The report step now emits the verdict first, then one annotation per failed test (not per retry), at most 8, each linked to its spec file and line. The github reporter is removed again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-3 review nits: - stop credential-parameter redaction at `:`, so redacting a finished annotation cannot swallow its `::` separator when a test title carries `?token=` - mask the URL-encoded form of a generated value too (the throwaway address prints as `local%40server` inside URLs) - build the Playwright arguments as a bash array - escape the project cell in the summary table like the others Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oad balancer Review feedback: three matching responses through a load balancer do not prove every node updated; with two equally routed nodes and one stale, three requests can all land on the fresh one. deploy_component already reports each peer it replicated to (stage: stage-2.studio.harperfabric.com, prod: studio-2.harperfabric.com, dev: none), and every node serves Studio under its own hostname. The deploy step now passes those peers on as a `nodes` output, and the e2e gate probes the public origin and each peer directly: a round passes only when every one serves the new version, still three rounds in a row. Nothing about the deploy itself changes; an unparseable report only leaves the list empty. The README also says outright that the trusted lane still runs stage's specs against dev until the harness tests the commit each site serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-4 review follow-ups: - a credential parameter value may contain `:` again, so `?token=user:session` is redacted whole; a value starting with `[` is skipped instead, which keeps a second pass from eating the `::` after an already-redacted value in an annotation - the "no verification link" error lists the links it saw without their query strings, so a code or password in some other link cannot reach the public log - a test runs the deploy step's exact sed program over the verbatim deploy_component output of dev, stage and prod - the docs name the peers in full and state the assumption the probe rests on: each peer's Harper node name is its public hostname Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review feedback: when no verification link matched, the error printed the email's links before any of them was masked, so a changed template could publish its verification URL and token in the public log. Stripping query strings did not close that, since a token can sit in the path. Every collected link and its token is now masked first; in CI the error then prints them masked, and locally in full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage added a set-password branch to the signup round-trip after this branch was written: a second generated password and a reset-password email. Both email helpers now collect their links through one function that masks every link and token before anything can print them, and the new password is masked when it is generated. Also moves studio-e2e to the pnpm/action-setup and setup-node pins the rest of the repo uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b358323 to
a58e900
Compare
Since the custom-regions work, the clusters overview fetches GET /OrganizationRegion/. Neither spec mocked it, so the request reached the real backend with no session, came back 401, and the app signed the fixture user out: all seven clusters tests and both help-card tests ended on the sign-in page. The trusted lane has failed these same 9 on every dev run since 2026-10-28, and stage (v2.215.0) fails them too. Both specs now answer any unmocked fetch or XHR with a 404, as account-menu already does. The region query reads a 404 as "no custom regions", and the next new endpoint cannot sign the fixture out either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same post-deploy job dev has, wired into deploy-stage.yaml: it waits until stage and its replicated peer serve the version the deploy just built (the semantic-release tag on a release push), runs e2e/ from that commit, and fails the deploy run on red. Stage is the steadier signal while dev integrates unfinished epics, so the two together show whether the front end holds as changes move forward. Its secrets come from a new e2e-stage environment restricted to the stage branch, which must exist before this merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups: - the job summary clipped long messages at a fixed length, which could cut a link in half; GitHub masks a link whole, so the cut-off prefix of a link that carries its token in the path would print. Messages are now clipped at a word boundary. - an empty peer list on a multi-node environment means deploy_component changed its output, not that there are no peers. A new expected-peers input fails the gate in that case; stage sets it to 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…proves Closing-round review follow-ups: - stage raised the signup round-trip to 840 s an attempt, retried once in CI, so it can need 28 minutes; the 25-minute global timeout would report a test still inside its own budget as not run. The suite cap is now 30 minutes and both e2e jobs get 50. - the deploy node is trusted on deploy_component's exit code, not probed (its hostname is part of a secret); the docs now say so, including that a restart deploy can leave it briefly down while the peers pass. - types nothing outside postDeploy.ts imports are no longer exported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
⊙ Problem
Nothing tells a deployer that a deploy broke Studio. The harness's trusted lane has been red on every
devrun since 2026-10-28, andstagefails the same 9 tests, but no CI job runs the e2e specs, so it went unreported for two weeks.💡 Solution
devandstagedeploy runs the suite against the build it just deployed, from the same commit, and goes red if it fails. Nothing rolls back; prod is not wired yet.GET /OrganizationRegion/, which 401s without a session and signs the fixture out. Both now answer unmocked calls with a 404, asaccount-menudoes.For the human reviewer
e2e-stagemust exist before merge (stage job), or every stage deploy goes red. The signup round-trip also needs stage's Mailosaur domain inALLOWLIST_EMAIL_DOMAINS; the stage job can exclude@roundtripinstead.judge), so a missing secret cannot read green.dev, cherry-pick: mergingstageintodevconflicts with the pricing epic.🔧 Changes
studio-deployexposes the deployedversionand the replicated peers asnodes(parsed fromdeploy_component).deploy-dev.yamlanddeploy-stage.yamladd ane2ejob: own runner, read-only, environment secrets only, the version and peers from their deploy job; stage setsexpected-peers: 1.studio-e2e/action.yamlinstalls Chromium, waits, runs Playwright and reports.postDeploy.ts: version match, per-node rounds, peer floor, verdict, summary clipped at word boundaries, annotations, redaction before formatting, credential params, annotation cap.clusters.anon.spec.ts,sidebar-help.anon.spec.ts: 404 for unmocked calls.maskInCi.ts,mail.ts,signup-verification.anon.spec.ts(set-password too): mask generated values and every email link.deploying.md(action list, setup),e2e/README.md.✅ Verification
dev_611d7e7andv2.215.0, every trace showing the 401; with the fix, 80/80 against stage and 13/13 against dev.stage@7e1dad9c8; four nits are open as review deferrals. The last commit was not re-reviewed.More evidence
sedover realdeploy_componentoutput from dev, stage and prod; today's output still yieldsstage-2.harpercalled by path.devorstage.Related PRs: #2054 overlaps (both edit .github/deploying.md), #2030 independent (merged; its mail.ts change is in the base)
Complexity: medium
Deferred findings
9ec1dc7d14031007nit maintainability —e2e/tests/mail.ts— maskedLinks masks every href, including non-secret links — NOT FILED22b088c6b7f977f6nit maintainability —.github/actions/studio-e2e/postDeploy.ts— unconsumed headline export, comments repeating deploying.md, extensionless imports — NOT FILEDdc4c6173665a6947nit reliability —.github/workflows/deploy-dev.yaml— 45-minute job limit could cut off the Report step — NOT FILEDfa4ba0778e60a4a8nit reliability —.github/actions/studio-e2e/postDeploy.ts— version check reads only the entry chunk; no real-build smoke test — NOT FILEDReview-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=2; full=2 @ 6d6c428
Review-Attention: study ~20m (sensitive: deploy-dev.yaml, deploy-stage.yaml; decisions: do-less-alternative, e2e-holds-deploy-group, red-after-deploy-no-rollback, skip-fails-gate, deploy-node-trusted-by-exit) @ 6d6c428