Repository navigation
test(e2e): clean up only this run's mail, and fail loudly when the throwaway account survives - #1845
Merged
Merged
Conversation
…rowaway account survives The signup round-trip's afterAll purged the whole Mailosaur server, which every lane and developer shares, so it could delete a concurrent run's verification email mid-wait. It now deletes only the messages sent to the address this run signed up with (search by sentTo, then delete by id). Account cleanup resolved GET /User/current against Playwright's baseURL. On a localhost target that is the dev server, which does not serve the API, so the cleanup no-opped with a console.warn and every local run leaked a real account on stage. It now sends the cleanup to the API base the signup request actually reached, requires a JSON content-type, and turns any failure into a soft assertion that names the leaked address. Adds a `unit` Playwright project (*.unit.spec.ts: no browser, target, or credentials) with tests for both helpers. Closes #1584 Closes #1583 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts without the dev server A pinned PLAYWRIGHT_TEST_EMAIL is shared by every run that pins it, so deleting the mail sent to it could still remove another run's verification email; the mail cleanup now skips it. The unit project needs no target, but a localhost PLAYWRIGHT_BASE_URL in .env.e2e still made the config-level webServer start `pnpm dev` first. Add `pnpm test:unit`, which sets PLAYWRIGHT_NO_WEBSERVER=1. Also closes every connection before stopping the unit test's HTTP fake, and drops two doc comments that narrated their functions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s API base The round-trip waited for any POST whose URL contained `/User/`, then parsed the API base with a stricter `<base>User/` pattern that threw on anything else. A mismatch would have failed the test before the cleanup knew where the account lived, leaking it unnamed. `signupApiBase` now returns undefined for a non-signup URL and is the waitForResponse predicate too, so a response it picked always has a base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-message loop stopped at the first failed delete and left the rest on the shared server. Delete them all, then report how many failed. Also reject the unit test fake's listen promise on a server error instead of leaving it unhandled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a unit testing project within the E2E test suite to test helper functions in isolation without launching a browser. It refactors the account and email cleanup processes in the signup verification tests to be more targeted and robust; instead of purging the entire Mailosaur inbox, it now deletes only the emails sent to the specific test address. Additionally, it extracts account deletion logic into a separate helper, adds unit tests for these helpers, and turns cleanup failures into soft test assertions to prevent silent account leaks. There are no review comments, so I have no feedback to provide.
Coverage Report
File CoverageNo changed files found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
The signup round-trip's cleanup did damage in one place and nothing in another. Its
afterAllcalled Mailosaur'sdeleteAll, which purges the whole server that every lane and developer shares, so it could delete a concurrent run's verification email mid-wait (#1584). And its account self-delete called/User/currentagainst Playwright'sbaseURL: against a localhost target that is the Vite dev server, which does not serve the API, so the cleanup only warned, the test stayed green, and every local run leaked a real account on stage (#1583).Closes #1584
Closes #1583
💡 Solution
afterAllnow deletes only the messages sent to the addresses this worker signed up with: onemessages.search({ sentTo }), then amessages.del(id)for each message, and if one delete fails, the rest still run. e2e: Mailosaur deleteAllMail() is server-wide and can delete a concurrent run's verification email #1584 suggestedmessages.list({ sentTo }), butlisttakes no recipient filter in mailosaur 11.1.1, sosearchis the call that has one. A pinnedPLAYWRIGHT_TEST_EMAILis never swept, since every run that pins it shares it.signupApiBasereads it offPOST <base>User/, and the same function picks which response is the signup, so a picked response always has a base. That base is the one the app itself used (VITE_CENTRAL_MANAGER_API_URL), so no new configuration is needed and none can drift from the app's. The cleanup sendsAccept: application/jsonand requires a JSON content-type before parsing. Any failure is now anexpect.softthat names the leaked address: the test fails, and if the flow has already thrown, its own error is still reported too.⚖️ Alternatives
PLAYWRIGHT_API_BASE_URL, defaulting tobaseURL(e2e: throwaway-account cleanup silently no-ops against a localhost target #1583's suggestion): this would still be wrong on every local run until someone set it, and it could point at a different backend than the one the app signed up on. Reading the URL off the signup request can't disagree with the app.console.warn". A soft assertion is the smallest change that makes the run go red without hiding the original failure.🔧 Changes
e2e/tests/accountCleanup.ts(new):signupApiBaseanddeleteSessionAccount(GETUser/current→ JSON content-type check → DELETEUser/{id}). It throws with the reason on any failure and leaves the policy to the caller.e2e/tests/signup-verification.anon.spec.ts: picks the signup response withsignupApiBase, records the account's API base only once signup succeeded, and infinallyturns a cleanup failure into a soft assertion. TheafterAllsweeps only the addresses the test used.e2e/tests/mail.ts:deleteAllMailis replaced bydeleteMailTo(address), which skips a pinned address, anddeleteMessagesSentTo, whosePick<…, 'search' | 'del'>parameter meansdeleteAllcan't even be called.e2e/playwright.config.ts: a fourthunitproject for*.unit.spec.ts.e2e/package.json:pnpm test:unitruns it withPLAYWRIGHT_NO_WEBSERVER=1, so a localhostPLAYWRIGHT_BASE_URLin.env.e2edoesn't start the dev server first.e2e/README.md: the layout and naming conventions list the unit project. The email round-trip section says mail cleanup is per address. The account-churn note says the delete follows the signup's API and that a failed cleanup fails the test and names the address.✅ Verification
Specs were not run against live services. This harness's Mailosaur and test-account credentials live in the separate
studio-e2e-harnessrepo, and this session didn't use them.origin/stage(dcaeb6c), a throwawayvite --port 5297from this worktree answersGET /User/currentwith 404 and an empty body (Accept: */*andapplication/json). That behaviour dates from commit 8d4d8a0'sappType: 'mpa'; when e2e: throwaway-account cleanup silently no-ops against a localhost target #1583 was filed it was a200index.html. Base's cleanup body, run with Playwright's realAPIRequestContextagainst both shapes, resolves without throwing, loggingGET /User/current -> 404; leaving accountandSyntaxError: Unexpected token '<'respectively. That is the silent no-op.pnpm test:unit(newunitproject; no browser, target, or credentials): 16 passed, exit 0. It also passed withPLAYWRIGHT_BASE_URL=http://localhost:5173, and no dev server started.accountCleanup.unit.spec.tsruns the helper over a realAPIRequestContextagainst an in-process HTTP server. It deletes on the given base (/api/User/…, not the origin root). It throws on the dev server's empty 404, on a 200text/htmlSPA shell, on a 401, on a body with no id, and on a refused DELETE. The 404, SPA-shell, and no-id cases also check that no DELETE was sent.signupApiBaseaccepts root, path-prefixed and query-string signup URLs and rejects/Login/,/Users/track,/OrganizationUser/and/User/usr-1/resend.mail.unit.spec.tschecks that the Mailosaur call searches{ sentTo: <address> }with no pollingtimeout, deletes exactly the returned ids, sends no delete when nothing matched, and keeps deleting after one delete fails, then reports it./User/currentfrom the origin root; searching{}instead of{ sentTo }; un-anchoring the signup-path regex; and going back to a sequential delete loop that stops at the first failure.expect.softin thefinallyreports both the body's error and the cleanup's, and a cleanup failure alone fails the test. With fake Mailosaur env andHTTPS_PROXYaimed at a closed local port,deleteMailToon the pinned address makes no request, and on a generated one it tries and warns.npx playwright test --list: 80 tests in 15 files, exit 0; the round-trip still collects inanon.tsc --noEmit -p e2e/tsconfig.jsonexit 0; rootnpx tsc -bexit 0;npx oxlint --format stylish .exit 0;npx dprint checkexit 0. The pre-commit hook passed on all four commits: vitest 395 files / 3,663 tests, oxlint, dprint, commitlint..tsextensions to relative imports (e2e and src both import without them, ande2e/tsconfig.jsonhas noallowImportingTsExtensions; codex ruled it by design) and trimming the remaining comments, which each state a constraint. The last commit (4fb6c88: every delete runs even after one fails, plus the fake server'slistenerror handler) is exactly the change both legs asked for in round 3, and no reviewer saw it after it was written.🤖 Generated with Claude Code
Related PRs: #1757 overlaps (same import line and adjacent lines in the round-trip spec; it CI-masks the address this PR's leak message prints)
Complexity: medium
Review-Coverage: authored=claude; ran=gemini,codex; blocked=cursor-composer(no-receipt),domain(auth); declined=cursor-grok,cursor-kimi,cursor-muse; rounds=3; full=1 @ 4fb6c88
Review-Attention: read ~3m (raised: degraded review) @ 4fb6c88