Skip to content

test(e2e): clean up only this run's mail, and fail loudly when the throwaway account survives - #1845

Merged
dawsontoth merged 4 commits into
stagefrom
claude/1583-e2e-cleanup
Oct 12, 2026
Merged

dawsontoth merged 4 commits into
stagefrom
claude/1583-e2e-cleanup

Conversation

@dawsontoth

@dawsontoth dawsontoth commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

⊙ Problem

The signup round-trip's cleanup did damage in one place and nothing in another. Its afterAll called Mailosaur's deleteAll, which purges the whole server that every lane and developer shares, so it could delete a concurrent run's verification email mid-wait (#1584). And its account self-delete called /User/current against Playwright's baseURL: against a localhost target that is the Vite dev server, which does not serve the API, so the cleanup only warned, the test stayed green, and every local run leaked a real account on stage (#1583).

Closes #1584
Closes #1583

❓ Your call: both fixes as the issues describe them, in one PR because they edit the same spec and helper. Was a separate Mailosaur server per lane (#1584's other option) wanted instead? That needs Mailosaur-console work this PR can't do, and scoped deletion makes it unnecessary. Reverting means restoring deleteAll plus a per-lane MAILOSAUR_SERVER_ID.

💡 Solution

⚖️ Alternatives

❓ Your call: a cleanup failure now fails the test, so under CI=1 (the trusted lane runs CI=1 pnpm test) it uses the round-trip's single retry, and the retry signs up a second account. Today the deployed target cleans up fine (#1583: "verified in the trusted lane"), so this only costs anything when cleanup really breaks, and then two named leaks beat one silent one. The other option is an annotation with no failure. Changing it touches one line.

🔧 Changes

✅ Verification

Specs were not run against live services. This harness's Mailosaur and test-account credentials live in the separate studio-e2e-harness repo, and this session didn't use them.

  • Base behaviour reproduced: on origin/stage (dcaeb6c), a throwaway vite --port 5297 from this worktree answers GET /User/current with 404 and an empty body (Accept: */* and application/json). That behaviour dates from commit 8d4d8a0's appType: 'mpa'; when e2e: throwaway-account cleanup silently no-ops against a localhost target #1583 was filed it was a 200 index.html. Base's cleanup body, run with Playwright's real APIRequestContext against both shapes, resolves without throwing, logging GET /User/current -> 404; leaving account and SyntaxError: Unexpected token '<' respectively. That is the silent no-op.
  • pnpm test:unit (new unit project; no browser, target, or credentials): 16 passed, exit 0. It also passed with PLAYWRIGHT_BASE_URL=http://localhost:5173, and no dev server started.
    • accountCleanup.unit.spec.ts runs the helper over a real APIRequestContext against an in-process HTTP server. It deletes on the given base (/api/User/…, not the origin root). It throws on the dev server's empty 404, on a 200 text/html SPA shell, on a 401, on a body with no id, and on a refused DELETE. The 404, SPA-shell, and no-id cases also check that no DELETE was sent. signupApiBase accepts root, path-prefixed and query-string signup URLs and rejects /Login/, /Users/track, /OrganizationUser/ and /User/usr-1/resend.
    • mail.unit.spec.ts checks that the Mailosaur call searches { sentTo: <address> } with no polling timeout, deletes exactly the returned ids, sends no delete when nothing matched, and keeps deleting after one delete fails, then reports it.
    • Mutation checks, each red then restored: dropping the content-type guard; resolving /User/current from the origin root; searching {} instead of { sentTo }; un-anchoring the signup-path regex; and going back to a sequential delete loop that stops at the first failure.
    • Throwaway checks, not committed: expect.soft in the finally reports both the body's error and the cleanup's, and a cleanup failure alone fails the test. With fake Mailosaur env and HTTPS_PROXY aimed at a closed local port, deleteMailTo on the pinned address makes no request, and on a generated one it tries and warns.
  • npx playwright test --list: 80 tests in 15 files, exit 0; the round-trip still collects in anon.
  • tsc --noEmit -p e2e/tsconfig.json exit 0; root npx tsc -b exit 0; npx oxlint --format stylish . exit 0; npx dprint check exit 0. The pre-commit hook passed on all four commits: vitest 395 files / 3,663 tests, oxlint, dprint, commitlint.
  • Cross-model review: three rounds by codex (graded) and gemini. The Cursor leg failed on the 1Password SSH agent (round 1) and was pruned after that. The Harper domain adjudicator failed all three rounds on an expired Claude OAuth session, so outside findings were triaged by the author, not adjudicated. Every finding was fixed except two, which were rejected on evidence: adding .ts extensions to relative imports (e2e and src both import without them, and e2e/tsconfig.json has no allowImportingTsExtensions; codex ruled it by design) and trimming the remaining comments, which each state a constraint. The last commit (4fb6c88: every delete runs even after one fails, plus the fake server's listen error handler) is exactly the change both legs asked for in round 3, and no reviewer saw it after it was written.

🤖 Generated with Claude Code

Related PRs: #1757 overlaps (same import line and adjacent lines in the round-trip spec; it CI-masks the address this PR's leak message prints)
Complexity: medium

Review-Coverage: authored=claude; ran=gemini,codex; blocked=cursor-composer(no-receipt),domain(auth); declined=cursor-grok,cursor-kimi,cursor-muse; rounds=3; full=1 @ 4fb6c88

Review-Attention: read ~3m (raised: degraded review) @ 4fb6c88

dawsontoth and others added 4 commits October 11, 2026 00:56
…rowaway account survives

The signup round-trip's afterAll purged the whole Mailosaur server, which every lane and
developer shares, so it could delete a concurrent run's verification email mid-wait. It now
deletes only the messages sent to the address this run signed up with (search by sentTo,
then delete by id).

Account cleanup resolved GET /User/current against Playwright's baseURL. On a localhost
target that is the dev server, which does not serve the API, so the cleanup no-opped with a
console.warn and every local run leaked a real account on stage. It now sends the cleanup to
the API base the signup request actually reached, requires a JSON content-type, and turns any
failure into a soft assertion that names the leaked address.

Adds a `unit` Playwright project (*.unit.spec.ts: no browser, target, or credentials) with
tests for both helpers.

Closes #1584
Closes #1583

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts without the dev server

A pinned PLAYWRIGHT_TEST_EMAIL is shared by every run that pins it, so deleting the mail sent
to it could still remove another run's verification email; the mail cleanup now skips it.

The unit project needs no target, but a localhost PLAYWRIGHT_BASE_URL in .env.e2e still made
the config-level webServer start `pnpm dev` first. Add `pnpm test:unit`, which sets
PLAYWRIGHT_NO_WEBSERVER=1.

Also closes every connection before stopping the unit test's HTTP fake, and drops two doc
comments that narrated their functions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s API base

The round-trip waited for any POST whose URL contained `/User/`, then parsed the API base with
a stricter `<base>User/` pattern that threw on anything else. A mismatch would have failed the
test before the cleanup knew where the account lived, leaking it unnamed. `signupApiBase` now
returns undefined for a non-signup URL and is the waitForResponse predicate too, so a response
it picked always has a base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-message loop stopped at the first failed delete and left the rest on the shared server.
Delete them all, then report how many failed. Also reject the unit test fake's listen promise
on a server error instead of leaving it unhandled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a unit testing project within the E2E test suite to test helper functions in isolation without launching a browser. It refactors the account and email cleanup processes in the signup verification tests to be more targeted and robust; instead of purging the entire Mailosaur inbox, it now deletes only the emails sent to the specific test address. Additionally, it extracts account deletion logic into a separate helper, adds unit tests for these helpers, and turns cleanup failures into soft test assertions to prevent silent account leaks. There are no review comments, so I have no feedback to provide.

@github-actions

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 67.53% 10098 / 14953
🔵 Statements 67.68% 10776 / 15920
🔵 Functions 60.78% 2599 / 4276
🔵 Branches 62.53% 7685 / 12289
File CoverageNo changed files found.
Generated in workflow #2087 for commit 4fb6c88 by the Vitest Coverage Report Action

@dawsontoth
dawsontoth marked this pull request as ready for review October 12, 2026 23:16
@dawsontoth
dawsontoth requested a review from a team as a code owner October 12, 2026 23:16
@dawsontoth
dawsontoth merged commit 383db9b into stage Oct 12, 2026
4 checks passed
@dawsontoth
dawsontoth deleted the claude/1583-e2e-cleanup branch October 12, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant