Skip to content

test(e2e): Add Gherkin acceptance tests with playwright-bdd and Screenplay actors - #57

Open
DetweilerRyan wants to merge 7 commits into
mainfrom
test/e2e-bdd
Open

DetweilerRyan wants to merge 7 commits into
mainfrom
test/e2e-bdd

Conversation

@DetweilerRyan

@DetweilerRyan DetweilerRyan commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

Adds Gherkin end-to-end tests, run by Playwright Test through playwright-bdd. The step definitions use a small Screenplay layer: actors with a browser and an MCP client, plus tasks and questions. The conventions are in testing conventions, and the reasons and owner decisions are in its sidecar.

.feature files are linted and formatted too. npm run lint runs gplint with .gplintrc, and npm run format and format:check run scripts/format-features.ts, which uses Cucumber's own formatter. Node 26 runs the script directly by stripping its types, and npm run typecheck covers it. The definition-of-done commands and CI steps don't change.

What to look at

  • src/store/yjsProvider.ts: the only app change. NEXT_PUBLIC_WEBRTC_SIGNALING_URLS replaces the hard-coded signaling servers when set. Unset or blank keeps the public servers, so Vercel behaves as before.

  • e2e/screenplay/actor.ts: each actor's browser context aborts every request to a host other than localhost. That is how the e2e run meets the offline rule, along with the local y-webrtc-signaling started in playwright.config.ts.

  • CI gains an "End-to-end tests" step after the production build, and the e2e run builds the app again with the signaling variable set. In CI the "Definition of done" job took 1 min 5 s. The new step took 9 s, its rebuild included, and the other steps did not change.

  • .gplintrc's vocabulary list (no-restricted-patterns): 52 patterns, approved in session on 2026-11-22, in six commented categories. The categories come from web research recorded in .gplintrc.meta.md. table, link, page, request, phone and MCP server stay allowed as domain words.

Decisions made

  • Gherkin tools: gplint 2.6.1 and @cucumber/gherkin-utils 12.0.2, approved in session on 2026-11-22. The rules are ported from gol's .gherkin-lintrc, with two changes:
    • only-one-when becomes max-keywords (when: 1, ignoreConjunctions: true).
    • Layout rules are off, following gol2, so only the formatter decides layout.
    • The linter's reasons, including the options not taken, are in .gplintrc.meta.md. The formatter's are in testing conventions' sidecar.
  • VS Code: the dev container installs Cucumber's extension, CucumberOpen.cucumber-official 1.12.1 (approved in session on 2026-11-22).
    • Its settings point it at e2e/features and e2e/steps, since its default paths miss e2e/, and make it the formatter for .feature files.
    • The kit's network policy already allows the marketplace.
  • e2e/features/, not a top-level features/: your decision on 2026-11-22, recorded in the sidecar.
  • A sidecar for a config file: at your suggestion, .gplintrc's reasons live in .gplintrc.meta.md. single-source-of-truth.md now allows a <config>.meta.md when the reasons need more than a comment. Until now it defined sidecars for documents only.
  • gplint 2.6.1, not 2.6.2: 2.6.2 was under two weeks old.
  • Into existing commands: the Gherkin checks join lint and format:check rather than getting new ones, so the definition-of-done table and CI stay as they are.
  • Rejection scenario: it uses a duplicated item id, not the tax line test(e2e): Add Gherkin acceptance tests with playwright-bdd and Screenplay actors #55 suggested. The schema doesn't reject tax lines; leaving them out is only an instruction to the model.
  • How actors claim: they tap their name's button on the item, not the identity picker. The header's identity button is named after whoever is active, so it has no stable name to find it by.
  • Phone screens: actors browse on a Pixel 7 profile with en-US locale, following the mobile-first guidance, with amounts in USD so no exchange rate is fetched.
  • No server reuse: reuseExistingServer is off. NEXT_PUBLIC_ variables are fixed at build time, so a server already running could be pointing at the public signaling servers.
  • No HTML report or trace upload in CI: uploading them would need a new action, which needs your approval. Failures print in the log, with traces kept locally in test-results/.

For the owner to confirm

Nothing.

Changed test expectations

Nothing. The new tests in yjsProvider.test.ts and scripts/format-features.real.test.ts are additions. The edits to the two .feature files leave every scenario's steps unchanged:

  • a reworded feature description, which the vocabulary rule flagged for naming create_receipt_room;
  • right-aligned prices from the formatter.

Verification

  • Definition of done: all six checks, the new npm run test:e2e included, exit 0 locally in the Docker Sandbox.
  • Stability: each scenario passed 10 times in a row (--repeat-each 10, 40 runs, 8 workers).
  • The tests can fail:
    • Pointing the app at a dead signaling port makes both live-split scenarios fail, waiting for "2 connections online".
    • Changing the "claimed by one person" assertion to expect "Unclaimed" makes it fail, with Bob's row reading "Solo".
  • Gherkin lint and format can fail:
    • A planted feature with two Whens, a Then that should be And, and button and clicks failed npm run lint with six findings. It passed again once the file was removed.
    • None of the 52 vocabulary patterns matches the current features. A planted scenario in implementation language (URL, snake_case table, database, field, a tap on a button, a wait in seconds, a GET request, an endpoint, a status code) got 13 findings.
    • format:check failed on both features before formatting and passes after.
    • scripts/format-features.real.test.ts covers formatted, unformatted, nested, empty and unparseable input.
  • Lockfile: no existing package's version changed. @cucumber/gherkin-utils moved to 12.0.2 at the top level, and playwright-bdd keeps its own 11.0.0. npm audit reports 0 vulnerabilities.
  • CI: the first commit's run 44965589382 passed, all seven steps. The second commit's run 44967711124 passed too, in 1 min 7 s. The third commit's run 44968358749 passed in 1 min 15 s. The fourth commit's run 44968554671 passed in 1 min 23 s. The fifth commit's run 44981613389 passed in 1 min 24 s. The sixth commit's run 44981972453 passed in 1 min 13 s. The seventh commit's run 44982171066, which converts the format script to TypeScript, passed in 1 min 23 s.
  • Dev container: devcontainer.json parses, and the extension package's own settings confirm its default paths miss e2e/. Not verified: opening the container in VS Code, the extension linking steps, and whether its bundled formatter matches npm run format. Its code is minified, so the version can't be read.

Links

Closes #55. Running against Vercel previews is deferred to #56.

🤖 Generated with Claude Code

Scenarios in e2e/features run on Playwright Test through playwright-bdd,
with step definitions written as actors performing tasks and answering
questions. An assistant actor calls create_receipt_room over /api/mcp
with the MCP client SDK, and people open the link in their own phone-sized
browser contexts and split the receipt live.

Runs stay offline: NEXT_PUBLIC_WEBRTC_SIGNALING_URLS points room sync at
y-webrtc's own signaling server, which playwright.config.ts starts beside
the app, and actors' browsers refuse requests to other hosts.

npm run test:e2e is a new definition-of-done check and CI step.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
split Ready Ready Preview Nov 22, 2026 3:47am UTC

@DetweilerRyan
DetweilerRyan marked this pull request as ready for review November 22, 2026 00:15
@DetweilerRyan
DetweilerRyan marked this pull request as draft November 22, 2026 00:37
npm run lint now also runs gplint over e2e/features with .gplintrc:
the structure rules from the gol repository, plus a list of
implementation words scenarios may not use. npm run format and
format:check now also run scripts/format-features.mjs, which formats
features with @cucumber/gherkin-utils, or checks them with --check since
its CLI has no check mode. Layout rules stay off in the linter so only
the formatter decides layout.

The first lint run caught create_receipt_room in a feature description,
reworded here, and the formatter right-aligned the price columns.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DetweilerRyan
DetweilerRyan marked this pull request as ready for review November 22, 2026 00:41
@DetweilerRyan
DetweilerRyan marked this pull request as draft November 22, 2026 00:46
agent and others added 2 commits November 21, 2026 19:47
.gplintrc's no-restricted-patterns grows from 15 words to 52 patterns
in six categories from web research on Gherkin style: code identifiers
(any snake_case name), URLs and HTTP, storage and sync internals, the
DOM, widgets and gestures, and the browser and test harness. Words that
belong to this app's domain (table, link, page, MCP server) stay allowed.

None matches the current features; a planted scenario in implementation
language gets 13 findings.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why the linter is gplint, where each rule comes from, and the research
behind the banned-word list now sit beside .gplintrc instead of in
testing-conventions.meta.md, which keeps the formatter choice, the
precedents and a pointer.

single-source-of-truth.md now lets a config file have a sidecar named for
the whole file name, when its reasons need more than a comment.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make this a ts file

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 09c2f98: scripts/format-features.mjs is now scripts/format-features.ts.

  • How it runs: Node 26 runs it directly by stripping the types, so no build step or new dependency is needed. npm run format and format:check call node scripts/format-features.ts.
  • Type checking: npm run typecheck now covers the script. On the first run it caught that readdirSync(path, { recursive: true }) with no encoding is typed to return Buffers, so the script now passes encoding: 'utf8'.
  • One limit: type stripping only handles TypeScript syntax that can simply be erased, with no enums, namespaces or parameter properties. A comment at the top of the script says so.
  • References updated: package.json, the script's test, AGENTS.md, .gplintrc, .gplintrc.meta.md, testing-conventions.meta.md and the PR description.
  • Self-review: I reread the whole branch diff, and no other file names the old .mjs script.
  • Checks: all six definition-of-done checks pass locally, including the script's five tests. CI is running.

@DetweilerRyan
DetweilerRyan marked this pull request as ready for review November 22, 2026 00:51
@DetweilerRyan
DetweilerRyan marked this pull request as draft November 22, 2026 03:37
CucumberOpen.cucumber-official highlights, formats and autocompletes the
e2e features and jumps from a step to its definition. Its default paths
don't include e2e/, so the dev container's settings point it at
e2e/features and e2e/steps, and make it the formatter for .feature files.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DetweilerRyan
DetweilerRyan marked this pull request as ready for review November 22, 2026 03:41
@DetweilerRyan
DetweilerRyan marked this pull request as draft November 22, 2026 03:43
agent and others added 2 commits November 21, 2026 22:44
Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As asked in review: scripts/format-features.mjs becomes
scripts/format-features.ts, which Node 26 runs directly by stripping its
types, so npm run typecheck now covers it. The typecheck caught that
readdirSync without an encoding is typed to return Buffers, so it now
asks for utf8.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DetweilerRyan
DetweilerRyan marked this pull request as ready for review November 22, 2026 03:48
@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22:

i noticed you haven't been writing comments to the pr with the feedback i've been giving directly to you in this session. have you not read the revising-work.md doc? was it not clear that you should have had a read trigger for it?

Correct.

  • What went wrong: after this PR was first marked ready, the session made five rounds of changes you asked for in the session without recording them here. Revising work says to post session feedback as a PR comment before acting on it, and to reply with what changed.
  • The trigger was clear. The workflow index's "Read when" for revising work is "When the owner gives feedback on submitted work, in a review or in the session". The session read the procedure only when your inline review comment arrived. It had treated your requests in the session as new tasks and approvals, not as feedback on submitted work.
  • The fix: the comments below record each request late, quoting your words, and each is followed by a reply saying what changed and in which commit.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22, recorded late (why):

approved, add both to PR #57

It answered a recommendation to lint and format .feature files with two new dev dependencies: @cucumber/gherkin-utils (formatter) and gplint (linter, with the rules from the owner's gol repository).

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22, recorded late (why):

i approve

It answered a recommended list of implementation words to ban in .feature files: about 40 more patterns for .gplintrc's no-restricted-patterns, in six categories from web research on Gherkin style, with domain words (table, link, page, MCP server) left allowed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22, recorded late (why):

maybe add a meta side car to .gplintrc (.gplintrc.meta.md) instead of cramming it all into testing-conventions.meta.md?

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22, recorded late (why):

add the VS Code Cucumber extension to the dev container

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Feedback given in a Claude Code session, 2026-11-22, recorded late (why):

is a top level e2e folder the standard place for feature files or should we go with features/ instead?

and, after the session compared the conventions:

keep e2e/features

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: done in baf20e6.

  • Dependencies: @cucumber/gherkin-utils 12.0.2 and gplint 2.6.1, both exact versions. 2.6.1 rather than 2.6.2, which was under two weeks old.
  • Lint: npm run lint also runs gplint over e2e/features with .gplintrc. It uses gol's structure rules, with only-one-when done through max-keywords, and layout rules off so only the formatter decides layout.
  • Format: npm run format and format:check also run the feature formatter script, which adds a --check mode the formatter's CLI lacks. It started as .mjs and became .ts in 09c2f98, after your review.
  • Feature fixes: the first lint run flagged create_receipt_room in a feature description, which was reworded. The formatter right-aligned the price columns.
  • Checks:
    • A planted feature got six findings.
    • format:check failed before formatting and passed after.
    • The lockfile changed no existing package's version, and npm audit reported 0 vulnerabilities.
    • CI run 44967711124 passed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: done in 9aae08d.

  • The list: .gplintrc's no-restricted-patterns went from 15 words to 52 patterns, in six commented categories:
    • code identifiers, meaning any snake_case name and function calls;
    • URLs and HTTP;
    • storage and sync internals;
    • the DOM;
    • widgets and gestures;
    • the browser and the test harness.
  • Left allowed: table, link, page, request, phone and MCP server, as proposed.
  • Checks:
    • None of the patterns matches the current features.
    • A planted scenario written in implementation language got 13 findings.
    • CI run 44968358749 passed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: done in 1eacfee, as suggested.

  • .gplintrc.meta.md now holds the linter's reasons: why gplint, where each rule comes from, the vocabulary research, the words left allowed, the checks, and the sources.
  • testing-conventions.meta.md keeps the precedents and the formatter choice, with a pointer to the new file.
  • The rule change this needed: single-source-of-truth.md defined sidecars for documents only, and put a config's reasons in comments. It now also allows a <config>.meta.md, named for the whole file name, when the reasons need sources, cases or options not taken. The config's header comment links to it once. Overrule it if you'd rather keep sidecars to documents.
  • Self-review: no links were left pointing at the moved sources.
  • Checks: CI run 44968554671 passed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: done in 087a6e3.

  • The extension: .devcontainer/devcontainer.json installs CucumberOpen.cucumber-official. Checked on 2026-11-22: version 1.12.1, updated 2026-11-07, about 473,000 installs, source in cucumber/vscode.
  • Settings: the extension's own package shows its default paths miss e2e/. So cucumber.features and cucumber.glue point it at e2e/features and e2e/steps, and it is the formatter for .feature files.
  • Network: the kit's network policy already allows the marketplace.
  • Not verified:
    • Opening the container in VS Code, and steps linking to their definitions.
    • Which formatter version the extension bundles, since its code is minified. If it disagrees, npm run format:check decides.
  • Checks: CI run 44981613389 passed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: kept e2e/features/, and recorded the decision in 075536d.

  • Options compared:
    • features/: Cucumber's default and gol's layout. cucumber-js and the VS Code extension need no paths configured for it.
    • e2e/: Next.js's own with-playwright example puts Playwright tests there.
  • Taken: e2e/features/, your choice.
  • Recorded: the reasons and the option not taken are in testing-conventions' sidecar, so a later session doesn't move the folder. No files moved.
  • Checks: CI run 44981972453 passed.

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Reply to this feedback: the five requests from the session are now recorded, each with a reply.

Request Recorded Reply Commit
Add gherkin-utils and gplint comment reply baf20e6
Approve the banned-word list comment reply 9aae08d
.gplintrc.meta.md sidecar comment reply 1eacfee
VS Code Cucumber extension comment reply 087a6e3
Keep e2e/features comment reply 075536d

@DetweilerRyan

Copy link
Copy Markdown
Owner Author

Follow-up to this feedback: the guard against it happening again is in #58. It adds a boundary in AGENTS.md, which every session loads, and the definition of in-session feedback in the workflow index and revising-work.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(e2e): Add Gherkin acceptance tests with playwright-bdd and Screenplay actors

1 participant