Skip to content

Store page stylesheets and fonts in the trace and replay them from it - #533

Open
vishnuv688 wants to merge 2 commits into
mainfrom
feat/499-archive-resources
Open

vishnuv688 wants to merge 2 commits into
mainfrom
feat/499-archive-resources

Conversation

@vishnuv688

Copy link
Copy Markdown
Member

What & why

Closes #499. Part of #494.

The player replayed a page from its captured DOM but loaded that page's stylesheets and fonts from the live site under test. As a result:

  • cross-origin fonts failed without CORS (the Font Awesome icon on the-internet's Login button rendered as an empty box);
  • the hosted player, being https, blocked an http site's styles;
  • a trace replayed wrongly once the site changed, and not at all for a site the viewer cannot reach;
  • opening a trace made requests to the tested site.

In trace mode the zip now stores the stylesheets and fonts the page used, and the player replays them from the archive, in show-trace and the hosted player alike. Anything not stored still loads from the site, as before. Live mode is unchanged.

Type of change

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Polish (an improvement to an existing feature)
  • Breaking change (existing behavior changes for users)
  • Documentation
  • Internal (build, CI, dependencies, tooling)

Packages touched

  • shared (types and contracts)
  • core (framework-agnostic capture/reporting)
  • elements (published element/snapshot API — @wdio/elements)
  • service (WebdriverIO adapter)
  • nightwatch-devtools (Nightwatch adapter)
  • selenium-devtools (Selenium adapter)
  • selenium-devtools-py (Selenium Python adapter)
  • backend (server)
  • app (UI)
  • script (page-injected runtime)
  • trace (Trace mode)

Plus conformance (smoke fixture site and expectations, capability matrix).

Capability matrix

  • packages/conformance/src/features.ts updated, and tables regenerated with pnpm docs:capabilities, if this adds or changes user-facing behaviour

New feature page-resources, backed per adapter by a test that the adapter passes the mode in its trace drain and keeps what comes back. The docs-site tables are refreshed with the docs-site PR.

Notes for reviewers

Why this touches every adapter, and why the work is not there: the capture is in the page collector (script/src/collectors/pageResources.ts), the storage and filtering in trace (shared by the JS adapters and the backend that builds Python's archives), and the replay in app. Each adapter only passes a mode in the drain it already runs and keeps what comes back (core page-resource-store.ts for the JS adapters, page_resources.py plus the backend store for Python).

How it works

  • Capture. The collector watches Resource Timing for stylesheet and font loads and re-reads each from the browser cache through the fetch the page had before the network patch (cache: 'force-cache', 1.8 s timeout). A same-origin stylesheet it cannot re-read falls back to its cssRules; a cross-origin one without CORS replays live. Bodies are kept only when the content type is a stylesheet or a font, skipping a small analytics host list. The drain carries the keys of what is already stored, so a new document does not re-send them.
  • No duplicate Network rows. The re-reads are claimed in the collector's own Resource Timing rows, and trace-network.ts drops the matching BiDi or performance-log row only when it can place it on the browser's clock and its Sec-Fetch-Dest is empty. When unsure it keeps the row, so the worst case is a leftover duplicate, never a lost page load.
  • Zip. Each body is stored once at resources/<sha1>, indexed by page-resources.ndjson. Each archive (per-test slices included) stores only what its own documents reference, following url() and @import inside stored stylesheets, capped at 1 MiB per resource and 8 MiB per archive. Old zips without the index open as before.
  • Replay. archived-resources.ts turns stored resources into blob: URLs, resolving each stylesheet's url()/@import against the stylesheet's own URL. The rewrite runs after replay-safety filtering and writes only quoted blob: or absolute URLs. The CSP adds blob: to style-src and font-src only, in player-csp.ts and vercel.json.
  • CSS scanning (shared/css-urls.ts) is one forward scan, no regex, tested linear on hostile input of a million characters.

Behaviour change for every trace: replay now honours a page's own <base href> (http/https only, otherwise the page URL), not only for stored resources. Before, the player always resolved against the page URL. This matches how the page itself resolved its URLs.

Options: traceResources: 'none' | 'styles', default 'styles', trace mode only. Python: enable(trace_resources=...), --devtools-trace-resources, ini devtools_trace_resources, env DEVTOOLS_TRACE_RESOURCES. Images are a follow-up ('all').

Not covered (README): images; cross-origin stylesheets without CORS; CSS added through the CSSOM; the page can observe the re-reads (its PerformanceObserver, RUM scripts, a service worker), and a CORS refusal logs a console error; a row without a browser-clock start time can leave a duplicate.

Review: two independent rounds. The first found one blocking issue (the end-of-test drains reset the WDIO service's "the last action navigated" flag, skipping the wait for the new page to load) and 11 more; all are fixed with tests that fail without them. The second confirmed the fixes; its four low findings are fixed too.

Testing

  • Frozen install, pnpm build, pnpm test (3731 passed, 1 skipped), pnpm test:ui (31 of 31), pnpm lint, pnpm docs:capabilities --check are green; no workspace import leaks in any bundle.
  • Python: 900 passed. Two test_instrumentation.py tests fail locally with and without this PR.
  • The examples smoke job gets a fixture stylesheet with an @font-face and checks every adapter's zips carry both.

Needs checking by hand (after pnpm build; pnpm smoke:fixtures and export DEVTOOLS_EXAMPLE_BASE_URL=http://127.0.0.1:4567)

  1. DEVTOOLS_MODE=trace with pnpm demo:wdio:mocha, pnpm demo:selenium, pnpm demo:nightwatch, pnpm demo:python:pytest; then each in live mode, which should look as before with no extra stylesheet rows in Network.
  2. Per zip: unzip -l <zip> | grep page-resources lists the index; pnpm show-trace <zip> renders styled with no duplicate fixture.css or font rows in Network.
  3. Against the public the-internet site in trace mode, the Login button's icon renders in show-trace.
  4. pnpm player:hosted, open the same zips at http://localhost:4173: styled, icon rendered, no CSP errors.
  5. traceResources: 'none' (Python DEVTOOLS_TRACE_RESOURCES=none): no page-resources.ndjson, styles load live.

Not in this PR: the images option and the docs-site mirror.

Screenshots / recordings

To be added: the-internet's login page replayed in the hosted player, before and after.

🤖 Generated with Claude Code

The player replayed a page from its captured DOM but loaded the page's
stylesheets and fonts from the live site: cross-origin fonts failed
without CORS, an https player blocked http styles, a trace broke once
the site changed, and opening one made requests to the tested site.

In trace mode the page collector re-reads each stylesheet and font the
page loaded from the browser cache and hands it over in the drain every
adapter already runs. The writer stores only what each archive's
documents use, each body once at resources/<sha1> with a
page-resources.ndjson index, under per-resource and per-archive caps,
and drops the collector's own re-reads from the network rows. The
player turns stored resources into blob: URLs, resolving url() and
@import against each stylesheet, and replay honours a page's own http(s)
<base href>. The CSP allows blob: for styles and fonts.

traceResources: 'none' | 'styles' (default 'styles', trace mode only),
with trace_resources / --devtools-trace-resources / DEVTOOLS_TRACE_RESOURCES
in Python. Live mode is unchanged.

Closes #499

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two network rows in the backend export test lacked the required
startTime and type, and the Selenium option-wiring test replaced the
context's options with an object missing required fields. CI's
typecheck covers test files; the build and test runs do not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trace: store page resources and replay from the archive

1 participant