Repository navigation
ci: reduce flaky tests and run Node and extension tests in parallel - #1943
Conversation
Wait for the jgd startup profile child to publish completion before reading its status file instead of polling a file the child replaces by rename, which fails on Windows with Permission denied or blocks the rename. Give the parallel sess setup follower and installer fixture budgets that cover a slow Windows source install.
Fake timers in the extension host also capture timers from VS Code and unrelated extension code, so clock.countTimers() intermittently reported foreign timers. Track timeouts scheduled from the session and terminal modules and assert that those are cleared.
Wait up to 45 seconds for languageserver diagnostics and client startup, and report the document text, version, client state and current diagnostics on timeout instead of a generic notebook execution message. Always clear the shared Interactive input after the lint test so a timeout does not cascade into later tests, and give the bulk session setup, which starts four arf sessions serially, its own timeout.
Each OS ran R integration checks, Node tests and extension-host tests back to back in one job, which made macOS the bottleneck. Add a suite matrix dimension so R integration and Node tests run in one job and extension-host tests in another, after the same R, sess and arf setup.
Replace the suite matrix dimension and per-step conditions with separate test-node and test-extension jobs. Move the shared setup into two local composite actions: setup-test-tools installs R, pnpm and Node.js, and prepare-test-runtime installs R test dependencies, builds and installs sess, compiles the tests and installs arf. sess bootstrap checks still run between the two, before R test dependencies exist. Re-include .github/actions in the path filter, and let Dependabot update actions pinned inside the composite actions.
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed commit 6d1c0a896cfa4437af8d768b58a499383e0766ed. The job split preserves the bootstrap-before-dependency-install order and existing suite coverage. The Windows completion marker and scoped timer tracking look sound. One issue remains: the new diagnostic wait budgets can exceed the enclosing Mocha test timeout and let shared-input cleanup run during a later test; see the inline comment.
Validation on macOS:
- Build/test compilation, TypeScript checking, and lint/format checks for the changed TypeScript files passed.
R/tests/attach_sess.RandR/tests/prepare-jgd.Rpassed.- All 28 terminal lifecycle tests passed in VS Code 1.145.2. An intentionally leaked pending timeout in a temporary copy was detected by the new tracker.
- All six affected Interactive language-service tests plus their notebook-creation prerequisite passed with the pinned arf 0.5.3 runtime (7 tests).
- A deterministic fake-clock probe using the compiled diagnostic wait helper reproduced the 60-second timeout and next-test-before-cleanup ordering.
- All 11 hosted checks are green on this commit.
…imeout Tests that wait for languageserver startup or diagnostics now raise their timeout to cover every sequential 45-second wait. Each wait is also capped by the running test's deadline minus a cleanup reserve, so a slow wait fails inside its own test and its cleanup cannot run during a later test that shares the Interactive input.
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed d789fe9a2b57b32888bd1cb531a0c2a292243077. The previous P2 finding is addressed: the six affected language-service tests now budget for all sequential waits, and the shared wait helper reads the current test timeout and caps its deadline with a five-second cleanup reserve. I found no additional actionable issues in the update.
Validation:
- Build/test compilation and lint/format checks passed.
- All six affected Interactive language-service tests and their notebook-creation prerequisite passed in VS Code 1.145.2 with the pinned arf 0.5.3 runtime (7 tests).
- Eight deterministic probe checks using the compiled helper passed: the original 35s + 35s case, four diagnostic waits just below 45s each, the enclosing-deadline cap, and the individual diagnostic timeout. In each case, the following test observed completed cleanup.
- All 11 hosted checks are green on this commit.
Approving the updated PR.
Summary
Over the past month, the
mainworkflow'stestjobs failed often on single OSes, and macOS took a median of 12 minutes because every suite ran back to back. This PR fixes the most frequent flaky failures that need only test changes, and splits each OS into two parallel jobs.Flaky test fixes
Verify managed jgd preparation(most frequent; every inspected failure isPermission deniedonprofile-declined/startup.status): the test polled the status file every 10 ms while the child replaced it withfile.rename. On Windows, an open reader makes the rename fail, and a reader that races the rename getsPermission denied. The child now publishes a completion marker atomically afterprofile.Rreturns, and the parent reads the final status once.attach_sess.Rparallel_approve: the follower timed out after 20 seconds, and the test timed out waiting 10 seconds for the owner's install. Both budgets cover the owner's whole source install, which is slow on Windows runners. Raise them to 60 seconds (and the outer child wait to 90 seconds). Passing runs take the same time.1 !== 0,4 !== 0):clock.countTimers()counts every fake timer in the extension host, including timers from VS Code and unrelated extension code. A newtrackModuleTimerstest helper counts only timeouts scheduled from the session and terminal modules and checks that they are cleared.Faster CI
testjob with two parallel jobs per OS:Node (OS)runs the R integration checks, sess tests, tmux and Node tests, andExtension (OS)runs the plot lifecycle and extension-host tests. On the latest successful macOS run, the extension-host part alone took about 4 minutes after roughly 2 minutes of setup, compared with 8.5 minutes for the whole job. Setup now runs twice per OS, so total runner minutes go up.mainhas no required status checks, so the new job names do not block merges.setup-test-tools(R, pnpm, Node.js and npm dependencies) andprepare-test-runtime(R test dependencies, sess build and install, test compilation and arf). The sess bootstrap checks still run between the two, before R test dependencies are installed..github/actions/**in the path filter, so changes to the composite actions alone still run the tests./.github/actions/*as well, sincedirectory: /covers only workflows and a rootaction.yml.Not included
interruptinstead of SIGINT), so it will be a separate PR.src/webViewer. It saves only about 10 seconds per run.