Repository navigation
Run scenarios in parallel #390
Description
Scenarios run strictly one after another, even though almost all of them are independent and the CI runners have several cores. In a local experiment with 213 entity-command scenarios, a serial run took 491 seconds. With 8 workers it took 164 seconds when split by feature file, where one large file set the floor. Split by scenario, it took 140 seconds (3.5×).
Steps:
- Give each worker its own state: temporary directory,
HOME, WP-CLI cache and test database.composer prepare-testsneeds to grant access to the additional databases. - Keep scenarios that rely on fixed shared resources, such as port 8080 or the scaffold test database, from running at the same time.
- Split the work by scenario rather than by file, balanced by the per-scenario timings from Measure Behat run times reliably #384.
- Merge the results into a single report and keep
behat-rerunworking.
Measuring: take the baseline from #384, then open a draft PR, run the same CI jobs against it with the parallel runner enabled and record both results here before we decide. CI runners have fewer cores than a development machine, so the gain there will be smaller than locally.
Activity
schlessera commentedon Oct 30, 2026
Measurement
Measured on GitHub Actions with the full test matrix of
core-commandandcache-command, run onmeasure/*branches of wp-cli/core-command#350 and wp-cli/cache-command#137. All runs use wp-cli-tests at the same base commit (9298472). The parallel run uses #405 withWP_CLI_TEST_PARALLEL=4, matching the 4 cores of the Linux runners.The metric is the Behat wall time of each job, from the start of the Behat step to its last summary, including any rerun of failed scenarios. "Total" is the sum over the Linux jobs present in all runs, "per job" the median of each job's change against the mean of the two baseline runs, which run identical code and show the noise level.
core-commandtotalcore-commandper jobcache-commandtotalcache-commandper jobThe comparison covers 48 Linux jobs for
cache-commandand 29 forcore-command, the ones whose logs could be read for all runs. Both ran the same scenarios as the baselines (1,138 and about 2,550 across these jobs).The run time of the Behat step roughly halves, and more than halves for
core-command, whose scenarios are longer and balance better over the workers.Two things came up along the way:
features/core-download.feature:524incore-commandfails whenever it runs with an empty WP-CLI cache, also on its own in a serial run without any of these changes. A serial run of the whole suite only passes because the scenario before it puts the same download in the cache. In a parallel run, it fails in whichever worker it does not follow that scenario, as in the PHP 7.2 job above. That needs a fix incore-command(orwp core download --no-extract) before parallel runs can be enabled there.composer behat-rerunafter a parallel run reran every scenario serially whenWP_CLI_TEST_PARALLELwas not set for the rerun as well. Reruns now always cover exactly the failed scenarios. The other failed job is the Windows one, where Chocolatey timed out installing Ghostscript before any scenario ran.schlessera commentedon Oct 31, 2026
Measurement after wp-cli/core-command#355
The
core-download.feature:524failure is fixed in wp-cli/core-command#355, which is merged. Thecore-commandmeasurements were repeated on top of it, with fresh baselines and the same wp-cli-tests commit:core-commandtotalcore-commandper jobEvery job of the matrix passes now, Windows and macOS included, and the parallel run executed the same 2,935 scenarios across the 33 Linux jobs compared here. The run time of the Behat step drops by about 58%, in line with the earlier measurement.
For #391, the run in containers on the same base (3,131 s, -64.2%) still fails the 16 MySQL jobs, for the reasons described there.