Repository navigation
Measure Behat run times reliably #384
Description
Every other performance change needs a trustworthy before/after number, and today the Behat timing data is incomplete.
WP_CLI_TEST_LOG_RUN_TIMES reports process run times, but with the default progress formatter its output is swallowed unless a scenario fails. The per-scenario part of the report is empty ("Top 20 (of 0) scenario run times"), and the scenario infix in RUN_DIR names is missing, because the current Behat version no longer exposes the feature file on the scenario node.
Steps:
- Make the run time report visible with the default formatter.
- Restore per-scenario timings.
- Record a baseline from GitHub CI on
mainfor a set of reference packages (for exampleentity-command,extension-command,core-command,cache-command), covering the MySQL, MariaDB and SQLite jobs.
The baseline lives in this issue, so the other issues in this epic can compare against it. Per-scenario timings are also what parallel runs need to balance their shards.
Activity
schlessera commentedon Oct 30, 2026
Measurement
Measured on GitHub Actions with the full test matrix of
core-commandandcache-command, run onmeasure/*branches of wp-cli/core-command#350 and wp-cli/cache-command#137. All runs use wp-cli-tests at the same base commit (9298472) and differ only in the run time report being enabled (WP_CLI_TEST_LOG_RUN_TIMES=stdout,40,40) with the changes of #399.The metric is the Behat wall time of each job, from the start of the Behat step to its last summary, for the 49 Linux jobs present in all runs. "Total" is the sum over these jobs, "per job" the median of each job's change against the mean of the two baseline runs. Both baselines run the identical code, so their spread is the noise level: about ±1% for
cache-commandand about ±5% forcore-command. All runs executed the same scenarios (about 4,518 forcore-commandand 1,162 forcache-commandacross the Linux jobs).core-commandtotalcore-commandper jobcache-commandtotalcache-commandper jobCollecting and printing the report adds no measurable cost; both differences are within the noise. The report now shows up in the job logs with the default progress formatter, including the slowest scenarios, which the other measurements in #394 can use from here on.