Skip to content

Fail CI retry chains on the job that actually failed - #59570

Open
GijsWeterings wants to merge 1 commit into
react:mainfrom
GijsWeterings:export-D128043004
Open

GijsWeterings wants to merge 1 commit into
react:mainfrom
GijsWeterings:export-D128043004

Conversation

@GijsWeterings

@GijsWeterings GijsWeterings commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary:
The E2E and Fantom retry chains reported their final failure through a separate report job. The test job always succeeded (its test step used continue-on-error: true), and report read its status output and exited 1 on the last attempt. This caused three problems:

  • The red check was a five-second status job, not the job that ran the tests, so finding the real failure meant opening a green job and looking for the step with a warning icon.
  • "Re-run failed jobs" re-ran only report. It read the previous attempt's cached status output and failed again without running any tests.
  • The Debug and Release matrix legs wrote the same status output, and GitHub keeps only the last value written to a given output name. If one flavor failed and the other passed and finished later, status came out empty, so no retry ran and the failure was dropped silently.

This change:

  • Removes the report jobs. The test step runs as two copies of the same step, gated on inputs.fail-on-error: earlier attempts use continue-on-error: true, so they stay green and record the failure in the status output, and the final attempt runs without it, so the test step itself fails the job. The two copies share their with: block through a YAML anchor. (A single step with continue-on-error: ${{ !inputs.fail-on-error }} doesn't work: GitHub evaluated it as true on the final attempt, so the failure stayed hidden. Step if: conditions read inputs correctly.)
  • Has each matrix leg write a flavor-specific output (debug_status / release_status). GitHub combines outputs with different names across matrix legs. The reusable workflow's status output is debug_status || release_status, or jobs.test.outputs.status for the single-job Fantom workflow.
  • Also triggers the retry jobs in test-all.yml on needs.<attempt>.result == 'failure'. The iOS report jobs counted a failed build or test job as a failure, and this keeps that behavior for every workflow.

The flavor is lowercased with tr rather than ${FLAVOR,,} because the macOS runners' default bash may be 3.2. A flavor with no matching output fails the step with an error instead of writing an output nothing reads; a red test job still triggers the retry through needs.<attempt>.result.

Changelog: [Internal]

Differential Revision: D128043004

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 28, 2026
@meta-codesync

meta-codesync Bot commented Oct 28, 2026

Copy link
Copy Markdown

@GijsWeterings has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128043004.

@GijsWeterings
GijsWeterings force-pushed the export-D128043004 branch 3 times, most recently from 4e79d81 to e682a84 Compare October 28, 2026 13:25
Summary:
The E2E and Fantom retry chains reported their final failure through a separate `report` job. The `test` job always succeeded (its test step used `continue-on-error: true`), and `report` read its `status` output and exited 1 on the last attempt. This caused three problems:

- The red check was a five-second status job, not the job that ran the tests, so finding the real failure meant opening a green job and looking for the step with a warning icon.
- "Re-run failed jobs" re-ran only `report`. It read the previous attempt's cached `status` output and failed again without running any tests.
- The Debug and Release matrix legs wrote the same `status` output, and GitHub keeps only the last value written to a given output name. If one flavor failed and the other passed and finished later, `status` came out empty, so no retry ran and the failure was dropped silently.

This change:

- Removes the `report` jobs. The test step runs as two copies of the same step, gated on `inputs.fail-on-error`: earlier attempts use `continue-on-error: true`, so they stay green and record the failure in the `status` output, and the final attempt runs without it, so the test step itself fails the job. The two copies share their `with:` block through a YAML anchor. (A single step with `continue-on-error: ${{ !inputs.fail-on-error }}` doesn't work: GitHub evaluated it as true on the final attempt, so the failure stayed hidden. Step `if:` conditions read `inputs` correctly.)
- Has each matrix leg write a flavor-specific output (`debug_status` / `release_status`). GitHub combines outputs with different names across matrix legs. The reusable workflow's `status` output is `debug_status || release_status`, or `jobs.test.outputs.status` for the single-job Fantom workflow.
- Also triggers the retry jobs in `test-all.yml` on `needs.<attempt>.result == 'failure'`. The iOS `report` jobs counted a failed `build` or `test` job as a failure, and this keeps that behavior for every workflow.

The flavor is lowercased with `tr` rather than `${FLAVOR,,}` because the macOS runners' default `bash` may be 3.2. A flavor with no matching output fails the step with an error instead of writing an output nothing reads; a red `test` job still triggers the retry through `needs.<attempt>.result`.

Changelog: [Internal]

Differential Revision: D128043004
@GijsWeterings
GijsWeterings force-pushed the export-D128043004 branch 3 times, most recently from 043a531 to 7da2f7a Compare October 28, 2026 15:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported p: Facebook Partner: Facebook Partner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant