Repository navigation
Demo: fail an Android Release E2E flow on the first attempt only (#59573) - #59573
Open
GijsWeterings wants to merge 2 commits into
Open
GijsWeterings wants to merge 2 commits into
GijsWeterings wants to merge 2 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
|
@GijsWeterings has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128054792. |
GijsWeterings
force-pushed
the
export-D128054792
branch
from
October 28, 2026 13:25
57ad580 to
8ad262d
Compare
Summary:
The E2E and Fantom retry chains reported their final failure through a separate `report` job. The `test` job always succeeded (its test step used `continue-on-error: true`), and `report` read its `status` output and exited 1 on the last attempt. This caused three problems:
- The red check was a five-second status job, not the job that ran the tests, so finding the real failure meant opening a green job and looking for the step with a warning icon.
- "Re-run failed jobs" re-ran only `report`. It read the previous attempt's cached `status` output and failed again without running any tests.
- The Debug and Release matrix legs wrote the same `status` output, and GitHub keeps only the last value written to a given output name. If one flavor failed and the other passed and finished later, `status` came out empty, so no retry ran and the failure was dropped silently.
This change:
- Removes the `report` jobs. The test step runs as two copies of the same step, gated on `inputs.fail-on-error`: earlier attempts use `continue-on-error: true`, so they stay green and record the failure in the `status` output, and the final attempt runs without it, so the test step itself fails the job. The two copies share their `with:` block through a YAML anchor. (A single step with `continue-on-error: ${{ !inputs.fail-on-error }}` doesn't work: GitHub evaluated it as true on the final attempt, so the failure stayed hidden. Step `if:` conditions read `inputs` correctly.)
- Has each matrix leg write a flavor-specific output (`debug_status` / `release_status`). GitHub combines outputs with different names across matrix legs. The reusable workflow's `status` output is `debug_status || release_status`, or `jobs.test.outputs.status` for the single-job Fantom workflow.
- Also triggers the retry jobs in `test-all.yml` on `needs.<attempt>.result == 'failure'`. The iOS `report` jobs counted a failed `build` or `test` job as a failure, and this keeps that behavior for every workflow.
The flavor is lowercased with `tr` rather than `${FLAVOR,,}` because the macOS runners' default `bash` may be 3.2. A flavor with no matching output fails the step with an error instead of writing an output nothing reads; a red `test` job still triggers the retry through `needs.<attempt>.result`.
Changelog: [Internal]
Differential Revision: D128043004
GijsWeterings
added a commit
to GijsWeterings/react-native
that referenced
this pull request
Oct 28, 2026
…ct#59573) Summary: Demo only, stacked on the retry-chain fix so the exported PR shows on real CI that a failure which passes on retry still leaves CI green. Do not land. This adds an RNTester Maestro flow that always fails, and makes `e2e-android-rntester.yml` exclude it (tag `demo-fail-first-attempt`) on retries (`retry-attempt > 0`). It behaves like a test that flakes once: - `android-release-only` excludes it from the Android Debug job. - `local-screenshot-baseline` keeps it out of the Maestro Cloud runs. - A `when: platform: Android` condition makes it a no-op in the iOS E2E job. It also tags `sectionlist-viewability.yml` `android-release-only`, because that flow currently fails in Android Debug on every attempt. Otherwise `retry_2 / test (debug)` would go red for an unrelated reason. What the PR run should show: - `test_e2e_android_rntester`: the Release "Run E2E Tests" step fails, but `test (release)` and `test (debug)` both finish green, and `retry_1` starts. - `test_e2e_android_rntester_retry_1`: Release re-runs only the unfinished flows. The demo flow is excluded and the rest passed in the first attempt, so the job passes. Debug skips the run because every flow already passed. - `test_e2e_android_rntester_retry_2` is skipped, and no check from this chain is red. Changelog: [Internal] Differential Revision: D128054792
GijsWeterings
force-pushed
the
export-D128054792
branch
from
October 28, 2026 15:58
8ad262d to
d3603c5
Compare
meta-codesync
Bot
changed the title
…ct#59573) Summary: Pull Request resolved: react#59573 Demo only, stacked on the retry-chain fix so the exported PR shows on real CI that a failure which passes on retry still leaves CI green. Do not land. This adds an RNTester Maestro flow that always fails, and makes `e2e-android-rntester.yml` exclude it (tag `demo-fail-first-attempt`) on retries (`retry-attempt > 0`). It behaves like a test that flakes once: - `android-release-only` excludes it from the Android Debug job. - `local-screenshot-baseline` keeps it out of the Maestro Cloud runs. - A `when: platform: Android` condition makes it a no-op in the iOS E2E job. It also tags `sectionlist-viewability.yml` `android-release-only`, because that flow currently fails in Android Debug on every attempt. Otherwise `retry_2 / test (debug)` would go red for an unrelated reason. What the PR run should show: - `test_e2e_android_rntester`: the Release "Run E2E Tests" step fails, but `test (release)` and `test (debug)` both finish green, and `retry_1` starts. - `test_e2e_android_rntester_retry_1`: Release re-runs only the unfinished flows. The demo flow is excluded and the rest passed in the first attempt, so the job passes. Debug skips the run because every flow already passed. - `test_e2e_android_rntester_retry_2` is skipped, and no check from this chain is red. Changelog: [Internal] Differential Revision: D128054792
GijsWeterings
force-pushed
the
export-D128054792
branch
from
October 28, 2026 16:03
d3603c5 to
1ac5529
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Demo only, stacked on the retry-chain fix so the exported PR shows on real CI that a failure which passes on retry still leaves CI green. Do not land.
This adds an RNTester Maestro flow that always fails, and makes
e2e-android-rntester.ymlexclude it (tagdemo-fail-first-attempt) on retries (retry-attempt > 0). It behaves like a test that flakes once:android-release-onlyexcludes it from the Android Debug job.local-screenshot-baselinekeeps it out of the Maestro Cloud runs.when: platform: Androidcondition makes it a no-op in the iOS E2E job.It also tags
sectionlist-viewability.ymlandroid-release-only, because that flow currently fails in Android Debug on every attempt. Otherwiseretry_2 / test (debug)would go red for an unrelated reason.What the PR run should show:
test_e2e_android_rntester: the Release "Run E2E Tests" step fails, buttest (release)andtest (debug)both finish green, andretry_1starts.test_e2e_android_rntester_retry_1: Release re-runs only the unfinished flows. The demo flow is excluded and the rest passed in the first attempt, so the job passes. Debug skips the run because every flow already passed.test_e2e_android_rntester_retry_2is skipped, and no check from this chain is red.Changelog: [Internal]
Differential Revision: D128054792