Skip to content

Split the Python tests over ten CI shards instead of eight - #2919

Open
MaxGhenis wants to merge 1 commit into
mainfrom
ci-ten-test-shards
Open

MaxGhenis wants to merge 1 commit into
mainfrom
ci-ten-test-shards

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Shards 1/8 to 7/8 of the Python tests now take 70 to 89 minutes against their 90-minute limit, and the limit cancelled a shard in 5 of the 14 recent runs counted below. This PR spreads the tests over ten shards. It changes no test and no model code.

What happened from 14 to 16 November

Why ten and not nine

The change

  • test-suite.yaml runs ten shards. Shards 1 to 9 share the tests that do not need the dataset. Shard 10 runs the tests that do, with one worker, as shard 8 did.
  • The workflow's history comment gains a "Ten, not eight, for time" paragraph with these runs.
  • Makefile, docs/engineering/skills/testing.md, pr_code_changes.yaml and docs/book/validation/ubuntu-vs-macos.md now give the new count. The last two had said seven.
  • The 90-minute limit and shard_durations.toml stay as they are. If a shard nears the limit on main, the next step is to regenerate the durations table from ten-shard logs with .github/test_durations.py.

Checks

  • The branch ruleset requires only Test / Result and Lint, so the renamed Test / Python K/10 checks need no settings change.
  • test_ci_test_shards.py checks that the matrix numbers its shards from 1 to N and gives the last shard one worker. Both still hold. Four of its properties draw at most eight shards; Let the shard-assignment properties draw up to 16 shards #2920 tracks raising that to 16.
  • Because this PR changes test-suite.yaml, the release workflow also builds and tests the wheel on it.

CI at dbca2c6be

  • All 43 checks passed. In run 43862308780, shards 1/10 to 9/10 took 59 to 74 minutes and shard 10/10 took 22. The release workflow's run of the same suite (run 43862308838) took 57 to 69 minutes on shards 1/10 to 9/10.

Open PRs that touch the same lines

axiom: n/a: CI test sharding only

🤖 Generated with Claude Code

Shards 1/8 to 7/8 took 70 to 89 minutes in 14 recent runs, against a
90-minute limit, and the limit cancelled a shard in 5 of them. Nine shards
left little room on #2670's and #2840's branches; ten shards on #2840 took
62 to 73 minutes. Shard 10/10 runs the tests that need the dataset, with one
worker, as shard 8/8 did.

The Makefile, the testing skill, the code-changes workflow and the Ubuntu
versus macOS page now give the new count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review status: COMPLETE at dbca2c6bec2133cde8669887fb03502ea66f9a4e — no findings or material evidence gaps. Approval is deferred while exact-head CI is unfinished.

The ten-shard wiring preserves the dataset-only last shard with one worker, dynamically numbers ordinary shards 1–9, and leaves the Result gate and required context names unchanged. The release-wheel workflow reuses the same ten-shard suite.

57 focused tests passed (54 existing tests plus three N10 diagnostics), zero failures. A real 53-node pytest sample was assigned exactly once across all ten current conftest shards, with dataset nodes only on shard 10. The actual Result checker accepted complete records and rejected missing/duplicate logs. The weighted and unknown-node fallback paths were also checked. These are routing/gate tests, not a claim that the whole country suite has finished.

Independent technical-source review preserved 29 official GitHub originals covering all eleven cited runs, six explicit 90-minute timeout annotations and the applied branch rules. The cited cancellation examples and rounded nine-/ten-shard timings are corroborated. The broader fourteen-run/81-minute statistic remains attributed rather than independently reconstructed; the review does not promise a future runtime bound.

No country-model formulas, tests, dependency lock or sharding algorithm change. Country-law review is not applicable; no private-data or population run was used. Required current-head checks, including Result and Lint, must pass before approval. Pending CI can be checked later without repeating this completed review.

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at dbca2c6bec2133cde8669887fb03502ea66f9a4e after the CI-readiness follow-up.

The completed independent code/source review remains applicable: 57 focused tests passed, including actual ten-shard collection and missing/duplicate outcome-gate checks; no findings or material gaps remain. The commit and code are unchanged, so no tests or model work were repeated.

Current official checks pass at this exact head, including required Lint and Test / Result; the PR is ready and mergeable. The added CI-results paragraph is supported by the two linked run payloads: test steps round to 59–74 minutes (dataset shard 22), and release test steps to 57–69. The current check set has 43 successful contexts plus a skipped Publish context. These measurements remain observations, not a future runtime guarantee.

@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Thanks, María. An independent review of dbca2c6be is queued and should start by about 15:30Z; its result will go here. Please hold the merge until then.

@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Thanks, María. An independent review of dbca2c6 approved it, with no finding above P3.

Its one P3 predates this PR. In docs/engineering/skills/testing.md, the example command runs four workers on every shard, while CI gives shard 10/10 one (test-suite.yaml:264). The routing note also leaves out the duration table, which places each listed test before hashing places the rest (conftest.py:257-262). Both can wait for a follow-up.

The reviewer couldn't reach GitHub, so I checked each run this PR cites. Every cancellation and timing matches its run within a minute. Shard 3/8 hit the 90-minute limit in 43617721320, 43805734007 and 43814986259, and shard 7/8 in 43752493693 and 43619418570. On #2670, shards 1/9 to 8/9 averaged 72 and 76 minutes (43715128471, 43796151601), and on #2840 shards 1/10 to 9/10 took 62 to 73 (43805104592).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants