Skip to content

Size QRF prediction workers from joblib.cpu_count() and stop comparing thread counts - #1684

Open
anth-volk wants to merge 11 commits into
mainfrom
claude/qrf-predict-workers-joblib-cpu-count-37170b
Open

anth-volk wants to merge 11 commits into
mainfrom
claude/qrf-predict-workers-joblib-cpu-count-37170b

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

QRF prediction threads now default to the CPUs the process may use, and QRF thread counts no longer have to match between a pool build, a release, or a checkpoint and its resume. Tests require byte-identical imputations for one and four prediction workers and for one and four fit jobs.

What changes

Default thread count

  • With POPULACE_FIT_PREDICT_WORKERS unset or blank, the draw's thread pool has joblib.cpu_count() threads instead of os.cpu_count(). This applies in qrf.py, worker_identity.py and stacked_spine.py. The variable still overrides the default.
  • POPULACE_FIT_N_JOBS unset stays n_jobs=-1, which joblib already resolves to joblib.cpu_count() threads.

Primary-QRF worker identity

  • The worker binding moves to schema 2. Both thread counts are recorded in a new audit_thread_controls field and are no longer part of semantic_identity.
  • Pool loading, checkpoint resume, the pool's checkpoint identity and the spec engine's worker resolution compare only semantic_identity.
  • Building a binding still refuses a malformed count. Schema-1 records still validate, and a pool holding one is refused because its worker identity changed.
  • If the live environment cannot produce a binding, the error names the live setting instead of calling the recorded binding malformed.

Resume

  • QRF.fit_draw_next no longer compares the fit-job count saved in QRFChainState. The field is excluded from dataclass equality and stays in to_dict, so saved states keep their bytes.
  • The ACS transfer target bank reuses a target saved under another fit-job count. It used to mark the target invalid and rebuild it.
  • tools/build_us_puf_support_base.py records both POPULACE_FIT_* variables and skips them on resume, through StageRuntime(resume_ignored_run_config_paths=...). BLAS and OpenMP thread counts and PYTHONHASHSEED stay compared.

Resume and identity effects

No resume migration is included.

  • A pool build interrupted before this change cannot resume after it, because its worker identity hashes the edited qrf.py, worker_identity.py and stacked_spine.py.
  • A base run interrupted before this change cannot resume after it, because its run configuration records a digest of the builder and the packaged source.
  • A compatibility attestation written before this change for the scoring-only relocation of a sealed schema-9 pool no longer validates, so that relocation needs a new one.
  • That relocation still compares thread counts: its frozen schema-4 record is unchanged.

Environment variables

None of these is a secret, and all are optional.

  • POPULACE_FIT_N_JOBS and POPULACE_FIT_PREDICT_WORKERS: no workflow, Dockerfile or committed Modal plan sets them. Tests, the graph parity tools and evidence scripts do. A Modal plan may set them because its environment allowlist accepts POPULACE_*. Primary-QRF workers inherit them from the parent process environment.
  • LOKY_MAX_CPU_COUNT: tools/build_us_fiscal_refresh_release.py sets it with setdefault to at most 16. Pool and base builders do not set it, and a Modal plan cannot, because the allowlist refuses the name.

Test evidence

  • The new tests were seen to fail first: 38 for the default and identity work, and 7 for the resume work. A further test proves the pool loaders fail if the worker comparison also requires equal thread counts.
  • All 140 tests added or changed by this branch pass, run individually by node ID. The full suite was not run locally.
  • Forests with a leaf-size cap (max_samples_leaf): two cases draw byte-identical values with 1 and 4 fit jobs and prediction workers. One sets max_samples_leaf=5; the other uses the automatic 4,096 cap on a target with 24 distinct values. Each fit wraps random.seed and random.shuffle and requires one seed per tree, then one shuffle per leaf of that tree, all on the thread that called the fit. A second test shows the draws change when other leaf samples are kept (984 and 86 of 1,001 rows). With a temporary change that made the leaf subsample depend on the thread count, these tests failed and the earlier tests passed. They also pass with Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682's _InPlaceLeafQRF merged in. Two more review rounds covered these tests.
  • Pins were regenerated with the repository's tools, and their --check modes pass on this merge of main.
  • Five rounds of independent review ran; the last found nothing.

Not verified, and for reviewers

  • No test or receipt shows what joblib.cpu_count() returns inside a Modal container. It reads a CPU quota only from cpu.max or cpu.cfs_quota_us.
  • Byte-identity was tested on small fixtures, not a full-size build.
  • microcosm-fit imports joblib but receives it only through its other dependencies. Declaring it would change uv.lock and the approved lock digest, so it is left for a separate change.
  • stacked_spine._legacy_primary_qrf_worker_execution_binding is unused and now records the joblib default. It could be deleted separately.
  • An ACS transfer first target that is rebuilt starts its chain again and records the live fit-job count. Its draws do not change.
  • This branch and Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682 both edit qrf.py, so whichever merges second needs a re-pin.

🤖 Generated with Claude Code

anth-volk and others added 9 commits November 17, 2026 14:52
With POPULACE_FIT_PREDICT_WORKERS unset, the QRF draw's thread pool,
the primary-QRF worker identity and the older stacked-spine binding now
take their width from microcosm.fit.qrf.default_predict_workers(), which
returns joblib.cpu_count(): the smallest of the machine's CPU count, the
affinity mask, a cgroup CPU quota and LOKY_MAX_CPU_COUNT. os.cpu_count()
counted only the machine's CPUs. The variable still overrides it.

The worker identity's cache keys an unset variable by the resolved
width, because joblib.cpu_count() can change within a process.

Tests compare 1 and 4 prediction workers and 1 and 4 fit jobs on fixed
seeds for a single-forest target, a three-sign gated target and a
fit_draw_next chain, and require byte-identical draws.

Re-pinned the attested fit.qrf hashes: repin, coverage (41/41), repin,
--check clean, then graph_parity_repin fit.qrf under Python 3.14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unset POPULACE_FIT_PREDICT_WORKERS now records resolution
"joblib_cpu_count_fallback". Validation still accepts the older
"os_cpu_count_fallback", so identities recorded before stay readable.
legacy_primary_qrf_worker_execution_binding maps the new label back to
the old one in the frozen schema-4 shape, as it already freezes policy,
overrides and bound_names, so the scoring-only relocation of schema-9
pools still finds only its two permitted differences. The older
stacked-spine binding builds that schema-4 shape and keeps the old label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The frozen schema-4 projection reports an unset prediction-worker
  count as schema-4 code recorded it, os.cpu_count() under
  "os_cpu_count_fallback", so the scoring-only relocation of schema-9
  pools compares what it compared before. The live identity still
  records joblib.cpu_count().
- The older stacked-spine binding records "joblib_cpu_count_fallback".
- The worker identity reads joblib.cpu_count() once per cache entry, so
  the identity records the count its key holds.
- A non-integer LOKY_MAX_CPU_COUNT fails with an error naming it.
- Three test caches key by the count the identity uses instead of
  os.cpu_count(), so a mid-session LOKY_MAX_CPU_COUNT change recomputes.
- Docstrings: the joblib count's floor of 1, cgroup v1 and v2 quotas,
  and joblib's serial mode.

Re-pinned the attested fit.qrf hashes: repin, coverage (41/41), repin,
--check clean, then graph_parity_repin fit.qrf under Python 3.14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With POPULACE_FIT_PREDICT_WORKERS unset, the prediction-worker count now
comes from the machine (joblib.cpu_count()). The worker identity compared
the resolved count, so a pool built on a host with more CPUs than the
release tool's LOKY_MAX_CPU_COUNT cap of 16, or a checkpoint resumed on
another host, was refused although the count changes no imputed value.

The worker binding is now schema 2. POPULACE_FIT_N_JOBS and
POPULACE_FIT_PREDICT_WORKERS are recorded in a new audit_thread_controls
field next to audit_aliases, outside the semantic identity that pool
loading, checkpoint resume, the pool's checkpoint identity and the spec
engine's worker resolution compare. Building a binding still refuses a
malformed count. Schema-1 bindings still validate and are refused as a
changed identity. The semantic identity is cached per lock only. The
spec template names schema 2 and the new field; pins re-pinned.

The scoring-only relocation of the sealed schema-9 pool keeps comparing
its frozen schema-4 shape, thread counts included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POPULACE_FIT_N_JOBS sets how many threads fit each forest and changes no
draw, so a resume no longer has to match it.

- QRF.fit_draw_next no longer compares the count saved in the chain state,
  and QRFChainState equality ignores it. The state still records the value
  in force when the chain started, so saved states keep their bytes.
- The ACS transfer target bank checks a saved target's predecessor digest
  with the count that target saved, so it reuses the target instead of
  rebuilding it.
- The PUF support base builder keeps recording both QRF thread variables
  in its run configuration and skips them on resume, through StageRuntime's
  new resume_ignored_run_config_paths. BLAS and OpenMP thread counts and
  PYTHONHASHSEED stay compared.

Also address review round 2: a live environment that cannot be bound now
reports its own error instead of a malformed recorded worker, and the
docstrings, operator docs and changelog describe the new behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the operator documents, the runbook, the changelog and the
docstrings in line with what is compared now: worker identity version 2,
sidecars that may differ in their audit fields, and run-config values a
resume skips. The runbook's staging plan is pinned to a commit that still
uses os.cpu_count(), and the changelog's CPU-quota claim now names the
cgroup files joblib reads.

Add a test that changes the fit-job count between chain steps with real
threads and checks each step fits with the live count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a test that writes a stacked pool under one pair of QRF thread counts
and opens it through the ready, release and scoring loaders under another.
It fails when the worker comparison also requires equal counts.

Correct what the documents claim: why an older base run cannot resume,
which thread variables a resume compares at which commit, the execution
config and legacy materializer versions, and that only the thread counts
stopped mattering, not the rest of the worker identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 5d584290f. Neither thread count changes a draw, so dropping them from the identity and resume checks is safe. Two things before merge, and one note for the first Modal run.

Why the counts change no draw

  • scikit-learn fixes each tree's seed before tree building and apply run in parallel. quantile-forest 1.4.2 then fills the leaves one tree at a time, in tree order, so its random.seed/random.shuffle on Python's global random doesn't depend on n_jobs.
  • The draw's thread pool only predicts. It computes each chunk's rows independently and writes them back by row range, and the draw takes its uniforms for all rows before the pool starts.
  • The gate classifier is a HistGradientBoostingClassifier, which uses OpenMP threads, and those counts stay compared.

Before merge

  1. Add a capped-leaf case to the byte-identity tests. None of test_qrf_worker_counts.py's comparisons sets max_samples_leaf, and capped leaves are the one path where quantile-forest 1.4.2, and #1682's _InPlaceLeafQRF, use Python's global random. It would pass today: on invented data, a forest with max_samples_leaf=5 and a near-discrete target draws the same bytes under 1 and 4 fit jobs and prediction workers.
  2. Check that no US pool build, PUF base run or scoring-only relocation is in flight. As the PR says, none of them can resume after this merges, and the relocation needs a new compatibility attestation.

If #1682 or #1667 merges first, re-run tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py, and after #1682 also tools/graph_parity_repin.py fit.qrf: all three PRs move the US seed-protocol pins, and #1682 also edits qrf.py and the fit.qrf parity pin.

For the first Modal run

audit_thread_controls in its worker binding will show what joblib.cpu_count() resolved to inside the container. That answers the PR's open question without a separate test.

Checked

  • 510 of 512 tests on the touched surface pass locally (macOS arm64, Python 3.14), including the 23 in test_qrf_worker_counts.py and the 37 changed cases in test_us_stacked_spine.py; the 2 skips need the US engine.
  • On macOS, joblib.cpu_count() equals os.cpu_count() (15 here), so builds on a Mac keep the same draw thread pool width.
  • The PUF base builder's resume ignores only thread_environment.POPULACE_FIT_N_JOBS and .POPULACE_FIT_PREDICT_WORKERS; BLAS, OpenMP and PYTHONHASHSEED stay compared.

A leaf-size cap (max_samples_leaf) makes the fit keep a seeded subsample of
each larger leaf, taken with Python's global random: for each tree, one
random.seed with the tree's seed, then one random.shuffle per leaf. None of
the existing byte-identity tests used a cap, so a subsample that changed with
the fit-job or prediction-worker count would have passed them.

The new tests fit and draw two capped cases under 1 and 4 fit jobs and
prediction workers and require equal float64 bytes:
- an explicit max_samples_leaf=5 on a continuous target;
- the automatic 4,096 cap the fit applies to a target with 24 distinct values.

Each fit wraps random.seed and random.shuffle and requires one seed per tree
followed by one shuffle per leaf of that tree, all on the thread that called
the fit. A second test shows the draws change when other leaf samples are
kept (984 and 86 of 1,001 rows), so equal bytes are not vacuous.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Contributor Author

Thanks. Item 1 is in 827be5a. There are two capped-leaf cases: an explicit max_samples_leaf=5, and the automatic 4,096 cap on a target with 24 distinct values. Each case requires byte-identical draws with 1 and 4 fit jobs and prediction workers. Each fit also checks that every leaf was subsampled (one random.seed per tree, then one random.shuffle per leaf) on the thread that called the fit. A further test shows the draws change when other leaf samples are kept, so equal bytes aren't vacuous. They also pass with #1682's _InPlaceLeafQRF merged in.

Item 2: Anthony will check before merge that no interrupted pool build, PUF support base run or scoring-only relocation would be resumed at a commit after this one.

On the first Modal run: as far as I can trace, audit_thread_controls is written only into the worker record that tools/build_us_multispine_pool.py builds. That builder isn't a stage in tools/modal_us_stage_plan.py. The Modal base stage runs tools/build_us_puf_support_base.py, which records the raw POPULACE_FIT_* values (null when unset) and no worker record. So that run won't show what joblib.cpu_count() returned. _cpu_visibility() in tools/modal_us_stage.py already records os.cpu_count() and the CPU affinity, and adding joblib.cpu_count() there would answer the question. That belongs in a separate change, not this PR.

origin/main now has #1682 (the in-place QRF leaf array), which also edits
microcosm.fit.qrf. The conflicts were the recorded digests both sides moved:
the US seed-protocol and seed-map pins, the two spec_sha256 pins in tests, the
spec-engine coverage report and the fit.qrf parity pin. They were regenerated
with tools/graph_parity_repin.py fit.qrf, tools/repin_us_spec_engine_contracts.py
and tools/spec_engine_coverage.py, and both comments are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants