Repository navigation
Conversation
With POPULACE_FIT_PREDICT_WORKERS unset, the QRF draw's thread pool, the primary-QRF worker identity and the older stacked-spine binding now take their width from microcosm.fit.qrf.default_predict_workers(), which returns joblib.cpu_count(): the smallest of the machine's CPU count, the affinity mask, a cgroup CPU quota and LOKY_MAX_CPU_COUNT. os.cpu_count() counted only the machine's CPUs. The variable still overrides it. The worker identity's cache keys an unset variable by the resolved width, because joblib.cpu_count() can change within a process. Tests compare 1 and 4 prediction workers and 1 and 4 fit jobs on fixed seeds for a single-forest target, a three-sign gated target and a fit_draw_next chain, and require byte-identical draws. Re-pinned the attested fit.qrf hashes: repin, coverage (41/41), repin, --check clean, then graph_parity_repin fit.qrf under Python 3.14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unset POPULACE_FIT_PREDICT_WORKERS now records resolution "joblib_cpu_count_fallback". Validation still accepts the older "os_cpu_count_fallback", so identities recorded before stay readable. legacy_primary_qrf_worker_execution_binding maps the new label back to the old one in the frozen schema-4 shape, as it already freezes policy, overrides and bound_names, so the scoring-only relocation of schema-9 pools still finds only its two permitted differences. The older stacked-spine binding builds that schema-4 shape and keeps the old label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The frozen schema-4 projection reports an unset prediction-worker count as schema-4 code recorded it, os.cpu_count() under "os_cpu_count_fallback", so the scoring-only relocation of schema-9 pools compares what it compared before. The live identity still records joblib.cpu_count(). - The older stacked-spine binding records "joblib_cpu_count_fallback". - The worker identity reads joblib.cpu_count() once per cache entry, so the identity records the count its key holds. - A non-integer LOKY_MAX_CPU_COUNT fails with an error naming it. - Three test caches key by the count the identity uses instead of os.cpu_count(), so a mid-session LOKY_MAX_CPU_COUNT change recomputes. - Docstrings: the joblib count's floor of 1, cgroup v1 and v2 quotas, and joblib's serial mode. Re-pinned the attested fit.qrf hashes: repin, coverage (41/41), repin, --check clean, then graph_parity_repin fit.qrf under Python 3.14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With POPULACE_FIT_PREDICT_WORKERS unset, the prediction-worker count now comes from the machine (joblib.cpu_count()). The worker identity compared the resolved count, so a pool built on a host with more CPUs than the release tool's LOKY_MAX_CPU_COUNT cap of 16, or a checkpoint resumed on another host, was refused although the count changes no imputed value. The worker binding is now schema 2. POPULACE_FIT_N_JOBS and POPULACE_FIT_PREDICT_WORKERS are recorded in a new audit_thread_controls field next to audit_aliases, outside the semantic identity that pool loading, checkpoint resume, the pool's checkpoint identity and the spec engine's worker resolution compare. Building a binding still refuses a malformed count. Schema-1 bindings still validate and are refused as a changed identity. The semantic identity is cached per lock only. The spec template names schema 2 and the new field; pins re-pinned. The scoring-only relocation of the sealed schema-9 pool keeps comparing its frozen schema-4 shape, thread counts included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POPULACE_FIT_N_JOBS sets how many threads fit each forest and changes no draw, so a resume no longer has to match it. - QRF.fit_draw_next no longer compares the count saved in the chain state, and QRFChainState equality ignores it. The state still records the value in force when the chain started, so saved states keep their bytes. - The ACS transfer target bank checks a saved target's predecessor digest with the count that target saved, so it reuses the target instead of rebuilding it. - The PUF support base builder keeps recording both QRF thread variables in its run configuration and skips them on resume, through StageRuntime's new resume_ignored_run_config_paths. BLAS and OpenMP thread counts and PYTHONHASHSEED stay compared. Also address review round 2: a live environment that cannot be bound now reports its own error instead of a malformed recorded worker, and the docstrings, operator docs and changelog describe the new behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the operator documents, the runbook, the changelog and the docstrings in line with what is compared now: worker identity version 2, sidecars that may differ in their audit fields, and run-config values a resume skips. The runbook's staging plan is pinned to a commit that still uses os.cpu_count(), and the changelog's CPU-quota claim now names the cgroup files joblib reads. Add a test that changes the fit-job count between chain steps with real threads and checks each step fits with the live count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a test that writes a stacked pool under one pair of QRF thread counts and opens it through the ready, release and scoring loaders under another. It fails when the worker comparison also requires equal counts. Correct what the documents claim: why an older base run cannot resume, which thread variables a resume compares at which commit, the execution config and legacy materializer versions, and that only the thread counts stopped mattering, not the rest of the worker identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed at 5d584290f. Neither thread count changes a draw, so dropping them from the identity and resume checks is safe. Two things before merge, and one note for the first Modal run.
Why the counts change no draw
- scikit-learn fixes each tree's seed before tree building and
applyrun in parallel. quantile-forest 1.4.2 then fills the leaves one tree at a time, in tree order, so itsrandom.seed/random.shuffleon Python's globalrandomdoesn't depend onn_jobs. - The draw's thread pool only predicts. It computes each chunk's rows independently and writes them back by row range, and the draw takes its uniforms for all rows before the pool starts.
- The gate classifier is a
HistGradientBoostingClassifier, which uses OpenMP threads, and those counts stay compared.
Before merge
- Add a capped-leaf case to the byte-identity tests. None of
test_qrf_worker_counts.py's comparisons setsmax_samples_leaf, and capped leaves are the one path where quantile-forest 1.4.2, and #1682's_InPlaceLeafQRF, use Python's globalrandom. It would pass today: on invented data, a forest withmax_samples_leaf=5and a near-discrete target draws the same bytes under 1 and 4 fit jobs and prediction workers. - Check that no US pool build, PUF base run or scoring-only relocation is in flight. As the PR says, none of them can resume after this merges, and the relocation needs a new compatibility attestation.
If #1682 or #1667 merges first, re-run tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py, and after #1682 also tools/graph_parity_repin.py fit.qrf: all three PRs move the US seed-protocol pins, and #1682 also edits qrf.py and the fit.qrf parity pin.
For the first Modal run
audit_thread_controls in its worker binding will show what joblib.cpu_count() resolved to inside the container. That answers the PR's open question without a separate test.
Checked
- 510 of 512 tests on the touched surface pass locally (macOS arm64, Python 3.14), including the 23 in
test_qrf_worker_counts.pyand the 37 changed cases intest_us_stacked_spine.py; the 2 skips need the US engine. - On macOS,
joblib.cpu_count()equalsos.cpu_count()(15 here), so builds on a Mac keep the same draw thread pool width. - The PUF base builder's resume ignores only
thread_environment.POPULACE_FIT_N_JOBSand.POPULACE_FIT_PREDICT_WORKERS; BLAS, OpenMP andPYTHONHASHSEEDstay compared.
A leaf-size cap (max_samples_leaf) makes the fit keep a seeded subsample of each larger leaf, taken with Python's global random: for each tree, one random.seed with the tree's seed, then one random.shuffle per leaf. None of the existing byte-identity tests used a cap, so a subsample that changed with the fit-job or prediction-worker count would have passed them. The new tests fit and draw two capped cases under 1 and 4 fit jobs and prediction workers and require equal float64 bytes: - an explicit max_samples_leaf=5 on a continuous target; - the automatic 4,096 cap the fit applies to a target with 24 distinct values. Each fit wraps random.seed and random.shuffle and requires one seed per tree followed by one shuffle per leaf of that tree, all on the thread that called the fit. A second test shows the draws change when other leaf samples are kept (984 and 86 of 1,001 rows), so equal bytes are not vacuous. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. Item 1 is in 827be5a. There are two capped-leaf cases: an explicit Item 2: Anthony will check before merge that no interrupted pool build, PUF support base run or scoring-only relocation would be resumed at a commit after this one. On the first Modal run: as far as I can trace, |
origin/main now has #1682 (the in-place QRF leaf array), which also edits microcosm.fit.qrf. The conflicts were the recorded digests both sides moved: the US seed-protocol and seed-map pins, the two spec_sha256 pins in tests, the spec-engine coverage report and the fit.qrf parity pin. They were regenerated with tools/graph_parity_repin.py fit.qrf, tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py, and both comments are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
QRF prediction threads now default to the CPUs the process may use, and QRF thread counts no longer have to match between a pool build, a release, or a checkpoint and its resume. Tests require byte-identical imputations for one and four prediction workers and for one and four fit jobs.
What changes
Default thread count
POPULACE_FIT_PREDICT_WORKERSunset or blank, the draw's thread pool hasjoblib.cpu_count()threads instead ofos.cpu_count(). This applies inqrf.py,worker_identity.pyandstacked_spine.py. The variable still overrides the default.POPULACE_FIT_N_JOBSunset staysn_jobs=-1, which joblib already resolves tojoblib.cpu_count()threads.Primary-QRF worker identity
audit_thread_controlsfield and are no longer part ofsemantic_identity.semantic_identity.Resume
QRF.fit_draw_nextno longer compares the fit-job count saved inQRFChainState. The field is excluded from dataclass equality and stays into_dict, so saved states keep their bytes.tools/build_us_puf_support_base.pyrecords bothPOPULACE_FIT_*variables and skips them on resume, throughStageRuntime(resume_ignored_run_config_paths=...). BLAS and OpenMP thread counts andPYTHONHASHSEEDstay compared.Resume and identity effects
No resume migration is included.
qrf.py,worker_identity.pyandstacked_spine.py.Environment variables
None of these is a secret, and all are optional.
POPULACE_FIT_N_JOBSandPOPULACE_FIT_PREDICT_WORKERS: no workflow, Dockerfile or committed Modal plan sets them. Tests, the graph parity tools and evidence scripts do. A Modal plan may set them because its environment allowlist acceptsPOPULACE_*. Primary-QRF workers inherit them from the parent process environment.LOKY_MAX_CPU_COUNT:tools/build_us_fiscal_refresh_release.pysets it withsetdefaultto at most 16. Pool and base builders do not set it, and a Modal plan cannot, because the allowlist refuses the name.Test evidence
max_samples_leaf): two cases draw byte-identical values with 1 and 4 fit jobs and prediction workers. One setsmax_samples_leaf=5; the other uses the automatic 4,096 cap on a target with 24 distinct values. Each fit wrapsrandom.seedandrandom.shuffleand requires one seed per tree, then one shuffle per leaf of that tree, all on the thread that called the fit. A second test shows the draws change when other leaf samples are kept (984 and 86 of 1,001 rows). With a temporary change that made the leaf subsample depend on the thread count, these tests failed and the earlier tests passed. They also pass with Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682's_InPlaceLeafQRFmerged in. Two more review rounds covered these tests.--checkmodes pass on this merge ofmain.Not verified, and for reviewers
joblib.cpu_count()returns inside a Modal container. It reads a CPU quota only fromcpu.maxorcpu.cfs_quota_us.microcosm-fitimportsjoblibbut receives it only through its other dependencies. Declaring it would changeuv.lockand the approved lock digest, so it is left for a separate change.stacked_spine._legacy_primary_qrf_worker_execution_bindingis unused and now records the joblib default. It could be deleted separately.qrf.py, so whichever merges second needs a re-pin.🤖 Generated with Claude Code