Skip to content

Add one thread setting, MICROCOSM_THREADS, to the US build commands - #1696

Draft
anth-volk wants to merge 10 commits into
claude/qrf-predict-workers-joblib-cpu-count-37170bfrom
claude/build-tool-thread-settings-9e62d0
Draft

anth-volk wants to merge 10 commits into
claude/qrf-predict-workers-joblib-cpu-count-37170bfrom
claude/build-tool-thread-settings-9e62d0

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Summary

Stacked on #1684: this pull request targets that branch, so its diff shows only this task. It adds one thread setting, MICROCOSM_THREADS, to the US release, base, ACS multispine staging and ACS local-release commands. When it is set, each command copies it into every library's thread variable before numpy loads. When it is not, nothing changes: the base and ACS commands set nothing, and the release tool keeps its default of min(os.cpu_count(), 16).

The base command now records the setting and the BLAS thread count actually in use in its run configuration, so a resume on a host where OpenBLAS would use a different count is refused. On Linux that count can change output bytes.

Why one setting, and nothing set by default

  • One value lets an operator, and later the --max-cores flag and thread planner of tasks 1.8 · 1 and 2.6 · 2, choose every library's thread count at once. The release tool and Modal already work this way: the release tool copies min(os.cpu_count(), 16) into six variables, and Modal copies the class's CPU request into four.
  • With nothing set, every library keeps choosing its own count. On Linux the OpenBLAS count can change output bytes, and the release calibration's exact weights depend on torch's thread count, so defaults that differed from today's would change outputs. Earlier versions of this branch filled the variables from the host's CPU count. Review found that changed scikit-learn's thread count (it stops following a CPU quota once OMP_NUM_THREADS is set) and torch's (it reads MKL_NUM_THREADS and OMP_NUM_THREADS).
  • POPULACE_FIT_N_JOBS and POPULACE_FIT_PREDICT_WORKERS are not set. Since Size QRF prediction workers from joblib.cpu_count() and stop comparing thread counts #1684 both follow joblib.cpu_count() when unset, and LOKY_MAX_CPU_COUNT, which the setting fills, limits that count.

What changes

tools/us_build_thread_environment.py (new, standard library only)

  • thread_setting() reads MICROCOSM_THREADS. A blank value counts as unset. A value that is not a positive whole number raises ValueError naming the setting.
  • set_build_thread_environment(default_count=None) copies the count into each unset or blank one of OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, BLIS_NUM_THREADS, VECLIB_MAXIMUM_THREADS, NUMEXPR_NUM_THREADS and LOKY_MAX_CPU_COUNT. Above 64 it also fills NUMEXPR_MAX_THREADS, because numexpr runs one thread when given more than its maximum.
  • Variables already set keep their values. The setting also replaces a blank value, since joblib fails on a blank LOKY_MAX_CPU_COUNT. default_count applies only without the setting, and keeps a blank value as the release tool always has.

Commands

  • The base, ACS staging (both files) and ACS local-release commands call the helper before numpy loads, only when run as __main__. Each loads it from its own folder, so python -m and python -P starts work.
  • The release tool's block now calls the helper with default_count=min(os.cpu_count() or 1, 16). It still runs on import, so the exact-k ladder, certification and scoring tools, which import it before numpy, read the setting too. Without the setting its values are unchanged; it now also fills BLIS_NUM_THREADS, which no installed library reads.
  • tools/build_us_acs_local_release.py used to get thread values only when it imported the release tool partway through a run. It now applies the setting at start. Without the setting it behaves as before.

Base command's run configuration

  • thread_environment also records MICROCOSM_THREADS and blas_threads_in_use: each loaded BLAS library with its thread count, read by threadpoolctl after numpy loads. The list is empty where numpy uses Accelerate (macOS).
  • A resume compares both, like the other recorded variables, as written: 007 and 7, or blank and unset, differ. POPULACE_FIT_* stay recorded and not compared.
  • compare-lineage reports thread_environment differences without refusing them, so a Modal run compared with a macOS reference reports blas_threads_in_use and does not refuse.

Modal plan

  • tools/modal_us_stage_plan.py refuses a plan whose MICROCOSM_THREADS is not a positive whole number when it is parsed. The base tool's check step loads the tool without running its start-up step, so the tool would otherwise refuse the value only after the inputs were staged.

Dependency

  • microcosm-build declares threadpoolctl>=3.6, which scikit-learn already installs. 3.6 reads the scipy-openblas library that numpy 2 wheels bundle. uv.lock gains the dependency edge only.

Docs

  • docs/us-modal-stage-runbook.md "Thread variables": what reads each variable, what each library does with nothing set, the Linux byte effect and where it was measured, the setting's other effects, when a change takes effect on resume, and Modal.

Resume and identity effects

  • A base run interrupted before this change cannot resume after it: the run configuration's code digest covers the tool file, pyproject.toml and uv.lock, which changed. The new thread_environment keys would also differ.
  • The helper file is not in that digest, but the values it sets are recorded and compared, except LOKY_MAX_CPU_COUNT, which changes no imputed value.
  • The ACS staging command keeps no run configuration. The ACS local-release command's settings check records no thread variables.

Environment variables

None is a secret, and all are optional. Wiring common to all: on Modal the runner's tool_environment takes the container's environment, removes credential-like names, PYTHONHASHSEED and runner-owned names, applies the plan's env last, and starts the tool with Popen(env=...). Base stage subprocesses get os.environ.copy(). Primary-QRF workers get {**os.environ, overrides}, and the overrides touch no thread variable. Route A starts commands with the operator's environment plus PYTHONUNBUFFERED.

  • MICROCOSM_THREADS: operator shell, or a Modal plan's env (the allowlist accepts MICROCOSM_*). No workflow, Dockerfile, committed plan or Route A file sets it. Read by the five command files and the release tool's importers. Recorded and compared by the base command.
  • OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS: the setting, the release default, the Modal container (4 in the recorded base run at 4b57d15a2; 2 in the check class according to the runbook), a plan's env, .github/workflows/test.yml (1), operator shell. Recorded and compared by the base command.
  • BLIS_NUM_THREADS: the same providers except test.yml; no installed library reads it. Recorded and compared.
  • VECLIB_MAXIMUM_THREADS: the setting, the release default, operator shell. A plan may not set it. Unset in the recorded Modal base run. Recorded and compared.
  • NUMEXPR_NUM_THREADS: the setting, the release default, a plan's env, operator shell. Unset in the recorded Modal base run. Recorded and compared.
  • NUMEXPR_MAX_THREADS: the setting above 64, operator shell. A plan may not set it. Not recorded.
  • LOKY_MAX_CPU_COUNT: the setting, the release default, operator shell. A plan may not set it. Not recorded.
  • GOTO_NUM_THREADS: operator shell only; not set by any tool, not recorded. Its effect shows in blas_threads_in_use.
  • POPULACE_FIT_N_JOBS, POPULACE_FIT_PREDICT_WORKERS: a plan's env (POPULACE_*) or operator shell. Not set by this change. Recorded and not compared.
  • PYTHONHASHSEED: unchanged by this change.

Unverified: what Modal sets in the heavy, light and acs-staging classes; whether the Modal container passes LOKY_MAX_CPU_COUNT, GOTO_NUM_THREADS or NUMEXPR_MAX_THREADS through (the receipt records only names ending in _NUM_THREADS); blas_threads_in_use on a Linux host (the tests check it on macOS, where it is empty, and with a stand-in for threadpoolctl); and the Linux byte measurements, which came from container probes on 18 November 2026 that are not stored in the repository.

Test evidence

  • Red then green: the failing tests were committed before each change (86b39fec6, 78699b813, fd3f0875f).
  • Three rounds of adversarial review, the first on an earlier design. Round 3 found no behaviour defect; its wording and edge-case findings are fixed.
  • All 68 cases added or changed by this branch pass on macOS, each run by node ID: test_us_build_tool_thread_defaults.py (every case), three tests in test_us_puf_support_base_builder.py and test_a_plan_s_thread_setting_must_be_a_positive_whole_number in test_us_modal_stage_plan_tool.py. The full suite was not run.
  • CI: the workflow runs only on pull requests that target main, so this stacked pull request gets no checks until it is retargeted and a commit is pushed.

Tracker scope

Task 1.2 · 4's title and completed state still describe setting POPULACE_FIT_* and filling every thread variable by default. This branch does neither, by the user's decision: one setting, nothing by default.

🤖 Generated with Claude Code

anth-volk and others added 10 commits November 19, 2026 02:46
The base and local-area commands now set OMP_NUM_THREADS,
OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, BLIS_NUM_THREADS,
VECLIB_MAXIMUM_THREADS and NUMEXPR_NUM_THREADS before numpy loads. Each gets
the count its library already uses, so a host's output bytes do not change.
Values already set are kept, and importing the tools changes nothing.

On Linux the OpenBLAS thread count changes output bytes, so the base command
now records real counts and refuses a resume with different ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy OMP_NUM_THREADS, GOTO_NUM_THREADS and NUMEXPR_MAX_THREADS only when each
is a plain positive integer. OpenBLAS and numexpr read other spellings
differently, so a copied value could change their thread counts.

Correct the runbook and changelog from review: name the ACS multispine staging
command, cite the recorded Modal base environment, and state that setting
OMP_NUM_THREADS changes scikit-learn's and torch's thread counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MICROCOSM_THREADS, when set, fills every OpenMP, BLAS, numexpr and joblib
thread variable that is unset; without it the base and ACS staging
commands set nothing and the release tool keeps min(os.cpu_count(), 16).
The base command's resume check compares the setting and the BLAS thread
count in use. These tests fail until the change that follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release, base and ACS multispine staging commands copy
MICROCOSM_THREADS, before numpy loads, into every unset OpenMP, BLAS,
numexpr and joblib thread variable, and refuse a value that is not a
positive whole number. Without it the base and ACS staging commands set
nothing, so each library keeps choosing its own count, and the release
tool keeps its default of min(os.cpu_count(), 16).

The base command records MICROCOSM_THREADS and each loaded BLAS
library's thread count (threadpoolctl) in thread_environment, so a resume
on a host where OpenBLAS would use a different count is refused: on Linux
that count changes output bytes. threadpoolctl, already installed through
scikit-learn, is now declared by microcosm-build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commands must find the helper without their folder on the import
path, the ACS local-release command applies the setting at start, a count
above 64 raises numexpr's maximum, the release default keeps a blank
variable, the library test needs two CPUs, and a Modal plan with an
unusable MICROCOSM_THREADS is refused when it is loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The base, ACS staging and new ACS local-release start-up steps load the
  helper from their own folder, so `python -m`, `python -P` and runpy
  starts work again.
- The ACS local-release command applies MICROCOSM_THREADS before numpy
  loads; before, it read the setting only when it imported the release
  tool partway through a run.
- A count above 64 also fills NUMEXPR_MAX_THREADS, since numexpr runs
  one thread when asked for more than its maximum.
- The release tool's default keeps a blank variable, as it always has.
- A Modal plan with an unusable MICROCOSM_THREADS is refused when loaded.
- threadpoolctl>=3.6, which reads numpy 2's scipy-openblas library.
- The runbook and changelog name every command that reads the setting,
  correct scikit-learn's, torch's and MKL's rules, and say where the Linux
  byte measurements came from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A MICROCOSM_THREADS value longer than int() parses must be refused with
an error that names the setting, by the helper and by the Modal plan.
The library test skips when joblib.cpu_count(), which also follows a CPU
quota, is under two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A MICROCOSM_THREADS value longer than int() parses is refused with an
  error that names the setting, by the helper and by the Modal plan.
- The library test skips when joblib.cpu_count() is under two.
- Comments, runbook and changelog: joblib fails on a blank
  LOKY_MAX_CPU_COUNT, the ACS local-release command still takes the
  release default when it imports the release tool, NUMEXPR_MAX_THREADS is
  filled above 64 and not recorded, the resume compares values as written,
  and blas_threads_in_use is empty only where numpy and scipy use
  Accelerate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 1684's branch merged origin/main. The one conflict was the release
tool's pathlib import: that branch now also imports PurePosixPath, and this
branch imports Path at the top of the file for the thread setting, so the
lower import keeps PurePosixPath only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant