Skip to content

Fill the QRF leaf array in place and pin quantile-forest 1.4.2 - #1682

Merged
anth-volk merged 16 commits into
mainfrom
qrf-leaf-array-in-place
Nov 19, 2026
Merged

anth-volk merged 16 commits into
mainfrom
qrf-leaf-array-in-place

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

After growing its trees, quantile-forest stores, for every tree, the training rows each leaf holds in one int64 array of shape (trees × nodes × outputs × width), zero-padded to the widest leaf, or to max_samples_leaf when that is set. quantile-forest 1.4.2's _get_y_train_leaves builds one zeroed array per tree and then copies the list into the final array with np.array(list). That copy writes every byte, padding included, so the whole padded array becomes resident memory, and the per-tree arrays and the copy exist together for a while. Pages of an np.zeros array that nothing has touched take no memory.

  • _InPlaceLeafQRF (packages/microcosm-fit/src/microcosm/fit/qrf.py) subclasses RandomForestQuantileRegressor and overrides the private _get_y_train_leaves. It allocates the final array once with np.zeros, and the per-tree fill (_fill_tree_leaves, the library's _get_y_train_leaves_slice writing into the caller's slice) writes each tree's entries straight into that tree's slice. The width computation, apply(), bootstrap replay, rank computation, the global random.seed()/random.shuffle() calls in tree order, the writes, dtype, shape and C order are the library's, so the array is byte-identical and every prediction is unchanged.
  • _fit_forest now builds _InPlaceLeafQRF. _fit_target is its only caller in package code, so every quantile forest in the US and UK builds is built this way. _Forest now declares and checks its model as _InPlaceLeafQRF. Package code names RandomForestQuantileRegressor only as the base class; the byte-comparison tests and the measurement script name it because they compare against it. Nothing in the repository builds ExtraTreesQuantileRegressor. The RandomForestClassifier in sipp_vehicles.py and the RandomForestRegressor in tools/validate_uk_housing_imputation.py are not quantile forests and are unchanged.
  • Exact pin. packages/microcosm-fit/pyproject.toml pins quantile-forest==1.4.2 (was >=1.4.2; uv.lock already resolved 1.4.2, and only its recorded specifier changes). Because the override replaces a private method, _InPlaceLeafQRF raises a RuntimeError naming both versions when the imported quantile_forest.__version__ is anything but _SUPPORTED_QUANTILE_FOREST_VERSION: in fit, before any tree is grown, and again in _get_y_train_leaves, which also covers a caller that runs the library's own fit on the subclass. The private module quantile_forest._quantile_forest is imported only after that check.

Known differences from the library class, none of which affects a draw:

  • scikit-learn's validation messages name _InPlaceLeafQRF; exception types are the same.
  • Forests pickle as microcosm.fit.qrf._InPlaceLeafQRF, and loading a forest pickled before this change raises TypeError; there is no compatibility path for the old class. The fitted-model cache in microcosm.fit.cache moves to schema version 2, so its older entries are cache misses rather than load errors. Only tests call that cache, and no pickle is tracked in the repository. No file pins model pickle bytes, and the fit.qrf@1 graph artifacts cannot mix across versions because node keys include the implementation hash.
  • A max_samples_leaf so large that the whole forest's array exceeds what numpy can describe raises numpy's ValueError where the library, allocating per tree, raises MemoryError.

Tests

packages/microcosm-fit/tests/engine_free/shared/test_qrf_leaf_array.py, 50 cases, written first and seen failing:

  • The subclass's leaf array equals the library class's in dtype, shape, C-contiguity and tobytes(), and so does the stored forest_.y_train_leaves, for:
    • a continuous target with no cap;
    • a near-discrete target whose leaves exceed the cap, so the shuffle and truncation run (integer, numpy-integer and fractional caps);
    • three outputs (uncapped, capped and one entry per leaf);
    • weighted samples, including leaves the weight filter empties and tiny, zero and negative weights, with bootstrapping on and off;
    • other forest settings (one sample per leaf, no bootstrap, max_samples, n_jobs=-1) and DataFrame predictors.
  • Python's global random state ends in the same place, both classes shuffle the same leaves, and both hand quantile-forest's compiled helper the same leaf lists.
  • _fit_forest builds the subclass, and its 203-quantile grid predictions and draws equal the library class's, including a near-discrete target with leaves over the production cap of 4,096. A chained public fit with an amount of either sign and a near-discrete ratio draws the same values.
  • The stored array is the one preallocated buffer the fill wrote. In a fresh process, mincore shows about a quarter of a capped forest's 71 MiB array in memory with 16 KiB pages (the library's copy: all of it); the test requires under 0.6 and skips where pages are over 16 KiB, huge pages are always on, or mincore is unavailable.
  • The overrides keep the library's signatures; fit returns the model; sparse_pickle passes through; warnings and warnings filters match; weighted fits the library refuses raise the same IndexError; the subclass pickles, unpickles and clones; a pickled forest holding the library class is refused.
  • With scikit-learn 1.9 reported, both classes pass the fit's own sample_weight_ to scikit-learn's private bootstrap helpers. The changed tests also pass under a real scikit-learn 1.9.1.
  • The installed quantile-forest, the pyproject.toml pin and uv.lock all name _SUPPORTED_QUANTILE_FOREST_VERSION, and fits under other versions, or a quantile-forest without __version__, are refused.

test_cache.py gains a test that a schema 1 entry holding a library-class forest is a cache miss. test_high_quantile_draw_reaches_the_observed_max in test_qrf.py now builds _InPlaceLeafQRF.

Memory measurement script

tools/measure_qrf_leaf_memory.py fits one synthetic forest through _fit_forest (weighted resample, near-discrete cap and forest), once with each class and each in a fresh Python process; for the library run the child swaps in quantile-forest's class and checks which class it fitted. It reports each process's peak memory just before and after the fit (on macOS the lifetime peak physical footprint, which counts compressed pages, beside the peak resident set; on Linux the peak resident set), the leaf array's shape and nominal size, the fit time, the platform, page size and huge-page settings, and whether the two arrays have the same shape, dtype and bytes.

  • --shape wage (default): shaped like the CPS-ORG hourly_wage fit. 100 trees, a positive whole-dollar target with hundreds of distinct values (so no cap), 10,000 rows by default (the real pool is 119,237), about 0.7 nodes per donor row. --rows-in-widest-leaf rows (default 250) share one predictor vector and set the padded width; the production width is not recorded anywhere.
  • --shape near-discrete: shaped like a base-stage near-discrete fit. 32 trees, five values dominated by zero, the 4,096 cap, 10,000 rows by default.
  • Also --rows, --trees, --seed, --implementation {library,in-place,both}, --json, --no-digest; POPULACE_FIT_N_JOBS fixes the thread count as in a build. A failed or killed measurement does not stop the other one, and the exit code is 1 then, or when the arrays differ.
uv run python tools/measure_qrf_leaf_memory.py
uv run python tools/measure_qrf_leaf_memory.py --shape near-discrete --rows 3000

It was run only at small sizes, not at its default sizes or at production size (macOS arm64, 16 KiB pages):

  • --shape near-discrete --rows 3000: leaf array 32 × 343 × 1 × 4,096, 343.0 MiB if every page were resident. The library fit added 691.6 MiB to the peak physical footprint; the in-place fit added 83.8 MiB.
  • --shape wage --rows 3000: leaf array 100 × 2,025 × 1 × 271, 418.7 MiB if every page were resident. The library fit added 865.5 MiB; the in-place fit added 427.6 MiB.
  • Both pairs of leaf arrays were byte-identical.

For the wage shape a leaf row of a few hundred entries is narrower than a page, so the in-place fill still writes nearly every page and the saving is mostly the library's second copy. For the capped shape each 32 KiB row spans several pages and most are never written. Linux uses 4 KiB pages, so its saving for the capped shape should be larger; where transparent huge pages are granted, a first write makes 2 MiB resident and the saving shrinks. Neither was measured here.

Re-pinned hashes

Every edit to qrf.py moves these, so they were re-pinned with the repository's tools after the last one:

  • APPROVED_UV_LOCK_SHA256 in us_runtime/worker_identity.py, for the changed uv.lock bytes.
  • The US spec pins that attest microcosm.fit.qrf's source (inventory_coverage.py's seed map and seed protocol, docs/evidence/spec-engine/us-f0-coverage.json, and the spec_sha256 pins in test_spec_engine_loader.py and test_us_multispine_pool_tool.py): tools/repin_us_spec_engine_contracts.py, tools/spec_engine_coverage.py (41/41), the repin again, then --check.
  • The fit.qrf parity fixture's implementation hash and three platform node keys: tools/graph_parity_repin.py fit.qrf, which refuses to write if the local draws differ from the recorded direct.csv; every direct.csv is unchanged.

Limits, not fixed here

  • ==1.4.2 keeps two 1.4.2 bugs that quantile-forest 1.5.0 fixes. The pin changes nothing today, because uv.lock already resolved 1.4.2.
    • 1.4.2 groups each tree's rows by leaf with an unstable np.argsort (zillow/quantile-forest#181). Where a leaf holds more rows than max_samples_leaf (the 4,096 cap for near-discrete targets), which rows it keeps can depend on the CPU, so builds on different machines can draw differently. _fill_tree_leaves calls the same library function, so this PR keeps that behaviour exactly.
    • predict's weighted-quantile search started at zero-weight inputs and rounded across exact ranks (zillow/quantile-forest#171). microcosm's grid stays strictly between 0 and 1 with linear interpolation; whether the fix changes any microcosm draw has not been checked.
    • 1.5.0 needs scikit-learn 1.9, changes the default interpolation and stores leaves without padding, so moving to it changes predictions and retires _InPlaceLeafQRF. Move microcosm's QRF to quantile-forest 1.5.0 #1691 tracks that.
  • Pickling and deep copies still write the full padded array. quantile-forest's QuantileForest.__reduce__ (_quantile_forest_fast.pyx, about lines 522–536) hands the whole array to pickle. microcosm does this in:
    • packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py, which deep-copies the fitted SSI disability model once per support channel (production);
    • packages/microcosm-fit/src/microcosm/fit/kernels.py, where the fit.qrf@1 kernel pickles the fitted model as its model artifact (used by the parity fixtures);
    • packages/microcosm-fit/src/microcosm/fit/cache.py (save_fitted_qrf_cache, which only tests call).
    • _Forest.__getstate__ in qrf.py defines the payload all of these use.
  • On macOS, reading a never-written page also makes it resident: prediction reads each leaf row it reaches, so part of the saving is given back during a draw.
  • Storing leaf members as compressed index lists is a later task.
  • The page-residency test was not run on Linux, and GitHub's runners' huge-page setting was not checked; if it is always, that test skips.
  • Locally, only the tests this branch adds or changes were run, each by node ID, on macOS arm64 with Python 3.14 and scikit-learn 1.8.0. The memory saving in a real build was not measured.

Review

Nine rounds of adversarial review by separate reviewers, each given the code and diff, plus one review of the commit that made _Forest require _InPlaceLeafQRF, which found the cache-schema problem fixed here. None found an input where the array, the random state, a stored attribute or a prediction differs from the library's (the largest comparisons covered 4,420 random combinations and 620 random configurations, 120 of the latter under scikit-learn 1.9.1). Mutation testing in those rounds (deliberate small bugs in a copy of the module) found test gaps, which were closed. The cache schema change and the removal of the memory script's result checks were made after the last review and have not been reviewed separately.

Overlap

This edits qrf.py only at the new subclass and helpers (placed before _SERIALIZED_FOREST_N_JOBS), the imports, and the forest construction in _fit_forest. Other open work on _predict_workers/_fit_n_jobs and on the CPS-ORG and ACS callers also edits qrf.py, and every change to qrf.py moves the pinned hashes above, so whichever merges second re-runs the repin tools.

🤖 Generated with Claude Code

anth-volk and others added 16 commits November 17, 2026 14:48
Failing tests for an in-place leaf array (task 1.2 · 1). After growing its
trees, quantile-forest 1.4.2 maps every leaf to its training rows in one
zero-padded int64 array (trees x nodes x outputs x width). It builds one
array per tree and copies the list with np.array(list), which writes every
padding byte. The tests expect microcosm.fit.qrf._InPlaceLeafQRF to build
the same array in one preallocated buffer:

- the array equals the library class's in dtype, shape, C order and
  tobytes() for a continuous uncapped target, a near-discrete target whose
  leaves exceed the cap (so the shuffle and truncation run), three outputs,
  weighted samples with zero weights, and other forest settings;
- Python's global random state ends where the library leaves it;
- _fit_forest builds the subclass, and its 203-quantile grid predictions and
  draws equal the library class's, including a leaf over the 4,096 cap;
- a chained, regime-gated public fit draws the same values;
- the subclass pickles and clones;
- quantile-forest is pinned to exactly 1.4.2 in pyproject.toml and uv.lock,
  and a fit refuses any other installed release before growing a tree.

All 25 cases fail: the subclass and the version constant do not exist yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_InPlaceLeafQRF subclasses RandomForestQuantileRegressor and overrides the
private _get_y_train_leaves. It allocates the final (trees x nodes x
outputs x width) int64 array once with np.zeros and has the per-tree fill
write into each tree's slice, instead of building one array per tree and
copying the list with np.array(list). The width, apply(), bootstrap replay,
rank computation, global random.seed()/shuffle() calls in tree order, and
the writes are the library's own, so the array is byte-identical and every
prediction is unchanged. np.zeros pages that are never written stay
unmapped, so internal-node rows and padding past each leaf's entries cost
no resident memory, and the second copy is gone. _fit_forest now builds
the subclass; no other non-test code constructs the library's forest.

quantile-forest is pinned to ==1.4.2 (uv.lock already resolved 1.4.2), and
_InPlaceLeafQRF.fit raises a RuntimeError naming both versions when any
other release is imported, before growing a tree.

tools/measure_qrf_leaf_memory.py fits one synthetic forest shaped like the
CPS-ORG wage fit (100 trees, continuous positive target, no cap) or a base
near-discrete fit (32 trees, 4,096 cap) with each class in a fresh process,
and reports each process's peak RSS before and after the fit, and whether
the two leaf arrays hash the same. At 3,000 near-discrete rows the library
fit added 0.93 GiB and the in-place fit 0.11 GiB; at 4,000 wage rows
(width 280) 1.33 GiB and 1.03 GiB. Both arrays were identical.

Re-pins for the changed bytes:
- APPROVED_UV_LOCK_SHA256, for the uv.lock specifier change;
- the US F0 seed map, seed protocol and spec_sha256 pins, which attest
  microcosm.fit.qrf's source: repin, coverage (41/41), repin, --check clean;
- the fit.qrf parity fixture's implementation hash and node keys
  (graph_parity_repin fit.qrf), which left every direct.csv unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- _InPlaceLeafQRF._get_y_train_leaves takes the library's own argument
  names (X, y) again, so keyword callers work; a test compares both
  overrides' signatures with the library's.
- The version check also runs at the top of _get_y_train_leaves, the only
  place the copied code runs, so calling quantile-forest's own fit on the
  subclass is refused too (new test, seen failing first). The private
  quantile_forest._quantile_forest module is imported after that check
  instead of when qrf.py loads, so a release that moved it gets the
  explanatory error instead of an ImportError from import microcosm.fit.
- The README and the pyproject comment no longer call the quantile-forest
  requirement a floor, and name BaseForestQuantileRegressor, where the
  method is defined.
- tools/measure_qrf_leaf_memory.py:
  - fits through _fit_forest with lognormal donor weights, so microcosm's
    weighted resample and near-discrete cap run as in a build (the wage
    shape now has about 0.85 nodes per donor row, not 1.2);
  - reads macOS's lifetime peak physical footprint, which counts pages
    compressed under memory pressure (ru_maxrss drops when they are), and
    warns when the library fit's growth is below the array's size;
  - validates --rows-in-widest-leaf, --seed and the near-discrete row
    count with argparse errors, and reports a failed child briefly;
  - records platform, page size, huge-page mode, package versions, seed,
    n_jobs and every input, and compares shape and dtype as well as the
    digest;
  - documents the platform dependence and when the cap's truncation runs.
- Re-pinned the US F0 spec pins (repin, coverage 41/41, repin, --check
  clean) and the fit.qrf parity pins, which every qrf.py edit moves;
  direct.csv is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the memory script

Tests (five deliberate code mutations survived round two; each now fails a test):
- emptied weighted leaves: a fit with nine in ten weights at zero records,
  through quantile-forest's compiled helper, that both classes hand it the
  same number of [0] placeholders, and more than none;
- fractional caps where ceil and floor differ: 0.021 of 400 rows (9), and
  0.0047 of 3,000 near-discrete rows (15), which also truncates leaves;
- the width-1 branch with three outputs;
- the shared-memory check, by making the helper return a copy;
- sparse_pickle reaching the library.
The weights helper's docstring no longer claims the placeholder runs.

qrf.py docstrings: the per-tree fill names all its differences from the
library's; the class says fit is wrapped for the version check, that
scikit-learn's messages name the subclass, and when untouched pages do
become resident (writes; reads on macOS, including prediction and
pickling; huge pages on Linux).

tools/measure_qrf_leaf_memory.py:
- whole-dollar wages bring the wage shape to about 0.7 nodes per donor row,
  the top of the build's 0.4-0.7;
- each child checks it fitted the class it was asked for, and the report
  prints the class;
- a failed or killed measurement is reported by signal or exit code, the
  other one still runs, and the exit code is 1;
- the unreliable low-reading warning is removed; the docstring explains
  when growth reads low;
- libproc is reached through libSystem, with a fallback to ru_maxrss;
- the report prints the resident-set peak, the thread count, numpy's huge
  page request and CPU count; --json carries failures and the identity
  result;
- --rows-in-widest-leaf defaults to half of --rows when smaller, and a wage
  target with 32 or fewer values is refused instead of silently capped.

Re-pinned the US F0 spec pins (repin, coverage 41/41, repin, --check clean)
and the fit.qrf parity pins; direct.csv is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… memory script

Round three found no defect in the leaf array, the random state or the
predictions; nine deliberate code changes that alter behaviour still passed
the tests. New tests cover each:
- the forest keeps the one preallocated buffer the fill wrote (a copy has
  the same bytes but makes every padding page resident);
- fit returns the model, and the default pickle format stays dense;
- DataFrame predictors give the library's warnings and array;
- emptied weighted leaves under a cap of 3 and of 1;
- weights that are tiny, zero or negative go through the library's
  positive-weight filter;
- a quantile-forest without __version__ is refused;
- with scikit-learn 1.9 reported, both classes pass the same sample_weight_
  to the private bootstrap helpers.
Docstrings now say the fill checks that the helper's result shares memory
with the slice, that the filter drops non-positive weights, and that an
oversized max_samples_leaf raises ValueError where the library raises
MemoryError; the pyproject comment names the private sklearn helpers the
copy calls.

tools/measure_qrf_leaf_memory.py: a child that exits without a JSON result
is a failure instead of a crash; differing leaf arrays set exit code 1;
only SIGKILL gets the out-of-memory note, and unnamed signals are printed
by number; the report gives trees grown at once rather than the pool size,
labels each result's own measure, and says when --no-digest skipped the
comparison; --json records the arguments and structured failures;
--rows-in-widest-leaf is refused for the near-discrete shape; the
docstring states the sizes from which the widest-leaf and node-density
statements hold, and why the wage shape's saving is mostly the second copy.

Re-pinned the US F0 spec pins (repin, coverage 41/41, repin, --check clean)
and the fit.qrf parity pins; direct.csv is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est filter wording

Round four found no input where the subclass's leaf array, random state,
attributes, predictions or exceptions differ from the library's (620
random cases, 120 of them under scikit-learn 1.9.1). It found:

- The simulated scikit-learn 1.9 test dropped the weights before calling
  the real helpers, so it failed under a real 1.9, and it compared only
  the weights' sum. Its wrappers now pass the weights on when the installed
  scikit-learn takes them, and it checks that both classes hand the helpers
  the fit's own sample_weight_ array, once per count and once per tree.
- quantile-forest 1.4.2's weight filter looks up each entry's weight by the
  draw's position minus one, not by the drawn row. The copy keeps that
  exactly; the comment and the test's name and docstring now say so
  instead of describing a positive-weight filter.
- New tests: a fit leaves warnings.filters as it found them; weighted fits
  the library refuses (several outputs, one row) raise the same IndexError;
  and in a fresh process, mincore shows that about a quarter of a capped
  forest's 71 MiB array is in memory after the fit, against all of it for
  the library's copy (the test requires under 0.6, and skips where huge
  pages are always on or mincore is unavailable).
- The class docstring notes that a failed whole-forest allocation leaves
  Python's random state where it was, unlike the library's.

tools/measure_qrf_leaf_memory.py: the child marks its result line and the
parent checks the result's fields, so other output cannot be mistaken for
or hide it; a one-valued target gets its own message; failures in --json
give the exit code, the signal number and its name (or null).

Re-pinned the US F0 spec pins (repin, coverage 41/41, repin, --check clean)
and the fit.qrf parity pins; direct.csv is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round five found no difference from quantile-forest 1.4.2 in the leaf
array, random state, predictions or exceptions, and every mutant that
changes behaviour is now caught. It found:

- Under scikit-learn 1.9, sklearn's own tree growing refuses negative
  weights, so the weight-filter test drops them there; its docstring says
  so. The weight-filter docstrings no longer call it a zero-weight filter.
- The class docstring's note on a failed allocation now says the random
  state can differ only when a leaf cap is set and the library gets past a
  tree before failing; the documented too-large case leaves both equal.
- The page-residency test skips with pages over 16 KiB or any huge page
  size set to always, unsets glibc's MALLOC_PERTURB_ in its child, and
  shows the child's error output when it fails.
- The warning-filter test reads the active filters through
  warnings._get_filters where it exists, so context-aware warnings
  (free-threaded builds) cannot hide a leaked filter.
- The memory script tells apart no result, a result that is not a JSON
  object, and one that lacks fields or has a null where a value is
  required; the wage distinct-value message names --seed.

Re-pinned the US F0 spec pins (repin, coverage 41/41, repin, --check clean)
and the fit.qrf parity pins; direct.csv is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ricter result checks

Round six found nothing that changes what _InPlaceLeafQRF computes. It
found:

- From scikit-learn 1.9 the forest's weighted bootstrap draw (not tree
  growing) refuses negative weights. The weight-filter test now runs with
  bootstrapping on and off; negative weights are left out only with it on
  under 1.9, so on every version they reach the copied filter.
- A numpy integer leaf cap (np.int64(16)) is now tested; it must count as
  an integer, as in the library.
- The page-residency test also unsets GLIBC_TUNABLES in its child and
  skips where os.sysconf is missing.
- tools/measure_qrf_leaf_memory.py: which result fields may be null now
  depends on the run (the digest only under --no-digest, the cap only for
  the uncapped wage forest), and numeric fields, the array shape and the
  environment record are type-checked, so a malformed result is reported
  as unusable with the reason instead of being accepted or crashing.

qrf.py is unchanged, so no pin moves. The tests pass under scikit-learn
1.8.0 (49 cases), and the changed ones under 1.9.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round seven found the test changes sound; every mutant that changes
behaviour is caught under scikit-learn 1.8.0, including the numpy-integer
cap and negative weights without bootstrapping. The remaining findings were
in tools/measure_qrf_leaf_memory.py, for malformed child results that a
real child never prints:

- numbers must be finite and fit in 64 bits (NaN, infinity and huge
  integers were accepted and then crashed the report);
- fields that must be null for the run's options are now required to be
  null, and the rest to hold a value (a wage result with a cap, or a
  near-discrete one with a widest-leaf count, was accepted);
- rows_in_widest_leaf and max_samples_leaf are type-checked, as are the
  environment's platform, page size and CPU count, the digest's format,
  the array shape (four non-negative integers), the memory measure's name
  and the implementation name;
- a null shape or environment is reported once.

Tests: the cap parametrisation has named cases, and the residency test's
child keeps GLIBC_TUNABLES entries other than glibc.malloc ones.

qrf.py is unchanged, so no pin moves. 49 cases pass under scikit-learn
1.8.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…esults

Round eight found every real child result accepted, but more malformed
results (a list in memory_measure, integers over Python's digit limit,
deep nesting, NaN in unchecked fields, unprintable strings) that could
still crash the parent or reach --json output. Instead of one check per
field, tools/measure_qrf_leaf_memory.py now:

- parses with parse_constant refusing NaN and Infinity, and treats any
  ValueError or RecursionError from parsing as a result that is not
  standard JSON;
- turns any exception while checking a result, or while formatting it
  for the report, into a reported failure for that measurement, so the
  other measurement still reports;
- checks that each result describes the run the parent asked for (class,
  implementation, shape, rows, trees, seed, widest-leaf rows);
- writes --json with allow_nan=False and escapes characters the terminal
  cannot encode in the text report.

The page-residency test's child also drops glibc.mem.tagging tunables,
since memory tagging makes glibc's calloc zero the whole block.

qrf.py is unchanged, so no pin moves. 49 cases pass under scikit-learn
1.8.0, and real runs of every shape and option exit 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng results

Round nine found one problem with real children: a caller's
PYTHONIOENCODING (cp500, utf-16, utf-8-sig) changed how the child wrote
its result, so the parent either failed to decode it or reported that no
result was printed. tools/measure_qrf_leaf_memory.py now runs each child
with PYTHONIOENCODING=utf-8 and reads its output as UTF-8, replacing any
byte it cannot decode; real runs under all three encodings report both
results.

For malformed results, which a real child cannot print:
- numbers too large for a float (1e400) are refused at parse time;
- the safeguard also writes each result as standard JSON in valid UTF-8,
  so deep nesting and lone surrogates are refused before --json output;
- leaf_array_bytes must match the shape for an int64 array;
- two results must share their platform record and fit settings
  (n_jobs, trees grown at once, the cap, the memory measure), or the run
  reports that they disagree and exits 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_Forest now declares and checks its model as _InPlaceLeafQRF, so loading a
forest pickled with quantile-forest's class raises TypeError. Package code
names RandomForestQuantileRegressor only as the base class. Regenerate the
source hash pins that cover microcosm.fit.qrf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forests pickled with quantile-forest's class are now refused on load, and
the cache key does not include the forest class, so an older entry raised
TypeError instead of being a miss. Schema version 2 changes the key, so
those entries are no longer looked up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The script's child process is the script itself, so validating every field
of its result guarded against output it cannot produce. Keep the memory
readings, the fitted-class check and the reporting of a failed or stopped
child.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Regenerate the spec coverage record and the US spec hash pin on the merged
source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8729ab5d4. _fill_tree_leaves matches quantile-forest 1.4.2's per-tree fill line for line, except that it writes into the preallocated slice. Three things before merge.

  1. Rerun the failed engine-free (3.13) job. Its one failure, test_graph_executor_structural_columns.py::test_a_structural_hit_holds_no_column_beside_its_frame (15.4 MB held against a 5.0 MB bound), is in code this PR doesn't touch. The test passes locally under Python 3.13, alone and after test_qrf_leaf_array.py in the same process.

  2. Build the UK spine before and after this change, on one machine. CI's UK integration job doesn't compare spine outputs, and the PR hasn't measured a real build. The spine draws from declared seeds, each forest keeps its seed, and quantile-forest stays at 1.4.2 and scikit-learn at 1.8.0, so the two spines should be payload-identical, not just close. Each build takes about 20 minutes and peaks at 10–14 GB.

    • Build at the merge base 087bcc6df and at 8729ab5d4, each from its own worktree after its own uv sync --all-packages --locked, with the same tools/build_uk_frs_spine.py inputs and flags.
    • Compare them with tools/compare_uk_h5_payload.py <before.h5> <after.h5> --json-out <receipt.json>, which should exit 0.
    • Record each build's peak memory with /usr/bin/time -l placed before the venv's python, not before uv run. That measures the saving at production size.
    • Post the tool's verdict and the two peaks. Any difference means the override is not equivalent.
  3. Say in the PR that ==1.4.2 keeps 1.4.2's known bugs, and open an issue for 1.5.0. The pin changes nothing today, because uv.lock already resolved 1.4.2. But quantile-forest 1.5.0 (7 Nov) fixes two bugs that reach microcosm's forests:

    • Upstream PR 181: 1.4.2 groups leaf rows with an unstable np.argsort. When a leaf exceeds max_samples_leaf (the 4,096 cap on near-discrete targets), which rows it keeps then depends on the CPU, so builds on different machines can draw differently.
    • Upstream PR 171 fixes the weighted-quantile search that microcosm's predict uses.

    The weight and global-random fixes don't reach microcosm's fits: no sample_weight reaches the forest, and nothing else reads the global random that 1.4.2 reseeds. 1.5.0 also stores leaf samples without padding, which is the memory problem this PR works around, but it needs scikit-learn 1.9 and changes the default interpolation, so predictions can change. Moving to it is a separate step that retires _InPlaceLeafQRF.

Checked

  • 179 tests on the touched surface pass locally under Python 3.14 and scikit-learn 1.8.0, including the 50 in test_qrf_leaf_array.py with the page-residency case. The re-pinned uv.lock hash, spec_sha256 and fit.qrf parity pins hold, and CI's US jobs pass.
  • Only _fit_forest builds a quantile forest, and it passes no sample_weight.
  • #1666's raw-write scan finds nothing in this PR's files, so the two PRs can merge in either order.

@anth-volk

Copy link
Copy Markdown
Contributor Author

@juaristi22, thanks for the review. All three points are done.

1. Rerun. I reran the failed jobs of run 44388590022. On attempt 2, engine-free (3.13) and ci both pass, so test_a_structural_hit_holds_no_column_beside_its_frame passed without any change.

2. UK spine at 087bcc6df and 8729ab5d4, on one machine

  • Setup

    • Machine: Apple M5 (Mac17,2), 32 GB, 10 CPUs, macOS 26.3.
    • Each commit was built from its own worktree after its own uv sync --all-packages --locked --extra uk, the UK integration job's sync. Both environments have quantile-forest 1.4.2 and scikit-learn 1.8.0.
    • Both builds used the same inputs, fetched by tools/run_uk_cut.py fetch and checked against the committed pins.
    • Both used the same tools/build_uk_frs_spine.py flags: every licensed tab spine_argv passes, the full sample, --staging-local-only and one --staging-run-id. Neither used --smoke or --release-candidate.
    • The builds ran one after the other.
  • Payload: identical. tools/compare_uk_h5_payload.py before.h5 after.h5 --json-out receipt.json exited 0 with "payload_identical": true.

    • Both spines have the same four tables, keys, column order, dtypes, indexes and values, and identical root attributes.
    • The two files are the same size (441,982,053 bytes) but have different sha256 values (d74c859d… and 33b08a1c…). The tool found no payload or attribute difference, and I didn't look into which bytes differ.
  • Checkpoint stores

    • Each store has 33,290 files, and every stored table has the same bytes in both. That covers all 22,394 values.npy files and every meta.json, offsets.npy and mask.npy.
    • The only files that differ are the stage-timing log, spine.graph.json, and the graph evidence files (binding.json, manifest.json, execution.evidence.json). These record node keys, and node keys include qrf.py's implementation hash.
    • The validation-check reports (spine.spine_gates.json) differ only in their timestamped release_id. All 56 stage records in spine.build.json are identical apart from their seconds.
  • Peak memory, from /usr/bin/time -l placed directly before each worktree's .venv/bin/python:

    wall time maximum resident set peak memory footprint
    087bcc6df 2,701 s 8,864,284,672 B (8.26 GiB) 66,117,125,560 B (61.6 GiB)
    8729ab5d4 1,326 s 12,888,113,152 B (12.0 GiB) 45,756,839,296 B (42.6 GiB)

    These figures don't isolate this PR's saving, because the two builds ran under different memory pressure.

    • Free system memory was 34% during the first build (sampled at 19:44 UTC) and 68% during the second (20:06 UTC).
    • At 20:06, 9.2 GB of the 10 GB swap was in use.
    • Under that pressure macOS compresses and swaps out a process's pages. That keeps the resident set down and adds to the footprint and the run time, which fits the first build's lower resident set and higher footprint.
    • I didn't control for other load on the machine. Measuring the saving cleanly would need a quiet machine, better a Linux one, and repeated runs. I haven't done that.

3. Pin and 1.5.0

🤖 Generated with Claude Code

@anth-volk
anth-volk merged commit 7ff5a26 into main Nov 19, 2026
22 of 24 checks passed
@anth-volk
anth-volk deleted the qrf-leaf-array-in-place branch November 19, 2026 21:20
anth-volk added a commit that referenced this pull request Nov 19, 2026
Brings in #1682 (the QRF leaf array filled in place). Both PRs move the
US seed-map and seed-protocol digests, so the four spec-pin files
conflicted; they are regenerated with tools/repin_us_spec_engine_contracts.py
and tools/spec_engine_coverage.py, keeping both PRs' comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
origin/main now has #1682 (the in-place QRF leaf array), which also edits
microcosm.fit.qrf. The conflicts were the recorded digests both sides moved:
the US seed-protocol and seed-map pins, the two spec_sha256 pins in tests, the
spec-engine coverage report and the fit.qrf parity pin. They were regenerated
with tools/graph_parity_repin.py fit.qrf, tools/repin_us_spec_engine_contracts.py
and tools/spec_engine_coverage.py, and both comments are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
#1682's in-place QRF leaf array in microcosm.fit.qrf and this branch's
calibrate return_frame option in solve both move the US seed-protocol
digests, so neither side's values were kept. Regenerated with
tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py
in the US engine environment; both --check runs pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 19, 2026
Five pin files conflicted, as Max listed: #1682's in-place QRF leaf array
and this branch's engine refresh each moved the same pins. Each takes this
branch's side, and the pins both sides moved are recomputed on the merged
tree:

- tools/repin_us_spec_engine_contracts.py (twice, so the inventory count is
  read with the new seed hashes in place) rewrote the seed map and seed
  protocol digests, the spec-engine loader's golden vector and the pool
  tool's spec_sha256; tools/spec_engine_coverage.py regenerated the F0
  coverage report (50,842 of 50,842 fields, 41 of 41 inventory checks), and
  both --check runs pass.
- APPROVED_UV_LOCK_SHA256 is the merged uv.lock's digest. The lock merged
  without a conflict (#1682 pins quantile-forest==1.4.2), and uv lock --check
  resolves it unchanged.

The US bundle generator needs policyengine-us, which the engine-free
environment lacks; #1666 and #1682 change none of its inputs, so the bundle
stands, and CI's engine lane checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants