Skip to content

Fit and draw one QRF target at a time in CPS-ORG wages and the ACS transfer - #1667

Open
anth-volk wants to merge 13 commits into
mainfrom
claude/qrf-fit-draw-per-target-d9ab13
Open

anth-volk wants to merge 13 commits into
mainfrom
claude/qrf-fit-draw-per-target-d9ab13

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

The CPS-ORG wage imputation (impute_us_org_wages) and the ACS input transfer (transfer_acs_inputs) now fit and draw one QRF target at a time. Each target's forests and sign classifier are freed before the next target is fitted. Imputed values are byte-identical to main.

  • CPS-ORG wages (org_wages.py).
    • Before: one QRF.fit over hourly_wage and is_paid_hourly, then one predict. Both targets' forests stayed referenced until the function returned.
    • Now: one RegimeGatedQRF.start_chain, then one fit_draw_next per target, through _fit_draw_next_org_target. That helper returns only the target name, its draw and the next chain state, so the step result that holds the forests is dropped before the next fit.
  • ACS transfer, in-memory path (_fit_family_patterns).
    • The same change, for each availability pattern (up to 8 targets per fit, max_targets_per_fit).
    • Before, the previous pattern's fitted model also stayed referenced while the next pattern's fit ran, so up to 16 targets' forests could be alive at once.
  • ACS transfer, checkpointed path (_fit_family_patterns_banked, used when a target bank is passed).
    • It already fitted one target at a time, but its loop variable kept the previous step's result, and so its forests, referenced while the next step fitted.
    • Both paths now go through one helper, _fit_draw_pattern_target. It keeps the existing target-name, weight-kind, regime and finite-value checks.
    • Both paths refuse a QRF without the chain methods, with one message (_require_chain_api).
  • _verify_fitted_target_regimes is removed. Regimes are now checked per step, by the _verify_chain_target_regime the checkpointed path already used.
  • qrf.py is not changed.
  • tools/_legacy/build_us_acs_multispine_base.py: the --max-targets-per-fit help now says the option sets how many targets are chained together. It no longer limits how many targets' forests are in memory.

Why these two callers first

  • CPS-ORG is where the spike was measured. The release stage's largest memory increase, +30–35 GB, was measured during the CPS-ORG wage imputation.
  • The ACS transfer kept the most targets' forests alive of the two, and runs in two places: the base stage (with a target bank) and the local-area staging tool (without one).
  • The chain methods take them as they are. Neither sets gate_level_columns, which start_chain refuses, so qrf.py needed no change. That refusal is what keeps the UK SPI income imputation, whose stage-1 model declares gate level columns, from moving the same way.
  • Targets' forests alive at once, before and now:
    • ACS transfer without a target bank: up to 16 (two availability patterns of up to 8 targets), now 1.
    • ACS transfer with a target bank: 2, now 1.
    • CPS-ORG: 2, now 1.
  • CPS-ORG's saving is small. It is about 60 MB off the peak, because hourly_wage's forest is the largest item and is fitted first in either order (see "Expected memory change"). Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682, which fills the leaf array in place, targets that peak.

Why the draws are unchanged

  • Separate random streams. RegimeGatedQRF.fit and start_chain both split the model seed with SeedSequence(seed).spawn(2) into a fit generator and a draw generator.
    • Fitting uses only the fit generator: weighted bootstrap, forest seeds and sign-classifier seeds.
    • Drawing uses only the draw generator: one uniform per recipient for the quantile and one for the sign.
    • Moving each target's draw to just after its fit therefore consumes both streams in the same order as before: targets 1, 2, … for fitting, and targets 1, 2, … for drawing.
    • The chain carries each generator's PCG64 state between steps as exact JSON integers, including the buffered 32-bit half.
  • Same conditioning. In both orders, target k is fitted on the predictors plus the donors' observed values of targets 1…k−1. It is drawn on the recipients' predictors plus their raw draws of targets 1…k−1.
  • Global random state. quantile-forest reseeds Python's random per tree, and only while fitting, so reordering doesn't shift it.

Tests

New file packages/microcosm-build/tests/engine_free/us/test_us_qrf_target_release.py. Helpers are in test_support/microcosm_build/qrf_chain_doubles.py and fixtures in test_support/microcosm_build/qrf_per_target.py.

Frozen bytes from main. SHA-256 digests of every output table, recorded by running main's code (3b75cf1) on small fixtures.

  • Fixtures.
    • CPS-ORG: 160 recipients, 400 donors, 6 trees. hourly_wage is positive-only; is_paid_hourly has zero and positive donors.
    • ACS: 48 recipients, 160 donors, 3 trees. Three families (two person, one tax unit) with negative, zero and positive targets and booleans, each with 4 availability patterns (8 with an opt-in predictor).
  • ACS runs.
    • The shared plan.
    • With a person predictor extension, as the local-area staging tool's income pass uses.
    • With families split into batches of two.
    • Through a target bank, cold and then resumed.
  • Per-platform records. QRF draws are bitwise reproducible only within one platform (docs/graph-qrf-cross-platform.md), and these fixtures' outputs differ between macOS arm64 and Linux x86_64 even on main. So the digests are kept per platform:
    • macOS arm64, recorded natively;
    • Linux x86_64 (CI's platform), recorded under Docker emulation on Apple silicon.
  • When they apply. On each platform, Python 3.13 and 3.14 gave the same bytes, as did single-threaded native libraries and QRF thread pools. The digests are checked only when the platform and the installed numpy, pandas, scipy, scikit-learn and quantile-forest match the recording; otherwise these tests skip. The calibration tests' frozen bytes work the same way.

Same-run reference. FitThenPredictQRF is the real RegimeGatedQRF whose chain methods fit every target and then predict: main's computation, run on the callers' own inputs. It runs on every platform. Exactly equal frames, imputed-input records and fit records with it for:

  • CPS-ORG;
  • the three ACS runs;
  • the checkpointed ACS path, cold and resumed;
  • a family whose two exported immigration columns share one joint model target, in memory and checkpointed.

Lifetimes. tracked_target_fits wraps RegimeGatedQRF._fit_target.

  • What it checks. When a target's fit starts, every object built for earlier targets must be dead. That covers the target model, the sign classifier with its binning and trees, and each forest with its regressor, trees and the numpy array of its leaves' training-row positions (the large, zero-padded allocation).
  • How. Weak references, with the garbage collector off, so an object counts as freed only once nothing references it.
  • Result. On main this fails for CPS-ORG, the in-memory ACS path and the checkpointed ACS path. With the change it passes for all three, and each target is fitted once per availability pattern.
  • Sensitivity. A separate test shows the check notices a kept leaf array or tree.
  • Limit. It can't see scikit-learn's compiled Tree node storage, which can't be weakly referenced and is freed with its tree.

Other new tests.

  • Both ACS paths refuse a chain step that reports a regime other than the donors'.
  • Both ACS paths refuse a QRF without chain methods.

Test doubles. The fit/predict stand-ins for the QRF in the ORG and ACS tests now inherit FitPredictChainDouble. That includes _ProducerDtypeQRF in test_support/microcosm_build/us_multispine_pool.py, which is patched in by a multi-line setattr. The adapter provides start_chain and fit_draw_next on top of the stand-in's own fit and predict, so the stand-ins' recorded calls and outputs are unchanged.

The local-area staging tool. tools/_legacy/build_us_acs_multispine_base.py reaches the QRF only through transfer_acs_inputs without a target bank: three calls in build_optional_acs_multispine, one of them with a person predictor extension. That is the path the tests above cover, extension included.

Spec pins

acs_transfer.py is one of the attested QRF seed-kernel modules, so its new bytes move the US seed map and seed protocol digests and the specs that contain them:

  • EXPECTED_HASHES in inventory_coverage.py;
  • docs/evidence/spec-engine/us-f0-coverage.json;
  • the loader's golden vector;
  • the multispine pool tool's spec_sha256.

These were regenerated with tools/repin_us_spec_engine_contracts.py, which now passes --check. tools/spec_engine_coverage.py passes (42401/42401 fields, 41/41 inventory checks). Re-deriving the pins with main's acs_transfer.py bytes gives back main's values, so nothing else moved. No draw, random-number rule or QRF target changes.

Expected memory change

Not measured: the project's rules allow no full-size fits. The estimates follow from which objects are alive at the same time.

  • ACS transfer, in-memory path (the path the local-area staging tool uses, 32 trees per forest). The resident forests go from the sum over up to 16 targets (two patterns of up to 8) to the largest single target's forests plus its draw. The saving grows with the number of targets per batch and with how evenly sized their forests are.
    • One small addition: the first pattern's recipient predictor table now exists during its fits, because fit_draw_next draws inside the same call. That is one float64 table of recipient rows × predictors, about 160 MB for 3.3 million people and 6 predictors. Later patterns already held the previous pattern's table on main.
  • ACS transfer, checkpointed path (the base stage). From two targets' forests at once to one.
  • CPS-ORG wages. Small at peak. The dominant item is hourly_wage's single uncapped forest (100 trees on about 119,000 donor rows; the measured spike is +30–35 GB), and it is fitted first in both orders. What changes:
    • is_paid_hourly's forests are no longer alive alongside it. That target's positive forest has pure single-node trees whose one leaf holds the whole bootstrap sample: roughly 100 trees × ~70,000 entries × 8 bytes ≈ 60 MB, plus a small classifier.
    • hourly_wage's forest is freed before is_paid_hourly is fitted and drawn, not when the function returns, so the large allocation is held for a shorter time.
    • Lowering the peak itself needs the leaf-array changes: the in-place fill in a separate pull request, and the compressed leaf index lists.

Behaviour differences

These arise only outside the callers' real configurations:

  • With POPULACE_FIT_N_JOBS set to an invalid value, a chain whose targets are all zero now fails when it starts. Before, it returned zeros, because the setting was read only when a forest was grown.
  • Inputs that only a broken QRF stand-in can produce now raise at different points. A wrong regime or a non-finite draw is reported per target, after earlier targets are drawn. The message names one target, not the full list.

Follow-ups (unchanged here)

Other production callers still fit several targets in one fit and then predict, so all those targets' forests are alive together at least until the fit ends. They are left as they are. Target counts come from the constants; the UK SPI counts were counted by hand.

  • US, in us_runtime/:
    • puf_support.py impute_us_puf_tax_detail_support: 52 targets. It uses predict(release_models=True), but all 52 targets' forests exist when the fit ends. This is only the non-checkpointed path; the checkpointed base stage already uses puf_qrf_chain.
    • scf_education_debt.py, the composition model: 10 targets, with release_models=True.
    • These keep the fitted model until the function returns:
      • retirement_contributions.py (5 targets);
      • retirement_distributions.py (4);
      • scf_wealth.py financial assets (3);
      • prior_year_income.py, other_health_insurance.py, child_support.py and scf_auto_loans.py (2 each).
    • sipp_financial_assets.py fits one target per fit in a loop, but the previous model stays bound while the next one fits, so two exist briefly.
  • UK, in uk_runtime/. These draw with predict_from_uniforms, which has no option to release models.
    • spi_income.py impute_uk_spi_income_support: a stage-1 model (about 21 targets), up to 5 reference fits (about 12 each) and a stage-2 model (38 targets), all held until the function returns.
    • spi_housing_shell.py (12 targets).
    • lcfs_consumption.py redraw_zero_road_fuel (2 targets).

Checks

  • Tests run. Only the added and modified tests, each selected by node ID, all passing:
    • the new tests: on macOS arm64, and under Docker emulation on Linux x86_64 with Python 3.13 and 3.14;
    • the 52 existing tests whose QRF stand-ins changed (one skip, which needs the US engine);
    • the repinned loader test;
    • the repinned engine-workflow spec_sha256 test, with the US engine installed.
  • Other commands. uv run ruff check ., python3 tools/ci_test_plan.py verify, tools/repin_us_spec_engine_contracts.py --check and tools/spec_engine_coverage.py pass.
  • Review. Six rounds of independent adversarial review; rounds 4–6 found no behaviour problems, only wording, which is fixed. Reviewers ran main's org_wages.py and acs_transfer.py side by side with the branch over many option sets and seeds, and found no byte difference. They also counted live trees, regressors and classifiers with the garbage collector off, and found none surviving.
  • CI. The engine-free jobs ran on GitHub's Ubuntu x86_64 runners with Python 3.13 and 3.14 and the locked library versions, and passed. That platform and those versions match the Linux recording, so the frozen Linux digests, recorded under emulation, were checked there. The job log doesn't print each test's result, so this rests on the tests' skip conditions.
  • Not verified. The memory change is estimated from which objects are alive at the same time, not measured (see "Expected memory change").
  • Changelog. changelog.d/qrf-fit-draw-per-target.changed.md.
  • No deployment, workflow or environment-variable changes.

🤖 Generated with Claude Code

anth-volk and others added 10 commits November 17, 2026 15:07
…ansfer

impute_us_org_wages and the ACS transfer's in-memory pattern fits used to fit
every target and then predict, so every target's forests stayed referenced
until prediction ended (and, in the ACS transfer, while the next pattern
fitted). Both now call RegimeGatedQRF.start_chain and fit_draw_next once per
target through helpers that return only the draw and the next chain state,
so each target's forests and sign classifier are freed before the next fit.
The checkpointed ACS path already fitted per target but its loop kept the
previous step result referenced during the next fit; it now uses the same
helper.

Fitting and drawing use separate random streams and the same conditioning
in both orders, so imputed values are byte-identical. New tests pin output
digests recorded from main's code, compare against fitting every target
first on the callers' own inputs, and check with weak references (garbage
collector off) that no earlier target's fitted objects are alive when the
next fit starts. Fit/predict QRF test doubles gain the chain methods through
a shared adapter. acs_transfer's new bytes move the attested seed-kernel
digests; pins regenerated with tools/repin_us_spec_engine_contracts.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for the one-target-at-a-time QRF change:

- QRF draws are bitwise reproducible only within one platform, and these
  fixtures' outputs differ between macOS arm64 and Linux x86_64 even on main.
  The frozen digests are now kept per operating system and machine, recorded
  from main's code on macOS arm64 natively and on Linux x86_64 under Docker
  emulation (Python 3.13 and 3.14 agree on each); other platforms skip them,
  and the in-process comparisons run everywhere.
- Add an in-process comparison of the checkpointed ACS path, cold and
  resumed, against fitting every target first.
- _ProducerDtypeQRF, patched into the ACS transfer by a multi-line setattr,
  gets the chain adapter.
- The legacy staging tool's --max-targets-per-fit help no longer says it
  bounds retained targets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 fixes:

- The frozen digests are checked only when the installed numpy, pandas,
  scipy, scikit-learn and quantile-forest match the versions they were
  recorded with, as the calibration tests' frozen bytes are; otherwise the
  tests skip and the in-process comparisons remain.
- The CPS-ORG fixture builds its people, frame and donor itself (copies of
  the shared ORG test helpers), so edits there for other tests cannot move
  its bytes.
- table_bytes_digest includes a categorical column's categories and order.
- New test: both ACS paths refuse a chain step that reports a regime other
  than the donors' (the per-target regime check had no test).
- The in-memory ACS path refuses a QRF without start_chain/fit_draw_next
  with the same message as the checkpointed path; a comment about test
  doubles now covers both; the changed lines follow ruff format, as the
  file did on main. acs_transfer's new bytes move the seed-kernel pins again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check

The lifetime check now also requires every tree of each earlier forest, the
numpy array that owns each quantile leaf store's values, and the sign
classifier's binning and trees to be freed before the next target's fit, so
code that kept only part of a forest alive would fail it. A new test shows
the check notices a kept leaf array or tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 additions:

- A family whose two exported immigration columns share one joint model
  target now has an in-process comparison against fitting every target
  first, in memory and with a target bank. The existing banked-versus-
  in-memory tests compare the chain with itself since both paths use it.
- Both ACS paths refuse a QRF without start_chain/fit_draw_next, tested.
- The lifetime check's docstring names the parts it cannot see: scikit-
  learn's compiled Tree node storage and the sign classifier trees' node
  arrays, which cannot be weakly referenced and are freed with their trees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lifetime check now also weakly references each sign-classifier tree's
node array, and its docstring names the parts it cannot see (scikit-learn's
compiled Tree storage and quantile-forest's leaf store object). The table
digest's docstring says which columns it hashes byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Say what a change that moves the outputs records; say which digested
tables have string columns; rename a test whose name said the batch split
bounds retained forests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The quantile leaf array holds each leaf's training-row positions, not
values; the digest is byte-exact for NumPy bool, integer and float columns;
the re-recording note names what to rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both sides moved the US spec digests: main through microcosm#1585's
us/gates.json row, this branch through acs_transfer's attested bytes. Keep
both explanatory comments and regenerate the digests and the coverage
evidence for the merged code with tools/repin_us_spec_engine_contracts.py
and tools/spec_engine_coverage.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 19d3d4b4a. The chaining is unchanged, and the outputs match main byte for byte. Three things before merge, and two for the follow-ups.

Chaining

Each chained target conditions on the same values, in the same order, as before:

  • fit_draw_next fits target k on the predictors plus the donors' observed targets 1…k−1. It draws it on the recipients' predictors plus their raw draws of 1…k−1, through the same _draw_target_with_rng and the same split fit and draw streams as fit then predict.
  • Both callers pass each step's raw_draw back unchanged. org_wages.py masks inactive workers, and acs_transfer.py decodes the joint codecs, only after the loop.
  • The chains cover the same targets as before: each availability pattern's model_targets, split by DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT = 8.
  • So later stages that use these columns as predictors see identical inputs.

Before merge

  1. Add a changelog fragment. Each of the last 25 PRs merged to main adds one.
  2. Say in the PR why these two callers move first. From the code, they are the ones the chain API takes as they are, with the most forests alive at once: up to 16 targets in the in-memory ACS path and 2 in the checkpointed one, against 1 now. For CPS-ORG, the PR's own estimate is about 60 MB off the peak, because hourly_wage's forest is fitted first either way; #1682 targets that peak.
  3. Update "Not verified". CI's engine-free jobs run on Linux x86_64 with the locked versions, so they checked the Linux digests recorded under emulation, and passed.

If #1682 merges first, re-run tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py: both PRs move the same US seed-protocol pins.

For the follow-ups

  • UK SPI income holds about 120 targets' forests until it returns, by the PR's counts. It can't move as it stands: its stage-1 model declares gate_level_columns (spi_age_band, main_source, has_pay, has_trade), and start_chain refuses them. fit_draw_next already draws from caller-owned uniforms, so the gate levels, not predict_from_uniforms, block SPI. spi_housing_shell.py and lcfs_consumption.py declare none.
  • fit_draw_next trusts raw_prior_draws. It checks their names, order, index, dtype and finiteness, but not that they are the chain's own raw draws. Before more callers move, consider recording a digest of each step's raw_draw in the chain state and refusing priors that don't match. A post-processed prior would then fail loudly instead of changing the conditioning.

Checked

  • 321 of 324 tests on the touched surface pass locally (macOS arm64, Python 3.14); the 3 skips need the US engine. They include all 21 in test_us_qrf_target_release.py, where main's macOS digests ran and matched.
  • main hasn't changed acs_transfer.py, org_wages.py or qrf.py since the digests were recorded at 3b75cf151.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Contributor Author

Thanks for the review. The three items before merge are done, in aa26137 and the PR description:

  1. Changelog fragment: changelog.d/qrf-fit-draw-per-target.changed.md.
  2. Why these two callers move first: a new section, "Why these two callers first". It covers:
    • the +30–35 GB increase measured during the CPS-ORG wage imputation;
    • the targets' forests alive at once, before and now: up to 16 → 1 in the ACS transfer without a target bank, 2 → 1 with one, and 2 → 1 in CPS-ORG;
    • why the per-target methods take both callers without a qrf.py change: neither sets gate_level_columns, which start_chain refuses;
    • that CPS-ORG's saving is about 60 MB, and Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682 is aimed at that peak.
  3. "Not verified": replaced by a CI entry. The engine-free jobs passed on GitHub's Ubuntu x86_64 runners with Python 3.13 and 3.14 and the locked library versions, so the Linux digests recorded under emulation were checked there. The memory change, which is estimated, is the only item left as not verified.

If #1682 merges first, this branch will re-run tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py after merging main.

Brings in #1682 (the QRF leaf array filled in place). Both PRs move the
US seed-map and seed-protocol digests, so the four spec-pin files
conflicted; they are regenerated with tools/repin_us_spec_engine_contracts.py
and tools/spec_engine_coverage.py, keeping both PRs' comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants