Repository navigation
Conversation
The UK SPI income imputation kept its stage-1 model and every reference fold's model referenced until it returned, while stage 2 fitted 38 targets at once. It now keeps only the receipts it reports from those models, frees each reference fold's forests before the next fold, frees the stage-1 forests after their draw, and fits and draws stage 2 one target at a time through the QRF chain. The person table and every receipt stay byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same-run comparison took its stage-2 inputs from the code under test, so it could not see a wrong salt, seed, group or prior draw, and the two stages' gate receipts shared one code path with their reference. The fit-everything- first model now records what the stage passes it, and the tests compare those inputs and the receipts with expressions written from the earlier code, also for runs whose two stages declare different gates. The lifetime check also covers compiled trees and the leaf sorter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stage-2 checks now also cover the chain's seed, tree count, gate calibration, predictor names and the recipient predictors stage 1 does not write, and spell out the released uniform salts. The digest command prints the library versions the digests depend on. A comment now says a reference fold without recipients adds no receipt entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cross-fitted gates are calibrated through BLAS, and the kernel BLAS loads on a given CPU moves the last bits of the transport receipt's statistics, so an exact digest would fail on machines other than the recording one. Digests for that scenario now round floats to ten significant digits and were recorded again from origin/main's code; they match under four OpenBLAS kernels. The in-sample digests and every same-run comparison stay exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests now compare the chain's weights, training table and training weights with the FRS rows, the recipients' stage-1 incomes and the stored draws with the returned person table, so these no longer rest on the frozen digests alone. The cross-fitted digests round floats to eight significant digits, which leaves more room for BLAS rounding on CPUs that were not recorded; they were recorded again from origin/main's code and match under four OpenBLAS kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge 2's weight kind comes from Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… off The check that no fitted model outlives the stage read its weak references after the tracking block had turned the collector back on, so a model held only by a reference cycle could be collected first and go unnoticed. The tracked run now records what is alive before leaving the block. The changelog and a test docstring also say more exactly what is freed and what is covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed at 754d17f28. Stage 2's chain conditions on the same values as fit then predict_from_uniforms, and freeing stage 1 and the reference folds changes no receipt. Two things before merge.
What I checked in the code
- Stage 2 stores each step's
raw_drawinstage2_drawsand passes it back as the next step's prior draws. Later steps rewrite four of its columns only after the loop, and the uniforms keep their keys, seed and salts. - Under the default calibration,
gate_outcomesandpooled_gate_outcomesreturn{}for the empty evidence, so the receipt is{}as onmain. Stage 2's outcomes are collected in target order. - The tape fixture is resampled from the invented four-record donor; the real tape isn't tracked.
Before merge
- Merge
mainfor a fresh CI run. #1682 changedqrf.pyand #1666 added the raw-write scan after CI last ran here. A local merge with7ff5a26bcis clean and passes the touched surface (below). - Build the UK spine before and after this change on one quiet machine, as for #1682, and compare the SPI stage's memory. Production fits both stages cross-fitted with stage-1 levels, the scenario whose frozen digest is rounded to eight digits, so the pair is the exact check at full size. It is also the clean memory measurement this PR needs: the #1682 pair's footprints (61.6 GiB against 42.6 GiB) ran under different memory pressure. Start both builds with nothing else running and the same free memory, check that
compare_uk_h5_payload.pyexits 0, and report each build's peak footprint duringhmrc_spi_income_spinefrom the build telemetry.
Notes
- If #1684 merges first, the first behaviour difference goes away: #1684 stops comparing the fit-job count between chain steps.
- The macOS digests also match on an M5 Pro with macOS 26.6.2, on the merge with
7ff5a26bc, so after #1682's in-place leaf fill. That covers one "Not verified" item.
Checked
- 292 of 293 tests on the touched surface pass on a local merge with
7ff5a26bc(macOS arm64, Python 3.14). They include the 48 intest_uk_spi_forest_release.py, both digest tests among them, and #1666's raw-write scan; the one skip needs the licensed tape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Thanks, Anthony. I traced the change against main and found nothing that changes output or receipts. Paths are under packages/microcosm-build/src/microcosm/build/uk_runtime/.
Checked in the code:
- Stage 1 is read for
_sign_class_mismatchesandpredict_from_uniformsbefore_fit_gate_evidencecopies its receipts anddel stage1runs (spi_income.py:1771-1778). Nothing reads it after that. - Reference folds.
cross_fit_transportdropsreferenceoncetransport_uniformshas located the fold's recipients (spi_rank_transport.py:388).SPITransportResultholds only arrays and the receipt, andfit_referencekeeps only a_FitGateEvidence, so no fold's model survives.- The folds use
stage1_gate_calibrationand the stage-1 level kwargs, so reading their outcomes and levels under the stage-1 flags matchesmain's conditions. reference_evidence[0].weight_kindfails on an empty list asreference_weight_kind["kind"]did.
- The folds use
- Receipts.
- Under the default calibration, the evidence holds
{}, andgate_outcomesandpooled_gate_outcomes(qrf_gates.py:170-195) then return{}, so the receipt is{}as onmain. gate_level_reportsgets{}from the evidence wheremaingot it from thegetattrfallback. That only matters when level columns are declared, and the evidence reads the reports then.
- Under the default calibration, the evidence holds
- Stage 2. Each target's uniforms use the same salt,
f"{FRS_ONLY_FIT_NAME}:{name}"and its:signvariant, withseed + 1. Each step'sraw_drawgoes intostage2_draws, which the next step reads as prior draws._fit_draw_stage2_targetreturns without the step result, so that target's forests can be freed.
Runs.
- CI passes all 12 checks at
2205f85. The engine-free (3.13) job ran this file: 32,018 passed, 0 failed. - I started the 48 tests locally, but they used every core and didn't finish a test within 11 minutes, so I stopped them. Their result here rests on CI and María's local merge run.
Still open from María's review: item 1, the fresh CI after merging main, is done. Item 2, the before-and-after UK spine pair on one quiet machine, isn't reported yet. It would check compare_uk_h5_payload.py at full size and give each build's peak footprint during hmrc_spi_income_spine. That is the one check left for the memory claim.
Review drafted with AI assistance and checked against the code at 2205f85.
What this changes
The UK SPI income imputation (
impute_uk_spi_income_support, build stagehmrc_spi_income_spine) now frees each fitted QRF as soon as it has finished using it. The person table and every receipt are unchanged frommain._FitGateEvidencerecord, then runsdel stage1before stage 2 is fitted.SPI_TRANSPORT_FOLDS= 5, 12 targets each, fitted on FRS rows). Nothing draws from them. Each fold's model is used to compute its recipients' uniforms for rank transport.fit_referenceappended every fold's fitted model toreference_fits, which the pooled receipts read when the stage returned.fit_referenceappends a_FitGateEvidencerecord instead.cross_fit_transport(spi_rank_transport.py) deletes the fold's model once the fold has located its recipients, before the next fold is fitted.FRS_ONLY_SPI_FILL_PERSON_COLUMNS, less the 2 inSPI_STAGE2_REVIEWED_ABSENT_OUTPUTS).fitover all 38 targets, then onepredict_from_uniforms, so all 38 targets' forests were in memory together.RegimeGatedQRF.start_chain, then onefit_draw_nextper target, through_fit_draw_stage2_target._FitGateEvidenceprovides the three methodsqrf_gates.pyreads:gate_outcomes(),gate_levels()andgate_level_reports(). It also has aweight_kindfield, which the stage reads forfit_weight_records._fit_gate_evidencereads only the receipts this run reports. Sign-classifier outcomes are read when that stage's calibration is notUK_DEFAULT_GATE_CALIBRATION. Level offsets and level-fit reports are read when stage 1 declares level columns. These are the same conditionsmainused when it built the result.qrf.pyandqrf_gates.pyare not changed.Tracker project 15 (US build memory, CPU and run time), task 1.2 · 5.
Why stage 2 moved to the chain, and stage 1 and the reference fits did not
start_chainandfit_draw_nextwith its existing per-person uniforms, without changing bytes. It can:fitandstart_chainboth split the model seed withSeedSequence(seed).spawn(2)into a fit generator and a draw generator. Fitting uses only the fit generator, and the chain saves and restores its full state between steps. Drawing here uses the uniforms the stage passes in. Fitting and drawing per target therefore consumes the fit stream in the same order as before.fit_draw_nextdraws supplied uniforms through_draw_target_from_uniforms, the functionpredict_from_uniformsuses. Each target's quantile and sign uniforms come fromstable_identity_uniformswith the same keys, seed and salts as before. Only the dict comprehension became a per-target call.start_chainrefuses a model constructed withgate_level_columns, which stage 1 and the reference fits use when level columns are declared. Stage 1 also draws through rank transport, and only after every fold has been fitted. So these keep onefiteach and are freed as a whole.Why the output is unchanged
fit_weight_records,gate_outcomes,gate_levelsandlevel_fitshold the same data in the same key order.start_chainreturns.Tests
New file
packages/microcosm-build/tests/engine_free/uk/test_uk_spi_forest_release.py, with its fixture and helpers intest_support/microcosm_build/uk_spi_forest_release.py. It holds 48 test cases.Scenarios.
in_sample: default sign-classifier calibration in both stages.cross_fitted_levels: cross-fitted calibration in both stages, stage-1 level columnsspi_age_band,has_payandmain_source, and conditioning on state pension receipt. Every receipt read from a model is non-empty here, and a test checks that.Frozen digests from
main. SHA-256 digests of the person table and of every other result field exceptdonor_path, which names the test's temporary directory. They were recorded by runningmain's (402c358)impute_uk_spi_income_supporton the fixture.docs/graph-qrf-cross-platform.md). The digests are therefore kept per platform: macOS arm64, recorded natively, and Linux x86_64, recorded under Docker emulation on Apple silicon.uv.lock, and so on CI, match the recording.in_sampleis compared to the bit.cross_fitted_levelscompares floats at eight significant digits. Its sign classifiers are calibrated through BLAS (@andnp.linalg.solve), and which OpenBLAS kernel loads on a given CPU moves the last bits of the transport receipt's statistics.maindoes this too.main's code are identical under four OpenBLAS kernels: the autodetected one (Nehalem), Sandybridge, Haswell, and Zen (which loads Haswell).Same-run comparison, every platform.
FitEveryTargetFirstQRFis the realRegimeGatedQRFwith the lifetimes and order the stage had onmain.start_chainfits all targets at once, asfitdoes, and itsfit_draw_nextonly draws, through the functionpredict_from_uniformsuses.Inputs and receipts, every platform.
mainused:self_employment_incomeis excluded, because the stage rewrites it after stage 2, so only the frozen digests cover it.employee_pension_contributions,employer_pension_contributions,trading_lossandowned_company_carries_on_trade.Lifetimes.
tracked_spi_forest_fitswrapsRegimeGatedQRF._fit_target.Treecannot be weakly referenced, so the check holds each one and counts its other references.Test doubles.
SPIChainDouble(test_support/microcosm_build/uk_spi_income.py) gives the existingfit/predictstand-insstart_chainandfit_draw_next. Both_FakeQRFclasses now inherit it: the one in that module and the one intest_uk_spi_spine.py.Runs. Only these tests were run, each selected by node ID.
b4445e2c5, before the last two commits changed a docstring and the after-stage lifetime check, all 48 also passed on macOS arm64:POPULACE_FIT_N_JOBSandPOPULACE_FIT_PREDICT_WORKERSunset);test_uk_spi_spine.pytests that use the changed_FakeQRFpass.Expected memory change (estimate, not measured)
No full-size fit was run. These estimates follow from which fitted models are in memory at the same time, counted in targets. Each target's model holds up to two forests and a sign classifier.
hmrc_spi_income_spine, while the build process's resident memory peaked at about 7 to 14 GiB. Which objects held that memory is not established. Retained forests are an inference from the code.Behaviour differences
These come from stage 2 now going through
start_chainandfit_draw_next. In each casemainreturns and this branch raises:POPULACE_FIT_N_JOBSchanges between stage-2 targets._validate_chain_configraisesValueError. Thetools/scripts set this variable only when the process starts._index_identityraisesUnicodeEncodeErrorwhile hashing the index. This is not plausible for valid person data.POPULACE_FIT_N_JOBSis invalid and the run fits no forest.start_chainreads the setting and raisesValueError. Onmainthe setting is read only when a forest is fitted, so any run that fits a forest already fails on it.Cached stage results and pins
spi_incomeorspi_rank_transport:uk.stage.hmrc_spi_income_spine@1uk.stage.property_components@1uk.stage.company_cars@1hmrc_spi_income_spineand everything after it, although the output should be identical.uk/specandspec_enginedo not reference these modules.fit.qrfparity pins depend onqrf.py, which is unchanged.Follow-ups (not changed here)
etb_services.py,lcfs_consumption.pyandwas_wealth.pykeep the previous target's result while the next target is fitted.spi_housing_shell.pyfits 12 targets in onefit.uv.lock, only the every-platform checks run until the digests are recorded again withpython -m test_support.microcosm_build.uk_spi_forest_release.Not verified
Review
Before this pull request was opened, five rounds of adversarial review by fresh subagents looked for output or receipt differences, surviving references to freed models, lost receipt data and weak tests. A sixth checked this description and the changelog against the code. Their findings were fixed, apart from those listed under Behaviour differences and Follow-ups. This was self-review and is not a pull request review.
🤖 Generated with Claude Code