Repository navigation
US household location v1: one census block per household, every geography derived - #1047
Conversation
|
The Connecticut CBSA defect noted in this PR's body is fixed in #1072. That PR maps each CT 2020 block to its 2022 planning region through its town (the BAF MCD layer plus Census's
|
b112560 to
09b798d
Compare
49f01e1 to
83287c4
Compare
83287c4 to
e778f05
Compare
…aphy derived (#696) Max's 2026-09-27 ruling: draw one 2020 census block per household with probability proportional to block population within the finest geography its source provides (ACS: observed PUMA; CPS ASEC: identified county, else its state outside the counties that are both on Census's List 4 for that ASEC year and coded in that year's file; else state), and look every other geography up from the block. One function, us_runtime/block_location.py, wired into the base-h5 line, the stacked multispine pool (with h5_io consumer validation) and the ACS local-release line behind --location-rule block_v1; the default 'legacy' path stays byte-identical. - Keyed draws (stable_identity_uniforms over household id and clone), exact inverse CDF, optional K clones at weight/K (refused on every line until Route A admits location clones). - The ACS line keeps each donor's block, so it requires a donor base that recorded block_v1 (each donor drawn within its CPS source geography) and refuses a legacy base before the ACS fetch, and any donor without a block. - Block-ladder builder adds an additive per-block 'puma' array. - Census's CPS ASEC identified-county lists (List 4, ASEC 2023-2026) packaged with PDF digests and a per-year whole-county-guarantee flag. - Property tests (Hypothesis) for consistency, containment, exactness, convergence, determinism, row-order invariance, independence from the other households and clone conservation on every weighted entity; a Connecticut planning-region test with independently written regions; differential tests against the PUMA ladder and the national artifacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e778f05 to
33969fd
Compare
- Clone frames by row position, so a duplicate pandas index no longer multiplies rows; the conservation property now runs on range, duplicate and reversed indexes. - Give the CBSA narrowing test two competing CBSAs in one state, so a draw that ignores the narrowing fails it. - Drop the removed state-draw fallback from --location-rule's help on the ACS staging tool, and name the block_v1 base it requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The Microcosm hub now owns this PR and lands it.
|
|
This landed as a60b164 on 2026-11-13 at 19:06 UTC, at the handed-off head da636f4. Main's tree equals the tree git computes for merging da636f4 into b647c54 (3bbb6526c). Since this branch's base, main and the branch both changed one file, Landing it puts three open pull requests in conflict: #779 in |
Main gained #1605, #1590 and #1047 (US household location v1) since the last merge (d5a9322). Two files conflict. us_runtime/h5_io.py, _validate_stacked_late_dag_manifest_binding: this branch made the expected operator order depend on the manifest's schema (an attested schema-9 pool binds the literal order it sealed, without the immigration composition gate), and #1047 made it depend on the manifest's geography rule (us_stacked_pool_operator_order). The merge keeps both: a current pool binds the order its geography rule selects, and a schema-9 pool binds the schema-9 literal. #1047 derives the block_v1 order from US_STACKED_POOL_OPERATOR_ORDER, so that order carries this branch's immigration gate too. tools/build_us_puf_support_base.py: both sides added functions after _write_policyengine_dataset. The merge keeps this branch's two immigration helpers and then #1047's location functions. Eight files change on both sides. Five functions change on both sides (_validate_stacked_geography_h5_binding, do_finalize, build_stacked_pool, _parse_args, _run_all); each merged body equals either side plus exactly the other side's lines. #1047 bumps no checkpoint or schema version, and the merged pool tool writes operator_order through #1047's _stacked_operator_order_for_contract at both sites. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…xture The merge of main after #1047 (a2f2f3f) brought in five block_v1 loader tests without a textual conflict. Their helper, _rewrite_as_block_location_pool, assigned three fixed blocks to the ready stacked fixture's households. On main that fixture has three households (two native and one clone). On this branch it has one ASEC household and one ACS household for each positive humanitarian draw, so the helper raised "Length of values (3) does not match length of index (13)" before the loader ran, and six test cases failed. The helper now gives each household row a block by clone lineage: the first native household and its clone share the first block, and every other native household sits in the second. The accepting test and the clone-drift test ask the fixture for its clone row, as #1047's fixture always carried one. Test code only. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…anifest _validate_stacked_late_dag_manifest_binding expects one operator order per kind of stacked manifest. A current manifest binds the order its geography rule selects: US_STACKED_POOL_OPERATOR_ORDER under the legacy rule and US_STACKED_POOL_BLOCK_LOCATION_OPERATOR_ORDER under block_v1 (microcosm#696), both ending with the immigration composition gate (microcosm#767). An attested schema-9 manifest binds _SCHEMA9_STACKED_POOL_OPERATOR_ORDER, the literal order it sealed, which predates the gate and the block_v1 rule. The merge of main a60b164 combined this branch's schema-9 rule with main's per-rule order (#1047), and no test held the combination at the function. The schema-9 cases ran only through the whole scoring loader, behind importorskip("tables"), and no test refused a current legacy-rule manifest that carries another order. test_late_dag_binding_expects_one_operator_order_per_manifest_kind is a nine-row table of direct calls, on manifests that carry only what the rule reads: - current, legacy rule: binds its order; refused with the schema-9 order (no gate) or the block order; - current, block_v1 rule: binds the block order; refused with the legacy rule's order or with the block order minus the gate; - attested schema-9: binds the order it sealed; refused with either current order. Each row also checks that the binding reaches the geography receipt only after it accepts the order. Mutation check, in memory (a pytest plugin compiles h5_io with the rule replaced; no file changed). With main's rule, the per-rule order for every manifest, the two schema-9 rows fail. With this branch's rule before the merge, the legacy-rule order for every current manifest, the two block_v1 rows and test_h5_io_accepts_a_block_manifest_round_trip fail. Test code only. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Main gained #1600, #1605, #1590, #1047, #1592, #1598 and #1545 since the 1e88319 this branch last merged. #1598 changed microcosm-calibrate (hierarchy value ranks) and an attested kernel module and regenerated the shared pins, so the five pin files conflicted. Each conflict hunk takes main's text, with this branch's #1096 comment sentence kept after main's, and the tools then wrote the values for the merged tree: - docs/evidence/spec-engine/us-f0-coverage.json: main's five hunks, then rewritten by tools/spec_engine_coverage.py (two map, two protocol, two implementation and two spec digests; 42394/42394 fields, 41/41 checks). - spec_engine/inventory_coverage.py: main's hunk and its hierarchy comment, plus the #1096 sentence; tools/repin_us_spec_engine_contracts.py wrote seed_map d29d95be and seed_protocol f3738669 (main: 0c19b903, 15e6cad1). - test_spec_engine_loader.py: main's hunk plus the #1096 comment line; the repinner wrote the golden 0b417bfc (main: 99f9b82e). - test_us_multispine_pool_tool.py: main's hunk plus the #1096 comment line; the repinner wrote spec_sha256 c79a1d3f (main: 98d58365). - parity/kernels/calibrate/pins.json: main's line, then tools/graph_parity_repin.py calibrate wrote implementation_hash 20180a24 and node_key 86ae2227 (main: 0c669ad2, fbab5f42). direct.csv, graph.json and inputs.csv did not move, so the default path's bytes are main's. The repinner and the coverage tool ran twice each, then both passed --check. ruff format changed only the quotes the repinner writes. tools/build_us_acs_local_release.py and its test merged without conflict. Main's #696 and this branch both changed do_package and _parse_args, in separate statements: main swapped the refresh recipe's staging value and added --block-ladder; this branch renamed one calibration key and reworded the --mass-parametrization help. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Main gained #1600, #1605, #1590, #1047 (US household location v1, the opt-in block_v1 rule), #1592, #1598 and #1545 since the 1e88319 this branch last merged. Seven files conflicted: - us_runtime/acs_multispine.py: the base_pool import block only. Union: main's ACS_POOL_BLOCK_LOCATION_METADATA_KEY and this branch's acs_pool_local_jurisdiction_rungs. The function body merged cleanly. - us_runtime/base_pool.py: __all__ only. Union of main's two new names and this branch's one. - tests/engine_free/us/test_us_spine_blindness.py: the pool tool's runtime-graph count. Main counted 83 (81 plus block_location.py and cps_source_geography.py), this branch 82 (81 plus local_jurisdictions.py). The test's own graph helper reaches 84 on the merged tree, with all three modules in it and none unclassified; the comment keeps both explanations. - spec_engine/inventory_coverage.py, tests/engine_free/shared/ test_spec_engine_loader.py, tests/engine_workflow/us/ test_us_multispine_pool_tool.py and docs/evidence/spec-engine/ us-f0-coverage.json: digests only. Each conflicted hunk takes main's value and main's comment, and keeps this branch's comment sentences beside it. The values are stale until the pin tools rewrite them in a later commit. Functions both sides changed and git merged without conflict: build_optional_acs_multispine, with_optional_acs_spine, the fiscal refresh builder's _main and the pool tool's _validate_stacked_geography_assignment_receipt. The last two are independent edits. The first two are not: under block_v1 the merged pool still picks the flag rungs from the PUMA ladder alone, so it removes every flag from a block-located pool, and the receipt can name rungs the pool did not use. A later commit fixes that. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
#1047's opt-in --location-rule block_v1 puts each household on one drawn 2020 block and looks county and place up from it. This pull request derives four engine inputs from county and place, and its ladder gates refuse a table that lacks one. Merged, the two did not meet: - with_household_us_block_location (the base-h5 line and the stacked pool) wrote no flag, so the base-h5 line's geography-ladder gate and the stacked pool's PUMA-ladder gate refused every block_v1 draw (17 of #1047's stacked-pool tests failed at the merge commit); - with_optional_acs_spine picked its flag rungs from the PUMA ladder alone. A block_v1 pool takes none, so it removed every flag, while build_optional_acs_multispine, which staging also hands a PUMA ladder, recorded the county-rule flags as derived. Now every line that draws derives all four flags from the drawn or kept block's county and place (BLOCK_GEOGRAPHY_RUNGS), after the draw, and overwrites any flag the frame carried before it: - with_household_us_block_location applies the flags to its tables once the block's geography is written, as the block ladder does; - acs_pool_local_jurisdiction_rungs takes the location rule and returns the block rungs under block_v1; the pool and the multispine receipt both pass it, and the block_v1 geography record carries the block-rung receipt in place of the PUMA summary's; - the stacked pool's block-location summary records local_jurisdiction_flags, as its legacy summary does. The default legacy rule is unchanged: acs_pool_local_jurisdiction_rungs returns what it did when location_rule is left at its default, and no legacy ladder, gate or receipt is touched. Tests: a shared synthetic ladder carrying Denver, Philadelphia, Yonkers, Wilmington and a decoy for each place code, and an expectation written out in the test support, not read from the registry. On each line (kernel with 1 and 3 clones, base-h5, stacked pool, ACS pool, ACS multispine) a frame that carries flags from before the draw comes out with flags that equal the rules on the drawn geography, and the line's receipt names the county and place rungs with matching counts. The module docstring, US_LOCAL_JURISDICTIONS.md and docs/us-location-v1.md now say this; they named a block-location draw that did not exist when the branch was written (review D1). The pool's static remaining-stage manifest still lists the place-rule flags as unavailable for every pool; its comment and the document say that this describes the default rule. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Main gained #1605, #1590, #1047, #1592, #1598, #1545 and #1612 since this branch's last merge. Git merged all of them without a conflict. #1592's delete-old-layout step calls publishes_to_build_folder to tell a folder named by a release from a folder named by one build's tag. With this branch, it no longer counts a UK local-area name the registry does not name as such a release. The next commit pins what the step reports for such a folder. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Main (52aea14) moved the same pins this branch moves: #1598 changes medicaid_take_up, an attested module. The five pin files conflicted. Both sides' reason comments are kept, and every value is recomputed on the merged tree: - the documented sequence (repin --legacy-only, the bundle generator, repin, spec_engine_coverage, repin again), with all three --check modes passing and 42395/42395 fields and 41/41 inventory checks; - tools/graph_parity_repin.py calibrate, with the case's direct output unchanged. Merged values, against main's: - seed_protocol 15e6cad1… -> a696ad74…, seed_map 0c19b903… -> 270cf649… - US spec_sha256 98d58365… -> bd526d8c… (pool tool pin) - loader golden 99f9b82e… -> 1ff3b42c… - calibrate parity implementation_hash 0c669ad2… -> 018974ad… The release tool and test_spec_engine_seeds.py (#1047) merged without conflict. The venv is resynced for main's microcosm-emitter-interface 0.15.0. Tests on the merged tree: seeds 13, loader 10, field usage 14, engine-free spec bundle 4, graph parity pins 11, keyed draws 64, recertification parity 4, post-export scoring 48, batched target materialization 37, the builder's signed-component tests 2, head-to-head scorer 54, coverage tool 7, engine-contract spec bundle 22, the pool tool's pin test 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new LOCATION_RULE setting picks how the base locates households. - legacy, the default, keeps the published run's command byte for byte: a congressional district drawn from SOI return counts, then a block within it. - block_v1 draws one 2020 census block per household within its CPS ASEC source geography. The base drops --ledger-facts, --assign-congressional-districts and --congressional-district-seed, which it refuses beside --location-rule block_v1, and passes --location-rule block_v1 --location-seed 0. It keeps the vintage crosswalk, which the release checks, and the block ladder. - Under block_v1, check_commit also requires the base tool at the commit to declare the two new flags. A legacy build of an older commit is checked as before. - Any other value stops the driver with exit 64. The base command now reads its location flags from base_location_args, which section 4 calls once W is set. Tests check that: - the legacy command still matches the Modal registration; - the block_v1 command differs only in the location flags; - the base tool's own parser accepts the block_v1 command; - an unknown value is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #696. This PR implements its block-first draw on all three US lines; the predictor-surface and lineage-spec items stay open there.
Summary
Max ruled on 27 September that each household draws one 2020 census block, with probability proportional to block population, within the finest geography its source provides, and takes every other geography from that block. Calibration supplies fidelity afterwards. This PR adds that rule as one function in
us_runtime/block_location.pyand wires it into every US line behind--location-rule block_v1. The default,legacy, keeps each line's current rule and bytes.It replaces three rules:
legacy)block_v1build_us_puf_support_base.py, feeds Route A)build_us_multispine_pool.py)_legacy/build_us_acs_multispine_base.py→build_us_acs_local_release.py)block_v1base drewThe rule
Candidate blocks come from each record's source:
GTCO≠ 0: the blocks of that county.GTCO= 0: the blocks of its state, minus counties that Census's identified-county list for that ASEC year names (List 4 of the technical documentation) and that year's file codes. A year whose List 4 drops the whole-county guarantee (ASEC 2026) excludes nothing.The draw then looks up tract, county, the 2023 county-equivalent (
county_fips_2023, which uses Connecticut's planning regions), place, SLDU/SLDL, CBSA, PUMA and every attached congressional-district plan from the block, so a household's geographies cannot contradict each other.The draw keys each uniform to the household, not to its row:
stable_identity_uniforms("{household_id}:{clone}", seed, salt=rule id)feeds an exact inverse CDF over block population. The draw also supports K clones, each with weight/K. Every line refuses K > 1 for now, because Route A's and the pool's duplicate-copy checks don't yet recognize location clones (follow-up below).The ACS line draws blocks for its ACS households only. Its donors come from a base-h5 file and keep that file's blocks, so staging reads the rule the base recorded in its root attributes and exits before loading the base if it finds any rule but
block_v1. The pool also refuses a donor without a block.docs/us-location-v1.mdhas the full write-up.Why both confirmations for GTCO = 0
Max's 16 August refinement on #696 says a record without a county code does not live in an identified county. Census's list and the files disagree in the same way in every year we checked.
census_cps_2022/2023/2024.h5match the rawhhpub25.csv, and we computed the table's ratios on those files:A data-derived set would rule out Tarrant's uncoded 85%. The list alone would rule out Lubbock, whose households carry GTCO = 0. Requiring both excludes 271 counties in each of ASEC 2023–2025.
The ASEC 2026 documentation drops the "entire county is identified" sentence. Its file also has 49 Delaware households with GTCO = 0, although List 4 names every Delaware county and the file codes each one for its whole population. So for that year GTCO = 0 draws from the whole state. The repo packages the official lists for 2023–2026 with each PDF's SHA-256, and the loader verifies the CSV against its provenance.
Evidence
pumaarray. Schema 1 keeps its arrays, so the legacy loader reads the new file unchanged. The national rebuild (us_block_ladder_2020_puma.npz, sha9f2dc986…) has 5,769,942 blocks, 2,462 PUMAs, 436 districts, 3,143 counties and 331,449,281 people.7ba39b95…) byte for byte. A rebuild with Give Connecticut blocks their CBSA through 2022 planning regions #1072's Connecticut fix changescbsa_codeon CT's 42,008 blocks and nothing else (09-30 comment).39a2ab2a…) cell for cell (2,462, 4,043 and 4,620 cells). Within a PUMA, the new draw reproduces the legacy district and county distributions and keeps the two consistent with each other.census_cps_2022–2025.h5with the packaged lists and the national ladder:Invariants and their tests
Hypothesis property tests run over nested synthetic ladders (
test_us_block_location.py,test_us_cps_source_geography.py):county_fips_2023, place, SLDs, CBSA, PUMA, every attached plan) equals the ladder lookup of the drawn block, and the household's state never changes.us_block_location_gatepasses, and fails with the column's name when a test tampers with any single column. Households the test draws in five real Connecticut tracts carry the planning region it names for each. Bridgeport and Stamford share Fairfield County but fall in different regions, so a mapping by county fails the test.test_cbsa_agreement_requires_every_block_of_the_county).Differential tests compare:
assemble_us_puma_ladderon the same synthetic sources;Mutation checks. We ran eleven hand-written mutants of
block_location.pyagainst the location tests (verify/mutate.pyin_build_artifacts/us-location-v1). The first run left two alive: the tract mutant and the inverse-CDF mutant. A direct prefix check inus_block_location_gate, synthetic tracts whose codes end in a nonzero digit, and a boundary test for the half-open intervals closed both. The rerun killed all eleven:We reran them on the 09-28 code. The new CBSA test (invariant 8) fails against the first-block rule it replaces.
Effect on certified builds
legacystays the default on every line and leaves certified builds byte-identical.65572aa4…), and the tests check its stage lists, run_config and child argv.spec_engine/seeds.py's kernel lists changes. The legacy-v1 seed-protocol digest matches main's (5840ac6c…), and a new engine-free test checks it against the inventory pin.block_v1.The stacked pool's
block_v1branch never reads the source spine (test_us_spine_blindness): a row with an observed PUMA draws within it, and any other row resolves through its CPS source key.Review
An independent five-dimension review found seven problems, and we fixed all seven:
A second round confirmed all seven fixes and raised six low findings: a stale doc command, stale docs and changelog, two fixes without tests, and invariant 6's wording. We fixed all six.
A Subfleet review of
e778f059cfound one P1 and three P2s, and we fixed all four in33969fdbe:block_v1label (P1). The ACS pool kept each donor's base block. A legacy base draws a district by SOI return counts and then a block within it, so its donor can sit outside the county its CPS file names. The reviewer gave a donor whose CPS record names county 01001 a legacy block in county 01003 (010030001011000), and the gate passed. The pool now takes the base's recorded rule (donor_location_rule), refuses any rule butblock_v1, and refuses a donor without a block where it used to draw one within the state. The staging builder applies the same check before it loads the base.county_fips_2023with the draw's own helper. The new test in invariant 1 writes down each tract's region and checks the draw against it.We ran each new test against the bug it targets: Connecticut keeping its 2020 county, the seed plus the household count, and cloning that splits only the household weight. Each new test fails on its bug, and the row-order test alone misses the seed mutant.
A second Subfleet review of
33969fdbeconfirmed all four fixes and raised three P2s, whichda636f440fixes:--location-rulestill promised a state draw for a donor without a block. It now names theblock_v1base the line requires.Each new test fails on the bug it targets, and its old version passes with that bug in place.
Rebase onto main (2026-11-11 to 11-13)
After the Microcosm hub's sweep kept this PR, we rebased it as one commit five times: onto
ecf87ee24, keeping both sides in ten conflicting files; onto2cfe3043dwith no conflicts; onto27e305d65; onto7b93dae1e; and ontod91415a48(#1582, NZ only), with no conflicts. The third rebase keeps main's district ESS gate (#1088) beside this PR'sblock_v1limitations intools/build_us_acs_local_release.py, so finalize fails when either gate fails, and takes main's rewrite of a donor-identity paragraph from which this PR had only dropped a line number. The fourth keeps #1311's writer binding on both writers: before calibrate writes any byte of the H5, legacy orblock_v1, it checks that the checkout still holds the commit and dirty state it started with. Main already carries the CT CBSA fix (#1072) and the CD registry (#1041, #1043).Beyond the conflicts, the rebase changes eight things:
county_fips_2023. Main's legacy ladders now write the 2023 county-equivalent (Connecticut planning regions,county_vintage).block_v1derives it from the block too, sous_block_location_gate, the stacked pool's clone-coherence check and its H5 loader check it like every other column.cbsa_vintage_agreementused each county's first block (09-30 comment); a pair now agrees only when every block in the county carries that CBSA. The new rule can only add disagreements, so narrowing stays off for the CPS.65572aa4…, the value main's own tool produces. The legacy ACS builder and the release writer have gained stages since September; the location tests stub those stages, and main's own tests cover them.block_location.pyandcps_source_geography.py.--location-rule,--location-seedand--location-clonesfrom Route A's base (PUF_SUPPORT_BASE_LOCATION_FLAGS), those three and--block-ladderfrom the ACS staging builder, and--block-ladderfrom the ACS release. No plan can move a Modal build offlegacy, and the runbook says so.block_v1writer tests pass the checkout's code identity to_write_calibrated_artifact, as main's tests do, and check thatconsumer_export.jsonrecords it. A new test has the checkout report a different commit at write time and checks that theblock_v1writer refuses and leaves no H5.Three decisions bear on this PR:
census_cps_2023–2025.h5, which hold ASEC 2024–2026. The packaged List 4 covers all three, and a test covers ASEC 2026's missing guarantee.block_v1behind its flag, and its branch never reads the source spine (above). A one-population build on that tool gets the rule from the flag.block_v1writescongressional_district_geoidfrom the block ladder's primary plan, so route (a), which switches the ladders' primary plan to the 120th, carries intoblock_v1with no change here. Certified builds stay onlegacy, so the 120th switch need not wait on this PR.Not in this PR
puf_capital_gains_tail.py.--cd-plan-registryflag that attaches its 117th–120th plans in each tool. This PR ships the attach interface.block_v1on for a certified build: that changes how we build the dataset, so it needs a from-scratch build receipt (docs/us-release-build-rule.md) and its own ruling. The ACS line also needs ablock_v1base-h5 build as its donor.axiom: n/a: microsimulation geography assignment, no policy rule changes.
🤖 Generated with Claude Code