Skip to content

Add the US one-population contract library - #1685

Draft
MaxGhenis wants to merge 24 commits into
mainfrom
us-one-population-contract
Draft

MaxGhenis wants to merge 24 commits into
mainfrom
us-one-population-contract

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

PR 2 of d2283 (one US population) adds the population contract as a library with no caller yet. PR 4's prepare step will call it between PR 1's selection (#1664) and the H5 write. The library joins every CPS row of the parent's Route A export with the ACS households PR 1 selects from a staging run. It rewrites both origins' provenance into one assembled form and refuses a joined frame that breaks any clause.

What it adds

packages/microcosm-build/src/microcosm/build/us_runtime/one_population_contract.py:

  • to_assembled_provenance joins the export's CPS rows and staging's ACS rows into one assembled Frame.
    • On CPS rows it keeps Route A's labels: asec on clone 0 and puf_tax_detail on clones 1 and 2. It sets their spine source id to the source id and writes the spine tag asec_puf.
    • On ACS rows it keeps acs_2024_1yr and clone index 0 and sets the spine source id to staging's native id.
    • It writes a clone-0, three-channel spine assembly record, which validate_assembly_provenance accepts.
    • It takes each origin's household weights by key: CPS by household_id, ACS by acs_unit_key(SERIALNO).
    • It refuses positional weights, incomplete support metadata, colliding ids, a repeated (source id, rank) pair, an ACS row whose SERIALNO/SPORDER lineage breaks, a group unit whose persons live in two households, and an ACS household with more than one SPM unit, each with its own message. It also refuses any data but integers, real numbers and strings in ids, clone indices, lineage values and person-to-group links, before it converts or matches them. The links tie each person to its household, SPM unit, tax unit, family and marital unit. For example, it refuses a source_year of 2024+1j or a source_year column of dictionary-encoded Arrow timestamps. It refuses complex and temporal household weights the same way.
    • It checks lineage person by person. It requires staging_lineage, which maps each staged ACS person's native (household id, person id) to its (SERIALNO, SPORDER). It compares the map with the ACS rows before it looks up any weight by SERIALNO, so it refuses two households that swap SERIALNO. require_one_population makes the same comparison in R1 only when the caller passes staging_lineage. Without the map, R1 checks the lineage's form only, and two households that swap SERIALNO pass R1.
  • require_one_population checks a joined frame clause by clause, in a fixed order, and each refusal message states its clause's rule. When a frame breaks two clauses, the contract reports the earlier one, so it refuses a real location clone at R1 before R11 and a repeated ACS SERIALNO at R1 before R2. It refuses:
    • incomplete assembled provenance, including a spine assembly record count that is not a nonnegative integer, an ACS SERIALNO that repeats across households, a group unit that crosses households, and an ACS household with more than one SPM unit (R1);
    • a location-clone column (R11);
    • a CPS key without exactly one native row, or key sets that differ from the selection manifest (R2);
    • a household without a congressional district in its state (R12);
    • an ACS household without TYPEHUGQ 1, 2 or 3 (T1);
    • a column the engine computes at 2024 (R3);
    • a lower-case column that neither the engine nor the register names (R9);
    • a null ESI employer premium (R4);
    • an SPM unit with no classified adult (R5);
    • a null on any row in an engine input outside the 18 in RELEASE_WRITTEN_INPUTS, or any null in one of those 18 other than None or a float NaN, such as pd.NA, pd.NaT or numpy's NaT (R6);
    • a null PUF tail column on any tax unit (R7);
    • a tail donor set unlike the parent's, one donor on two tax units, or a donor on an ACS tax unit (R8);
    • a register column null on some ACS row while some CPS row holds it, unless the owner declares the absence (R10).
  • RELEASE_WRITTEN_INPUTS lists the 18 engine inputs that may stay null on ACS rows. For each, a release tool producer rewrites every row, or stops on a null, before any stage computes from it, and the entry cites the lines that show it. PR 4 fills every other engine input on ACS rows, including eleven producer outputs that engine passes need before their producers write them. The release tool's first engine pass refuses a null at load in the nine float and integer ones, and later passes compute from is_self_employed and takes_up_medicaid_if_eligible. Among the 18, R6 accepts a null only as None or a float NaN, on either origin's rows. That first pass cannot load a pd.NA in the twelve flags it reads before their producers write them, and R6 also refuses a pd.NA in the other six, whose producers run before that pass. R6 refuses pd.NaT, numpy's NaT and any other null too, since this PR does not trace how the release loads them.
  • combined_spm_role_record recounts the joined frame and builds its SPM role record from the CPS rows' record and staging's ACS receipt. It refuses a CPS record that differs from its recount of the CPS rows, and a receipt that names another rule or counts fewer than the selected ACS rows. It also refuses an ACS person whose RELSHIPP is missing, holds anything but an integer, real number or string, falls outside 20 to 38, or gives, under staging's rule, a role other than the one the row holds. The receipt states counts only, so it cannot say which persons hold the role. Nothing ties the row's RELSHIPP to staging, so a swap of both RELSHIPP and the role between two persons passes.
  • absences and write_absences list what each origin lacks, in both directions and by support channel. Each entry has one of six kinds: owner-declared, release-written, raw source column, reviewed default fill, zero written, or origin-native (the six ACS-native acs_* amounts that CPS rows lack). absences records a declaration the frame does not use instead of refusing it, so one mapping passes both R10 and absences.
  • cps_unit_ids derives each CPS household's selection key with PR 1's cps_unit_key, so the manifest's digest and R2 share one rule. The PR renames one_population_selection.py's private _keys_sha256 to the public keys_sha256.
  • A caller can pass the index of the engine's variable classes at 2024. The default index reads the installed policyengine-us metadata on first use, never at import.

The PR also adds a generator of valid populations (test_support/microcosm_build/us_one_population_contract.py) and a changelog fragment, and classifies the module in test_us_spine_blindness.py.

What later PRs owe

The module docstring lists each obligation in full.

  • PR 3 flips six strict xfails in the whole-frame reader differential. Four compare statistics (weeks unemployed under the UC rule, Medicare take-up, retirement contributions, retirement distributions). Two compare verdicts on a frame with one clone-only defect: Route A's gate fails it, and the assembled gate passes it.
  • PR 4 fills every engine input outside RELEASE_WRITTEN_INPUTS on ACS rows, and writes all fourteen raw predictor columns of the ACS predictor join. It passes keyed household weights, the parent's applied tail donor ids, a CPS role record it re-derives on the selected rows, selection keys from cps_unit_ids, and a staging_lineage it builds from staging's output and keys by each ACS person's native ids. It keeps RELSHIPP on every ACS person row, and joins it from the ACS PUMS person source by (SERIALNO, SPORDER) if staging's pooled tables drop it. For health_insurance_premiums and is_self_employed on ACS rows it runs the producer's own pinned rule, as the US data owner decided on 11-19. It calls attribute_us_se_health_premiums_from_manifest with the operation it takes by kind from the packaged stage, after the reviewed default fill (which holds both columns out) and before the contract. It writes ACS rows only, and refuses unless the rule reproduces every CPS row's carried values. It also writes the spine assembly record and the SPM role record into the population manifest, since the H5 writer keeps no frame metadata.
  • PR 5's loader restores both records from the population manifest and checks each on the loaded frame: the spine assembly record with validate_assembly_provenance, and the role record by recomputing it after the load. It also joins the absences file to the register consistency gate and compares the CPS rows with the parent export.

Upstream, R4 refuses every real child until PR 0 (#1670) lands. PR 4 re-derives the CPS role record from raw ASEC role inputs on the export; if the export lacks them, the role record needs a new design.

Tests

Local runs on 8f21661's tree gave these results:

  • test_us_one_population_contract.py: 522 passed, 1 skipped, 6 xfailed (strict).
    • A table of corruptions, most of one cell and at least one per clause, each asserting its own clause's message, plus a test of the clause order.
    • A Hypothesis property over null cells of the checked columns.
    • 27 acceptance cases pin dtype forms the contract accepts for ids, lineage values and RELSHIPP: numpy, nullable and Arrow integers and floats, pandas and Arrow strings, Arrow dictionaries of integers and of strings, categoricals, and object columns of numbers.
    • Tests pin RELEASE_WRITTEN_INPUTS to us/source_stages.json, to the release tool's stage order in its source text, and to the engine versions its evidence names.
    • The whole-frame differential runs eight provenance readers (seven gates and the SSI take-up reporter lineage) on an ASEC-only population before and after the rewrite. All but the four PR 3 owns return the same verdicts and statistics. Two one-cell clone defects flip verdicts, and PR 3 owns those too. The differential leaves out acs_local_hours, which has no release gate and whose donor prep refuses the assembled population.
    • The skip waits for Bind national ACS household types and SOI filing statuses (#880) #1610's household-type materializer.
  • test_us_spine_blindness.py: 505 passed.
  • ruff check and ruff format --check pass on the module, its tests and the generator.

engine_contract/us/test_us_one_population_contract_index.py checks the installed engine's inputs against the lock's 926 names, and the stand-in index's answers against the default index's; CI runs it.

Review

  • In the first round, reviewers read the implementation against the design, and one seeded 68 faults; the tests caught 56. The round's fixes, 9f0019e to 729f0d1, added a test for 11 of the other 12. The twelfth dropped a condition of R10 that lead call L2 later replaced, and the tests catch the same drop in the new rule. Eight lead calls, L1 to L8 in the code, then settled the open items.
  • A second round checked those calls and rated one defect blocking. to_assembled_provenance read household weights by position, while PR 1's ACS weights come in selection order and staging's households in id order. With keyed weights in another order than the tables, the old code gave 75 of 78 households in a generated population another household's weight. 05aeb34 takes weights by key and refuses positional weights and repeated, missing or extra keys. It also moves takes_up_medicaid_if_eligible off RELEASE_WRITTEN_INPUTS, since the ACA stage's engine pass computes from it before the Medicaid stage writes it. In policyengine-us 2.2.1, Medicaid eligibility masks that read, but the contract does not check the mask. At 3f58101, that round's tests reviewer seeded 13 faults in the lead calls' code, two readers and the generator, and the tests caught every one.

Independent review

An independent review of 1161db1 found three blocking issues (B1 to B3), eight should-fix issues (SF1 to SF8) and two nits (N1, N2). 74ca478 answers each:

Finding Fix Tests
B1: a pd.NA in a release-written boolean stops the release's first engine pass R6 refuses a pd.NA in any of the 18 RELEASE_WRITTEN_INPUTS columns; None and NaN still pass test_a_pd_na_in_a_release_written_input_refuses (3 cases), test_a_release_written_null_may_be_none_or_nan (3 cases)
B2: two ACS households that swap SERIALNO trade weights staging_lineage maps each ACS person's native (household id, person id) to its (SERIALNO, SPORDER); P7, the lineage check, compares the frame's pairs with it key by key, before any weight lookup, in to_assembled_provenance, and in R1 when the caller passes staging_lineage test_a_serialno_swap_cannot_trade_weights; corruption-table rows for a SERIALNO swap, a SPORDER swap and malformed lineage entries
B3: an ACS tax unit can span two households a new check, P6c, refuses a tax unit, SPM unit, family or marital unit whose persons live in two households, and an ACS household with more than one SPM unit, in to_assembled_provenance and in R1 corruption-table rows on both paths; test_a_cps_group_unit_across_households_refuses over all four group entities
SF1: the ACS receipt can vouch for swapped roles combined_spm_role_record re-derives each ACS person's role from RELSHIPP by staging's rule and refuses a mismatch, a missing code or an off-domain code role-record refusals "acs roles traded", "relshipp missing", "relshipp off-domain"; test_the_relshipp_codes_are_staging_constants
SF2: a nullable lineage value passes P7 or raises TypeError P7 refuses a null source_year, source_household_id or source_person_id before comparing, parses the year and household id as integers, and requires a string person id three pd.NA corruption-table rows in to_assembled_provenance, one in R1
SF3: R2 drops CPS clone-0 rows that hold digit-string clone indices R2 parses CPS household ids and clone indices as R1 does before the join test_r2_parses_cps_clone_indices_as_r1_does, test_r2_parses_string_household_ids
SF4: an injected index can leave an engine input out of R6's null check the contract refuses an index whose inputs omit a known, non-structural column the engine does not compute at 2024 test_incoherent_answers_are_refused[omitted], test_an_index_that_omits_an_input_cannot_hide_its_null
SF5: a fill receipt can record a negative row count absences refuses a per-spine fill count that is not a nonnegative integer corruption-table row "fill negative count"; test_a_zero_spine_count_files_no_entry
SF6: a nullable numeric column crashes zero-written accounting the zero mask counts a null as not zero in any dtype test_zero_written_on_a_nullable_numeric_column
SF7: the main merge moved the release tool lines the module cites each citation now names its function or statement and gives its line at this branch's release tool none: the stage-order test already pins the eleven producer and pass names in the release tool's source, and a line-number test would fail on each main merge that shifts the release tool's lines
SF8: the handoff to PRs 4 and 5 left out the source call and the record persistence the module docstring states the 11-19 source call for the two columns, PR 4's write of both frame records to the population manifest, and PR 5's restore and check none (docstring)
N1: R1 accepts a fractional assembly record count R1 refuses a native row count that is not a nonnegative integer before the shared validator reads it corruption-table row for a fractional count; test_r1_refuses_a_record_count_the_validator_would_coerce (digit string, integral float, negative)
N2: no clause-order test covers R11 order cases pin R1 before R11 and R11 before R2; the docstring states the precedence; R2's repeated-key branch stays as a guard test_clauses_run_in_their_fixed_order (two new cases), test_a_real_location_clone_refuses_at_r1_first, test_a_repeated_serialno_refuses_at_r1_before_r2

A second independent review confirmed that 74ca478 fixes the 13 earlier findings in the cases they describe, and raised nothing blocking: one should-fix issue and five nits. 26c519f answers each:

Finding Fix Tests
SF1: P7 reads a source_year of 2024+1j as 2024, and the role check reads a complex RELSHIPP as its real part the shared id parser and the RELSHIPP check refuse complex and temporal data, as a dtype or as values in an object column, before converting to numbers, so P7 also refuses a timedelta of 2024 units in source_year, which it read as 2024; the staging_lineage parser refuses a numpy timedelta64, which numpy counts as an integer test_a_complex_or_temporal_source_year_refuses (4 cases), test_integral_ids_refuse_complex_and_temporal_data (15), test_integral_ids_still_read_real_ids (5), four RELSHIPP role-record refusals, corruption-table row "staging_key_timedelta"
Nit 1: R6 lets pd.NaT through, though the module's docstrings and this body allow only None and NaN R6 accepts only None and a float NaN in the 18 columns and refuses every other null on either origin's rows; pd.NA keeps its message test_a_null_other_than_none_or_nan_refuses (5 cases: pd.NaT and numpy's NaT in object columns, and NaT in a datetime64 and a timedelta64 column); acceptance cases for a float32 column, a numpy float32 NaN in an object column and an unmasked NaN in a nullable float column
Nit 2: no test pins the lineage refusal ahead of a missing weight tests only test_the_lineage_refusal_precedes_a_missing_weight
Nit 3: the nesting tests miss an ACS SPM unit merger and a CPS crossing at R1 tests only corruption-table rows for the merger in to_assembled_provenance and R1; test_a_cps_group_unit_across_households_refuses_at_r1 (4 group entities)
Nit 4: the null tests miss a CPS-only pd.NA and an ACS pd.NA beside written zeros tests only test_a_lone_pd_na_on_a_cps_row_refuses; test_zero_written_on_a_nullable_numeric_column adds a pd.NA on a third ACS row, and absences still counts two zeros
Nit 5: this body and 74ca478's commit message leave out R1's condition on staging_lineage wording, in "What it adds", the B2 row above and the new commit's message none

Against a scratch copy of 74ca478's module, 29 of the new refusal cases fail: 22 raise nothing and 7 raise an older message. The nine nit 2 to 4 tests pass there, since 74ca478 already behaves as they require. Against a copy of the new module with five seeded faults, one per behavior they pin, all nine fail, and so does nit 1's CPS NaT case. A copy that lets a datetime64 or timedelta64 column's NaT pass by its dtype fails the two temporal-column cases, and a copy whose object scan reads only the first value fails the source_year case with a numpy timedelta64 on the last ACS person.

A third, scoped review of 26c519f raised nothing blocking either: two should-fix issues and four nits. 8f21661 answers each.

Finding Fix Tests
SF1: an Arrow dictionary of timestamps or durations passes both temporal guards, so P7 reads a source_year of 2024 nanoseconds as 2024, and the role check reads RELSHIPP codes in that encoding the shared guard reads the value type behind a categorical or an Arrow dictionary, accepts integer, unsigned, floating and string types and object columns of real numbers, strings, None and pd.NA, and refuses every other dtype by name; P7, the id parser, the RELSHIPP check, the weights and the person-to-group links use it two Arrow cases in test_a_complex_or_temporal_source_year_refuses (to_assembled_provenance); corruption-table rows "p7_source_year_arrow_timestamps" and "p7_source_year_arrow_durations" (require_one_population); two Arrow RELSHIPP role-record refusals (combined_spm_role_record); test_integral_ids_refuse_any_data_but_integers_reals_and_strings (8 cases); acceptance: test_integral_ids_read_the_pinned_id_dtypes (17), test_a_source_year_in_a_real_dtype_passes (6), test_relshipp_in_a_real_dtype_passes (4)
SF2: R8 takes np.timedelta64 parent donor ids as integers a real-number predicate, and an integer predicate built on it, exclude booleans and numpy's timedelta64 and datetime64; the staging_lineage parser and the checks on the parent donor ids, the role-record, fill-receipt and R1 record counts, T1's TYPEHUGQ and R8's applied flag use them; _keyed_weights refuses complex and temporal weights through the shared guard test_timedelta_parent_donor_ids_refuse; role-record refusals "timedelta count" and "receipt timedelta count"; absence refusal "fill rows timedelta"; corruption-table rows "t1_typehugq_timedelta", "r8_applied_flag_timedelta", "acs_weight_complex", "cps_weight_numpy_complex" and "cps_weight_timedelta"
Nit 1: under pandas 2.x with mode.use_inf_as_na, R6's float path accepts an inf that pandas counts as missing no code change: uv.lock pins pandas 3.0.3, which has no such option none
Nit 2: this body's wording covered more ids than assembly checked: a complex person-to-household link resolved against its integer household id P6b, where the five person-to-group links resolve on both paths, runs each link through the shared guard before it matches them test_a_complex_membership_refuses (to_assembled_provenance) and test_a_complex_membership_refuses_at_r1, 10 cases each: the five links, on an ACS and on a CPS person
Nit 3: no fixture pins R6's pd.NA message ahead of another null tests only test_a_pd_na_is_reported_before_another_null
Nit 4: the staging_lineage parser's docstring says int() reads any timedelta as its count the docstring says "for some units" ("ns" reads, "us" raises) none (docstring)

Against a scratch copy of 26c519f's module, 42 of the 43 new refusal cases fail, and so do the 12 earlier cases whose expected message changed: 39 raise nothing, 14 raise another message, and one crashes on pandas' NotImplementedError for an Arrow string_view. The new case that passes there, a categorical of Arrow durations, pins a refusal 26c519f already made by dtype. The acceptance cases and the nit 3 fixture pass there too. Against a copy of the new module with four seeded faults, each fault fails its tests. R6 reporting another null before pd.NA fails the nit 3 fixture, and a real-number predicate that admits timedelta64 fails ten timedelta cases. A guard that skips the Arrow dictionary unwrap fails the four Arrow dictionary acceptance cases, and one that drops pandas string dtypes fails five string acceptance cases. A copy that guards only ACS persons' links fails the ten CPS link cases.

Merge

Two merges bring in main with no conflicts: 250fd6a (main bcb5cba, with #1664, #1559, #1635, #1519 and #1662) and 1cc4b14 (main cfd5a2a, with #1672, #1677 and #1681). This PR certifies no numbers, since no build calls the contract yet. It stays a draft until #1676 merges, since its engine-bound tests assume policyengine-us 2.2.1 and #1676 moves the engine to 2.145.3.

🤖 Generated with Claude Code

MaxGhenis and others added 24 commits November 17, 2026 13:26
d2283 PR 2 joins the parent Route A export's CPS rows with staging's
selected ACS rows and checks the result. This commit lays the foundation;
the clause bodies follow in later commits:

- one_population_contract.py: the module, its public signatures
  (to_assembled_provenance, require_one_population with keyword-only
  injected inputs, combined_spm_role_record, absences, cps_unit_ids) and
  OnePopulationContractError. Unbuilt bodies raise NotImplementedError.
- EngineVariableIndex: the injected answers to "known", "computed at
  2024" and "input at 2024". The default adapter wraps the installed
  metadata index, built on first use, never at import, and reuses its
  _engine_computed_columns(tables, period=2024), so the contract and the
  release's export guard cannot disagree. A variable whose only formula
  starts after 2024 (weeks_worked) counts as an input at 2024.
- The spine-blindness test classifies the module as a provenance owner.
- keys_sha256: PR 1's key digest made public, so the contract digests
  frame keys by the manifest's rule.
- Test support: a stand-in index built from the engine ABI lock's 926
  committed inputs (count and digest checked against the lock's receipt),
  and a seeded generator of one valid input pair. CPS tables come in the
  export's form, with clones 1 and 2 labelled puf_tax_detail and applied
  tail donors. ACS tables come in staging's form, with ids offset past the
  CPS maximum and SERIALNO/SPORDER lineage.

Tests show the uncorrupted generated population passes the SPM composition
check, the SPM role signal gate (share bands included) with a record built
by the gate's rule, and the assembly validator with a clone-0,
three-channel record. Its stored columns are all engine variables or
registered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staging pools the donor's and the ACS rows over the union of their
columns, so its ACS rows carry every CPS-only column all null, and an
ACS-only integer column (SPORDER, TYPEHUGQ) comes back as float64. The
generator returns only the ACS rows' own columns. with_staging_union_columns
gives the pooled form, so a clause that reads "absent" and "all null"
differently can be tested on both. Tests show the union form still passes
the composition check, the role gate and the assembly validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
to_assembled_provenance now joins the parent export's CPS rows and
staging's ACS rows into one assembled Frame. CPS rows keep Route A's
labels, gain the spine source id and the asec_puf spine tag; ACS rows
take assemble_spines' rule without calling it. It refuses incomplete
support metadata on either origin, a null spine tag, colliding ids or
unresolved memberships, a repeated (source id, rank) pair and a broken
SERIALNO/SPORDER lineage, each with its own message, and writes the
clone-0, three-channel spine assembly record. The household weights
come in per origin and follow the household sort.

cps_unit_ids derives each CPS household's PERIDNUM[:20] key with PR 1's
cps_unit_key, refusing anything but digit strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shuffled household and tax-unit inputs must give the same frame and
id-to-weight map as sorted ones, and a person whose channel disagrees
with its household's must come back as the contract's own error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
require_one_population checks a joined frame clause by clause, in a
fixed order, each refusal naming its own rule: assembled provenance
(R1, with P5's value rules, the spine tag, P6b, P6, P7 lineage and the
clone-0 record), no location-clone column (R11), one native row per
CPS key and one row per ACS household against the selection manifest
(R2), a congressional district on every household (R12), TYPEHUGQ on
every ACS household (T1), no column computed at 2024 (R3), no
unregistered stored column (R9), the ESI premium (R4), a classified
adult per SPM unit (R5), no null engine input outside the release-
written list (R6), the six tail columns (R7), the tail donor set
(R8, ids only) and declared absences for register columns (R10).

RELEASE_WRITTEN_INPUTS holds the 26 exempt engine inputs with the
lines that show each producer rewrites every row or raises on a null.
takes_up_snap_if_eligible is left out: SNAP take-up passes ACS nulls
to a gate that reads them through to_numpy(dtype=bool).

Tests: a table of single-cell corruptions (a few add or drop a
column) with at least one per clause, the clause order, a Hypothesis
property over null cells of the checked columns, the exempt list
pinned to source_stages.json, and a weeks_worked-like column refused
under R6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
combined_spm_role_record checks the CPS rows' role record against a recount
of the frame (the role gate's counts and digest), requires the role from its
source column, checks staging's ACS receipt by keys, rule and coverage, and
returns the gate's keys over the whole frame with per-channel and per-origin
counts and both inputs cited. absences lists what each origin lacks, both
directions, by support channel, with one of five kinds (owner-declared,
release-written, raw source column with its release readers, reviewed
default fill from the fill manifest, zero written from a receipt);
write_absences writes it as deterministic JSON. Tests cover the real role
gate on the generated population, one refusal per check, N-2's kinds with
an injected register (X2), and X7 skipped until #1610's materializer lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
absences now collects every model-named column one origin lacks without a
kind and refuses once, naming each with its rows and direction, so a first
live run lists them all. A test accepts a CPS role record derived on the CPS
tables before the join (seeds 0-2, both forms). The module docstring now
says which rule moves each of PR 3's four readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
X5: on an ASEC-only population built by the production clone and tail
operators, each provenance-reading gate (alimony, Medicare take-up, both
retirement families, SSI take-up reporters, weeks unemployed, WIC,
workers' compensation) returns the same verdict and statistics before
and after to_assembled_provenance with no ACS row. The four readers
Route A's clone label moves are strict expected failures naming PR 3,
each with a companion check that only its named fields move;
acs_local_hours is left out, its donor prep refusing the assembled
population.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pytest reports a fixture's AssertionError as an expected failure of a
test marked xfail(raises=AssertionError), so the X5 fixture re-raises
the builder's self-check failures as a RuntimeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(a) The six ACS-native acs_* amounts (operator_boundary's
_ACS_NATIVE_INPUT_CONTRACTS) that CPS rows lack get a sixth absence kind,
origin-native, with no declaration. The names are a module constant
pinned to that contract by reading its source. Any other column one
origin lacks still refuses, in both directions.

(b) absences records stale and needless declarations as
unused_declarations and needless_declarations instead of refusing them,
so one mapping passes require_one_population and absences. R10 is
unchanged.

(c) Tests for the role record's missing age and channel refusals, the
absences channel refusals, each public function's TypeErrors and the
US-schema refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R8 refuses a parent donor applied on more than one tax unit. P7 checks
staging's lineage on a frame with no ACS row. The role record checks
staging's receipt rule and partition by value, from pinned copies of the
acs_inputs constants. R1 and R2 messages name their clause. Tests cover
R2's digest and orphaned clone key, T1's missing and boolean values, R8's
missing-only and non-flag cases, non-finite household weights, and a
column null on every row. An engine-contract test pins the stand-in index
to the installed engine's. Docstrings record PR 4's lineage and fill
obligations and the two upstream conditions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine-contract test also checks that the installed metadata index's
input list is the lock's 926 names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A column both declared and named by the fill manifest or the zero-written
receipts no longer refuses: the receipt files its entry and the
declaration goes to needless_declarations, so the mapping R10 accepts
passes absences too. A column both receipts name still refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
L1: ten producer outputs leave RELEASE_WRITTEN_INPUTS, because a release
engine pass built from the frame loads each before its producer writes it
(policyengine-core refuses a null float or integer input at load, and the
ACA, Medicaid and SNAP-state passes compute from health_insurance_premiums
and is_self_employed). R6 now refuses their nulls on ACS rows. Each of the
19 remaining entries states its earlier readers, and the obligations name
PR 4's fill source for each moved column.

L2: R10 and absences share one gap predicate (null on some row of one
origin while the other origin holds a value), so one declaration answers
both and a CPS null no longer switches R10 off.

L4: R8 refuses a tail donor applied on an ACS tax unit.

L3, L5, L6: the obligations add the ACS weight reorder, the id-dtype
tightening and PR 5's continuity anchor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lead call L7: the reader generator gains READER_CLONE_DEFECTS, two
one-cell defects on one clone row (a Medicare clone's MCARE that disagrees
with its take-up flag, and a clone with weeks unemployed and no
unemployment compensation). Each fails Route A's gate and passes the
assembled one, so X5 holds two more strict xfails, on verdicts, naming PR
3, which now owns six. The builder checks that the defect's reader fails
with exactly its one failure and every other reader passes; a test outside
the xfails checks that each defect frame differs in one clone cell, and a
companion checks that only the verdict and the named fields move.

The changelog fragment names R8's ACS refusal and R10's shared gap rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The companion to the two L7 verdict flips now also requires the assembled
gate's failures to be at most the defect's one failure, which holds before
and after PR 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1664 (the selection library this branch stacks on, merged as
4d287f4), #1559 (Route A default pool) and #1635 (the US gate battery
in shadow). Git merged every file cleanly, the spine-blindness
classification lists included: this branch adds one_population_contract.py
to the provenance owners and the other-runtime list, #1635 adds
battery_bindings.py, gate_scopes.py and gate_shadow.py to the other-runtime
list and re-pins the pool tool's graph at 86 modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1635 wrapped the release tool's gate calls in the gate battery's
shadow, which moved its lines by 21 to 495 between 4d287f4 and
bcb5cba. No stage moved and no engine pass was added: shadow_gate runs
today's gate in place and the battery's check on the same frame, so
lead call L1's verdicts stand. The release-written evidence and the
module's other citations now point at the same statements on the merged
tree: the SSI December pass (15416), the SNAP take-up gate, now inside
_shadow_frame_gate (14819), the ORG, ACA, Medicaid, SNAP-state, other
health insurance and SIPP vehicle stages, the SSI pass docstring, the ACA
output write and pass, and the CD-vintage preflight. The assembly
validator's callers in multispine_pool.py and the pool tool moved with
#1635's shadow imports (8 and 34 lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- to_assembled_provenance takes each origin's household weights keyed:
  CPS by household_id, ACS by acs_unit_key(SERIALNO), as a Series or a
  mapping, and refuses weights given by position, a repeated key, a
  household without a weight and a weight without a household. PR 1's
  ACS starting weights come in selection order and staging's households
  in id order; a keyed Series read by position gave 75 of 78 households
  another household's weight.
- takes_up_medicaid_if_eligible leaves RELEASE_WRITTEN_INPUTS (18 left):
  the ACA pass computes medicaid_enrolled from it before Medicaid writes
  it, by two routes, both masked today but unchecked, so it is filled
  like the other moved inputs.
- New tests pin what the exempt list rests on: the release tool's stage
  order and its frame-built engine passes through other health
  insurance (read from source text, no import), and the engine pins
  (policyengine-us 2.2.1, policyengine-core 3.32.5). Each failure says to
  re-run L1's earlier-reader trace.
- The moved inputs are checked on whole generated populations: filled
  on every row they pass and add no absence; null on ACS rows, R6 and
  absences both name each one.
- PR 4 obligations name all fourteen predictor-join raw columns, the
  owner and a candidate source for health_insurance_premiums and
  is_self_employed (the choice is still the lead's), and the keyed
  weights. R10's message says "held on some CPS row". The ACS-native
  copy's comment says it stays by lead call L8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings #1672, #1677 and #1681. None of them touches a file this branch changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The module docstring and two test comments gave "an engine pass loads
each before its producer writes it" as the reason lead call L1 moved ten
producer outputs off RELEASE_WRITTEN_INPUTS. Every pass built from the
frame loads every engine variable, so that reason also fits the twelve
booleans that stay on the list. The real split, as the list's own comment
says: the release's first engine pass refuses a null in the nine float
and integer ones at load, and later passes compute from is_self_employed
before its producer writes it. Comments only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The independent review of 1161db1 found three blocking issues, eight
should-fix issues and two nits. This commit answers each:

Blocking
- B1: R6 refuses a pd.NA in any RELEASE_WRITTEN_INPUTS column present,
  on any row; None and NaN still pass. The release's first engine pass
  cannot cast a pd.NA in the twelve flags it loads before their
  producers write them, and R6 also refuses one in the other six.
- B2: staging_lineage maps each ACS person's native (household id,
  person id) to its (SERIALNO, SPORDER), in place of lead call L3's set
  of pairs. P7, the lineage check, compares it key by key before any
  weight lookup, in to_assembled_provenance and in R1, so two
  households that swap SERIALNO no longer trade weights. Passing a set
  raises TypeError.
- B3: a new check, P6c, refuses a tax unit, SPM unit, family or marital
  unit whose persons live in two households, and an ACS household with
  more than one SPM unit. It runs after the assembly validator in
  to_assembled_provenance and in R1. The generator gives each ACS
  household one SPM unit.

Should-fix
- SF1: combined_spm_role_record re-derives each ACS person's role from
  RELSHIPP by staging's rule and refuses a missing or off-domain code
  or a mismatch. The generator carries RELSHIPP on ACS persons. The
  module docstring tells PR 4 to keep RELSHIPP, joining it from the ACS
  PUMS person source by (SERIALNO, SPORDER) if staging drops it, and
  says what the check misses: a swap of both RELSHIPP and the role
  passes.
- SF2: P7 refuses a null source_year, source_household_id or
  source_person_id before comparing, parses the year and household id
  as integers, and requires a string person id.
- SF3: R2 parses CPS household ids and clone indices as R1 does before
  the join.
- SF4: the contract refuses a variable index whose inputs omit a known,
  non-structural column the engine does not compute at 2024.
- SF5: absences refuses a per-spine fill count that is not a
  nonnegative integer.
- SF6: zero-written accounting builds its mask with
  to_numpy(dtype=bool, na_value=False), so a null in a nullable numeric
  column counts as not zero.
- SF7: each release tool citation names its function or statement and
  its line at this branch's tool. No line-number test: the stage-order
  test pins the producer and pass names.
- SF8: the module docstring states the 11-19 source call for
  health_insurance_premiums and is_self_employed, PR 4's write of both
  frame records to the population manifest, and PR 5's restore and
  check of both.

Nits
- N1: R1 refuses a spine assembly record count that is not a
  nonnegative integer before validate_assembly_provenance reads it
  through int(). The shared validator still reads counts through int()
  for its other callers.
- N2: clause-order cases pin R1 before R11 and R11 before R2. New
  tests show the contract refuses a real clone_frame_for_location
  frame at R1, and a repeated ACS SERIALNO at R1 before R2. The
  docstring states the precedence, and R2's repeated-key branch stays
  as a guard.

The changelog fragment adds the refusals from B1 to B3, SF1, SF5 and
N1. On this tree, local harness runs gave 405 passed, 1 skipped and
6 xfailed for test_us_one_population_contract.py, and 505 passed for
test_us_spine_blindness.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second independent review confirmed that 74ca478 fixes the 13
earlier findings in the cases they describe, and raised one should-fix
issue and five nits. This commit answers each:

- SF1: pd.to_numeric keeps a complex value complex, and the float64
  conversion after it drops the imaginary part, so P7 read a
  source_year of 2024+1j as 2024 and the role check read a complex
  RELSHIPP as its real part. pd.to_numeric also reads a timedelta64 as
  its count of units. The shared id parser and the RELSHIPP check now
  refuse complex and temporal data (a complex, datetime, timedelta or
  period dtype, a categorical of one, or such a value anywhere in an
  object column) before any conversion. The staging_lineage parser
  refuses a numpy timedelta64, which numpy registers as an integer.
- Nit 1: R6 accepts a null in a RELEASE_WRITTEN_INPUTS column only as
  None or a float NaN (Python or numpy), on either origin's rows. A
  numpy float column passes by its dtype. pd.NA keeps its own message;
  pd.NaT, numpy's NaT and any other null refuse with a new one, in an
  object column or in a datetime64 or timedelta64 column.
- Nit 2: a test pins P7's moved-pair message ahead of a missing weight
  when the two corruptions come together.
- Nit 3: new tests cover an ACS SPM unit merger, which only the forward
  nesting check sees, on both paths, and a CPS crossing at R1.
- Nit 4: a new test puts a lone pd.NA on a CPS row at R6, and the
  zero-written test adds a pd.NA on a third ACS row; absences still
  counts two zeros.
- Nit 5: to_assembled_provenance requires staging_lineage and compares
  it with each ACS person's lineage before it looks up any weight.
  require_one_population compares it in R1 only when the caller passes
  staging_lineage; without the map, R1 checks the lineage's form only,
  and two households that swap SERIALNO pass R1. No code change.

The changelog fragment adds the SF1 and nit 1 refusals. On this tree,
local harness runs gave 451 passed, 1 skipped and 6 xfailed for
test_us_one_population_contract.py, and 505 passed for
test_us_spine_blindness.py.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The third, scoped review of 26c519f raised nothing blocking: two
should-fix issues and four nits. This commit answers each:

- SF1: an Arrow dictionary of timestamps or durations has dtype kind O
  but fails is_object_dtype, so the shared guard skipped it, and
  pd.to_numeric read a source_year or RELSHIPP in that encoding as its
  count. The guard now reads the value type behind a categorical and
  behind an Arrow dictionary, accepts integer, unsigned, floating and
  string types and object columns that hold only real numbers, strings,
  None and pd.NA, and refuses every other dtype by name. P7, the id
  parser, the RELSHIPP check, the household weights and the
  person-to-group links use it. 27 acceptance cases pin dtype forms
  the guard reads: numpy, nullable and Arrow integers and floats,
  pandas and Arrow strings, Arrow dictionaries of integers and of
  strings, categoricals, and object columns of numbers.
- SF2: numpy registers timedelta64 as an integer, so R8 accepted
  np.timedelta64 parent donor ids and read each as its count. One
  real-number predicate, and an integer predicate built on it, exclude
  booleans, timedelta64 and datetime64. The staging_lineage parser and
  the checks on the parent donor ids, the role-record, fill-receipt
  and R1 record counts, T1's TYPEHUGQ and R8's applied flag use them.
  _keyed_weights refuses complex and temporal weights through the
  shared guard. It now also refuses a bool weight Series and a mapping
  of 0-d numpy float arrays, which 26c519f accepted; no build calls
  the contract yet.
- Nit 1: no code change. uv.lock pins pandas 3.0.3, which has no
  mode.use_inf_as_na option.
- Nit 2: P6b resolves the five person-to-group links on both paths
  before any other clause reads them. It now runs each link through the
  shared guard before isin matches it, so a complex 5+0j on an ACS or
  a CPS person no longer matches the id 5.
- Nit 3: a test puts pd.NaT on an ACS row of one release-written input
  and pd.NA on a CPS row of another, and pins the pd.NA message.
- Nit 4: the docstrings of the staging_lineage parser and of
  _is_real_number say int() reads a timedelta as its count for some
  units.

The changelog fragment names the new refusals. On this tree, local
harness runs gave 522 passed, 1 skipped and 6 xfailed for
test_us_one_population_contract.py, and 505 passed for
test_us_spine_blindness.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 21, 2026
PR 2 (#1685, branch us-one-population-contract) now ends at
8f21661, one commit on 26c519f that answers its third review. It
changes three files: one_population_contract.py, its engine-free test
file and the changelog fragment.

In one_population_contract.py the shared dtype guard now accepts only
integer, real-number and string data. It reads the value type behind a
categorical or an Arrow dictionary, so an Arrow dictionary of times
refuses. A real-number predicate, and an integer predicate on top of
it, exclude booleans, timedelta64 and datetime64; _is_count, the
staging lineage parser and the checks on donor ids, TYPEHUGQ and the
tail's applied flag use them. _keyed_weights refuses complex and
temporal weights, and the five person-to-group links pass the same
guard before isin matches them.

Git merged the two heads with no conflict, and nothing needed a hand
fix. Both branches edit test_us_one_population_contract.py: PR 3 at
its imports and in the whole-frame differential section, PR 2 in its
docstring, its imports and tests above that section. No hunk sits on
both sides, and the merged file keeps every line each side added or
removed.

On the merged tree the engine-free harness gave 525 passed, 1 skipped
and 4 xfailed for test_us_one_population_contract.py (PR 3's 454
passed plus PR 2's 71 new cases). It gave 26 passed for
test_us_pr3_route_a_differential.py, 21 for test_us_medicare_take_up.py,
9 for test_us_retirement_contributions.py and 505 for
test_us_spine_blindness.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 21, 2026
The weeks-unemployed and both retirement gate commits (a682ea7,
ab5d9ec, 1348434) change three modules in the seed protocol's
source inventory, so four digests move:

- seed protocol 05fb4052 -> 24e52e42: EXPECTED_HASHES["seed_protocol"],
  and the F0 report's protocol_sha256 and implementation_sha256, twice each
- seed map afbe93eb -> d79f736a: EXPECTED_HASHES["seed_map"], and the F0
  report's map_sha256 twice
- US spec_sha256 f5eb16a0 -> 8c3ab04a: the F0 report twice, and the pool
  tool test's pin
- the loader's minimal-bundle golden 30d83d44 -> 176e72e2

A ledger sentence above EXPECTED_HASHES and a comment line above the
loader golden name the change. Every changed hunk in the three modules
sits in a gate or summary function, so no draw, RNG rule or QRF target
changes.

The values were read on this branch with main f156478 merged,
through the engine-free harness: on unedited main the same reads give
main's four pins. CI checks all twelve sites.

This is an interim commit. It is reverted before each merge of main or
of a new #1685 head, and the final pin commit reads the values again
at landing, on the main PR 3 merges from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 21, 2026
The weeks-unemployed and both retirement gate commits (a682ea7,
ab5d9ec, 1348434) change three modules in the seed protocol's
source inventory, so four digests move, at twelve sites:

- seed protocol 05fb4052 -> 24e52e42: EXPECTED_HASHES["seed_protocol"]
  in spec_engine/inventory_coverage.py, and protocol_sha256 and
  implementation_sha256 in the F0 coverage report
  (docs/evidence/spec-engine/us-f0-coverage.json), twice each
- seed map afbe93eb -> d79f736a: EXPECTED_HASHES["seed_map"], and
  map_sha256 in the F0 coverage report, twice
- US spec_sha256 f5eb16a0 -> 8c3ab04a: the F0 coverage report, twice,
  and the pin in tests/engine_workflow/us/test_us_multispine_pool_tool.py
- the minimal-bundle spec_sha256 30d83d44 -> 176e72e2: the golden in
  tests/engine_free/shared/test_spec_engine_loader.py

A ledger entry above EXPECTED_HASHES and a comment above each of the two
test pins name the change. Each changed hunk in the three modules sits
in a gate or summary function, in a mask or scope helper that only
those call, or in a module constant that only those read. None is a
draw site, so no draw, RNG rule or QRF target changes.

The engine-free harness gave these values on this branch with main
f156478 merged. On unedited main the same reads return main's four
pinned values, so the harness's package versions do not shift these
digests. CI checks all twelve sites.

The plan reverts this interim commit before each merge of main or of
a new #1685 head. At landing, a fresh read on the main that PR 3
merges from gives the final pin commit its values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant