Skip to content

County Files PoC: donor patch, R1 allowance, R2 pre-checks and fixes - #1660

Draft
juaristi22 wants to merge 26 commits into
mainfrom
claude/us-acs-poc-donor-patch-dd6a9d
Draft

juaristi22 wants to merge 26 commits into
mainfrom
claude/us-acs-poc-donor-patch-dd6a9d

Conversation

@juaristi22

@juaristi22 juaristi22 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #922. Plan: #1647. Gate report from Max's agent: #1647 comment 6937139936. Amended ruling 1: #1647 comment 6945072508.

Temporary: proof of concept only

Tracked in #1661: remove before the final County Files build.

Important

Remove the always-on drop flag before the final County Files build. Passing --county-drop-empty-soi-rows on every run is for tonight's proof-of-concept run only, so that one empty AGI row cannot stop R2 in a window with no room for a rerun (María's ruling, #1647 comment 6950739270). It is not part of the County Files product. In the final build, every county must be supported well: a county whose AGI row would be empty needs a real fix, for example more household support (minimum support that also covers counties whose SOI rows would be empty) or a declared cell with its own ruling, not a dropped target.

Also for the proof of concept only:

  • The patched donor (tools/patch_us_acs_donor_columns.py): the zero strike_benefits column, the zero partnership/S-corp split (passive_partnership_s_corp_income added as 0, s_corp_income all 0), and R1's two --allow-default-donor-column allowances. The final donor is the new national build from main, which carries the real split and the work columns.
  • Files only, with no donor manifest: R1 runs without --donor-release-manifest, so the release records donor_release: null and is not published.
  • The drop flag's report marks (e49d53997): the run identity's zero_measure_soi_rows_dropped key, and the reliability report's cell key and CSV column of that name. Only a flagged run has them, so they go with the flag.

Not temporary: keep these.

  • The strike_benefits reviewed fill on ACS rows (commit 3, an entry in acs_local_reviewed_fill_consumers.yaml). It is general and correct for any current donor: nothing transfers strike_benefits to ACS rows, so every ACS person reaches R2 with a null, and R2 fills the engine default, 0. Without the entry, R2 refuses at finalize for a staging built from any current national donor. Do not remove it with the proof-of-concept setup.
  • The two pre-checks (commits 5 and 7, with 3b28e315c, a5ca4df65, 28e7b7ce6 and f8b1039f0). In the final build, the zero-AGI check lists the counties that need a real fix before R2 runs.

The help text of both flags, R1's module docstring, the agent guide, the county rung doc and both changelog fragments carry the same note, with "removal tracked in microcosm#1661" (f96ff8e1b and 65b2155c0, help text and docs only). This PR does not finish #1661.

Purpose

R1, the ACS staging (tools/_legacy/build_us_acs_multispine_base.py), refuses 766b7c7's national donor populace-us-2024-spm-receipts-20260923 (sha256 00946972…) at three donor gates. Under the amended ruling, the County Files proof of concept runs R1 on a patched copy of that donor. This PR adds the patch tool, one explicit R1 option, the reviewed-fill register entry that R2 needs for the copy, an optional release manifest for the copy, a pre-R2 check for counties with an empty SOI AGI row, and an opt-in R2 path that drops such a row with a receipt.

Decisions confirmed by María (17 November, #1647 comment 6946488444):

  • The strike_benefits fill verdict stays harmless (commit 3).

  • Tonight's proof of concept is files only: R1 runs without --donor-release-manifest, R2 records donor_release: null, and the release is not published. The manifest option (commit 4) stays opt-in and is not used tonight.

  • The drop flag is on for every run tonight (#1647 comment 6950739270). R2's materialize always runs with --county-drop-empty-soi-rows (commit 6). This replaces the earlier pre-authorization (#1647 comment 6947604927), which passed the flag only when the zero-AGI check listed a county, and it answers Max's question in review comment 6949865318.

    • The flag drops only rows that are really empty, and records each one in county_surface.json (zero_measure_soi_rows_dropped), the run identity and the reliability report. A county with no households is still refused.
    • The zero-AGI check (commit 5) still runs after R1. Post its list on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647, together with any drop receipt from R2. The PoC report names every dropped row.
  • The published donor is read only. It does not change.

  • The patch is for the proof of concept only. The real County Files donor is a new national build from main, located with block_v1.

  • R1 and R2 can run from this PR's head tonight. Do not merge before review.

Commits

1. 44a11af9d: donor-patch tool, tools/patch_us_acs_donor_columns.py

  • It reads the donor in the layout that R1's _load_base_frame reads. It starts from a byte copy of the input and rewrites only the entity tables that gain columns, through the shared frame-table codec (read_frame_table / put_frame_table), in the stored format. A table-format table keeps its data columns. The published donor's person table is a pandas table-format compound table.
  • It keeps every other table, column, dtype and HDF5 root attribute. It verifies the copy against the input before the copy replaces the output path.
  • It adds strike_benefits and passive_partnership_s_corp_income as 0 on the entity that the US engine gives them (person). It refuses a column whose engine default is not 0.
  • It refuses a column that is already present with any other value (nonzero or null). It keeps an existing all-zero column.
  • It refuses an input on the sealed pool deny-list (refuse_denied_pool_h5).
  • It writes a JSON receipt (default <output>.receipt.json): input and output SHA-256, rows per entity, each added column with its entity, dtype and reason, and the tool's commit.
  • The tool's HDF writer joins the frame-serializer registry, with a round-trip adapter. The registry now has 11 serializers.

2. 601cf86cc: R1 option --allow-default-donor-column COLUMN=REASON (repeatable)

  • _require_dense_donor_coverage skips only the "every observed donor value equals the engine default" failure for a listed column.
  • It still refuses a listed column that is absent or has no observed value, a name that is not transfer-consumed, a name that the engine has no default for, and a stale allowance (the column has donor signal).
  • The staging summary records each allowance with its reason: allowed_default_donor_columns (entity, engine default, reason), orchestration.allow_default_donor_columns, and the reviewed limitation allowed_default_valued_donor_columns. R2's finalize copies the staging limitations into the release's reviewed_limitations.
  • tools/modal_us_stage_plan.py owns the flag and never passes it, as for the location flags (US household location v1: one census block per household, every geography derived #1047). The runbook says so, and the agent guide names this PoC as the one reviewed exception to "rebuild historical donors".

3. 77f5fa163: reviewed-fill register entry for person.strike_benefits

  • The local income pass's OIP predictor uses strike_benefits as a component (since d5aa4e2), so R1 requires every donor to carry it. Nothing transfers it to ACS rows, so every ACS person reaches R2 with a null value. R2 fills the engine default, 0, and finalize's reviewed-fill consumer gate then refuses the fill as unregistered (package refuses it too). Without this entry, R2 would fail at finalize, after materialize, calibrate and qa.
  • This is not specific to the proof of concept: a staging built from any current national donor hits the same refusal.
  • The entry names the 13 means-tested consumers that the engine-tier test recomputes from policyengine-us 2.2.1, with the verdict harmless. The verdict is a judgment for María to confirm. The reason: no pinned ASEC respondent reports code 12, so every donor value is 0, and a transfer from the donor would also draw 0. The alternative is known_bias / overstates; the coupling rules would then also apply to SNAP (via TANF) and to the PTC (via Medicaid and CHIP).
  • The register has no hash pin. Its run-time SHA-256 moves from 36601cb3e6ff… to 692564a40b30…. R2 binds the code commit (calibrate records it, package refuses another commit or a dirty tree), so run R2 from a head that includes this commit.
  • The engine-free test that pins the register's column set now lists the fills added since release 767312d.

4. b9e7f4010: optional release manifest for the copy (the ruling's "manifest for the copy")

  • --release-manifest PATH --manifest-release RELEASE --manifest-build BUILD (and --manifest-repo-id, default policyengine/populace-us-staging) writes a release manifest in the build-folder layout. Its one microdata artifact is the copy; the receipt is a second artifact.
  • R1's --donor-release-manifest then verifies the copy's SHA-256 against it and records the copy's build as donor_release. referenced_build and the local-area release contract's donor check accept that record (tested), so the publisher no longer refuses the R2 release for a null donor_release.
  • --input-release-manifest checks the input against the published donor's release manifest and records that build in the receipt and in the copy's manifest (derived_from).
  • The copy's dataset_role is proof_of_concept_donor_copy, which is not a release role, so the copy's folder cannot pass as a release itself. The manifest names files, not local paths.
  • Both options are opt-in. Without them, the copy and the receipt are the same as before.

5. 06b53f625: pre-R2 check for counties with an empty SOI AGI row, tools/check_us_acs_county_agi_support.py (#1630 review, finding 1)

  • R2's attach_surface builds each county's SOI rows from the state rows' stored entries, and the state AGI row stores no zero. A county whose households all have AGI of exactly 0 gets an empty AGI row, and R2 refuses after the materialize engine pass (about 1.5 h in). Minimum support fills only counties with no household.
  • Since 3b28e315c (item 9) the check computes the AGI that R2 stores, with the engine. The first version read only the AGI income columns and counted any person's nonzero value, so it missed income that the engine does not count.
  • With --puma-ladder it takes the county list from the ladder and applies R2's own minimum-support moves first. It exits 2 when a county is at risk and writes a JSON report.

6. 78acba5f5: R2 option --county-drop-empty-soi-rows (off by default)

  • With it, R2 drops an empty county SOI row whose county has households, instead of refusing. A county without households, and an empty ACS households row, are still refused.
  • Receipts: county_surface.json (zero_measure_soi_rows_dropped: row, county cell, concept, target value, households), the run identity (county_targets_dropped, county_drop_empty_soi_rows), the reliability report and the refresh recipe. Without the flag nothing changes, the identity bytes included.
  • Passing it is a ruling for María. She ruled it on for every run tonight (#1647 comment 6950739270). The other remedy, a declared SOI cell like Kalawao's (DECLARED_SOI_COMBINED_CELLS), covers only a county absent from the SOI facts; a county present in them would need new code as well as a ruling.

7. 46674e01f: post-R1 check of R2's reviewed-fill rules, tools/check_us_acs_local_reviewed_fills.py

  • R2's finalize gate refuses an unregistered fill, a different fill value or spine, or another policyengine-us, hours into the run.
  • The check reads R1's staging summary (reviewed_engine_input_nulls, names and counts only). It refuses what materialize refuses: a never-default-filled null off the ACS spine, and a null ESI premium. It builds the fill records R2 writes, with the defaults R2 resolves, and runs R2's own finalize gate on them, in seconds. It exits 2 when R2 would refuse.

8. 3bffc4aed: CI fix for commit 2

  • The ACS staging telemetry test builds R1's arguments by hand, so it had no value for --allow-default-donor-column. It now carries the parser's default (None), as it does for the location defaults. Test-only change. 2737c32fb then merges main (Belgium and Route A changes only; no file in this PR).

9. 3b28e315c: the zero-AGI check computes the AGI that R2 stores (review fix, comment 6949865318)

  • policyengine-us 2.2.1 adds only the positive income sources of a tax unit's head and spouse to irs_gross_income, and only an adult can be either. Income held by a dependent, or by a tax unit without an adult (one 16-year-old with wages), adds nothing, so such a household has an AGI of 0. The first version counted it, and could exit 0 for a county that R2 then refuses.
  • The check now loads the staging frame and summary as R2's materialize stage does. It holds back the formula-owned columns (project_input_only), fills the reviewed nulls (fill_reviewed_nulls), builds each household chunk's dataset (_dataset_from_frame) and computes the household sum of adjusted_gross_income. An engine-tier test checks that sum against the production SOI materializer's AGI measure on six households: a dependent's wages, a no-adult tax unit, a spouse's wages and a negative AGI included.
  • R2's take-up stages fill some never-default columns before its fill step. The check holds back such a column while it is null, as the AGI does not read it. A null in a never-default column that the AGI reads is refused, as R2 refuses it.
  • --staging-summary defaults to the H5 path with .summary.json, as in R2. The check needs memory for the full staging frame.
  • 72b7072bf merges the work-column head (000532170) into this commit's branch.

10. f96ff8e1b: the drop flag and the donor allowance are marked proof-of-concept only (María's request with the ruling of #1647 comment 6950739270)

  • The help text of --county-drop-empty-soi-rows and --allow-default-donor-column now says: temporary for the County Files proof of concept (microcosm#1647); do not use for the final County Files build. The county rung doc describes the drop flag with the same note, and R1's module docstring, the agent guide and both changelog fragments carry it.
  • Help text and docs only; no logic change. A run from 72b7072bf and a run from f96ff8e1b are the same.

11. 65b2155c0: the notes name the removal issue

12–15: review fixes for comment 6955224433. They change no R1 code, so R1's staging from 72b7072bf stays valid.

12. a5ca4df65: the zero-AGI check exits 3 when R2 would refuse whatever the drop flag

  • R2's check_support refuses a county outside the ladder and a ladder county without a household-universe record, with or without the drop flag, in materialize after the staging load and the SSI/Medicaid pre-pass. With --puma-ladder, the check now lists both (counties_outside_ladder, unsupported_counties) with R2's set rules. As R2 does, it moves no household when a county is outside the ladder.
  • Exit 3 takes precedence over exit 2, and the report records exit_status.

13. e49d53997: each dropped SOI row is named in its county's reliability cell and CSV row

  • With the flag, calibrate grades a county without its dropped row. Materialize now records each dropped row's county cell in the run identity (zero_measure_soi_rows_dropped). The bound reliability report names the row in every member county's cell, and in a zero_measure_soi_rows_dropped column of county_reliability.csv.
  • Only a flagged run has the identity key, the cell key and the column; every other run keeps its bytes. A flagged identity from older code that dropped rows but lacks their cells is refused at calibrate's report step (re-run materialize). With no row dropped, an older identity needs nothing.
  • The flag's test now runs on through calibration_evidence, write_reports and package_reports.

14. 28e7b7ce6: the fill pre-check accepts an ACS-only never-default null only where an R2 stage fills it

  • R2's stages fill only the take-up stage's SPM-unit take-up flags and engine-free fills, and the SSI and Medicaid take-up flags. For the immigration, disability, income-transfer and SSI disability-criteria columns, R2's reason says to re-run staging. An ACS-only null there now exits 2 with that message, not under natively_filled. A test ties the set to R2's own reasons.

15. 2e9bf301d: the patch tool refuses to replace an existing copy, receipt or copy release manifest unless --overwrite is passed

  • A re-run writes another SHA-256, so a staging summary that pinned the first copy would name bytes no longer at its path. With --overwrite, re-run R1 on the new copy.

16. f8b1039f0: the zero-AGI check holds back a null never-default column only where an R2 stage fills it (review fix for comment 6956136956; it changes only step 4a's check)

  • The check held back every null never-default column that the AGI does not read, on either spine, so run alone it passed a file that R2's first fill refuses. It now takes the stage-filled set from the fill pre-check (stage_filled_never_default). It holds back a null column only when an R2 stage fills it, every null is on an ACS row and the AGI does not read it, and refuses every other never-default null.
  • An engine-tier test checks that no stage-filled column is an input of the AGI closure, so an engine release that changes that fails in CI.

17. c990afa56: the first draft of the PoC report for LWI (comment 6987263616)

  • experiments/us-county-poc-20261117/lwi-poc-report.md, draft v0.1 in pe-doc Markdown, with a short README. It states the fit and the limits of the 2024 county file, from the run report on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 and the release's own reports. It changes no code.
  • Still open: the methods link (twice) and the schema documentation, as placeholders. The file is shared through the private Hugging Face dataset policyengine/lwi-county-files.

Ask: the work-column commit from Max's agents (pushed as 36bd0dafa)

Please add your verified work-column step to this tool as the next commit, on top of 2737c32fb:

  • Implement asec_work_column_step in tools/patch_us_acs_donor_columns.py. It now raises NotImplementedError; its docstring gives the contract. It restores detailed_industry_recode, major_industry_recode and worked_last_year from the pinned ASEC person files, matched on source year and PERIDNUM, and returns one ColumnPatch per column with its source in the receipt.
  • Register it in build_patch_steps before the zero step, with its command-line options in _parse_args.
  • Add a test that R1's _require_donor_work_experience passes on the patched fixture (the file has the same check for _require_income_donor).
  • Post the copy's SHA-256 and its receipt in this PR.

Run result (17–18 November)

Every stage passed, at the heads this description names. Package ended at 01:18:36Z on 18 November. Sources: Max's agent, comment 6979361950, and for the zero-AGI check and materialize, #1647 comment 6964605746.

  • Heads. R1 ran at 72b7072bf. The fill pre-check through package ran at f8b1039f0. The tree was clean before every stage, and build_manifest.json records build_dirty false.
  • Wall time and peak memory (the guard's figure): R1 241.6 min, 48.0 GiB; fill pre-check 1.3 min; zero-AGI check 3.6 min, 40.7 GiB; materialize 177.3 min, 59.7 GiB; calibrate 31.1 min, 47.7 GiB; qa 54.2 min, 23.9 GiB; finalize 2.0 min, 29.2 GiB; package 1.8 min, 25.8 GiB. Every stage exited 0.
  • Zero-AGI check: exit 0, with no county at risk and none that R2 refuses. Materialize dropped no SOI row: zero_measure_soi_rows_dropped is empty.
  • Calibrated file: populace_us_2024_acs_local.h5, SHA-256 82314a3b5a6a95762faf6eb4f5b499ce56608ad6a39a27b75e11c75b4945bd3c, 10,637,811,669 bytes. Build id populace-us-2024-buildo-acs-local-f8b1039f00-20261118T011728Z. It is not published.
  • Gates: all 23 gates in gate_summary.json passed.
    • calibration: final loss 0.015904 against an initial 0.020786; 75.7% of 16,440 trained rows within 10%; realized largest weight ratio 4.93 (cap 5).
    • district_ess_collapse is report-only and records 271 of 436 districts collapsed.
    • cd_holdout has 0 targets. This is expected: the holdout fraction is 0 for every --soi-mode except state_cd, and the run used state_county.
  • Counties: county_reliability.json grades 3,144 counties, under a provisional rule: A 0, B 3, C 103, D 3,038, U 0. No target is unsupported. Minimum support made 9 moves, into 15005, 31009 and 48301.
  • Disk: the lowest free disk was 64.64 GB, during qa, so no stage went under the 50 GB stop line.
  • Two hand steps on the run machine:
    • The launcher's shell wait returned 19 at R1's end, but the guard recorded exit 0 and every staging gate passed. So R1's done record was written by hand.
    • The release root copy is an APFS clone of the calibrated file, placed by hand during qa. Package found the same SHA-256 and did not copy.
  • Uploads (request, comment 6973462140): Max gave the go at about 13:05Z. The run folder goes to the private policyengine/populace-us-staging, at runs/lwi-county-poc-20261117/, with the SHA-256 list report/sha256-outputs.txt. The publish command will not be run.
    • Batch 1 is in (commit d8a772ce, 13:28Z, comment 6980005693): 73 files, 1.12 GB. They are the release folder (15 files, with the county reliability JSON and CSV), package_result.json, the 23 checkpoint files, both pre-check reports, the staging and output summaries, the donor receipt and every stage log.
    • Batches 2–4 are in (comment 6982412846): the calibrated file county-poc/populace_us_2024_acs_local.h5 (518c509713b3, 13:58Z), R1's staging county-poc/acs_multispine_staging.h5 (7ba16a15413f, 15:00Z), and the donor copy, the feed and the county facts (159a42f8a619, 66aaed953c40 and c88bfbf16ae4, 15:01–15:03Z). The feed and the county facts are in inputs/.
    • Checked from the repo's listing at c88bfbf16ae4, with no download: the folder holds 78 files, 25.5 GB. The LFS SHA-256 of the calibrated file (82314a3b…), the donor copy (438f47f4…), the feed (1d45a89c…) and the county facts (67b0998a…) equal the values pinned above. R1's staging (11,472,744,092 bytes, 083290dd…) has no pinned value here; Max's agent checked it against the run's SHA-256 list.
    • The release root copy is not uploaded: it has the same bytes as the calibrated file.

Tonight's run: exact commands

Run everything from one frozen head, the final head after the work-column commit, in a fresh worktree, while other work commits elsewhere. R2's calibrate and package must run at the same commit, and the checkout must not change while calibrate runs. uv run --no-sync keeps uv from touching the lock mid-run. Keep at least 60 GB of disk free.

# 0. Setup.
git worktree add <checkout> <final head SHA> && cd <checkout>
uv sync --all-packages --locked --extra us      # policyengine-us 2.2.1 exactly
export RUN=/abs/path/county-poc                  # outside the checkout
export LADDER=/abs/path/us_puma_ladder_2020.npz  # SHA-256 39a2ab2abeab07a88362af7ab2940e0e1d50a297c919e4bbc6fb65bab51147d8; one file for R1, the checks and R2
#   (a copy is in the private policyengine/populace-us-staging dataset at
#   inputs/us-puma-ladder/39a2ab2a/us_puma_ladder_2020.npz, HF commit c6718cc1)
export FEED=/abs/path/consumer_facts_us_41b2fb2.jsonl
export FEED_SHA256=1d45a89c28177a95371b4cc4068d3657c7bf12bd8edad7c82fe25bf453f9ea07            # us/chronicle_feed.json
export COUNTY_FACTS=/abs/path/consumer_facts_us_county_3e9d3a0.jsonl
export COUNTY_FACTS_SHA256=67b0998aa2ee1aeed6637dab6a846f405246706508074b1c1b5bab7bd98c8bea    # us/chronicle_county_facts.json
export PYTHONUNBUFFERED=1

# 1. The patched copy (files only: no manifest options). From 2e9bf301d
#    an existing output is refused; --overwrite replaces it, and R1 must
#    then re-run on the new copy (it has another SHA-256).
uv run --no-sync python tools/patch_us_acs_donor_columns.py \
  --input-h5 <populace_us_2024_receipt_qualified.h5, tag populace-us-2024-spm-receipts-20260923, SHA-256 00946972…> \
  --output-h5 $RUN/donor/donor_patched_poc.h5 \
  --receipt $RUN/donor/donor_patched_poc.receipt.json \
  <the work-column commit's options>

# 2. R1, the ACS staging, legacy location. The two limits replace 30 GB
#    defaults that refuse a full staging after the QRF transfer.
POPULACE_ACS_POOL_PEAK_LIMIT_BYTES=100000000000 \
POPULACE_STAGING_EXPORT_PEAK_LIMIT_BYTES=100000000000 \
uv run --no-sync python tools/build_us_acs_multispine_base.py \
  --base-h5 $RUN/donor/donor_patched_poc.h5 \
  --out-h5 $RUN/acs_multispine_staging.h5 \
  --summary $RUN/acs_multispine_staging.summary.json \
  --inputs-dir <ACS archive cache: csv_hus.zip 8281008e…, csv_pus.zip afdc6d90…> \
  --puma-ladder $LADDER \
  --period 2024 --seed 0 --geography-seed 0 --acs-share 0.5 \
  --n-estimators 32 --max-targets-per-fit 8 --chunksize 100000 \
  --donor-channel auto --location-rule legacy \
  --hours-under15-policy us_hours_under15_zero_completion_v1 \
  --allow-default-donor-column "s_corp_income=the donor holds partnership and S-corp income as one combined amount in partnership_income; the split is unknown" \
  --allow-default-donor-column "passive_partnership_s_corp_income=the donor holds partnership and S-corp income as one combined amount in partnership_income; the split is unknown"
  # optional: --sipp-donor <pinned pu2023.csv>; else the 3.73 GB SIPP file is read from, or fetched to, its cache

# 3. Reviewed-fill pre-check (seconds). Exit 2 = stop. A missing register
#    entry needs María's ruling (committed before calibrate). From 28e7b7ce6,
#    a never-default null that staging should have filled needs a staging fix.
uv run --no-sync python tools/check_us_acs_local_reviewed_fills.py \
  --staging-summary $RUN/acs_multispine_staging.summary.json \
  --out $RUN/reviewed_fills_precheck.json

# 4a. Zero-AGI county check (about 7 minutes and a 12 GB peak on the
#     766b7c7 file; it loads the staging frame as materialize does).
#     Exit 0 = no county listed; exit 2 = it lists counties in "at_risk".
#     On 0 or 2, go on to 4b and post the list on #1647: exit 2 does not
#     stop tonight's run, since materialize always passes the drop flag.
#     Exit 3 (from a5ca4df65) = stop: R2's materialize would refuse whatever
#     the flag (see counties_outside_ladder and unsupported_counties).
uv run --no-sync python tools/check_us_acs_county_agi_support.py \
  --h5 $RUN/acs_multispine_staging.h5 \
  --staging-summary $RUN/acs_multispine_staging.summary.json \
  --county-column county_fips_2023 \
  --puma-ladder $LADDER --county-min-support 3 \
  --out $RUN/county_agi_support.json

# 4b. R2, the 2024 county release, one stage per process (restartable).
R2="uv run --no-sync python tools/build_us_acs_local_release.py"
C="--staging-h5 $RUN/acs_multispine_staging.h5 --staging-summary $RUN/acs_multispine_staging.summary.json \
   --checkpoint-dir $RUN/checkpoints --ladder $LADDER --out-h5 $RUN/populace_us_2024_acs_local.h5 \
   --families snap,medicaid,soi --geographies state,cd,county --soi-mode state_county \
   --county-column county_fips_2023 --county-holdout-fraction 0.1 --county-holdout-salt acs-local-county-v1 \
   --county-min-support 3 --county-target-loss-cap 10 --county-max-weight-ratio 5 --hh-chunk 20000 --seed 0"
M="--feed $FEED --feed-sha256 $FEED_SHA256 \
   --county-facts $COUNTY_FACTS --county-facts-sha256 $COUNTY_FACTS_SHA256 --sample-fraction 1.0 --batch 5000"

# Materialize, always with the drop flag (form B; María's ruling, #1647
# comment 6950739270). The flag is read only at materialize and recorded
# in the run identity; it drops only rows that are really empty.
$R2 --stage materialize $C $M --county-drop-empty-soi-rows

# R2 records the checkout's commit and dirty state, and package refuses a
# dirty tree (an untracked file counts). Check before calibrate and package.
clean() { test -z "$(git status --porcelain)" || { git status --porcelain; echo "dirty tree: commit or remove these first"; return 1; }; }

clean && $R2 --stage calibrate $C --epochs 800 --epoch-batch 400
$R2 --stage qa $C
$R2 --stage finalize $C
clean && $R2 --stage package $C --out $RUN/release
  • --epochs 800, --epoch-batch 400, --batch 5000, --seed 0 and --sample-fraction 1.0 are R2's defaults, named for the record. --geographies, --soi-mode and --county-column are not defaults and must be passed. The county flags match the county rung doc; --stage all with the same flags is its single-command form.
  • --county-drop-empty-soi-rows goes on materialize only, on every run tonight. It drops only rows that are really empty; a county the check lists but whose row has an entry keeps its row. The receipt is county_surface.json → zero_measure_soi_rows_dropped, an empty list when no row is empty. In that case the data do not change, and the run identity, materialize_rss.json and the refresh recipe record the flag.
  • git status --porcelain must print nothing before calibrate and package. R2 counts an untracked file as a dirty tree: calibrate records it without refusing, and package then refuses (comment 6955224433). Keep $RUN and the logs outside the checkout.
  • Not passed unless María rules: --allow-dirty (package, records build_dirty: true), --district-ess-gate-blocking, and any manifest option.
  • Expected peaks (22–23 Sep measurements): R1 about 42 GB and 2–3 h; materialize about 78–95 GB and 1.5–2.5 h; calibrate about 68 GB and 10–20 min; qa, finalize and package about 22–23 GB and minutes each. R2's SSI/Medicaid pre-pass is new since then and unmeasured. If tools/route_a/supervise.py is reused, raise its 72 GiB RSS cap. Newer figure from Max's agent (comment 6950580220): the sparse-era materialize peaked at 48.9 GB RSS on 16 Oct, and tonight's run caps materialize at 112 GiB.
  • Disk (comment 6959825920): R1's staging is about 10.7 GB. After materialize, calibrate writes the release file at --out-h5 (9.8 GB in both earlier full builds), and package copies it to $RUN/release/microcosm_us_2024_acs_local.h5, a second 9.8 GB. Package skips that copy when the path already holds a file with the same SHA-256, such as a hard link made before package (tools/build_us_acs_local_release.py:6788-6793). QA and finalize write only reports. Measured in the run: the calibrated file was 10.6 GB, the lowest free disk was 64.64 GB, and package did not copy because an APFS clone already held the same SHA-256 (comment 6979361950).

Pre-check results (17 November)

  • R1 donor gates, on the real donor. The published file (SHA-256 00946972…) was patched (copy b3c57fbd…, 108 s, 3.3 GB peak), then R1's donor gates ran in R1's order. All pass except _require_donor_work_experience, which names exactly detailed_industry_recode, major_industry_recode and worked_last_year: the work-column commit fixes it. The coverage gate passes with the two allowances and refuses without them, naming only s_corp_income and passive_partnership_s_corp_income.
  • Donor inputs R2 reads. On the patched copy, every engine input is complete. The SSI and Medicaid take-up flags, SNAP and TANF take-up, the four engine-free fill inputs (ABAWD exemption, housing receipt and take-up, Medicare and Head Start take-up) and would_claim_wic are complete Booleans. The ESI premium column is absent, so its refusal cannot trigger.
  • Reviewed fills. No donor-spine null is expected. Every donor engine input that is not a transfer target, a register entry or a never-default column is ACS-native or structural. Run the fill pre-check after R1 for the full answer.
  • Zero-AGI early warning, on the 766b7c7 ACS local file (populace_us_2024_acs_local.h5, SHA-256 3ce781c4…, county_fips basis, pinned ladder), rerun with the engine check of 3b28e315c (438 s, 11.9 GB peak, 16 engine chunks): no county at risk. R2's minimum support would move 9 households into 15005, 31009 and 48301. Ten counties have fewer than three households with a nonzero AGI: 31005, 31103, 31183, 35021, 38087, 48033, 48261, 48263, 48269 and 48311, the same ten as the first version found, with the same counts. The engine's rule lowers the count in 312 other counties, none to zero. That file is R2's output and has no null engine input, so the run built its fill register from the file. Tonight's staging draws ACS counties again, so run the check on it.
  • County basis. With county_fips_2023, the pinned ladder gives 3,144 counties, all with PUMA overlaps, including Connecticut's nine planning regions and Kalawao.
  • Repeat patch runs. Two runs of the patch tool on the same input give different file hashes. On the test fixtures, every differing byte (14 of 9.7 MB) is a 4-byte Unix time in the HDF5 object headers that PyTables writes when the tool rewrites the person table; the data is identical. A run therefore pins its copy by SHA-256, as tonight's does.

What remains for Max's agents

  1. Push the work-column commit to this branch; fetch and merge first. Done: 36bd0dafa.
  2. Freeze a head that includes 3b28e315c (now 72b7072bf) in a fresh worktree. Patch the donor there and post the copy's SHA-256 and receipt here. Done (comment 6950580220): frozen head 72b7072bf; copy 438f47f4… made at 000532170 (receipt b6430abc…). Only the AGI pre-check's four files change from 000532170 to 72b7072bf. f96ff8e1b and 65b2155c0 (items 10 and 11) add help text and docs only, so 72b7072bf, f96ff8e1b and 65b2155c0 all give the same run. Items 12–16 (head f8b1039f0) change the two pre-checks, the patch tool's re-run refusal and R2's reliability marks on a flagged run, and no R1 code. R1's staging from 72b7072bf stays valid; to use the fixes, run steps 3–4b from f8b1039f0, every R2 stage from that one head.
  3. Run R1 with the two memory limits and --hours-under15-policy. Done at 72b7072bf: 241.6 min, exit 0.
  4. Run the fill pre-check. If it exits 2, stop: a missing register entry needs María's ruling, committed before calibrate; a never-default null that staging should have filled needs a staging fix. Done at f8b1039f0: exit 0.
  5. Run the zero-AGI check and post its list on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647. If it exits 3, stop: R2 would refuse in materialize. Otherwise run materialize with --county-drop-empty-soi-rows whether the check exits 0 or 2 (María's ruling, #1647 comment 6950739270), and post the county_surface.json drop receipt on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 with the list. Done (#1647 comment 6964605746): exit 0, and materialize dropped no row.
  6. Run the rest of R2 stage by stage from the same head, with an empty git status --porcelain before calibrate and package. Done (comment 6979361950). Every stage passed.
  7. Upload the run folder, and the county file for inspection, to policyengine/populace-us-staging (comment 6973462140, items 2 and 3). Max gave the go at about 13:05Z. Done: all four batches are in, the last at c88bfbf16ae4 (15:03Z, comment 6982412846). Nothing is published.

What each patch means for the data

  • strike_benefits = 0 on every donor person. This is the true value: no row in the donor or in ASEC pppub23–26 has OI_OFF 12. The column is a component of the income pass's __acs_transfer_other_income predictor. It is not a transfer target, so ACS rows carry it as null in staging, and R2 fills the engine default, 0. Commit 3 registers that fill, so finalize accepts it.
  • passive_partnership_s_corp_income = 0 and s_corp_income = 0 on every donor person. Here 0 means "split unknown", not "no income". partnership_income holds the combined partnership and S-corp amount (by the code that built it, the PUF's nonpassive-only sum that US frame partnership/S-corp income is nonpassive-only: the donor builds it from E25980-E25960 + E26190-E26180, not line 32 (E26270) #1239 describes). Both columns are QRF targets (puf_tax_itemization). A QRF fit on an all-zero target returns only 0, so ACS rows get no split either: their S-corp income stays inside partnership_income.
  • The SEMP-covered rule (ACS rows carry a donor partnership_income draw ($337.7B weighted) that can stack on SEMP #1387) also removes the S-corp share on some ACS rows. R1 runs the transfer with semp_covers_partnership_income=True. That rule sets the partnership_income draw to 0 on the ACS rows whose SEMP already holds own-partnership income. With this donor, the draw is the combined amount, so on those rows the S-corp share goes too. The transfer's docstring names this case: a caller whose donor carries the combined column with an all-zero s_corp_income "must refuse that donor itself (the legacy builder's dense-donor gate does)". The allowance on s_corp_income lifts that guard for this run only, and the reviewed limitation records it.
  • Work columns (the work-column commit). Donor rows get their own ASEC values, not defaults. ACS rows already map these columns natively, so they need no fill entries.

Defaults

  • Without --allow-default-donor-column, R1 calls the gate exactly as before, and its outputs and summary are unchanged. The tests check this.
  • Without --county-drop-empty-soi-rows, R2 is unchanged, its run identity bytes included.
  • No build calls the patch tool or the two checks. Modal plans pass none of the new flags.

donor_release: null

Confirmed for tonight: R1 runs without --donor-release-manifest. The staging summary's base block records the copy's path, its sha256 (the patch receipt's output.sha256), rows, household_weight_total and donor_release: null. The patch receipt links the copy to the published donor (input.sha256 00946972…).

Nothing refuses a null donor_release before publication. Finalize names the donor "unknown donor id" and still adds the inherited microcosm#1178 Social Security limitation. Package writes null into build_manifest.json, us_source_coverage.json and release_manifest.json. The publisher's local-area contract (validate_release_dir → _validate_local_area_release_dir) refuses a null donor_release, so this release is a file hand-over, as ruled. The manifest option of commit 4 would record the copy's build instead; it is not used tonight.

R2: later gates and the reviewed-fill register

  • All-zero s_corp_income and passive_partnership_s_corp_income: nothing in R2 refuses them.
    • Both are transfer targets. A QRF fit on an all-zero donor target is a degenerate zero, so every ACS cell holds 0.0, not null. R2's reviewed-null fill acts only on nulls, so these columns never reach the reviewed-fill gate.
    • The register's existing s_corp_income entry predates US PUF donor carries Schedule E line 32 as one number: s_corp_income is an exact-zero universe, so Table 1.4's partnership and S-corp legs cannot bind by entity #1258. With no s_corp_income fill in this run, the gate reports it as unused; it does not fail.
    • The self-employment identity gate leaves s_corp_income unconstrained and fails the passive column only where it is nonzero without line-32 income, so all-zero passes.
    • R2 has no degenerate-input gate. That gate and its strike_benefits exclusion belong to the national fiscal-refresh lane.
    • The stored-input check compares column names with engine variables. All three columns are inputs of policyengine-us 2.2.1.
  • strike_benefits: the reviewed-fill gate refused it. Commit 3 adds the entry.
    • Checked by grading one synthetic person.strike_benefits fill (value 0, ACS spine) against both registers. main's register fails it ("a reviewed default fill (0) the release applied has no entry"); this branch's passes.

Records that stay misleading for this run (none refuses; possible follow-ups):

  • Finalize's input_coverage entry says it "binds donor_certified_release" and that coverage is enforced on the certified donor release. With two allowances on a patched donor, that is not true.
  • The donor_sparse_selection_training_set limitation calls the donor "the certified Build O sparse release (unknown donor id)".
  • The packaged staging refresh recipe names --donor-release-manifest and no allowances, so it does not reproduce this staging.
  • The build manifest's staging_orchestration does not copy allow_default_donor_columns. The allowances reach the release only through R1's allowed_default_valued_donor_columns limitation.
  • R1's native_acs_source_universe_blanks limitation lists strike_benefits as a native ACS blank. It is a donor-only column that nothing transfers. This is true of any current staging, not only this one.

Tests

Run locally on Python 3.14 (the new tools' tests also on 3.13). CI runs the full suite.

  • All focused suites for the files this PR touches: 1161 passed, 2 skipped (-n 4), before commit 7; commit 7's tests: 6 passed.
  • Donor-patch tool: 16 tests (40 in the file now, with the work-column step and the overwrite refusal). R1 allowance: refusal unchanged without the flag; allowed columns pass and are recorded; absent, no-observed-value, unknown, not-transfer-consumed and stale allowances refused; the end-to-end summary.
  • Reviewed-fill register entry: 59 engine-free tests and 184 engine-tier tests with policyengine-us 2.2.1.
  • Zero-AGI check: 15 engine-free tests (exit 3 and R2's support refusals from a5ca4df65; the stage-filled hold-back from f8b1039f0), and 2 engine-tier tests with policyengine-us 2.2.1 (its AGI equals the production SOI AGI measure; no stage-filled column is an AGI input). Drop flag: the default refuses an empty AGI row; the flag drops it with a receipt, the identity key and the recipe flag; a county without households is never dropped. From e49d53997 the test runs on through calibration and packaging: the dropped row is named in its county's cell and CSV row; a combined cell marks every member; an older flagged identity that dropped rows is refused.
  • Fill pre-check: 8 tests, plus a real-engine run.
  • Review fixes (items 12–15), the touched files' tests on Python 3.14: AGI check 12 passed; county surface and production 196 passed, 1 skipped (-n 4); fill pre-check 8 passed; patch tool 40 passed (-n 4). Item 16: AGI check 15 engine-free and 2 engine-tier tests passed.
  • ruff check, ruff format --check and tools/ci_test_plan.py verify: clean.

🤖 Generated with Claude Code

juaristi22 and others added 3 commits November 17, 2026 10:55
The County Files proof of concept (#922, plan #1647) stages its ACS rows
from 766b7c7's national donor, which R1 refuses at three donor gates.
tools/patch_us_acs_donor_columns.py writes a patched copy of that donor
for the proof of concept only:

- It adds reviewed all-zero columns on the entity the US engine gives
  them, after checking that the engine default is 0:
  strike_benefits (no donor or ASEC pppub23-26 row has OI_OFF == 12) and
  passive_partnership_s_corp_income (the donor holds partnership and
  S-corp income as one combined partnership_income).
- It refuses a column that is already present with any other value.
- It starts from a byte copy of the input and rewrites only the changed
  entity tables, through the shared frame-table codec, in their stored
  format (fixed, or table with the same data columns). Every other
  table, column, dtype and root attribute is kept and verified.
- It writes a receipt: both SHA-256 digests, row counts per entity,
  each added column with its entity and reason, and the tool's commit.

asec_work_column_step is reserved for the next commit on this tool,
which restores the three ASEC work columns from the pinned ASEC person
files. The tool's writer joins the frame-serializer registry with a
round-trip adapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1's dense-donor coverage gate refuses a transfer-consumed donor column
whose every value equals the engine default, with no exception path.
The County Files proof of concept's patched donor (#922) holds
s_corp_income at 0 and gets passive_partnership_s_corp_income as 0,
because it stores partnership and S-corp income combined in
partnership_income.

--allow-default-donor-column COLUMN=REASON (repeatable) names a column
the gate may accept although every donor value equals the engine
default. The gate skips only that failure for listed columns. It still
refuses an absent column or one with no observed value, a name that is
not transfer-consumed, a name the engine has no default for, and a
stale allowance for a column with donor signal.

The staging summary records each allowance and its reason
(allowed_default_donor_columns, orchestration.allow_default_donor_columns)
and a reviewed limitation, allowed_default_valued_donor_columns, which
R2's finalize carries into the release. The Modal stage plan owns the
flag and never passes it, as for the location flags (#1047). Without
the flag the gate call, the outputs and the summary are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since d5aa4e2 the local income pass's OIP predictor uses
strike_benefits as a component, so R1 requires every donor to carry
it. Neither the shared plan nor the local income pass transfers it, so
every ACS person reaches R2 with a null strike_benefits, R2 fills the
engine default 0, and finalize's reviewed-fill consumer gate refuses
the fill as unregistered (package refuses it too). This blocks the
County Files proof of concept's R2 (#922) and any staging built from a
current national donor.

The register entry names the 13 means-tested consumers that the
engine-tier test recomputes from policyengine-us 2.2.1, with the
verdict harmless: no pinned ASEC respondent reports other-income
code 12, so every donor value is 0, and a transfer would draw 0 too.
The engine-free test that pins the register's column set now lists
fills added since release 767312d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22 juaristi22 changed the title County Files PoC: donor-patch tool and an R1 allowance for all-default donor columns County Files PoC: donor-patch tool, R1 allowance for all-default donor columns, strike_benefits reviewed fill Nov 17, 2026
Without --donor-release-manifest, R1 records donor_release as null,
R2 copies the null into us_source_coverage.json, and the local-area
release contract refuses the release at publication. The amended
ruling (#1647) allows a manifest for the copy instead.

- --release-manifest PATH with --manifest-release and --manifest-build
  (and --manifest-repo-id, default policyengine/populace-us-staging)
  writes a build-folder release manifest whose one microdata artifact
  is the copy, plus the receipt as a second artifact. R1's
  --donor-release-manifest then verifies the copy's SHA-256 against it
  and records the copy's build, which referenced_build and the release
  contract accept. Its dataset_role is proof_of_concept_donor_copy, not
  a release role, so the copy's folder cannot pass as a release.
- --input-release-manifest checks the input against the published
  donor's release manifest and records that build in the receipt and in
  the copy's manifest (derived_from).
- The uploaded manifest names files, not local paths.

Both options are opt-in; the copy and the receipt are unchanged
without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22 juaristi22 changed the title County Files PoC: donor-patch tool, R1 allowance for all-default donor columns, strike_benefits reviewed fill County Files PoC: donor-patch tool, R1 default-column allowance, strike_benefits reviewed fill, copy manifest Nov 17, 2026
juaristi22 and others added 3 commits November 17, 2026 11:42
R2's attach_surface builds each county's IRS SOI rows from the state
rows' stored entries, and the state AGI row stores no zero. A county
whose households all have AGI of exactly 0 therefore gets an empty AGI
row, and attach_surface refuses the run after the materialize engine
pass (#1630 review, finding 1). Minimum support fills only counties
with no household.

tools/check_us_acs_county_agi_support.py runs on the staging H5 before
R2, in minutes and with no engine pass:
- It reads the household table and only the AGI income columns of the
  person and tax-unit tables (fixed-format blocks are streamed in row
  chunks).
- The AGI income columns are the money inputs of policyengine-us's
  irs_gross_income closure, read from the installed engine's source
  without importing it, less Social Security (taxable share only) and
  deductions, expenses and contributions. A null cell counts as 0, the
  default R2 fills. The check errs towards listing a county.
- With --puma-ladder it takes the county list from the ladder and
  applies R2's own minimum-support moves first.
- It writes a JSON report and exits 2 when a county is at risk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A county whose households all have a zero measure (AGI exactly 0) gets
an empty county SOI row, and attach_surface refuses the run after the
engine pass. The #1630 review leaves the remedy to a ruling: drop the
row with a receipt, or give the county a declared SOI cell. The
declared-cell path (DECLARED_SOI_COMBINED_CELLS) covers only a county
absent from the SOI facts, so it cannot take a county that is present.

--county-drop-empty-soi-rows (off by default) drops such a row instead
of refusing: only an SOI row whose county cell has households. A county
without households, and an empty ACS households row, are refused as
before. county_surface.json records each dropped row (target, county
cell, concept, target value, households) under
zero_measure_soi_rows_dropped; the run identity records the flag and
county_targets_dropped, the reliability report lists the dropped rows,
and the refresh recipe repeats the flag. Without the flag nothing
changes, the identity bytes included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R2's finalize stage grades every reviewed default fill against
acs_local_reviewed_fill_consumers.yaml and refuses an unregistered fill,
a different fill value or spine, or another policyengine-us. That
refusal comes after materialize and calibrate, hours into the run.

tools/check_us_acs_local_reviewed_fills.py runs in seconds after R1:
- it reads the staging summary's reviewed_engine_input_nulls (names and
  counts by spine);
- it refuses what materialize refuses: a NEVER_DEFAULT_FILLED null off
  the ACS spine, and a null ESI employer premium;
- it builds the fill records R2 writes, with the fill values R2's
  _resolve_engine_default gives, and runs R2's own finalize gate on
  them. A null that an R2 stage fills first counts as a fill, so it
  errs towards a refusal.
It exits 2 when R2 would refuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22 juaristi22 changed the title County Files PoC: donor-patch tool, R1 default-column allowance, strike_benefits reviewed fill, copy manifest County Files PoC: donor patch, R1 allowance, R2 pre-checks and fixes Nov 17, 2026
juaristi22 and others added 3 commits November 17, 2026 13:43
The telemetry test builds the staging builder's arguments by hand, so it
had no value for --allow-default-donor-column and the builder stopped on
an AttributeError before its first stage. Add the parser's default (None:
no default-valued donor column is allowed), as the test already does for
the household-location defaults.

Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1's _require_donor_work_experience refuses 766b7c7's donor: it predates
the #719 work columns. asec_work_column_step now restores them on the
patched copy, ported from the route 2 step reviewed on #1647, and runs
before the zero step.

- It reads each donor income year's pinned ASEC person member (survey
  year source_year + 1, via _resolve_pin) through the pinned reader,
  which refuses a file whose size or SHA-256 is not the pin's. Only the
  donor's income years are read.
- WEIND and WEMIND join on (source_year, PERIDNUM as exact 22-digit
  text), many donor rows (an ASEC row and its PUF-support clones) to one
  member row. It refuses a donor person with no member row, a member
  that repeats a PERIDNUM (a person would match twice), and any row whose
  source_household_id (Census PH_SEQ), P_SEQ, A_LINENO, A_AGE, WKSWORK,
  POCCU2, PRDTRACE or PRDTHSP differs from the member's.
- The restoration's domains and work-experience identities run on each
  member and on the joined rows. The columns are the output of
  org_wages.derive_us_org_occupation_inputs (int16, int16, bool), whose
  other outputs must equal the donor's own columns.
- One ColumnPatch per column on person, with its source item (WEIND,
  WEMIND, WKSWORK > 0) in the record. The step receipt holds the files,
  their pinned digests, the match counts per income year, the clone
  counts, the carry checks and the nonzero shares beside the carry's
  gate bands. A work column on another entity is refused; one already
  on person must equal the restored values.
- build_patch_steps takes asec_sources; the command always passes it, so
  the copy always carries the columns. --asec-source-cache (default
  ~/.cache/microcosm/cps/asec_education) and --asec-fetch (download a
  missing pinned archive, verified) are its options.

Tests (engine-free, invented members with measured pins, no Census read
or download): the year-scoped join with clones, dtypes and the receipt;
R1's _require_donor_work_experience and _require_income_donor pass on
the patched fixture; a table-format donor keeps its layout; exact-text
keys; each refusal; the cache, archive and fetch paths; the command.

Written by Claude Code, Max Ghenis's agent, at María's request on #1660.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to 2737c32fb.

tools/check_us_acs_county_agi_support.py can exit 0 and R2's materialize can still refuse an empty county AGI row. The check counts a household when any person holds a nonzero value in a gross-income leaf (:231-235), a tax-unit dependent included. Version 2.2.1 of policyengine-us adds only a non-dependent's positive sources to irs_gross_income. A household whose only such income sits on a dependent then has no gross income, and its AGI stays 0 unless a deduction makes it negative. A county whose counted households all look like that passes the check. R2 then finds no nonzero entry in the row and stops (tools/us_acs_local_county_surface.py:2749-2754, :2771-2774). That refusal comes after the engine pass (tools/build_us_acs_local_release.py:2331-2357), so materialize would run again in full.

The check reports the exposed counties under fewer_than_min_support_with_agi_income, which leaves its exit status at 0 (:297-301, :467). Your note on #1647 counts ten of them on the 766b7c7 file.

With --county-drop-empty-soi-rows and no empty row, R2 drops nothing (:2761-2769, :2777-2785). county_surface.json gains an empty zero_measure_soi_rows_dropped list, and the run identity, materialize_rss.json and the refresh recipe record the flag (:2831-2832, :3091-3092).

Tonight's materialize starts in the evening UTC. María, which do you want for it: the flag only when the check lists a county (today's ruling), also when fewer_than_min_support_with_agi_income holds a county, or always?

The rest of this review follows in a second comment.

MaxGhenis and others added 3 commits November 17, 2026 09:00
Written by Claude Code, Max Ghenis's agent, before pushing the work-column commit to #1660.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check counted a household when any person held a nonzero gross-income
input. policyengine-us 2.2.1 adds only the positive sources of a tax
unit's head and spouse to irs_gross_income, and only an adult can be
either, so a household whose income sits on a dependent, or on a tax unit
without an adult, has an AGI of 0. A county of such households passed the
check, and R2 would refuse its empty AGI row after the engine pass (#1660
review).

The check now loads the staging frame and summary as R2's materialize
stage does, holds back the formula-owned columns, fills the reviewed
nulls with R2's own fill step, and runs the engine for the household sum
of adjusted_gross_income in household chunks. An engine-tier test checks
that sum against the production SOI materializer's AGI measure on six
households, the dependent and no-adult cases included. Null take-up flags
that R2's own stages fill are held back, as the AGI does not read them; a
null never-default column that the AGI reads is refused, as R2 refuses it.

Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6a9d' into claude/us-acs-poc-donor-patch-dd6a9d
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Confirmed, and fixed in 3b28e315c (head now 72b7072bf).

Confirmed with the engine. On six test households, policyengine-us 2.2.1 gives an AGI of 0 to a household whose only wages belong to a 10-year-old dependent. It also gives 0 to a household that is one tax unit of a 16-year-old with wages: only an adult can be a tax unit's head or spouse, so that unit's income adds nothing. The first version counted both households.

Fix: the check now computes the value R2 stores, not a guess from the income columns.

  • It loads the staging frame and its summary as materialize does. It holds back the formula-owned columns (project_input_only), fills the reviewed nulls (fill_reviewed_nulls) and builds each household chunk's dataset with _dataset_from_frame.
  • It sums adjusted_gross_income over each household's tax units. The engine-tier test test_us_acs_county_agi_support_engine.py compares that sum with the production SOI materializer's AGI measure (_materialize_target_frame) on the six households: they are equal, a negative AGI included.
  • R2's take-up stages fill some never-default columns before its fill step. The check holds such a column back while it is null, since the AGI does not read it. A null never-default column that the AGI reads is refused, as R2 refuses it.
  • A county is at risk when, after R2's minimum-support moves, none of its households has a nonzero AGI as float32.

Cost. The check now loads the full staging frame. On the 766b7c7 ACS local file it took 438 s with an 11.9 GB peak (16 engine chunks).

Rerun on the 766b7c7 file (county_fips, pinned ladder): no county at risk. fewer_than_min_support_with_agi_income lists the same ten counties as before, with the same counts. The engine's rule lowers the count in 312 other counties, none to zero. That file is R2's output and has no null engine input, so the run built its fill register from the file.

The flag question stays with María. One fact for it: the check now lists a county whenever R2 would build its AGI row empty, so the first option (today's ruling) covers that refusal. Your note on the flag's footprint stands: with the flag and no empty row, R2 drops nothing, and the run identity, materialize_rss.json and the refresh recipe record the flag.

The PR body is updated: item 9, step 4a (which now passes --staging-summary; the default is the same path) and the pre-check results.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Written by Claude Code, Max's agent for the county work, not by Max. Facts only: this comment asks for no decision.

The work-column commit is on this branch. 36bd0dafa implements asec_work_column_step in tools/patch_us_acs_donor_columns.py, and 000532170 merges 2737c32fb. The push went from 2737c32fb to 000532170 as a fast-forward, with no force-push. The tool's 39 tests pass on 000532170, and ruff is clean. An independent review of 36bd0dafa is running; anything it finds will come as a later commit, not part of tonight's head.

Tonight's frozen head is 72b7072bffa94fa0933f9d0a9621fceae403ed70, this branch's head after 3b28e315c, in a detached worktree with a fresh venv from uv sync --all-packages --locked --extra us: Python 3.14.7, policyengine-us 2.2.1. From 000532170 to 72b7072bf only the AGI pre-check, its two test files and its changelog fragment change. The patch tool, R1, R2, the reviewed-fill pre-check and uv.lock are unchanged, so the copy below, made at 000532170, stands.

The patched copy, files only. Step 1 of the PR's block ran with no manifest options and no work-column options, because the default ASEC cache holds pppub23-25 at the pinned hashes.

  • donor_patched_poc.h5: SHA-256 438f47f49b4e6e0c72c3adc3bb3156cbef40b4c0160b71738ac939be1d49a67b, 1,208,766,365 bytes.
  • Receipt: SHA-256 b6430abcb432e108a998dc6dd289157a440cb9ae948234460782f201d374a585. It records tool_commit 000532170 and tool_dirty false.
  • Input: 009469727409e58c0c9c1252581292497b688b76f1d59b51947483fb0f89221c. Row counts per entity are unchanged, with 166,321 persons.
  • Added on person:
    • detailed_industry_recode int16, from WEIND;
    • major_industry_recode int16, from WEMIND;
    • worked_last_year bool, true where WKSWORK > 0;
    • strike_benefits float64, all 0;
    • passive_partnership_s_corp_income float64, all 0.
  • Join on (source_year, PERIDNUM): 166,321 of 166,321 donor rows matched once each. There were no unmatched or multiply matched rows, and 0 identity mismatches in each income year, 2022-2024.

Checks.

  • This copy's three work columns are byte-identical to the local route 2 copy, SHA-256 ce121718c4cc1caa827c0347e453999a3637c4279b74fad07945621ca2c23a71. Values, dtypes and HDF field bytes all match. That copy replaced the one the agent comment on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 cites (60207e8a…); the two have the same work columns byte for byte, and the later one adds the zero strike_benefits column.
  • On a scratch copy made at 36bd0dafa, R1's donor checks, run in R1's order, all pass with the PR's two --allow-default-donor-column allowances. Without the allowances, the dense-coverage gate refuses on exactly s_corp_income and passive_partnership_s_corp_income. The merge 000532170 changes neither the patch tool nor R1's module.
  • Two runs of the tool at 36bd0dafa on the same input gave different file hashes (dc5351ee…, 6119543e…), so the run pins this copy's hash.

Materialize memory. The PR's 78-95 GB estimate for materialize is the 9/23 pre-sparse figure (materialize.peak_rss_gb 77.9). The sparse-era materialize on 10/16 peaked at 48.9 GB RSS. Tonight's run caps materialize at 112 GiB of memory footprint, and the measured peak will be posted after the run.

Receipt (home directory shown as ~)
{
  "added_columns": [
    {
      "action": "added",
      "column": "detailed_industry_recode",
      "derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
      "dtype": "int16",
      "entity": "person",
      "source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
      "source_item": "WEIND",
      "step": "asec_work_columns"
    },
    {
      "action": "added",
      "column": "major_industry_recode",
      "derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
      "dtype": "int16",
      "entity": "person",
      "source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
      "source_item": "WEMIND",
      "step": "asec_work_columns"
    },
    {
      "action": "added",
      "column": "worked_last_year",
      "derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
      "dtype": "bool",
      "entity": "person",
      "source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
      "source_item": "WKSWORK > 0",
      "step": "asec_work_columns"
    },
    {
      "action": "added",
      "column": "strike_benefits",
      "dtype": "float64",
      "engine_default": 0,
      "engine_dtype": "float",
      "entity": "person",
      "reason": "no donor or ASEC pppub23\u201326 row has OI_OFF == 12",
      "step": "reviewed_zero_columns",
      "value": 0
    },
    {
      "action": "added",
      "column": "passive_partnership_s_corp_income",
      "dtype": "float64",
      "engine_default": 0.0,
      "engine_dtype": "float",
      "entity": "person",
      "reason": "the donor holds partnership and S-corp income as one combined amount in `partnership_income`; the split is unknown",
      "step": "reviewed_zero_columns",
      "value": 0
    }
  ],
  "engine": {
    "package": "policyengine-us",
    "version": "2.2.1"
  },
  "input": {
    "path": "~/reviews/lwi-county-build-2026-11-13/inputs/donor-receipts-20260923/populace_us_2024_receipt_qualified.h5",
    "release": null,
    "rows": {
      "family": 65206,
      "household": 57240,
      "marital_unit": 130729,
      "person": 166321,
      "spm_unit": 59900,
      "tax_unit": 79729
    },
    "sha256": "009469727409e58c0c9c1252581292497b688b76f1d59b51947483fb0f89221c"
  },
  "issue": "microcosm#922",
  "kept_columns": [],
  "output": {
    "path": "~/reviews/lwi-county-build-2026-11-13/runs/poc-run/county-poc/donor/donor_patched_poc.h5",
    "rows": {
      "family": 65206,
      "household": 57240,
      "marital_unit": 130729,
      "person": 166321,
      "spm_unit": 59900,
      "tax_unit": 79729
    },
    "sha256": "438f47f49b4e6e0c72c3adc3bb3156cbef40b4c0160b71738ac939be1d49a67b"
  },
  "plan": "microcosm#1647",
  "preserved": {
    "root_attributes": [
      "CLASS",
      "PYTABLES_FORMAT_VERSION",
      "TITLE",
      "VERSION"
    ],
    "tables": [
      "/_time_period",
      "/family",
      "/household",
      "/marital_unit",
      "/person",
      "/spm_unit",
      "/tax_unit"
    ]
  },
  "purpose": "County Files proof of concept only: a patched copy of the published national donor that the ACS staging (R1) accepts. The real County Files donor is a new national build from main.",
  "release_manifest": null,
  "rewritten_tables": [
    "person"
  ],
  "steps": [
    {
      "carry": {
        "donor_work_experience_universe": {
          "weind_to_wemind_rows_verified": 166321,
          "worker_code_iff_weeks_worked": "verified",
          "worker_rows": 85521
        },
        "function": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
        "inputs": [
          "PRDTRACE",
          "PRDTHSP",
          "POCCU2",
          "WKSWORK",
          "WEIND",
          "WEMIND"
        ],
        "reemitted_columns_absent_from_donor": [],
        "reemitted_columns_verified_equal": [
          "cps_race",
          "is_hispanic",
          "detailed_occupation_recode",
          "has_never_worked",
          "is_military",
          "is_computer_scientist",
          "is_farmer_fisher",
          "is_executive_administrative_professional"
        ],
        "require_us_occupation_and_work_experience_inputs": "passed"
      },
      "clones": {
        "distinct_keys": 135833,
        "handling": "Every row sharing a key joins the same member row and must match it on every identity column, so clones get identical codes.",
        "keys_with_more_rows": 0,
        "keys_with_one_row": 105345,
        "keys_with_two_rows": 30488,
        "repeated_key_channel_composition": {
          "asec+puf_tax_detail": 30488
        },
        "rows": 166321,
        "rows_by_support_channel": {
          "asec": 66001,
          "puf_tax_detail": 100320
        }
      },
      "columns": [
        "person.detailed_industry_recode",
        "person.major_industry_recode",
        "person.worked_last_year"
      ],
      "files": {
        "2022": {
          "archive_sha256": null,
          "form": "csv",
          "income_year": 2022,
          "member": "pppub23.csv",
          "member_sha256": "19b56537e50e7663f954361ef2bb5ce9cef8d9d45f156fe1a69a99b654198ffe",
          "member_size_bytes": 281065733,
          "official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip",
          "path": "~/.cache/microcosm/cps/asec_education/pppub23.csv",
          "persons": 146133,
          "survey_year": 2023,
          "work_experience_universe": {
            "weind_to_wemind_rows_verified": 146133,
            "worker_code_iff_weeks_worked": "verified",
            "worker_rows": 73186
          }
        },
        "2023": {
          "archive_sha256": null,
          "form": "csv",
          "income_year": 2023,
          "member": "pppub24.csv",
          "member_sha256": "21a2b9e0e4b08534563578a45acad77868af4ae9a7d46f23776b707d4a559aa7",
          "member_size_bytes": 277250415,
          "official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asecpub24csv.zip",
          "path": "~/.cache/microcosm/cps/asec_education/pppub24.csv",
          "persons": 144265,
          "survey_year": 2024,
          "work_experience_universe": {
            "weind_to_wemind_rows_verified": 144265,
            "worker_code_iff_weeks_worked": "verified",
            "worker_rows": 73471
          }
        },
        "2024": {
          "archive_sha256": null,
          "form": "csv",
          "income_year": 2024,
          "member": "pppub25.csv",
          "member_sha256": "06921fe83fc66c907e6c7b86b82255dc70458ee7d76258fc48297cb34f0c06b5",
          "member_size_bytes": 277882549,
          "official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2025/march/asecpub25csv.zip",
          "path": "~/.cache/microcosm/cps/asec_education/pppub25.csv",
          "persons": 142125,
          "survey_year": 2025,
          "work_experience_universe": {
            "weind_to_wemind_rows_verified": 142125,
            "worker_code_iff_weeks_worked": "verified",
            "worker_rows": 72460
          }
        }
      },
      "join": {
        "by_income_year": {
          "2022": {
            "distinct_keys_matched": 44725,
            "donor_rows": 54464,
            "donor_rows_matched_once": 54464,
            "identity_mismatches": 0,
            "member_rows": 146133,
            "member_rows_not_in_donor": 101408
          },
          "2023": {
            "distinct_keys_matched": 44812,
            "donor_rows": 54654,
            "donor_rows_matched_once": 54654,
            "identity_mismatches": 0,
            "member_rows": 144265,
            "member_rows_not_in_donor": 99453
          },
          "2024": {
            "distinct_keys_matched": 46296,
            "donor_rows": 57203,
            "donor_rows_matched_once": 57203,
            "identity_mismatches": 0,
            "member_rows": 142125,
            "member_rows_not_in_donor": 95829
          }
        },
        "donor_rows": 166321,
        "donor_rows_matched_once": 166321,
        "identity_columns": {
          "A_AGE": "A_AGE",
          "A_LINENO": "A_LINENO",
          "POCCU2": "POCCU2",
          "PRDTHSP": "PRDTHSP",
          "PRDTRACE": "PRDTRACE",
          "P_SEQ": "P_SEQ",
          "WKSWORK": "WKSWORK",
          "source_household_id": "PH_SEQ"
        },
        "key": "(source_year, PERIDNUM as exact 22-digit text)",
        "multiply_matched_donor_rows": 0,
        "unmatched_donor_rows": 0
      },
      "name": "asec_work_columns",
      "source": "pinned Census ASEC person members (spm_role_source.ASEC_SPM_ROLE_SOURCES), read through spm_role_source.read_pinned_asec_person_columns",
      "vintage_rule": "source_year is the ASEC income year; its member is survey year source_year + 1 (asec_census_person_columns._resolve_pin)",
      "work_experience": {
        "detailed_industry_recode_counts": {
          "0": 37248,
          "1": 1710,
          "10": 1401,
          "11": 3839,
          "12": 1670,
          "13": 6940,
          "14": 4175,
          "15": 7453,
          "16": 11679,
          "17": 2142,
          "18": 6374,
          "19": 350,
          "2": 335,
          "20": 4037,
          "21": 3881,
          "22": 390,
          "23": 43552,
          "3": 6491,
          "4": 4583,
          "5": 2751,
          "6": 1509,
          "7": 9036,
          "8": 4132,
          "9": 643
        },
        "major_industry_recode_counts": {
          "0": 37248,
          "1": 1710,
          "10": 19132,
          "11": 8516,
          "12": 4387,
          "13": 3881,
          "14": 390,
          "15": 43552,
          "2": 335,
          "3": 6491,
          "4": 7334,
          "5": 10545,
          "6": 4775,
          "7": 1401,
          "8": 5509,
          "9": 11115
        },
        "nonzero_shares": {
          "detailed_industry_recode": {
            "by_source_year_weighted": {
              "2022": 0.807369122927107,
              "2023": 0.8163839632749547,
              "2024": 0.8453736523065007
            },
            "gate_band": [
              0.7,
              0.92
            ],
            "unweighted_nonzero_share": 0.7760475225617932,
            "weighted_nonzero_share": 0.8244304858584061,
            "within_band": true
          },
          "major_industry_recode": {
            "by_source_year_weighted": {
              "2022": 0.807369122927107,
              "2023": 0.8163839632749547,
              "2024": 0.8453736523065007
            },
            "gate_band": [
              0.7,
              0.92
            ],
            "unweighted_nonzero_share": 0.7760475225617932,
            "weighted_nonzero_share": 0.8244304858584061,
            "within_band": true
          },
          "worked_last_year": {
            "by_source_year_weighted": {
              "2022": 0.5427875993708351,
              "2023": 0.5408310303625891,
              "2024": 0.5511269424079637
            },
            "gate_band": [
              0.4,
              0.62
            ],
            "unweighted_nonzero_share": 0.514192435110419,
            "weighted_nonzero_share": 0.5453081901706771,
            "within_band": true
          }
        },
        "weights": "household_weight resolved to persons",
        "worked_last_year_rows": 85521,
        "worked_rows_with_worker_industry": 85521,
        "worker_industry_rows_without_weeks_worked": 0
      }
    },
    {
      "columns": [
        "person.strike_benefits",
        "person.passive_partnership_s_corp_income"
      ],
      "name": "reviewed_zero_columns"
    }
  ],
  "tool": "tools/patch_us_acs_donor_columns.py",
  "tool_commit": {
    "commit": "0005321705f539ef0a031fe345062da9afff7a04",
    "tool_dirty": false
  },
  "version": 1
}

@juaristi22

Copy link
Copy Markdown
Collaborator Author

On the two file hashes: I ran the patch tool twice on the same input with the test fixtures (zero-column step, fixed and table layouts). The hashes differ each time, but every differing byte (14 of 9.7 MB) is a 4-byte Unix time in the HDF5 object headers that PyTables writes when the tool rewrites the person table. The data is identical. So a re-run gives a new file hash with the same data; compare tables, not file hashes. No code change for tonight. The PR body now records this, the frozen head, the copy and the 48.9 GB materialize figure.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

On when to pass --county-drop-empty-soi-rows tonight: always. María ruled that R2's materialize runs with the flag on every run tonight, not only when the zero-AGI check lists a county: #1647 comment 6950739270.

The PR description now says so. Its commands keep one materialize line, form B, with the flag. Step 4a, the zero-AGI check, still runs after R1, and its exit 2 does not stop the run. Post its list on #1647 with any drop receipt from R2.

juaristi22 and others added 2 commits November 17, 2026 14:30
María ruled that tonight's County Files proof of concept passes
--county-drop-empty-soi-rows on every run (#1647 comment 6950739270), and
asked that the code say clearly that this is a temporary setup. The help
text of --county-drop-empty-soi-rows and --allow-default-donor-column now
says: temporary for the County Files proof of concept (#1647); do not use
for the final County Files build. There every county must be supported
well: a county whose AGI row would be empty needs a real fix (more
household support, or a declared cell with its own ruling), not a dropped
target, and the donor is a new national build.

The county rung doc now describes the drop flag with the same note, and
R1's module docstring, the agent guide and both changelog fragments carry
it too. Help text and docs only; no logic changes.

Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The removal of the County Files proof-of-concept setup now has its own
tracking issue, #1661, a sub-issue of #922. Each note from the last
commit now says "removal tracked in microcosm#1661": the help text of
--county-drop-empty-soi-rows and --allow-default-donor-column, R1's
module docstring, the agent guide, the county rung doc and both
changelog fragments. This PR does not finish #1661. Help text and docs
only; no logic changes.

Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

juaristi22 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Upload list for tonight's run, so that a failure can be fixed and re-run elsewhere

If a stage fails tonight, María (on GCP) or Pavel (on his machine) should be able to fix it and re-run from the failed stage, not from the start. That needs files that today exist only on the run machine. Please upload them as listed below to the private dataset repo policyengine/populace-us-staging (or to a bucket that both can read), in one run folder with a SHA-256 list.

Before the run

  1. The feed consumer_facts_us_41b2fb2.jsonl (SHA-256 1d45a89c…) and the county facts consumer_facts_us_county_3e9d3a0.jsonl (67b0998a…, 922 MB). They are hosted nowhere else. Without them, a re-run must first rebuild both from chronicle at the pinned commits.
  2. If it is easy, the PUMA ladder Done: the PUMA ladder us_puma_ladder_2020.npz (39a2ab2a…, 446,791 bytes) and its build summary are now in the staging repo at inputs/us-puma-ladder/39a2ab2a/ (HF commit c6718cc1). A download at that commit matches the pinned SHA-256. The staging repo also holds the F18 block ladder (inputs/us-block-ladder/528085de/, #1559 comment 6946720912), but that is a different file: tonight's legacy-location run reads only the PUMA ladder, in R1, the zero-AGI check and R2.

After each stage
3. Patch step: the donor copy and its receipt (438f47f4…, b6430abc…). A rebuild is quick, but it gets a new file hash because the HDF5 timestamps change.
4. R1: acs_multispine_staging.h5 and acs_multispine_staging.summary.json, with the R1 log. This upload matters most: it saves 2–3 h, and R2's checkpoint only accepts this exact staging file.
5. Pre-checks: reviewed_fills_precheck.json and county_agi_support.json.
6. Materialize: the whole $RUN/checkpoints folder. It saves 1.5–2.5 h.
7. Calibrate: the $RUN/checkpoints folder again, and the calibrated $RUN/populace_us_2024_acs_local.h5.
8. With each stage: the log and any traceback, the exact command and environment variables (the two R1 memory limits, the 112 GiB cap), the commit, and the peak memory.

Why these files are enough to restart at the failed stage (from the code at 65b2155c0)

  • After materialize, the calibrate, qa, finalize and package stages check only that the staging H5 has the SHA-256 that run_identity.json records. They do not check paths, or the commit that ran materialize. Only materialize reads --feed and --county-facts.
  • Package refuses any commit other than calibrate's, and a dirty tree. So after a code fix in a later stage, commit the fix and re-run calibrate, qa, finalize and package (about 30–60 min). A fix in materialize code re-runs materialize and the later stages. A fix in R1 or in the patch tool re-runs everything from R1.

To re-run on another machine

  • Use the same commit, or a fix commit on top of it, with uv sync --all-packages --locked --extra us (Python 3.14, policyengine-us 2.2.1).
  • Where the ASEC cache is empty, the patch step needs --asec-fetch. R1 downloads the ACS archives and the SIPP file itself.
  • Use 128 GB of RAM: materialize peaked at 48.9 GB on 16 October and at 78–95 GB in September, and calibrate used about 68 GB. Do not use a preemptible VM. Keep at least 60 GB of disk free, plus about 20 GB for the staging and calibrated files.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to 65b2155c0.

tools/check_us_acs_county_agi_support.py:455 sets its exit status from at_risk alone, and step 4a goes on to 4b on either status. The report also holds counties_outside_ladder, no_household and each county's supporting_households (:280-305). R2's check_support refuses a county outside the ladder and a ladder county without a household-universe record, with or without the drop flag (tools/us_acs_local_county_surface.py:2633-2635, :2656-2659). It runs in materialize, after the staging load and the SSI/Medicaid pre-pass (tools/build_us_acs_local_release.py:2284-2330). Give those cases their own exit status, and have step 4a stop on it.

When R2 drops a county's AGI row, calibrate grades that county without it (tools/us_acs_local_county_surface.py:1823-1889). In county_reliability.json, only the top-level unsupported_targets names the row (:3011); the county's cell and the packaged CSV carry no mark (:1924-1950, :3036-3069). Add the dropped rows to both, and extend the flag's test past load_checkpoint_surface (packages/microcosm-build/tests/engine_free/us/test_us_acs_local_county_surface.py:1817-1822) through calibrate and package_reports.

tools/build_us_acs_local_release.py:3926 counts an untracked file as a dirty tree. Calibrate records that without refusing (:2983), and package then refuses (:3927, :3907-3914). The run section could require an empty git status --porcelain before calibrate.

tools/check_us_acs_local_reviewed_fills.py:107-113 files any ACS-only null in a never-default column under natively_filled. R2's reasons for the immigration, disability, income and SSI disability columns say to re-run staging (tools/build_us_acs_local_release.py:827-848, :873-905), so such a null would pass step 3 and stop materialize (:1043-1052). Limit that branch to the columns R2's stages fill.

tools/patch_us_acs_donor_columns.py:1360-1377 never tests whether --output-h5 or the receipt exists, so a second run replaces both (:1090, :1127). Because two runs give different file hashes, a re-run after R1 leaves the staging summary's base.sha256 naming bytes that no longer sit at its path (tools/_legacy/build_us_acs_multispine_base.py:2231-2233). Refuse an existing output unless a flag allows it.

juaristi22 and others added 3 commits November 17, 2026 17:22
…check

The zero-AGI pre-check set its exit status from at_risk alone. R2's
check_support refuses a county outside the ladder and a ladder county
without a household-universe record, with or without
--county-drop-empty-soi-rows, in materialize after the staging load and
the SSI/Medicaid pre-pass. With --puma-ladder the check now lists both
(counties_outside_ladder, unsupported_counties) with R2's set rules,
moves no household when a county is outside the ladder (as R2 does), and
exits 3, which takes precedence over 2. The report records its exit
status.

Part of #922; review of #1660 (comment 6955224433).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --county-drop-empty-soi-rows, calibrate grades a county without its
dropped row, and only the reliability report's top-level
unsupported_targets named it. Materialize now records each dropped row's
county cell in the run identity (zero_measure_soi_rows_dropped; only a
flagged run has the key), and the bound report names the row in every
member county's cell and in a zero_measure_soi_rows_dropped column of
county_reliability.csv. A run without the flag keeps its identity, report
and CSV bytes. A flagged identity from older code that dropped rows but
lacks their cells is refused (re-run materialize), never left unmarked.
The flag's test now runs on through calibration_evidence, write_reports
and package_reports.

Part of #922; review of #1660 (comment 6955224433).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reviewed-fill pre-check filed any ACS-only null in a never-default
column under natively_filled. R2's stages fill only the take-up stage's
SPM-unit take-up flags and engine-free fills and the SSI and Medicaid
take-up flags; R2's reasons for the immigration, disability, income
transfer and SSI disability-criteria columns say to re-run staging, so
such a null passed the pre-check and then stopped materialize. The check
now takes the stage-filled set from R2's own constants
(stage_filled_never_default), refuses every other never-default null on
any spine with a re-run-staging message, and records the set in its
report. A test ties the set to R2's own reasons.

Part of #922; review of #1660 (comment 6955224433).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The donor-patch tool never tested whether --output-h5, the receipt or
the copy release manifest existed, so a second run replaced them. Two
runs give different file hashes (the HDF5 object headers record write
times), so a re-run after R1 left the staging summary's base.sha256
naming bytes no longer at its path. The tool now refuses an existing
output before any work, and --overwrite allows the replacement; its help
says to re-run R1 on the new copy.

Part of #922; review of #1660 (comment 6955224433).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

All five findings confirmed. Four are fixed in code (head 2e9bf301d) and one in the run section of the description.

  1. Exit status for what R2 always refuses. a5ca4df65: with --puma-ladder, the check now lists counties_outside_ladder (compared with every ladder county, as check_support compares) and unsupported_counties (ladder counties without a household-universe record after the minimum-support moves). It exits 3 for either, ahead of 2, and the report records exit_status. As check_support does, it plans no moves when a county is outside the ladder. Step 4a now stops on exit 3.
  2. Dropped rows in the county cell and the CSV. e49d53997: under the flag, materialize records each dropped row's cell (target, county_fips, cell_members, concept) in the run identity. _bound_report names the row in every member county's cell (zero_measure_soi_rows_dropped), and county_reliability.csv gets a column of that name. A run without the flag keeps its identity, report and CSV bytes. A flagged identity from older code that dropped rows but lacks their cells is refused in write_reports and package_reports, never left unmarked; with no row dropped, it needs nothing. The flag's test now runs on through calibration_evidence, write_reports and package_reports, and a unit test covers a combined cell.
  3. Dirty tree. The run section now has a clean guard (empty git status --porcelain) before calibrate and before package, and says to keep $RUN and the logs outside the checkout.
  4. Fill pre-check. 28e7b7ce6: natively_filled now takes only the never-default columns that R2's stages fill on ACS rows (the take-up stage's SPM-unit flags and engine-free fills, and SSI and Medicaid take-up), built from R2's own constants. Every other never-default null is refused, on any spine, with a re-run-staging message. A test checks the set against _NEVER_DEFAULT_FILLED_REASONS: each stage-filled reason says "stage first", and each other reason says "re-run staging".
  5. Patch re-run. 2e9bf301d: an existing --output-h5, receipt or --release-manifest is refused before any work. --overwrite allows the replacement, and its help says to re-run R1 on the new copy.

None of the four commits changes R1 code, so R1's staging from 72b7072bf stays valid. The description's commands and remaining steps now say to run steps 3–4b from 2e9bf301d, with every R2 stage from that one head.

Tests on Python 3.14, the touched files only: AGI check 12 passed; county surface and production 196 passed, 1 skipped; fill pre-check 8 passed; patch tool 40 passed. ruff check and ruff format --check are clean on the touched files.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to 2e9bf301d.

tools/check_us_acs_county_agi_support.py:202 still passes all of NEVER_DEFAULT_FILLED to hold_back_null_runtime_columns, which drops every null column in that set that AGI doesn't read, on either spine (:148-161). Since 28e7b7c, step 3 refuses a null in the columns no R2 stage fills, so the two steps in order catch it. Run alone, the AGI check passes that file, and R2 refuses it at its first fill (tools/build_us_acs_local_release.py:1043-1052). Pass stage_filled_never_default(release) here as well, and hold back only nulls confined to the ACS spine.

The engine-tier test holds back a null takes_up_snap_if_eligible against the real closure (packages/microcosm-build/tests/engine_workflow/us/test_us_acs_county_agi_support_engine.py:83, :127-128). No test covers the other seven stage-filled columns, and test_agi_engine_inputs_are_the_static_closure_of_agi uses a stand-in index (packages/microcosm-build/tests/engine_free/us/test_us_acs_county_agi_support.py:244-250). On policyengine-us 2.2.1, the metadata index's closure of adjusted_gross_income has 68 input leaves and none of the eight, so the refusal at :163 can't fire on them today. An engine-tier assertion that the two sets share no column would catch an engine release that changes that.

Neither point bears on tonight's run.

The zero-AGI check held back every null NEVER_DEFAULT_FILLED column that
the AGI does not read, on either spine. Run alone, it passed a file that
R2's first fill refuses. It now takes the stage-filled set from the
reviewed-fill pre-check (stage_filled_never_default) and holds back a
null column only when an R2 stage fills it, every null is on an ACS row
(the <entity>_spine tag) and the AGI does not read it. It refuses every
other never-default null, as R2's fill step does: a null on a donor row,
in a column no R2 stage fills, in one the AGI reads, or without a spine
tag to place it. An engine-tier test checks that no stage-filled column
is an input of the AGI closure, so an engine release that changes that
fails in CI.

Part of #922; review of #1660 (comment 6956136956).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Both points confirmed and fixed in f8b1039f0 (head now). Item 16 in the description; it changes only step 4a's check.

  1. Hold-back scope. household_agi now passes stage_filled_never_default(release) (imported from the fill pre-check, so the two steps share one set). hold_back_null_runtime_columns holds back a null never-default column only when an R2 stage fills it, every null is on an ACS row (the <entity>_spine tag) and the AGI does not read it. Everything else is refused, as R2's first fill refuses it: a null on a donor row, a null in a column no R2 stage fills, a null the AGI reads, or a null with no spine tag to place it. Run alone, the check now refuses the file that step 3 refuses. Engine-free tests cover each case: the hold-back, the three refusals and the missing tag.
  2. Engine-tier guard. A new engine-tier test (test_no_column_an_r2_stage_fills_is_an_agi_input) builds the AGI closure from PolicyEngineUSVariableMetadataIndex and asserts that it shares no column with the stage-filled set. It passes on policyengine-us 2.2.1. The existing engine-tier test now tags its SPM units with the ACS spine, so its null takes_up_snap_if_eligible is still held back.

Tests on Python 3.14: AGI check 15 engine-free tests passed; 2 engine-tier tests passed (103 s). ruff check and ruff format --check are clean.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Written by Claude Code, Max's agent for the county work, not by Max. Facts only: this comment asks for no decision and recommends none.

Tonight's run will start agi and materialize at 75 GB of free disk instead of 80, on Max's ruling. It will probably end under the 60 GB floor in your comment above.

  • The 80 GB line. The launcher's line for the zero-AGI check (step 4a) and for materialize was 80 GB: your "at least 60 GB of disk free, plus about 20 GB for the staging and calibrated files".
  • Why it was short. Free disk was 89 GB at 20:05Z. R1 was still running and had not yet written its staging, about 11 GB (September's measured 10.7 GB). So the check would have started at about 76-78 GB. Nothing safe could be freed on this Mac before then.
  • The ruling. Max ruled at 20:04Z that both stages may start at 75 GB. The launcher's settings now say so. A read-only probe confirms free >= 75 GB for both stages, with every other setting unchanged.
  • Expected low point. About 54-58 GB free after materialize. The run's own stop line is unchanged: the guard stops a stage under 50 GB.
  • Everything else runs as County Files PoC: donor patch, R1 allowance, R2 pre-checks and fixes #1660's description (18:03Z) says: R1's staging from 72b7072bf, and steps 3-4b and every R2 stage from f8b1039f0, with --county-drop-empty-soi-rows.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Facts only, on the disk figures in comment 6959513812. This reply asks for no decision and recommends none.

After materialize, R2 writes about 19.6 GB more with the PR's commands. So free disk is lowest after package, not after materialize.

  • Calibrate writes the release file at --out-h5. It was 9.8 GB in both earlier full builds: 9,821,668,812 bytes in September, and 9,819,603,807 bytes for the 766b7c7 file. Tonight's run is a legacy-location run, so calibrate writes the file once, with no temporary copy (_write_dataset). Its other files (weights and JSON) are small.
  • Package copies that file to $RUN/release/microcosm_us_2024_acs_local.h5 (tools/build_us_acs_local_release.py:6788-6793). The PR's --out-h5 is $RUN/populace_us_2024_acs_local.h5, another path, so the copy is a second 9.8 GB.
  • QA and finalize write only JSON and CSV reports. In September the largest kept report was calibration_diagnostics.json, 290 KB.
  • Free disk, from your 54-58 GB after materialize: about 44-48 GB after calibrate and about 34-38 GB after package. Both are under the 50 GB stop line.

What the code does about the copy: package skips it when $RUN/release/microcosm_us_2024_acs_local.h5 already holds a file with the calibrated file's SHA-256 (lines 6790-6793). A hard link on the same volume is such a file, and it uses no more disk. For example, after calibrate and before package:

mkdir -p "$RUN/release" && ln "$RUN/populace_us_2024_acs_local.h5" "$RUN/release/microcosm_us_2024_acs_local.h5"

Package refuses an existing path only at $RUN/release/releases/<release> (lines 6251-6256, and the parser check at 7075-7083), so the folder and the link do not stop it. With the link, the low point is about 44-48 GB, after calibrate, which is still under the 50 GB line.

These figures need no change to the code or to tonight's head. The stop line and the choice of what to do are Max's.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Request to Max and Max's agent: the run's outcome, the restart files, and the county file

Thank you for the 23:57Z update on #1647. The zero-AGI check exited 0, and materialize passed at 23:49Z with no SOI row dropped. We have seen nothing since then about calibrate, qa, finalize or package.

1. Please post the outcome of each later stage (calibrate, qa, finalize and package):

  • Passed or stopped. If a stage stopped: the stage, the log tail or traceback, and the exact command.
  • Wall time, peak memory and the lowest free disk. Comment 6959825920 expected free disk to go under the 50 GB stop line in calibrate or package. If the guard stopped a stage, please say so.
  • The commit, and whether the tree was clean (package refuses a dirty tree).
  • If package passed: the SHA-256 of the calibrated file, the result of each gate in gate_summary.json, and the counts in county_reliability.json (counties graded, unsupported targets).

2. Please upload the restart files, so that María (on GCP) or Pavel can fix a failure and re-run from the stopped stage. At 08:40Z on 18 Nov, the private staging repo policyengine/populace-us-staging had no commit after the PUMA ladder (15:02Z on 17 Nov). The full list and the reasons are in comment 6951750167. Please put these in one run folder with a SHA-256 list:

  1. The feed consumer_facts_us_41b2fb2.jsonl and the county facts consumer_facts_us_county_3e9d3a0.jsonl. They are on no other host.
  2. The patched donor copy and its receipt (438f47f4…, b6430abc…).
  3. R1's acs_multispine_staging.h5, its .summary.json and the R1 log. This upload is the most important: it saves 2–3 h, and R2 accepts only this exact staging file.
  4. reviewed_fills_precheck.json and county_agi_support.json.
  5. The whole $RUN/checkpoints folder after the last stage that ran.
  6. For each stage: the log, any traceback, the exact command and environment variables, and the peak memory.

If you used a bucket instead, please post its location.

3. If package passed, please also upload the county file for inspection, in the same run folder:

  • The calibrated file $RUN/populace_us_2024_acs_local.h5 (about 9.8 GB). $RUN/release/microcosm_us_2024_acs_local.h5 has the same bytes, so one of the two is enough.
  • The release folder $RUN/release/releases/<release>/ (manifests, gate summary, county reliability JSON and CSV, sha256sums.txt) and $RUN/release/package_result.json.

Please upload these files to the staging repo only, and do not run the publish command that package prints. This proof-of-concept release records donor_release: null and is not for publication, as the PR description says.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Written by Claude Code, Max's agent for the county work, not by Max. Facts only.

The run's outcome: every stage passed; package ended at 2026-11-18T01:18:36Z

Answering comment 6973462140, item 1.

Stage Result Start → end (UTC) Wall time Peak memory Lowest free disk
calibrate passed, exit 0 11-17 23:49:14 → 11-18 00:20:19 31.1 min 47.7 GiB 64.71 GB
qa passed, exit 0 00:20:25 → 01:14:35 54.2 min 23.9 GiB 64.64 GB
finalize passed, exit 0 01:14:41 → 01:16:40 2.0 min 29.2 GiB 64.85 GB
package passed, exit 0 01:16:46 → 01:18:36 1.8 min 25.8 GiB 64.80 GB
  • Peak memory is the guard's figure: the larger of summed RSS and summed footprint over the process tree, sampled every second. Free disk is decimal GB on the data volume.

  • The earlier stages: R1 241.6 min (48.0 GiB), fills 1.3 min, agi 3.6 min (40.7 GiB) and materialize 177.3 min (59.7 GiB). All exited 0.

  • Disk. The guard stopped no stage. Free disk never went under the 50 GB stop line. The lowest was 64.64 GB, during qa, and the launcher logged no disk hold. Swap was 0 wherever it was sampled (agi, materialize, calibrate).

  • Commit and tree.

    • R1 ran at 72b7072bffa94fa0933f9d0a9621fceae403ed70.
    • Fills through package ran at f8b1039f0099454da2b3e3d3ba17b674c5a488d9.
    • Before every stage the launcher logged "tree clean (git status --porcelain --untracked-files=all: nothing)". build_manifest.json records build_sha "f8b1039f00" and build_dirty false.
  • Calibrated file. populace_us_2024_acs_local.h5: sha256 82314a3b5a6a95762faf6eb4f5b499ce56608ad6a39a27b75e11c75b4945bd3c, 10,637,811,669 bytes. The release root copy has the same sha256. Build id populace-us-2024-buildo-acs-local-f8b1039f00-20261118T011728Z. It is not published, and the publish and promote commands that package_result.json lists were not run.

  • gate_summary.json. All 23 gates record passed: true, with no failures:

    • acs_local_hours_signal, us_puma_ladder_gate, hours_worked_signal, acs_local_take_up_signal, acs_local_immigration_signal, acs_local_work_disability_signal, acs_local_income_transfer_signal, acs_local_ssi_disability_signal, acs_local_ssi_medicaid_take_up_signal, acs_local_receipt_anchor_signal and acs_local_reviewed_fill_consumers;
    • calibration, cd_holdout, weight_origin, input_coverage, spine_composition, spine_ssi_qa, consumer_ready and district_ess_collapse;
    • work_experience_inputs, acs_self_employment_identity, stored_inputs and spm_measurement_composition.

    Three record more:

    • calibration: final loss 0.015904 against an initial 0.020786; 0.7566 of 16,440 trained rows within 10%; realized max weight ratio 4.9297 (cap 5).
    • district_ess_collapse is report-only (blocking: false) and records criteria_met: false: 271 of 436 districts collapsed.
    • cd_holdout records n_targets: 0.
  • county_reliability.json. 3,144 counties graded, with grade_rule.provisional true:

    A B C D U
    0 3 103 3,038 0

    unsupported_targets is empty, and zero_measure_soi_rows_dropped is empty. The minimum-support step made 9 moves, into 15005, 31009 and 48301.

  • Two hand steps.

    • The launcher's shell wait returned 19 at R1's end although R1 had passed: the guard recorded exit 0, and every staging gate passed. So R1's done record was written by hand, and the hand-over then armed the f8b1039 launcher.
    • The release root copy is an APFS clone of the calibrated file, placed by hand during qa. Package found the same sha256 and did not copy.

Uploads (items 2 and 3). Nothing is uploaded yet. The upload needs Max's go, which has been asked for. Once he gives it, the run folder goes to policyengine/populace-us-staging with a SHA-256 list, in four batches, each usable alone:

  1. Reports, checkpoints, the release folder, package_result.json, the stage logs (R1's included), commands, summaries and the pre-checks: 1.1 GB.
  2. The calibrated file: 10.6 GB.
  3. R1's staging file: 11.5 GB.
  4. The donor copy with its receipt, the feed and the county facts: 2.3 GB.

At this morning's measured rate the whole upload takes about 2.5 h. The publish command will not be run.

A fuller run report for the PoC report, with the fit numbers, follows on #1647.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor

Written by Claude Code, Max's agent for the county work, not by Max. Facts only.

Upload: batch 1 of 4 is in the staging repo

This follows up comment 6979361950, which said the upload awaited Max's go. He gave it at about 13:05Z.

  • Folder: policyengine/populace-us-staging → runs/lwi-county-poc-20261117/, a private dataset. Commit d8a772ced64e, 13:28Z.

  • Batch 1: 73 files, 1.12 GB. Paths follow the run folder:

    • county-poc/release/releases/microcosm-us-2024-local/ (all 15 files) and county-poc/release/package_result.json;
    • the whole county-poc/checkpoints/ folder after the last stage (23 files);
    • county-poc/reviewed_fills_precheck.json and county-poc/county_agi_support.json;
    • the staging and output summaries;
    • the donor receipt and the patch step's logs;
    • every stage log, with launcher.log and handover.log;
    • rss.jsonl, which holds each stage's exact argv;
    • the stage-*.done records.
  • SHA-256 list: report/sha256-outputs.txt, one line per file for all 78 files, with paths relative to the run folder. The three input files listed under ../ go to inputs/. The PUMA ladder is skipped, since it is already at inputs/us-puma-ladder/39a2ab2a/.

  • Draft report: report/REPORT.md is a draft of the run report. The version posted on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 supersedes it.

  • Checked: for each of the 73 files, the repo's LFS sha256 (large files) or git blob id (small files) equals the local file's and the list's.

  • Still uploading:

    • the calibrated file county-poc/populace_us_2024_acs_local.h5 (10.6 GB), started 13:28Z;
    • then R1's county-poc/acs_multispine_staging.h5 (11.5 GB);
    • then the donor copy county-poc/donor/donor_patched_poc.h5 and the feed and county facts in inputs/ (2.3 GB).

    At the measured rate, that's about 2 h more. A note follows here when they are in.

  • Not uploaded: county-poc/release/microcosm_us_2024_acs_local.h5, which has the same bytes as the calibrated file. Nothing is published, and the publish and promote commands are not run.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor

Written by Claude Code, Max's agent for the county work, not by Max. Facts only.

Upload: all four batches are in the staging repo

This follows up comment 6980005693 (batch 1), answering comment 6973462140, items 2 and 3.

  • Folder: policyengine/populace-us-staging → runs/lwi-county-poc-20261117/, a private dataset.
  • Commits:
    • batch 1, d8a772ced64e (13:28Z): the 73 files the earlier comment lists;
    • batch 2, 518c509713b3 (13:58Z): the calibrated file county-poc/populace_us_2024_acs_local.h5;
    • batch 3, 7ba16a15413f (15:00Z): R1's staging file county-poc/acs_multispine_staging.h5;
    • batch 4, one commit per file: 159a42f8a619 (15:01Z) the donor copy county-poc/donor/donor_patched_poc.h5; 66aaed953c40 (15:02Z) the feed inputs/consumer_facts_us_41b2fb2.jsonl; c88bfbf16ae4 (15:03Z) the county facts inputs/consumer_facts_us_county_3e9d3a0.jsonl.
  • Checked at c88bfbf16ae4: all 78 files in the folder are there and none else. For each, the repo's LFS sha256 (large files) or git blob id (small files) equals the local file's and, for the 76 on the SHA-256 list, the list's. The other two are report/REPORT.md (a draft, superseded by the County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 post) and the list itself.
  • Not uploaded: the PUMA ladder, already at inputs/us-puma-ladder/39a2ab2a/, and county-poc/release/microcosm_us_2024_acs_local.h5, which has the same bytes as the calibrated file. Nothing is published, and the publish and promote commands are not run.

🤖 Generated with Claude Code

Draft v0.1 of the proof-of-concept report: the fit and the limits of the
2024 county file from the 17-18 November run. It is pe-doc Markdown, with
the numbers from the run report on #1647 and the release's own reports,
and three placeholders still open (the methods link twice, the schema
documentation). Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

PoC report: first draft pushed (c990afa56)

The first draft of the PoC report for the Living Wage Institute is on this branch: experiments/us-county-poc-20261117/lwi-poc-report.md. It states the fit and the limits of the 2024 county file from the 17–18 November run. Its numbers come from the run report on #1647 (comment 6982547034 and part 2) and from the release's own reports.

What it contains

  • Summary and deliverables. A summary, and the deliverables set against the agreement and the stretch goals.
  • The file, and how we built it. This part is kept short. It links the imputation working paper and leaves the detail to the methods doc.
  • Fit. It covers:
    • states and the nation, and congressional districts;
    • county targets used in calibration;
    • held-out county targets, against the population-share baseline;
    • ACS county statistics never used in calibration;
    • the earlier state-and-district file.
  • Reliability. The provisional grades, and why the county effective samples are small: the donor weight share, the weight concentration and the loss weight on state targets.
  • Use and limits. How to use the file, and its limits.
  • Next steps. The county effective sample, the contribution of clones to the calibration, the calibration fit, the stretch goals and the final build.
  • Appendix. The file's provenance.
  • A loader test from today. policyengine-us 2.2.1's standard loader, Microsimulation(dataset=<file>), built the national simulation in 90 s with a peak of 17.9 GiB on a 32 GiB Mac. A one-state run computed income tax, SNAP and household net income with no missing values.

What it is missing

  • The methods documentation. The draft links it in two places, which are placeholders for now. Reminder to Max's agents: please post the methods draft (#1647 comment 6983733024).
  • The schema documentation. The agreement asks for schema docs, and Max lists the schema manifest as a stretch goal.
  • The review and the final build. I will review the draft. The final PDF is built with the template below.

Decision: we share the file through a new Hugging Face repo. The file is in the private dataset policyengine/lwi-county-files.

  • The 10.6 GB file was copied server-side from the staging run folder, and its SHA-256 82314a3b… was checked.
  • The repo also has a README with download and load instructions, county_reliability.csv and .json, and a SHA-256 list.
  • Hugging Face gives access per repository, so LWI gets access to this repo only.
  • Pending: the token. LWI needs a read-only fine-grained token for this one repo. I will create it, send it to LWI, and revoke it after the download.

Question for Max: do you have a report template? If you have one you would like us to use, please point us to it. If not, we will build the report with pe-doc from PolicyEngine/document-templates, the branded template for outbound PolicyEngine documents. It has no report kind, so the draft uses its proposal format with the fiscal-sponsor line off. The draft is already in its Markdown format.

juaristi22 and others added 5 commits November 19, 2026 20:16
Takes the corrections that part 2 of the methods draft lists (#1647
comment 7005741386). The ACS county tables are 2020-2024 5-year; only
some targets equal published figures; the ACS checks compare with the
file's state totals split in the ACS county proportions; county counts
are of households in housing units; and the report now names three
temporary shortcuts. The checks line says what gate_summary.json
records: all 17 blocking checks passed, and the district evidence
check flagged 275 of the 436 districts. The schema documentation is a
deliverable, not a stretch goal, and the placeholders are gone: the
report names the methods and schema documents instead of linking into
the private repository. Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds, next to the PoC report, the three documents microcosm#1647
comment 7006535601 asks for:

- schema_manifest.json, as tools/us_release_schema_manifest.py wrote it
  for the 2024 county file (SHA-256 82314a3b...bd3c);
- schema_manifest.md, the tool's output with a notes section added at
  the top. The notes say that six household geography columns hold
  empty strings, not nulls, for all 1,529,668 ACS households (and for
  some donors, including every Connecticut donor's cbsa_code); which
  columns are raw CPS ASEC or ACS fields, build identifiers or derived
  columns, and which 198 policyengine-us 2.2.1 reads; and why three
  columns were removed and 134 dtypes changed;
- lwi-poc-methods.md, part 1 of the methods draft as posted in #1647
  comment 7005740491, without the comment's preamble.

Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max's agents committed part 1 of the methods draft as posted (#1647
comment 7005740491). This edits it for the Living Wage Institute:
pe-doc front matter and title block; an appendix of build identifiers,
settings and input digests in place of the links to GitHub comments,
which a reader with only a download token cannot open; flags, run-file
names and imputation family names out of the text; and the gate
section reworded. The donor households' geography and tax-detail
records are now described from the national build's code at cae8640,
in place of "we could not establish". Max's agents are checking that
account against the 28 July build's run logs (#1647 comment
7022836029). The experiments README lists the three documents. Part
of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 254 columns policyengine-us 2.2.1 ignores are 213 raw survey
fields, 29 build identifiers and 12 derived columns, among them
county_fips_2023, as the notes in schema_manifest.md count them. The
report said "raw survey fields and build identifiers" (#1647 comment
7022836029). Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent check of the notes found three points to fix:

- medicare_part_b_premiums: drop the claim that it held the CPS ASEC
  field PEMCPREM. On tax-detail copies the donor build may have set it
  from the PUF instead. policyengine-us 2.2.1 computes the Part B
  premium itself (medicare_part_b_premium).
- A_EXPRRP is the Census value only for income-year 2024 donors. For
  income years 2022 and 2023 the donor build derived it from line,
  spouse and parent numbers (base summary relationship_recode_source).
- CENSUS_TAX_ID is the Census TAX_ID, which PolicyEngine's CPS extractor
  copies before it overwrites TAX_ID with the tax unit it builds.

The tables and schema_manifest.json are unchanged. Part of #922.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants