Repository navigation
County Files PoC: donor patch, R1 allowance, R2 pre-checks and fixes - #1660
juaristi22 wants to merge 26 commits into
Conversation
The County Files proof of concept (#922, plan #1647) stages its ACS rows from 766b7c7's national donor, which R1 refuses at three donor gates. tools/patch_us_acs_donor_columns.py writes a patched copy of that donor for the proof of concept only: - It adds reviewed all-zero columns on the entity the US engine gives them, after checking that the engine default is 0: strike_benefits (no donor or ASEC pppub23-26 row has OI_OFF == 12) and passive_partnership_s_corp_income (the donor holds partnership and S-corp income as one combined partnership_income). - It refuses a column that is already present with any other value. - It starts from a byte copy of the input and rewrites only the changed entity tables, through the shared frame-table codec, in their stored format (fixed, or table with the same data columns). Every other table, column, dtype and root attribute is kept and verified. - It writes a receipt: both SHA-256 digests, row counts per entity, each added column with its entity and reason, and the tool's commit. asec_work_column_step is reserved for the next commit on this tool, which restores the three ASEC work columns from the pinned ASEC person files. The tool's writer joins the frame-serializer registry with a round-trip adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1's dense-donor coverage gate refuses a transfer-consumed donor column whose every value equals the engine default, with no exception path. The County Files proof of concept's patched donor (#922) holds s_corp_income at 0 and gets passive_partnership_s_corp_income as 0, because it stores partnership and S-corp income combined in partnership_income. --allow-default-donor-column COLUMN=REASON (repeatable) names a column the gate may accept although every donor value equals the engine default. The gate skips only that failure for listed columns. It still refuses an absent column or one with no observed value, a name that is not transfer-consumed, a name the engine has no default for, and a stale allowance for a column with donor signal. The staging summary records each allowance and its reason (allowed_default_donor_columns, orchestration.allow_default_donor_columns) and a reviewed limitation, allowed_default_valued_donor_columns, which R2's finalize carries into the release. The Modal stage plan owns the flag and never passes it, as for the location flags (#1047). Without the flag the gate call, the outputs and the summary are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since d5aa4e2 the local income pass's OIP predictor uses strike_benefits as a component, so R1 requires every donor to carry it. Neither the shared plan nor the local income pass transfers it, so every ACS person reaches R2 with a null strike_benefits, R2 fills the engine default 0, and finalize's reviewed-fill consumer gate refuses the fill as unregistered (package refuses it too). This blocks the County Files proof of concept's R2 (#922) and any staging built from a current national donor. The register entry names the 13 means-tested consumers that the engine-tier test recomputes from policyengine-us 2.2.1, with the verdict harmless: no pinned ASEC respondent reports other-income code 12, so every donor value is 0, and a transfer would draw 0 too. The engine-free test that pins the register's column set now lists fills added since release 767312d. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without --donor-release-manifest, R1 records donor_release as null, R2 copies the null into us_source_coverage.json, and the local-area release contract refuses the release at publication. The amended ruling (#1647) allows a manifest for the copy instead. - --release-manifest PATH with --manifest-release and --manifest-build (and --manifest-repo-id, default policyengine/populace-us-staging) writes a build-folder release manifest whose one microdata artifact is the copy, plus the receipt as a second artifact. R1's --donor-release-manifest then verifies the copy's SHA-256 against it and records the copy's build, which referenced_build and the release contract accept. Its dataset_role is proof_of_concept_donor_copy, not a release role, so the copy's folder cannot pass as a release. - --input-release-manifest checks the input against the published donor's release manifest and records that build in the receipt and in the copy's manifest (derived_from). - The uploaded manifest names files, not local paths. Both options are opt-in; the copy and the receipt are unchanged without them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R2's attach_surface builds each county's IRS SOI rows from the state rows' stored entries, and the state AGI row stores no zero. A county whose households all have AGI of exactly 0 therefore gets an empty AGI row, and attach_surface refuses the run after the materialize engine pass (#1630 review, finding 1). Minimum support fills only counties with no household. tools/check_us_acs_county_agi_support.py runs on the staging H5 before R2, in minutes and with no engine pass: - It reads the household table and only the AGI income columns of the person and tax-unit tables (fixed-format blocks are streamed in row chunks). - The AGI income columns are the money inputs of policyengine-us's irs_gross_income closure, read from the installed engine's source without importing it, less Social Security (taxable share only) and deductions, expenses and contributions. A null cell counts as 0, the default R2 fills. The check errs towards listing a county. - With --puma-ladder it takes the county list from the ladder and applies R2's own minimum-support moves first. - It writes a JSON report and exits 2 when a county is at risk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A county whose households all have a zero measure (AGI exactly 0) gets an empty county SOI row, and attach_surface refuses the run after the engine pass. The #1630 review leaves the remedy to a ruling: drop the row with a receipt, or give the county a declared SOI cell. The declared-cell path (DECLARED_SOI_COMBINED_CELLS) covers only a county absent from the SOI facts, so it cannot take a county that is present. --county-drop-empty-soi-rows (off by default) drops such a row instead of refusing: only an SOI row whose county cell has households. A county without households, and an empty ACS households row, are refused as before. county_surface.json records each dropped row (target, county cell, concept, target value, households) under zero_measure_soi_rows_dropped; the run identity records the flag and county_targets_dropped, the reliability report lists the dropped rows, and the refresh recipe repeats the flag. Without the flag nothing changes, the identity bytes included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R2's finalize stage grades every reviewed default fill against acs_local_reviewed_fill_consumers.yaml and refuses an unregistered fill, a different fill value or spine, or another policyengine-us. That refusal comes after materialize and calibrate, hours into the run. tools/check_us_acs_local_reviewed_fills.py runs in seconds after R1: - it reads the staging summary's reviewed_engine_input_nulls (names and counts by spine); - it refuses what materialize refuses: a NEVER_DEFAULT_FILLED null off the ACS spine, and a null ESI employer premium; - it builds the fill records R2 writes, with the fill values R2's _resolve_engine_default gives, and runs R2's own finalize gate on them. A null that an R2 stage fills first counts as a fill, so it errs towards a refusal. It exits 2 when R2 would refuse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The telemetry test builds the staging builder's arguments by hand, so it had no value for --allow-default-donor-column and the builder stopped on an AttributeError before its first stage. Add the parser's default (None: no default-valued donor column is allowed), as the test already does for the household-location defaults. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1's _require_donor_work_experience refuses 766b7c7's donor: it predates the #719 work columns. asec_work_column_step now restores them on the patched copy, ported from the route 2 step reviewed on #1647, and runs before the zero step. - It reads each donor income year's pinned ASEC person member (survey year source_year + 1, via _resolve_pin) through the pinned reader, which refuses a file whose size or SHA-256 is not the pin's. Only the donor's income years are read. - WEIND and WEMIND join on (source_year, PERIDNUM as exact 22-digit text), many donor rows (an ASEC row and its PUF-support clones) to one member row. It refuses a donor person with no member row, a member that repeats a PERIDNUM (a person would match twice), and any row whose source_household_id (Census PH_SEQ), P_SEQ, A_LINENO, A_AGE, WKSWORK, POCCU2, PRDTRACE or PRDTHSP differs from the member's. - The restoration's domains and work-experience identities run on each member and on the joined rows. The columns are the output of org_wages.derive_us_org_occupation_inputs (int16, int16, bool), whose other outputs must equal the donor's own columns. - One ColumnPatch per column on person, with its source item (WEIND, WEMIND, WKSWORK > 0) in the record. The step receipt holds the files, their pinned digests, the match counts per income year, the clone counts, the carry checks and the nonzero shares beside the carry's gate bands. A work column on another entity is refused; one already on person must equal the restored values. - build_patch_steps takes asec_sources; the command always passes it, so the copy always carries the columns. --asec-source-cache (default ~/.cache/microcosm/cps/asec_education) and --asec-fetch (download a missing pinned archive, verified) are its options. Tests (engine-free, invented members with measured pins, no Census read or download): the year-scoped join with clones, dtypes and the receipt; R1's _require_donor_work_experience and _require_income_donor pass on the patched fixture; a table-format donor keeps its layout; exact-text keys; each refusal; the cache, archive and fetch paths; the command. Written by Claude Code, Max Ghenis's agent, at María's request on #1660. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Lines refer to
The check reports the exposed counties under With Tonight's materialize starts in the evening UTC. María, which do you want for it: the flag only when the check lists a county (today's ruling), also when The rest of this review follows in a second comment. |
The check counted a household when any person held a nonzero gross-income input. policyengine-us 2.2.1 adds only the positive sources of a tax unit's head and spouse to irs_gross_income, and only an adult can be either, so a household whose income sits on a dependent, or on a tax unit without an adult, has an AGI of 0. A county of such households passed the check, and R2 would refuse its empty AGI row after the engine pass (#1660 review). The check now loads the staging frame and summary as R2's materialize stage does, holds back the formula-owned columns, fills the reviewed nulls with R2's own fill step, and runs the engine for the household sum of adjusted_gross_income in household chunks. An engine-tier test checks that sum against the production SOI materializer's AGI measure on six households, the dependent and no-adult cases included. Null take-up flags that R2's own stages fill are held back, as the AGI does not read them; a null never-default column that the AGI reads is refused, as R2 refuses it. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6a9d' into claude/us-acs-poc-donor-patch-dd6a9d
|
Confirmed, and fixed in Confirmed with the engine. On six test households, policyengine-us 2.2.1 gives an AGI of 0 to a household whose only wages belong to a 10-year-old dependent. It also gives 0 to a household that is one tax unit of a 16-year-old with wages: only an adult can be a tax unit's head or spouse, so that unit's income adds nothing. The first version counted both households. Fix: the check now computes the value R2 stores, not a guess from the income columns.
Cost. The check now loads the full staging frame. On the 766b7c7 ACS local file it took 438 s with an 11.9 GB peak (16 engine chunks). Rerun on the 766b7c7 file ( The flag question stays with María. One fact for it: the check now lists a county whenever R2 would build its AGI row empty, so the first option (today's ruling) covers that refusal. Your note on the flag's footprint stands: with the flag and no empty row, R2 drops nothing, and the run identity, The PR body is updated: item 9, step 4a (which now passes |
|
Written by Claude Code, Max's agent for the county work, not by Max. Facts only: this comment asks for no decision. The work-column commit is on this branch. Tonight's frozen head is The patched copy, files only. Step 1 of the PR's block ran with no manifest options and no work-column options, because the default ASEC cache holds pppub23-25 at the pinned hashes.
Checks.
Materialize memory. The PR's 78-95 GB estimate for materialize is the 9/23 pre-sparse figure ( Receipt (home directory shown as ~){
"added_columns": [
{
"action": "added",
"column": "detailed_industry_recode",
"derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
"dtype": "int16",
"entity": "person",
"source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
"source_item": "WEIND",
"step": "asec_work_columns"
},
{
"action": "added",
"column": "major_industry_recode",
"derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
"dtype": "int16",
"entity": "person",
"source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
"source_item": "WEMIND",
"step": "asec_work_columns"
},
{
"action": "added",
"column": "worked_last_year",
"derivation": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
"dtype": "bool",
"entity": "person",
"source": "pinned ASEC person member of the row's income year, joined on (source_year, PERIDNUM)",
"source_item": "WKSWORK > 0",
"step": "asec_work_columns"
},
{
"action": "added",
"column": "strike_benefits",
"dtype": "float64",
"engine_default": 0,
"engine_dtype": "float",
"entity": "person",
"reason": "no donor or ASEC pppub23\u201326 row has OI_OFF == 12",
"step": "reviewed_zero_columns",
"value": 0
},
{
"action": "added",
"column": "passive_partnership_s_corp_income",
"dtype": "float64",
"engine_default": 0.0,
"engine_dtype": "float",
"entity": "person",
"reason": "the donor holds partnership and S-corp income as one combined amount in `partnership_income`; the split is unknown",
"step": "reviewed_zero_columns",
"value": 0
}
],
"engine": {
"package": "policyengine-us",
"version": "2.2.1"
},
"input": {
"path": "~/reviews/lwi-county-build-2026-11-13/inputs/donor-receipts-20260923/populace_us_2024_receipt_qualified.h5",
"release": null,
"rows": {
"family": 65206,
"household": 57240,
"marital_unit": 130729,
"person": 166321,
"spm_unit": 59900,
"tax_unit": 79729
},
"sha256": "009469727409e58c0c9c1252581292497b688b76f1d59b51947483fb0f89221c"
},
"issue": "microcosm#922",
"kept_columns": [],
"output": {
"path": "~/reviews/lwi-county-build-2026-11-13/runs/poc-run/county-poc/donor/donor_patched_poc.h5",
"rows": {
"family": 65206,
"household": 57240,
"marital_unit": 130729,
"person": 166321,
"spm_unit": 59900,
"tax_unit": 79729
},
"sha256": "438f47f49b4e6e0c72c3adc3bb3156cbef40b4c0160b71738ac939be1d49a67b"
},
"plan": "microcosm#1647",
"preserved": {
"root_attributes": [
"CLASS",
"PYTABLES_FORMAT_VERSION",
"TITLE",
"VERSION"
],
"tables": [
"/_time_period",
"/family",
"/household",
"/marital_unit",
"/person",
"/spm_unit",
"/tax_unit"
]
},
"purpose": "County Files proof of concept only: a patched copy of the published national donor that the ACS staging (R1) accepts. The real County Files donor is a new national build from main.",
"release_manifest": null,
"rewritten_tables": [
"person"
],
"steps": [
{
"carry": {
"donor_work_experience_universe": {
"weind_to_wemind_rows_verified": 166321,
"worker_code_iff_weeks_worked": "verified",
"worker_rows": 85521
},
"function": "microcosm.build.us_runtime.org_wages.derive_us_org_occupation_inputs",
"inputs": [
"PRDTRACE",
"PRDTHSP",
"POCCU2",
"WKSWORK",
"WEIND",
"WEMIND"
],
"reemitted_columns_absent_from_donor": [],
"reemitted_columns_verified_equal": [
"cps_race",
"is_hispanic",
"detailed_occupation_recode",
"has_never_worked",
"is_military",
"is_computer_scientist",
"is_farmer_fisher",
"is_executive_administrative_professional"
],
"require_us_occupation_and_work_experience_inputs": "passed"
},
"clones": {
"distinct_keys": 135833,
"handling": "Every row sharing a key joins the same member row and must match it on every identity column, so clones get identical codes.",
"keys_with_more_rows": 0,
"keys_with_one_row": 105345,
"keys_with_two_rows": 30488,
"repeated_key_channel_composition": {
"asec+puf_tax_detail": 30488
},
"rows": 166321,
"rows_by_support_channel": {
"asec": 66001,
"puf_tax_detail": 100320
}
},
"columns": [
"person.detailed_industry_recode",
"person.major_industry_recode",
"person.worked_last_year"
],
"files": {
"2022": {
"archive_sha256": null,
"form": "csv",
"income_year": 2022,
"member": "pppub23.csv",
"member_sha256": "19b56537e50e7663f954361ef2bb5ce9cef8d9d45f156fe1a69a99b654198ffe",
"member_size_bytes": 281065733,
"official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip",
"path": "~/.cache/microcosm/cps/asec_education/pppub23.csv",
"persons": 146133,
"survey_year": 2023,
"work_experience_universe": {
"weind_to_wemind_rows_verified": 146133,
"worker_code_iff_weeks_worked": "verified",
"worker_rows": 73186
}
},
"2023": {
"archive_sha256": null,
"form": "csv",
"income_year": 2023,
"member": "pppub24.csv",
"member_sha256": "21a2b9e0e4b08534563578a45acad77868af4ae9a7d46f23776b707d4a559aa7",
"member_size_bytes": 277250415,
"official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asecpub24csv.zip",
"path": "~/.cache/microcosm/cps/asec_education/pppub24.csv",
"persons": 144265,
"survey_year": 2024,
"work_experience_universe": {
"weind_to_wemind_rows_verified": 144265,
"worker_code_iff_weeks_worked": "verified",
"worker_rows": 73471
}
},
"2024": {
"archive_sha256": null,
"form": "csv",
"income_year": 2024,
"member": "pppub25.csv",
"member_sha256": "06921fe83fc66c907e6c7b86b82255dc70458ee7d76258fc48297cb34f0c06b5",
"member_size_bytes": 277882549,
"official_archive_url": "https://www2.census.gov/programs-surveys/cps/datasets/2025/march/asecpub25csv.zip",
"path": "~/.cache/microcosm/cps/asec_education/pppub25.csv",
"persons": 142125,
"survey_year": 2025,
"work_experience_universe": {
"weind_to_wemind_rows_verified": 142125,
"worker_code_iff_weeks_worked": "verified",
"worker_rows": 72460
}
}
},
"join": {
"by_income_year": {
"2022": {
"distinct_keys_matched": 44725,
"donor_rows": 54464,
"donor_rows_matched_once": 54464,
"identity_mismatches": 0,
"member_rows": 146133,
"member_rows_not_in_donor": 101408
},
"2023": {
"distinct_keys_matched": 44812,
"donor_rows": 54654,
"donor_rows_matched_once": 54654,
"identity_mismatches": 0,
"member_rows": 144265,
"member_rows_not_in_donor": 99453
},
"2024": {
"distinct_keys_matched": 46296,
"donor_rows": 57203,
"donor_rows_matched_once": 57203,
"identity_mismatches": 0,
"member_rows": 142125,
"member_rows_not_in_donor": 95829
}
},
"donor_rows": 166321,
"donor_rows_matched_once": 166321,
"identity_columns": {
"A_AGE": "A_AGE",
"A_LINENO": "A_LINENO",
"POCCU2": "POCCU2",
"PRDTHSP": "PRDTHSP",
"PRDTRACE": "PRDTRACE",
"P_SEQ": "P_SEQ",
"WKSWORK": "WKSWORK",
"source_household_id": "PH_SEQ"
},
"key": "(source_year, PERIDNUM as exact 22-digit text)",
"multiply_matched_donor_rows": 0,
"unmatched_donor_rows": 0
},
"name": "asec_work_columns",
"source": "pinned Census ASEC person members (spm_role_source.ASEC_SPM_ROLE_SOURCES), read through spm_role_source.read_pinned_asec_person_columns",
"vintage_rule": "source_year is the ASEC income year; its member is survey year source_year + 1 (asec_census_person_columns._resolve_pin)",
"work_experience": {
"detailed_industry_recode_counts": {
"0": 37248,
"1": 1710,
"10": 1401,
"11": 3839,
"12": 1670,
"13": 6940,
"14": 4175,
"15": 7453,
"16": 11679,
"17": 2142,
"18": 6374,
"19": 350,
"2": 335,
"20": 4037,
"21": 3881,
"22": 390,
"23": 43552,
"3": 6491,
"4": 4583,
"5": 2751,
"6": 1509,
"7": 9036,
"8": 4132,
"9": 643
},
"major_industry_recode_counts": {
"0": 37248,
"1": 1710,
"10": 19132,
"11": 8516,
"12": 4387,
"13": 3881,
"14": 390,
"15": 43552,
"2": 335,
"3": 6491,
"4": 7334,
"5": 10545,
"6": 4775,
"7": 1401,
"8": 5509,
"9": 11115
},
"nonzero_shares": {
"detailed_industry_recode": {
"by_source_year_weighted": {
"2022": 0.807369122927107,
"2023": 0.8163839632749547,
"2024": 0.8453736523065007
},
"gate_band": [
0.7,
0.92
],
"unweighted_nonzero_share": 0.7760475225617932,
"weighted_nonzero_share": 0.8244304858584061,
"within_band": true
},
"major_industry_recode": {
"by_source_year_weighted": {
"2022": 0.807369122927107,
"2023": 0.8163839632749547,
"2024": 0.8453736523065007
},
"gate_band": [
0.7,
0.92
],
"unweighted_nonzero_share": 0.7760475225617932,
"weighted_nonzero_share": 0.8244304858584061,
"within_band": true
},
"worked_last_year": {
"by_source_year_weighted": {
"2022": 0.5427875993708351,
"2023": 0.5408310303625891,
"2024": 0.5511269424079637
},
"gate_band": [
0.4,
0.62
],
"unweighted_nonzero_share": 0.514192435110419,
"weighted_nonzero_share": 0.5453081901706771,
"within_band": true
}
},
"weights": "household_weight resolved to persons",
"worked_last_year_rows": 85521,
"worked_rows_with_worker_industry": 85521,
"worker_industry_rows_without_weeks_worked": 0
}
},
{
"columns": [
"person.strike_benefits",
"person.passive_partnership_s_corp_income"
],
"name": "reviewed_zero_columns"
}
],
"tool": "tools/patch_us_acs_donor_columns.py",
"tool_commit": {
"commit": "0005321705f539ef0a031fe345062da9afff7a04",
"tool_dirty": false
},
"version": 1
} |
|
On the two file hashes: I ran the patch tool twice on the same input with the test fixtures (zero-column step, fixed and table layouts). The hashes differ each time, but every differing byte (14 of 9.7 MB) is a 4-byte Unix time in the HDF5 object headers that PyTables writes when the tool rewrites the person table. The data is identical. So a re-run gives a new file hash with the same data; compare tables, not file hashes. No code change for tonight. The PR body now records this, the frozen head, the copy and the 48.9 GB materialize figure. |
|
On when to pass The PR description now says so. Its commands keep one materialize line, form B, with the flag. Step 4a, the zero-AGI check, still runs after R1, and its exit 2 does not stop the run. Post its list on #1647 with any drop receipt from R2. |
María ruled that tonight's County Files proof of concept passes --county-drop-empty-soi-rows on every run (#1647 comment 6950739270), and asked that the code say clearly that this is a temporary setup. The help text of --county-drop-empty-soi-rows and --allow-default-donor-column now says: temporary for the County Files proof of concept (#1647); do not use for the final County Files build. There every county must be supported well: a county whose AGI row would be empty needs a real fix (more household support, or a declared cell with its own ruling), not a dropped target, and the donor is a new national build. The county rung doc now describes the drop flag with the same note, and R1's module docstring, the agent guide and both changelog fragments carry it too. Help text and docs only; no logic changes. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The removal of the County Files proof-of-concept setup now has its own tracking issue, #1661, a sub-issue of #922. Each note from the last commit now says "removal tracked in microcosm#1661": the help text of --county-drop-empty-soi-rows and --allow-default-donor-column, R1's module docstring, the agent guide, the county rung doc and both changelog fragments. This PR does not finish #1661. Help text and docs only; no logic changes. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Upload list for tonight's run, so that a failure can be fixed and re-run elsewhere If a stage fails tonight, María (on GCP) or Pavel (on his machine) should be able to fix it and re-run from the failed stage, not from the start. That needs files that today exist only on the run machine. Please upload them as listed below to the private dataset repo Before the run
After each stage Why these files are enough to restart at the failed stage (from the code at
To re-run on another machine
|
|
Lines refer to
When R2 drops a county's AGI row, calibrate grades that county without it (
|
…check The zero-AGI pre-check set its exit status from at_risk alone. R2's check_support refuses a county outside the ladder and a ladder county without a household-universe record, with or without --county-drop-empty-soi-rows, in materialize after the staging load and the SSI/Medicaid pre-pass. With --puma-ladder the check now lists both (counties_outside_ladder, unsupported_counties) with R2's set rules, moves no household when a county is outside the ladder (as R2 does), and exits 3, which takes precedence over 2. The report records its exit status. Part of #922; review of #1660 (comment 6955224433). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --county-drop-empty-soi-rows, calibrate grades a county without its dropped row, and only the reliability report's top-level unsupported_targets named it. Materialize now records each dropped row's county cell in the run identity (zero_measure_soi_rows_dropped; only a flagged run has the key), and the bound report names the row in every member county's cell and in a zero_measure_soi_rows_dropped column of county_reliability.csv. A run without the flag keeps its identity, report and CSV bytes. A flagged identity from older code that dropped rows but lacks their cells is refused (re-run materialize), never left unmarked. The flag's test now runs on through calibration_evidence, write_reports and package_reports. Part of #922; review of #1660 (comment 6955224433). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reviewed-fill pre-check filed any ACS-only null in a never-default column under natively_filled. R2's stages fill only the take-up stage's SPM-unit take-up flags and engine-free fills and the SSI and Medicaid take-up flags; R2's reasons for the immigration, disability, income transfer and SSI disability-criteria columns say to re-run staging, so such a null passed the pre-check and then stopped materialize. The check now takes the stage-filled set from R2's own constants (stage_filled_never_default), refuses every other never-default null on any spine with a re-run-staging message, and records the set in its report. A test ties the set to R2's own reasons. Part of #922; review of #1660 (comment 6955224433). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The donor-patch tool never tested whether --output-h5, the receipt or the copy release manifest existed, so a second run replaced them. Two runs give different file hashes (the HDF5 object headers record write times), so a re-run after R1 left the staging summary's base.sha256 naming bytes no longer at its path. The tool now refuses an existing output before any work, and --overwrite allows the replacement; its help says to re-run R1 on the new copy. Part of #922; review of #1660 (comment 6955224433). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
All five findings confirmed. Four are fixed in code (head
None of the four commits changes R1 code, so R1's staging from Tests on Python 3.14, the touched files only: AGI check 12 passed; county surface and production 196 passed, 1 skipped; fill pre-check 8 passed; patch tool 40 passed. |
|
Lines refer to
The engine-tier test holds back a null Neither point bears on tonight's run. |
The zero-AGI check held back every null NEVER_DEFAULT_FILLED column that the AGI does not read, on either spine. Run alone, it passed a file that R2's first fill refuses. It now takes the stage-filled set from the reviewed-fill pre-check (stage_filled_never_default) and holds back a null column only when an R2 stage fills it, every null is on an ACS row (the <entity>_spine tag) and the AGI does not read it. It refuses every other never-default null, as R2's fill step does: a null on a donor row, in a column no R2 stage fills, in one the AGI reads, or without a spine tag to place it. An engine-tier test checks that no stage-filled column is an input of the AGI closure, so an engine release that changes that fails in CI. Part of #922; review of #1660 (comment 6956136956). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both points confirmed and fixed in
Tests on Python 3.14: AGI check 15 engine-free tests passed; 2 engine-tier tests passed (103 s). |
|
Written by Claude Code, Max's agent for the county work, not by Max. Facts only: this comment asks for no decision and recommends none. Tonight's run will start agi and materialize at 75 GB of free disk instead of 80, on Max's ruling. It will probably end under the 60 GB floor in your comment above.
|
|
Facts only, on the disk figures in comment 6959513812. This reply asks for no decision and recommends none. After materialize, R2 writes about 19.6 GB more with the PR's commands. So free disk is lowest after package, not after materialize.
What the code does about the copy: package skips it when mkdir -p "$RUN/release" && ln "$RUN/populace_us_2024_acs_local.h5" "$RUN/release/microcosm_us_2024_acs_local.h5"Package refuses an existing path only at These figures need no change to the code or to tonight's head. The stop line and the choice of what to do are Max's. |
|
Request to Max and Max's agent: the run's outcome, the restart files, and the county file Thank you for the 23:57Z update on #1647. The zero-AGI check exited 0, and materialize passed at 23:49Z with no SOI row dropped. We have seen nothing since then about calibrate, qa, finalize or package. 1. Please post the outcome of each later stage (calibrate, qa, finalize and package):
2. Please upload the restart files, so that María (on GCP) or Pavel can fix a failure and re-run from the stopped stage. At 08:40Z on 18 Nov, the private staging repo
If you used a bucket instead, please post its location. 3. If package passed, please also upload the county file for inspection, in the same run folder:
Please upload these files to the staging repo only, and do not run the publish command that package prints. This proof-of-concept release records |
|
Written by Claude Code, Max's agent for the county work, not by Max. Facts only. The run's outcome: every stage passed; package ended at 2026-11-18T01:18:36Z Answering comment 6973462140, item 1.
Uploads (items 2 and 3). Nothing is uploaded yet. The upload needs Max's go, which has been asked for. Once he gives it, the run folder goes to
At this morning's measured rate the whole upload takes about 2.5 h. The publish command will not be run. A fuller run report for the PoC report, with the fit numbers, follows on #1647. 🤖 Generated with Claude Code |
|
Written by Claude Code, Max's agent for the county work, not by Max. Facts only. Upload: batch 1 of 4 is in the staging repo This follows up comment 6979361950, which said the upload awaited Max's go. He gave it at about 13:05Z.
🤖 Generated with Claude Code |
|
Written by Claude Code, Max's agent for the county work, not by Max. Facts only. Upload: all four batches are in the staging repo This follows up comment 6980005693 (batch 1), answering comment 6973462140, items 2 and 3.
🤖 Generated with Claude Code |
Draft v0.1 of the proof-of-concept report: the fit and the limits of the 2024 county file from the 17-18 November run. It is pe-doc Markdown, with the numbers from the run report on #1647 and the release's own reports, and three placeholders still open (the methods link twice, the schema documentation). Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
PoC report: first draft pushed ( The first draft of the PoC report for the Living Wage Institute is on this branch: What it contains
What it is missing
Decision: we share the file through a new Hugging Face repo. The file is in the private dataset
Question for Max: do you have a report template? If you have one you would like us to use, please point us to it. If not, we will build the report with |
Takes the corrections that part 2 of the methods draft lists (#1647 comment 7005741386). The ACS county tables are 2020-2024 5-year; only some targets equal published figures; the ACS checks compare with the file's state totals split in the ACS county proportions; county counts are of households in housing units; and the report now names three temporary shortcuts. The checks line says what gate_summary.json records: all 17 blocking checks passed, and the district evidence check flagged 275 of the 436 districts. The schema documentation is a deliverable, not a stretch goal, and the placeholders are gone: the report names the methods and schema documents instead of linking into the private repository. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds, next to the PoC report, the three documents microcosm#1647 comment 7006535601 asks for: - schema_manifest.json, as tools/us_release_schema_manifest.py wrote it for the 2024 county file (SHA-256 82314a3b...bd3c); - schema_manifest.md, the tool's output with a notes section added at the top. The notes say that six household geography columns hold empty strings, not nulls, for all 1,529,668 ACS households (and for some donors, including every Connecticut donor's cbsa_code); which columns are raw CPS ASEC or ACS fields, build identifiers or derived columns, and which 198 policyengine-us 2.2.1 reads; and why three columns were removed and 134 dtypes changed; - lwi-poc-methods.md, part 1 of the methods draft as posted in #1647 comment 7005740491, without the comment's preamble. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max's agents committed part 1 of the methods draft as posted (#1647 comment 7005740491). This edits it for the Living Wage Institute: pe-doc front matter and title block; an appendix of build identifiers, settings and input digests in place of the links to GitHub comments, which a reader with only a download token cannot open; flags, run-file names and imputation family names out of the text; and the gate section reworded. The donor households' geography and tax-detail records are now described from the national build's code at cae8640, in place of "we could not establish". Max's agents are checking that account against the 28 July build's run logs (#1647 comment 7022836029). The experiments README lists the three documents. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 254 columns policyengine-us 2.2.1 ignores are 213 raw survey fields, 29 build identifiers and 12 derived columns, among them county_fips_2023, as the notes in schema_manifest.md count them. The report said "raw survey fields and build identifiers" (#1647 comment 7022836029). Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent check of the notes found three points to fix: - medicare_part_b_premiums: drop the claim that it held the CPS ASEC field PEMCPREM. On tax-detail copies the donor build may have set it from the PUF instead. policyengine-us 2.2.1 computes the Part B premium itself (medicare_part_b_premium). - A_EXPRRP is the Census value only for income-year 2024 donors. For income years 2022 and 2023 the donor build derived it from line, spouse and parent numbers (base summary relationship_recode_source). - CENSUS_TAX_ID is the Census TAX_ID, which PolicyEngine's CPS extractor copies before it overwrites TAX_ID with the tax unit it builds. The tables and schema_manifest.json are unchanged. Part of #922. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #922. Plan: #1647. Gate report from Max's agent: #1647 comment 6937139936. Amended ruling 1: #1647 comment 6945072508.
Temporary: proof of concept only
Tracked in #1661: remove before the final County Files build.
Important
Remove the always-on drop flag before the final County Files build. Passing
--county-drop-empty-soi-rowson every run is for tonight's proof-of-concept run only, so that one empty AGI row cannot stop R2 in a window with no room for a rerun (María's ruling, #1647 comment 6950739270). It is not part of the County Files product. In the final build, every county must be supported well: a county whose AGI row would be empty needs a real fix, for example more household support (minimum support that also covers counties whose SOI rows would be empty) or a declared cell with its own ruling, not a dropped target.Also for the proof of concept only:
tools/patch_us_acs_donor_columns.py): the zerostrike_benefitscolumn, the zero partnership/S-corp split (passive_partnership_s_corp_incomeadded as 0,s_corp_incomeall 0), and R1's two--allow-default-donor-columnallowances. The final donor is the new national build frommain, which carries the real split and the work columns.--donor-release-manifest, so the release recordsdonor_release: nulland is not published.e49d53997): the run identity'szero_measure_soi_rows_droppedkey, and the reliability report's cell key and CSV column of that name. Only a flagged run has them, so they go with the flag.Not temporary: keep these.
strike_benefitsreviewed fill on ACS rows (commit 3, an entry inacs_local_reviewed_fill_consumers.yaml). It is general and correct for any current donor: nothing transfersstrike_benefitsto ACS rows, so every ACS person reaches R2 with a null, and R2 fills the engine default, 0. Without the entry, R2 refuses at finalize for a staging built from any current national donor. Do not remove it with the proof-of-concept setup.3b28e315c,a5ca4df65,28e7b7ce6andf8b1039f0). In the final build, the zero-AGI check lists the counties that need a real fix before R2 runs.The help text of both flags, R1's module docstring, the agent guide, the county rung doc and both changelog fragments carry the same note, with "removal tracked in microcosm#1661" (
f96ff8e1band65b2155c0, help text and docs only). This PR does not finish #1661.Purpose
R1, the ACS staging (
tools/_legacy/build_us_acs_multispine_base.py), refuses 766b7c7's national donorpopulace-us-2024-spm-receipts-20260923(sha25600946972…) at three donor gates. Under the amended ruling, the County Files proof of concept runs R1 on a patched copy of that donor. This PR adds the patch tool, one explicit R1 option, the reviewed-fill register entry that R2 needs for the copy, an optional release manifest for the copy, a pre-R2 check for counties with an empty SOI AGI row, and an opt-in R2 path that drops such a row with a receipt.Decisions confirmed by María (17 November, #1647 comment 6946488444):
The
strike_benefitsfill verdict staysharmless(commit 3).Tonight's proof of concept is files only: R1 runs without
--donor-release-manifest, R2 recordsdonor_release: null, and the release is not published. The manifest option (commit 4) stays opt-in and is not used tonight.The drop flag is on for every run tonight (#1647 comment 6950739270). R2's materialize always runs with
--county-drop-empty-soi-rows(commit 6). This replaces the earlier pre-authorization (#1647 comment 6947604927), which passed the flag only when the zero-AGI check listed a county, and it answers Max's question in review comment 6949865318.county_surface.json(zero_measure_soi_rows_dropped), the run identity and the reliability report. A county with no households is still refused.The published donor is read only. It does not change.
The patch is for the proof of concept only. The real County Files donor is a new national build from
main, located withblock_v1.R1 and R2 can run from this PR's head tonight. Do not merge before review.
Commits
1.
44a11af9d: donor-patch tool,tools/patch_us_acs_donor_columns.py_load_base_framereads. It starts from a byte copy of the input and rewrites only the entity tables that gain columns, through the shared frame-table codec (read_frame_table/put_frame_table), in the stored format. A table-format table keeps its data columns. The published donor's person table is a pandas table-format compound table.strike_benefitsandpassive_partnership_s_corp_incomeas 0 on the entity that the US engine gives them (person). It refuses a column whose engine default is not 0.refuse_denied_pool_h5).<output>.receipt.json): input and output SHA-256, rows per entity, each added column with its entity, dtype and reason, and the tool's commit.2.
601cf86cc: R1 option--allow-default-donor-column COLUMN=REASON(repeatable)_require_dense_donor_coverageskips only the "every observed donor value equals the engine default" failure for a listed column.allowed_default_donor_columns(entity, engine default, reason),orchestration.allow_default_donor_columns, and the reviewed limitationallowed_default_valued_donor_columns. R2's finalize copies the staging limitations into the release'sreviewed_limitations.tools/modal_us_stage_plan.pyowns the flag and never passes it, as for the location flags (US household location v1: one census block per household, every geography derived #1047). The runbook says so, and the agent guide names this PoC as the one reviewed exception to "rebuild historical donors".3.
77f5fa163: reviewed-fill register entry forperson.strike_benefitsstrike_benefitsas a component (since d5aa4e2), so R1 requires every donor to carry it. Nothing transfers it to ACS rows, so every ACS person reaches R2 with a null value. R2 fills the engine default, 0, and finalize's reviewed-fill consumer gate then refuses the fill as unregistered (package refuses it too). Without this entry, R2 would fail at finalize, after materialize, calibrate and qa.harmless. The verdict is a judgment for María to confirm. The reason: no pinned ASEC respondent reports code 12, so every donor value is 0, and a transfer from the donor would also draw 0. The alternative isknown_bias/overstates; the coupling rules would then also apply to SNAP (via TANF) and to the PTC (via Medicaid and CHIP).36601cb3e6ff…to692564a40b30…. R2 binds the code commit (calibrate records it, package refuses another commit or a dirty tree), so run R2 from a head that includes this commit.4.
b9e7f4010: optional release manifest for the copy (the ruling's "manifest for the copy")--release-manifest PATH --manifest-release RELEASE --manifest-build BUILD(and--manifest-repo-id, defaultpolicyengine/populace-us-staging) writes a release manifest in the build-folder layout. Its one microdata artifact is the copy; the receipt is a second artifact.--donor-release-manifestthen verifies the copy's SHA-256 against it and records the copy's build asdonor_release.referenced_buildand the local-area release contract's donor check accept that record (tested), so the publisher no longer refuses the R2 release for a nulldonor_release.--input-release-manifestchecks the input against the published donor's release manifest and records that build in the receipt and in the copy's manifest (derived_from).dataset_roleisproof_of_concept_donor_copy, which is not a release role, so the copy's folder cannot pass as a release itself. The manifest names files, not local paths.5.
06b53f625: pre-R2 check for counties with an empty SOI AGI row,tools/check_us_acs_county_agi_support.py(#1630 review, finding 1)attach_surfacebuilds each county's SOI rows from the state rows' stored entries, and the state AGI row stores no zero. A county whose households all have AGI of exactly 0 gets an empty AGI row, and R2 refuses after the materialize engine pass (about 1.5 h in). Minimum support fills only counties with no household.3b28e315c(item 9) the check computes the AGI that R2 stores, with the engine. The first version read only the AGI income columns and counted any person's nonzero value, so it missed income that the engine does not count.--puma-ladderit takes the county list from the ladder and applies R2's own minimum-support moves first. It exits 2 when a county is at risk and writes a JSON report.6.
78acba5f5: R2 option--county-drop-empty-soi-rows(off by default)county_surface.json(zero_measure_soi_rows_dropped: row, county cell, concept, target value, households), the run identity (county_targets_dropped,county_drop_empty_soi_rows), the reliability report and the refresh recipe. Without the flag nothing changes, the identity bytes included.DECLARED_SOI_COMBINED_CELLS), covers only a county absent from the SOI facts; a county present in them would need new code as well as a ruling.7.
46674e01f: post-R1 check of R2's reviewed-fill rules,tools/check_us_acs_local_reviewed_fills.pyreviewed_engine_input_nulls, names and counts only). It refuses what materialize refuses: a never-default-filled null off the ACS spine, and a null ESI premium. It builds the fill records R2 writes, with the defaults R2 resolves, and runs R2's own finalize gate on them, in seconds. It exits 2 when R2 would refuse.8.
3bffc4aed: CI fix for commit 2--allow-default-donor-column. It now carries the parser's default (None), as it does for the location defaults. Test-only change.2737c32fbthen mergesmain(Belgium and Route A changes only; no file in this PR).9.
3b28e315c: the zero-AGI check computes the AGI that R2 stores (review fix, comment 6949865318)irs_gross_income, and only an adult can be either. Income held by a dependent, or by a tax unit without an adult (one 16-year-old with wages), adds nothing, so such a household has an AGI of 0. The first version counted it, and could exit 0 for a county that R2 then refuses.project_input_only), fills the reviewed nulls (fill_reviewed_nulls), builds each household chunk's dataset (_dataset_from_frame) and computes the household sum ofadjusted_gross_income. An engine-tier test checks that sum against the production SOI materializer's AGI measure on six households: a dependent's wages, a no-adult tax unit, a spouse's wages and a negative AGI included.--staging-summarydefaults to the H5 path with.summary.json, as in R2. The check needs memory for the full staging frame.72b7072bfmerges the work-column head (000532170) into this commit's branch.10.
f96ff8e1b: the drop flag and the donor allowance are marked proof-of-concept only (María's request with the ruling of #1647 comment 6950739270)--county-drop-empty-soi-rowsand--allow-default-donor-columnnow says: temporary for the County Files proof of concept (microcosm#1647); do not use for the final County Files build. The county rung doc describes the drop flag with the same note, and R1's module docstring, the agent guide and both changelog fragments carry it.72b7072bfand a run fromf96ff8e1bare the same.11.
65b2155c0: the notes name the removal issue12–15: review fixes for comment 6955224433. They change no R1 code, so R1's staging from
72b7072bfstays valid.12.
a5ca4df65: the zero-AGI check exits 3 when R2 would refuse whatever the drop flagcheck_supportrefuses a county outside the ladder and a ladder county without a household-universe record, with or without the drop flag, in materialize after the staging load and the SSI/Medicaid pre-pass. With--puma-ladder, the check now lists both (counties_outside_ladder,unsupported_counties) with R2's set rules. As R2 does, it moves no household when a county is outside the ladder.exit_status.13.
e49d53997: each dropped SOI row is named in its county's reliability cell and CSV rowzero_measure_soi_rows_dropped). The bound reliability report names the row in every member county's cell, and in azero_measure_soi_rows_droppedcolumn ofcounty_reliability.csv.calibration_evidence,write_reportsandpackage_reports.14.
28e7b7ce6: the fill pre-check accepts an ACS-only never-default null only where an R2 stage fills itnatively_filled. A test ties the set to R2's own reasons.15.
2e9bf301d: the patch tool refuses to replace an existing copy, receipt or copy release manifest unless--overwriteis passed--overwrite, re-run R1 on the new copy.16.
f8b1039f0: the zero-AGI check holds back a null never-default column only where an R2 stage fills it (review fix for comment 6956136956; it changes only step 4a's check)stage_filled_never_default). It holds back a null column only when an R2 stage fills it, every null is on an ACS row and the AGI does not read it, and refuses every other never-default null.17.
c990afa56: the first draft of the PoC report for LWI (comment 6987263616)experiments/us-county-poc-20261117/lwi-poc-report.md, draft v0.1 inpe-docMarkdown, with a short README. It states the fit and the limits of the 2024 county file, from the run report on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 and the release's own reports. It changes no code.policyengine/lwi-county-files.Ask: the work-column commit from Max's agents (pushed as
36bd0dafa)Please add your verified work-column step to this tool as the next commit, on top of
2737c32fb:asec_work_column_stepintools/patch_us_acs_donor_columns.py. It now raisesNotImplementedError; its docstring gives the contract. It restoresdetailed_industry_recode,major_industry_recodeandworked_last_yearfrom the pinned ASEC person files, matched on source year andPERIDNUM, and returns oneColumnPatchper column with its source in the receipt.build_patch_stepsbefore the zero step, with its command-line options in_parse_args._require_donor_work_experiencepasses on the patched fixture (the file has the same check for_require_income_donor).Run result (17–18 November)
Every stage passed, at the heads this description names. Package ended at 01:18:36Z on 18 November. Sources: Max's agent, comment 6979361950, and for the zero-AGI check and materialize, #1647 comment 6964605746.
72b7072bf. The fill pre-check through package ran atf8b1039f0. The tree was clean before every stage, andbuild_manifest.jsonrecordsbuild_dirtyfalse.zero_measure_soi_rows_droppedis empty.populace_us_2024_acs_local.h5, SHA-25682314a3b5a6a95762faf6eb4f5b499ce56608ad6a39a27b75e11c75b4945bd3c, 10,637,811,669 bytes. Build idpopulace-us-2024-buildo-acs-local-f8b1039f00-20261118T011728Z. It is not published.gate_summary.jsonpassed.calibration: final loss 0.015904 against an initial 0.020786; 75.7% of 16,440 trained rows within 10%; realized largest weight ratio 4.93 (cap 5).district_ess_collapseis report-only and records 271 of 436 districts collapsed.cd_holdouthas 0 targets. This is expected: the holdout fraction is 0 for every--soi-modeexceptstate_cd, and the run usedstate_county.county_reliability.jsongrades 3,144 counties, under a provisional rule: A 0, B 3, C 103, D 3,038, U 0. No target is unsupported. Minimum support made 9 moves, into 15005, 31009 and 48301.waitreturned 19 at R1's end, but the guard recorded exit 0 and every staging gate passed. So R1's done record was written by hand.policyengine/populace-us-staging, atruns/lwi-county-poc-20261117/, with the SHA-256 listreport/sha256-outputs.txt. The publish command will not be run.d8a772ce, 13:28Z, comment 6980005693): 73 files, 1.12 GB. They are the release folder (15 files, with the county reliability JSON and CSV),package_result.json, the 23 checkpoint files, both pre-check reports, the staging and output summaries, the donor receipt and every stage log.county-poc/populace_us_2024_acs_local.h5(518c509713b3, 13:58Z), R1's stagingcounty-poc/acs_multispine_staging.h5(7ba16a15413f, 15:00Z), and the donor copy, the feed and the county facts (159a42f8a619,66aaed953c40andc88bfbf16ae4, 15:01–15:03Z). The feed and the county facts are ininputs/.c88bfbf16ae4, with no download: the folder holds 78 files, 25.5 GB. The LFS SHA-256 of the calibrated file (82314a3b…), the donor copy (438f47f4…), the feed (1d45a89c…) and the county facts (67b0998a…) equal the values pinned above. R1's staging (11,472,744,092 bytes,083290dd…) has no pinned value here; Max's agent checked it against the run's SHA-256 list.Tonight's run: exact commands
Run everything from one frozen head, the final head after the work-column commit, in a fresh worktree, while other work commits elsewhere. R2's calibrate and package must run at the same commit, and the checkout must not change while calibrate runs.
uv run --no-synckeeps uv from touching the lock mid-run. Keep at least 60 GB of disk free.--epochs 800,--epoch-batch 400,--batch 5000,--seed 0and--sample-fraction 1.0are R2's defaults, named for the record.--geographies,--soi-modeand--county-columnare not defaults and must be passed. The county flags match the county rung doc;--stage allwith the same flags is its single-command form.--county-drop-empty-soi-rowsgoes on materialize only, on every run tonight. It drops only rows that are really empty; a county the check lists but whose row has an entry keeps its row. The receipt iscounty_surface.json→zero_measure_soi_rows_dropped, an empty list when no row is empty. In that case the data do not change, and the run identity,materialize_rss.jsonand the refresh recipe record the flag.git status --porcelainmust print nothing before calibrate and package. R2 counts an untracked file as a dirty tree: calibrate records it without refusing, and package then refuses (comment 6955224433). Keep$RUNand the logs outside the checkout.--allow-dirty(package, recordsbuild_dirty: true),--district-ess-gate-blocking, and any manifest option.tools/route_a/supervise.pyis reused, raise its 72 GiB RSS cap. Newer figure from Max's agent (comment 6950580220): the sparse-era materialize peaked at 48.9 GB RSS on 16 Oct, and tonight's run caps materialize at 112 GiB.--out-h5(9.8 GB in both earlier full builds), and package copies it to$RUN/release/microcosm_us_2024_acs_local.h5, a second 9.8 GB. Package skips that copy when the path already holds a file with the same SHA-256, such as a hard link made before package (tools/build_us_acs_local_release.py:6788-6793). QA and finalize write only reports. Measured in the run: the calibrated file was 10.6 GB, the lowest free disk was 64.64 GB, and package did not copy because an APFS clone already held the same SHA-256 (comment 6979361950).Pre-check results (17 November)
00946972…) was patched (copyb3c57fbd…, 108 s, 3.3 GB peak), then R1's donor gates ran in R1's order. All pass except_require_donor_work_experience, which names exactlydetailed_industry_recode,major_industry_recodeandworked_last_year: the work-column commit fixes it. The coverage gate passes with the two allowances and refuses without them, naming onlys_corp_incomeandpassive_partnership_s_corp_income.would_claim_wicare complete Booleans. The ESI premium column is absent, so its refusal cannot trigger.populace_us_2024_acs_local.h5, SHA-2563ce781c4…,county_fipsbasis, pinned ladder), rerun with the engine check of3b28e315c(438 s, 11.9 GB peak, 16 engine chunks): no county at risk. R2's minimum support would move 9 households into 15005, 31009 and 48301. Ten counties have fewer than three households with a nonzero AGI: 31005, 31103, 31183, 35021, 38087, 48033, 48261, 48263, 48269 and 48311, the same ten as the first version found, with the same counts. The engine's rule lowers the count in 312 other counties, none to zero. That file is R2's output and has no null engine input, so the run built its fill register from the file. Tonight's staging draws ACS counties again, so run the check on it.county_fips_2023, the pinned ladder gives 3,144 counties, all with PUMA overlaps, including Connecticut's nine planning regions and Kalawao.What remains for Max's agents
36bd0dafa.3b28e315c(now72b7072bf) in a fresh worktree. Patch the donor there and post the copy's SHA-256 and receipt here. Done (comment 6950580220): frozen head72b7072bf; copy438f47f4…made at000532170(receiptb6430abc…). Only the AGI pre-check's four files change from000532170to72b7072bf.f96ff8e1band65b2155c0(items 10 and 11) add help text and docs only, so72b7072bf,f96ff8e1band65b2155c0all give the same run. Items 12–16 (headf8b1039f0) change the two pre-checks, the patch tool's re-run refusal and R2's reliability marks on a flagged run, and no R1 code. R1's staging from72b7072bfstays valid; to use the fixes, run steps 3–4b fromf8b1039f0, every R2 stage from that one head.--hours-under15-policy. Done at72b7072bf: 241.6 min, exit 0.f8b1039f0: exit 0.--county-drop-empty-soi-rowswhether the check exits 0 or 2 (María's ruling, #1647 comment 6950739270), and post thecounty_surface.jsondrop receipt on County Files: a proof-of-concept county-calibrated dataset by 23 November #1647 with the list. Done (#1647 comment 6964605746): exit 0, and materialize dropped no row.git status --porcelainbefore calibrate and package. Done (comment 6979361950). Every stage passed.policyengine/populace-us-staging(comment 6973462140, items 2 and 3). Max gave the go at about 13:05Z. Done: all four batches are in, the last atc88bfbf16ae4(15:03Z, comment 6982412846). Nothing is published.What each patch means for the data
strike_benefits= 0 on every donor person. This is the true value: no row in the donor or in ASEC pppub23–26 hasOI_OFF12. The column is a component of the income pass's__acs_transfer_other_incomepredictor. It is not a transfer target, so ACS rows carry it as null in staging, and R2 fills the engine default, 0. Commit 3 registers that fill, so finalize accepts it.passive_partnership_s_corp_income= 0 ands_corp_income= 0 on every donor person. Here 0 means "split unknown", not "no income".partnership_incomeholds the combined partnership and S-corp amount (by the code that built it, the PUF's nonpassive-only sum that US frame partnership/S-corp income is nonpassive-only: the donor builds it from E25980-E25960 + E26190-E26180, not line 32 (E26270) #1239 describes). Both columns are QRF targets (puf_tax_itemization). A QRF fit on an all-zero target returns only 0, so ACS rows get no split either: their S-corp income stays insidepartnership_income.semp_covers_partnership_income=True. That rule sets thepartnership_incomedraw to 0 on the ACS rows whoseSEMPalready holds own-partnership income. With this donor, the draw is the combined amount, so on those rows the S-corp share goes too. The transfer's docstring names this case: a caller whose donor carries the combined column with an all-zeros_corp_income"must refuse that donor itself (the legacy builder's dense-donor gate does)". The allowance ons_corp_incomelifts that guard for this run only, and the reviewed limitation records it.Defaults
--allow-default-donor-column, R1 calls the gate exactly as before, and its outputs and summary are unchanged. The tests check this.--county-drop-empty-soi-rows, R2 is unchanged, its run identity bytes included.donor_release: nullConfirmed for tonight: R1 runs without
--donor-release-manifest. The staging summary'sbaseblock records the copy'spath, itssha256(the patch receipt'soutput.sha256),rows,household_weight_totalanddonor_release: null. The patch receipt links the copy to the published donor (input.sha25600946972…).Nothing refuses a null
donor_releasebefore publication. Finalize names the donor "unknown donor id" and still adds the inherited microcosm#1178 Social Security limitation. Package writes null intobuild_manifest.json,us_source_coverage.jsonandrelease_manifest.json. The publisher's local-area contract (validate_release_dir→_validate_local_area_release_dir) refuses a nulldonor_release, so this release is a file hand-over, as ruled. The manifest option of commit 4 would record the copy's build instead; it is not used tonight.R2: later gates and the reviewed-fill register
s_corp_incomeandpassive_partnership_s_corp_income: nothing in R2 refuses them.s_corp_incomeentry predates US PUF donor carries Schedule E line 32 as one number: s_corp_income is an exact-zero universe, so Table 1.4's partnership and S-corp legs cannot bind by entity #1258. With nos_corp_incomefill in this run, the gate reports it as unused; it does not fail.s_corp_incomeunconstrained and fails the passive column only where it is nonzero without line-32 income, so all-zero passes.strike_benefitsexclusion belong to the national fiscal-refresh lane.strike_benefits: the reviewed-fill gate refused it. Commit 3 adds the entry.person.strike_benefitsfill (value 0, ACS spine) against both registers.main's register fails it ("a reviewed default fill (0) the release applied has no entry"); this branch's passes.Records that stay misleading for this run (none refuses; possible follow-ups):
input_coverageentry says it "binds donor_certified_release" and that coverage is enforced on the certified donor release. With two allowances on a patched donor, that is not true.donor_sparse_selection_training_setlimitation calls the donor "the certified Build O sparse release (unknown donor id)".--donor-release-manifestand no allowances, so it does not reproduce this staging.staging_orchestrationdoes not copyallow_default_donor_columns. The allowances reach the release only through R1'sallowed_default_valued_donor_columnslimitation.native_acs_source_universe_blankslimitation listsstrike_benefitsas a native ACS blank. It is a donor-only column that nothing transfers. This is true of any current staging, not only this one.Tests
Run locally on Python 3.14 (the new tools' tests also on 3.13). CI runs the full suite.
-n 4), before commit 7; commit 7's tests: 6 passed.a5ca4df65; the stage-filled hold-back fromf8b1039f0), and 2 engine-tier tests with policyengine-us 2.2.1 (its AGI equals the production SOI AGI measure; no stage-filled column is an AGI input). Drop flag: the default refuses an empty AGI row; the flag drops it with a receipt, the identity key and the recipe flag; a county without households is never dropped. Frome49d53997the test runs on through calibration and packaging: the dropped row is named in its county's cell and CSV row; a combined cell marks every member; an older flagged identity that dropped rows is refused.-n 4); fill pre-check 8 passed; patch tool 40 passed (-n 4). Item 16: AGI check 15 engine-free and 2 engine-tier tests passed.ruff check,ruff format --checkandtools/ci_test_plan.py verify: clean.🤖 Generated with Claude Code