Skip to content

Make the county rung a production county calibration for the County Files - #1630

Merged
juaristi22 merged 21 commits into
mainfrom
lwi-county-rung-production
Nov 17, 2026
Merged

juaristi22 merged 21 commits into
mainfrom
lwi-county-rung-production

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Draft, stacked on #1614 (the county facts pin); its base is lwi-county-facts-pin. Part of #922, the Living Wage Institute County Files: the 2024 county calibration that the 2026 and 2027 files are projected from.

What this does

Makes the ACS-local county rung (--geographies state,cd,county --soi-mode state_county) a production county calibration. Logic lives in tools/us_acs_local_county_surface.py; tools/build_us_acs_local_release.py only calls its hooks.

  • Training targets:

    • County population from the ladder, scaled within each state to the state's PUMA total.
    • IRS SOI county shares of each materialized state concept. County rows split the state's stored row, so they need no new engine columns.
    • ACS county households (B11001) under new ACS state household targets that sum the county rows.
    • Every county row set reconstructs its state row.
  • Held out, scored only: four ACS county families:

    • earnings (B19061)
    • households with earnings (B19051)
    • approximate household income (B19025)
    • its brackets (B19001)

    Each is scored against a pro-rata baseline (the state target times the county's population share). They never enter the loss, the target registry, the matrix or the weights; a test checks that changing them moves no loss input or weight.

  • Reliability: every county gets an A-D grade (U when it has no household support) in county_reliability.json and a packaged county_reliability.csv. A grade comes from supporting rows, distinct source households, Kish ESS, and training and holdout error. ACS group-quarters records count toward neither support nor ESS. The A-D thresholds are provisional and recorded as such.

  • Minimum support: --county-min-support K (default 3, 0 off, at most 50) fills each county the staging frame leaves empty with exactly K same-PUMA households. Households are ranked by a salted hash, so the result does not depend on row order. Only the county columns are rewritten, and every move is recorded in county_support.json and bound into the run identity. Counties still without support, including group-quarters-only counties, are refused before the engine pass.

  • Kalawao (15005) shares one declared SOI cell with Maui (15009), for the SOI families only.

  • Connecticut planning regions are the county equivalents. Under --county-column county_fips_2023, the county populations and the minimum support step's PUMA overlaps come from the ladder's tract overlaps, regrouped by each tract's 2023 county-equivalent (c5c7b4697; see below).

  • docs/us-acs-local-county-rung.md documents the surface, the evidence, the grades and the sizes. That adds 12,625 county and ACS state household rows (about 17,492 with population and district rows) and at most 7,944,270 matrix nonzeros (about 63.6 MB) for 1,588,854 households.

Round-1 review fixes (ddc49578b, edc871ab8)

The independent review returned REQUEST_CHANGES: one P1, one P2, and one pre-existing P2 that is out of scope.

  • P1: group quarters re-entered reliability after the checkpoint reload. Fixed.
    • The lean checkpoint dropped TYPEHUGQ, so support_universe() counted every ACS record. In the reviewer's case, 40 households and 360 group-quarters records went from ESS 40 and grade C to ESS 400 and grade A.
    • A run with county targets now keeps TYPEHUGQ in its lean checkpoint. A run without them writes the same columns as before, so its checkpoint bytes do not change.
    • calibration_evidence grades with reliability_universe(). It refuses a frame whose ACS records lack TYPEHUGQ, such as a checkpoint written before this change, and checks the mask against the run's identity-bound validation measures.
    • The new test round-trips the real checkpoint writer and loader.
  • P2: no driver integration test covers nonzero support moves. Fixed in edc871ab8.
    • test_support_moves_reach_materialize_the_checkpoint_and_the_export puts eight eligible households and one group-quarters record in 01001, none in 01003, and sets K=3.
    • It checks the moved counties in the frame the engine pass sees, in the support receipt and run identity, in the lean checkpoint after a reload, in reliability after the reload (01003 counts 3, 01001 counts 5 without the group-quarters record), and in the consumer export, which reloads unmoved staging.
    • The same commit fixes the P1 test: it built its frame with the ids-only group tables after the household table, so the frame kept household_id alone and two of its assertions failed.
    • Mutation check: dropping check_support's returned frame, skipping the export's replay, dropping TYPEHUGQ from the lean checkpoint, and grading with support_universe each fail these tests.
  • Out of scope: a malformed-geography refusal leaves an old county_surface.json success receipt. Filed as County rung: a malformed-geography refusal leaves an old county_surface.json success receipt #1638.

Local runs at edc871ab8 (this PR has no CI because its base is not main):

  • production suite, -n 0: 124 passed, 1 deselected (run with the next group);
  • the county surface file plus that test, -n 4: 67 passed;
  • no-county regression, properties, solver profile and county facts: passed at ddc49578b in a 156-test run whose only failures were the two P1 test cases fixed above;
  • release tool, district ESS gate, sparse CD and the Modal county-facts test, -n 3: 394 passed, 2 skipped (both need the real facts feed);
  • ruff check and ruff format --check on the changed files, and git diff --check, pass.

Fix from the round-2 attempt (118f96f93)

The first round-2 run failed three times on a provider transport error before it delivered, so it does not count as a round. Its working notes confirmed one more path for the round-1 P1:

  • A completed run resumed with its old evidence. With --resume and every epoch done, do_calibrate reused the old calibration_summary.json and called write_reports without calibration_evidence or reliability_universe. A checkpoint from before TYPEHUGQ was kept, with a matching old summary, therefore kept its inflated grade (40 households and 360 group-quarters records at A, not C). The shortcut now recomputes calibration_evidence from the checkpoint and the saved weights before writing anything, and refuses if any key differs from the summary's; a checkpoint the current code refuses is refused there too.

Test: test_a_completed_resume_ships_only_evidence_this_code_computes (regraded evidence; refused checkpoint; nothing written in either case). Comparing nothing fails the first case and skipping the recomputation fails both. Release tool, county surface and sparse CD suites: 409 passed, 2 skipped. The same notes found nothing in consistency, held-outs or minimum support.

2023 county populations (c5c7b4697)

A full county run refused at prepare_run. The PUMA ladder's county overlaps are on 2020 counties, so a national ladder built on 11/16 with tools/build_us_puma_ladder_artifact.py declares 3,143 county-equivalents, with Connecticut's eight legacy counties. The rung requires 3,144, and the county facts use the nine planning regions.

  • The fix: ladder_county_overlaps regroups the ladder's (PUMA, tract) overlaps by county_fips_2023_from_tracts(tract) when the county column is county_fips_2023. It feeds both with_county_populations (through ladder_population) and county_puma_overlaps.
  • What it preserves:
    • every PUMA's population is conserved;
    • every county outside Connecticut keeps its 2020 overlaps;
    • the default county_fips path still reads county_overlap_county unchanged.
  • Against that national ladder: 3,144 county-equivalents, the nine regions 09110 to 09190, and the same national total. An opt-in test reruns this check when MICROCOSM_US_PUMA_LADDER_ARTIFACT is set.
  • Tests: county surface, production and properties: 200 passed with that variable set (-n 4). The three new tests fail against 118f96f93's tools. ruff check, ruff format --check, tools/ci_test_plan.py verify and git diff --check pass.

Known gaps and open questions

  • Block-location staging: the minimum-support step needs households with no block_geoid/tract_geoid. Under --location-rule block_v1 no household is eligible, so a full build refuses when a county is empty. Filling such a county would mean redrawing a block and re-deriving every block-based geography. That is not implemented. The planned R1 staging (build_us_acs_multispine_base.py) uses legacy location, where the step fills the four empty counties.
  • The tax-target cost of the county rows, and full-size memory and time, are unmeasured until the joint solve. The recommended county loss cap 10 and weight cap 5 (comparison: 20) still need a real run.
  • Grades apply to the 2024 base. The 2026 and 2027 files (Project the county build to 2026 and 2027 for the County Files #1627) inherit them; nothing regrades projected counties yet.

Verification

  • Built by unit B in session and by Subfleet units B2 and B3. An in-session review (four dimensions, a verifier each) found 6 distinct findings, all minor. Five were fixed in code or docs and one, block_v1, is documented above.
  • B3 (d05559c16, 5c5515b08, 88ab158b5, a770ba7f7, written by a Codex lane whose sandbox could not commit, imported from its bundle): it fixes the support fixture inventory, refuses unprepared county facts with a recorded refusal instead of reopening the file, excludes group quarters from reliability, and validates every B19001 bracket label. Its lane completed no pytest run (host load ~147).
  • 921d86625 merges Pin US county facts beside the main feed for the county rung #1614 at 8f71d34a7 (main 429d7a0de and Pin US county facts beside the main feed for the county rung #1614's round-3 fix), with no conflicts.
  • Local verification at 921d86625: the three spec-engine --check modes, tools/ci_test_plan.py verify, ruff check ., ruff format --check on the four changed Python files and git diff --check pass. County suites (production, properties, solver profile, surface, no-county regression, county facts; -n 4): 277 passed, 1 failed. Release-tool suites (release tool, district ESS gate, sparse CD, Modal stage plan; -n 4): 676 passed, 2 skipped, 1 failed. Both failures were tests that could not pass. One validation-values test wrote into a folder it never created. The calibration-outputs pin did not list the new county_reliability.csv. Both are fixed in 44ddec5a8; each passes on rerun.
  • Not verified: any real county build, the joint solve, or real-data grades.

Plan

  1. Local verification. Done; see above.
  2. Independent hard review of the whole change. Round 1 returned REQUEST_CHANGES; P1 fixed in ddc49578b, P2 in edc871ab8. The first round-2 run did not deliver; the gap its notes confirmed is fixed in 118f96f93. Round 2 at that head (8f71d34a7..118f96f93) returned APPROVE with no findings: both round-1 findings closed, the completed-resume fix sound, and the no-county byte differences traced to HDF modification-time messages that also differ base against base. The review has finished. The malformed-geography stale receipt stays tracked in microcosm#1638.
  3. Pin US county facts beside the main feed for the county rung #1614 lands first. The microcosm hub landed it at 20:54Z on 11/16 (merge c7a5f9679) and retargeted this PR to main. It is mergeable, and CI is running at c5c7b4697.
  4. Scoped review round 3 of 118f96f93..c5c7b4697 (the 2023 county populations). APPROVE with no findings: population conserved across 2,462 PUMAs and 51 state totals, non-Connecticut overlaps identical, 876 Connecticut tracts each in one of nine regions, 3,144 counties, and default-column outputs byte-identical to the older tools. Handed to the hub.
  5. R2: the 2024 county release build with the flags in the doc. It waits on a national donor carrying detailed_industry_recode, major_industry_recode and worked_last_year, which R1 requires and no published national build has (microcosm#922).

🤖 Generated with Claude Code

MaxGhenis and others added 17 commits November 15, 2026 09:42
Extend tools/us_acs_local_county_surface.py and its hooks in
tools/build_us_acs_local_release.py so `--geographies state,cd,county
--soi-mode state_county` calibrates every county: ACS county households
(B11001) as trained levels under new state parents, the IRS SOI county split
with Kalawao in a declared {15005, 15009} cell, held-out ACS validation
families scored after calibration, minimum support, and a reliability grade
and CSV for every county. Adds the production county test file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new engine-free file covers ACS fact parsing and refusals, the trained
acs_households family, the declared Kalawao/Maui SOI cell, minimum county
support, validation scoring and grades for 3,144 counties, the reliability
CSV and its packaging check. It copies the SOI fixture helpers instead of
importing the older test module, whose imports pull in policyengine_us.

The county smoke test now feeds ACS rows in the pinned shape and person
earnings, expects the 12-row matrix and scores validation end to end. The
refresh recipe now replays --county-min-support 0 for a run that recorded no
support receipt, instead of falling back to the default of 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The county rung doc now describes the trained ACS households family and its
new state targets, the four held-out ACS validation families and their
within-state share scoring, minimum county support, the declared Kalawao/Maui
SOI cell, schema-3 reliability grades with the packaged CSV, the updated row
and nnz estimates, and --county-min-support in the full-run command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The county smoke test's materialization becomes a fixture helper, and a new
test materializes it twice with different public values for the four held-out
ACS validation tables. The target registry, roles and matrix are
byte-identical, only the validation design digest differs, and the solved
weights are equal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ACS households family gives the nine planning regions 09110-09190 one state 09 parent and nine rows that partition it, and a legacy Connecticut county code outside the planning-region inventory is refused by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#922)

Review finding: in a capped smoke with Kalawao left empty and Hawaii's SOI
block held out, Kalawao took grade D from Maui's error through their combined
SOI cell. A county without household support now always grades U; its
training and holdout counts stay in the cell for information. The doc now
says Kalawao keeps its own ladder population row (not PEP) and ACS rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on minimum county support:

- check_support rebuilt a set of every household's county once per county
  (3,144 times over about 1.6M codes in a full build). It now asks
  unsupported_counties once.
- The donor floor counted ACS group-quarters records, so a move could take a
  county's only housing unit and leave its acs_households row empty, turning
  a build that passed into a refusal. Support and the donor floor now count
  only households in the household universe, so a county with group-quarters
  records alone is filled like an empty one.

The support invariants property now defines support the same way and checks
each county's receipt; a new example covers both cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…block_v1 (#922)

Review findings: in a state_county run a county whose staged households are
all ACS group quarters has an empty acs_households row, which attach_surface
refused only after the engine pass. check_support now counts household-universe
records as support in a state_county run, so that county is filled by minimum
support or refused before any measure is built. Other runs keep counting every
household.

The doc now says minimum support fills nothing on block_v1 staging, where
every household carries block and tract geoids; filling counties there needs
a block redrawn inside the PUMA and county, which is not implemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Materialize writes county_validation.json with allow_nan=False, and the first and last B19001 brackets have infinite bounds, so every state_county materialize refused. The county smoke test caught it; the design's bracket list now writes an open end as null, and the production tests check that the design and the validation scores are strict JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review finding: the earnings and households-with-earnings families omitted
partnership_self_employment_net_earnings on donor records, though ACS SEMP
(B19061, B19051) counts partnership self-employment income. The pilot left it
out because it built income as earnings plus partnership_income, which
already holds most of it. Earnings now add it on donor records, and the
income approximation starts from the four base leaves, so neither measure
counts it twice. The receipt lists the donor earnings column when staging
stores it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Brings in #1614's merge of main 429d7a0 (and its repin) and its
round-3 test fix. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_validation_values_change_no_loss_input_and_no_weight wrote its
  fixtures into tmp_path/scale_<n>, which it never created, so it failed
  with FileNotFoundError before reaching any assertion. It now creates
  each scale folder. The test then passes: registry, roles and matrix
  digests and the solved weights are equal across the two validation
  value sets, and only the validation design digest differs.
- test_calibration_outputs_are_bound_to_the_materialization pins the
  calibration outputs a rematerialization clears. This branch added
  county_reliability.csv to CALIBRATION_OUTPUT_FILENAMES without
  updating the pin.

County suites (six files, -n 4): 277 passed, 1 failed before this fix;
release-tool suites (four files, -n 4): 676 passed, 2 skipped, 1 failed.
Both failing tests pass after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad (#922)

Round-1 review of #1630 found that the lean checkpoint drops TYPEHUGQ, so
after a reload support_universe() counted every ACS record as a household:
40 households and 360 group-quarters records went from ESS 40 and grade C
to ESS 400 and grade A, and a county of group quarters alone from U to A.

A run with county targets now keeps TYPEHUGQ in its lean checkpoint; a run
without them writes the same columns as before. calibration_evidence grades
with reliability_universe(), which refuses a frame whose ACS records lack
TYPEHUGQ (such as a checkpoint written before this change) instead of
counting it whole, and checks the mask against the run's identity-bound
validation measures. The regression test round-trips the checkpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 16, 2026 09:26
…'s frame (#922)

Round-1 review of #1630, P2: the driver fixture recorded no support move,
so a driver that dropped check_support's moved frame or skipped the
export's replay still passed. The new test puts eight eligible households
and one group-quarters record in 01001 and none in 01003, with K=3, and
checks the moved counties in the frame the engine pass sees, the support
receipt and run identity, the lean checkpoint after a reload, reliability
after the reload, and the consumer export, which reloads unmoved staging.
_materialize_county_fixture takes a household table and the county-PUMA
overlaps for it.

The checkpoint-reload test from ddc4957 built its frame with the
ids-only group tables after the household table, so the frame kept
household_id alone. It now adds the household table last and checks the
frame keeps its columns.

Mutation check: dropping the moved frame, skipping the export replay,
dropping TYPEHUGQ from the lean checkpoint, and counting every record in
calibration_evidence each fail these tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An earlier round-2 review run of #1630 found that the completed-resume
shortcut in do_calibrate reused the old calibration_summary.json and
called write_reports without calibration_evidence or
reliability_universe. A checkpoint from before TYPEHUGQ was kept, with a
matching old summary, therefore kept its inflated county grades: 40
households and 360 group-quarters records stayed at A instead of C.

The shortcut now recomputes calibration_evidence from the checkpoint and
the saved weights before it writes anything, and refuses if any key
differs from the summary's (compared as canonical JSON). A checkpoint
the current code refuses, such as one without TYPEHUGQ, is refused there
as it is by a fresh solve.

test_a_completed_resume_ships_only_evidence_this_code_computes covers
regraded evidence and a refused checkpoint, and checks that nothing is
written. Comparing nothing fails the first case; skipping the
recomputation fails both. Release tool, county surface and sparse CD
suites: 409 passed, 2 skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator

A full county run on this branch stops in prepare_run: the population targets use 2020 counties

For the County Files PoC (#922), I read what a full run needs with --county-column county_fips_2023, as docs/us-acs-local-county-rung.md runs it. At 118f96f93:

  • with_county_populations (lines 106–107) and the minimum-support PUMA order (lines 1975–1977) read ladder.county_overlap_county and ladder.county_overlap_population, whatever --county-column says.
  • The PUMA ladder tool keys those overlaps by each block's 2020 county (us_runtime/puma_ladder_sources.py). They hold 3,143 counties, with Connecticut's eight old counties.
  • The county facts from Pin US county facts beside the main feed for the county rung #1614 use Connecticut's nine planning regions (09110–09190). So a full run stops in prepare_run (lines 2428–2433): the ladder gives 3,143 county-equivalents, not 3,144. Past that check, population_rows would refuse the Connecticut households, because their county_fips_2023 has no target.

No new ladder is needed. The PUMA ladder already carries tract_overlap_* (its loader requires them), and county_vintage.county_fips_2023_from_tracts (#1376) maps each tract to its 2023 county-equivalent. puma_county_vintage.py already uses those overlaps for Connecticut's draws. A ladder whose county overlaps held 2023 codes would be the wrong fix: staging's legacy county draw reads the same arrays and writes the 2020 county_fips.

Proposed change:

  • Pass the county column from tools/build_us_acs_local_release.py (line 1578) into with_county_populations and into the minimum-support overlap.
  • For county_fips_2023, sum the tract overlaps by 2023 county-equivalent, then scale within each state as now. Keep county_overlap_* for county_fips.
  • Record the mapping in the population receipt. Its source reads us_puma_ladder_2020 today.
  • Tests: 3,144 county-equivalents; Connecticut's nine regions sum to its state total; every other county equals its 2020 overlap; prepare_run passes at 3,144.

I'm scoping this today so that it lands with this PR, before R2. I'll say in this thread whether it comes as a commit here or from a branch merged into this one.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator

The county-equivalent fix comes as commits on this branch. Max's agents write it from my spec before review round 2, and I review it in round 2. It changes only the two county tools, their tests and the doc. No pin moves, and no ladder is rebuilt.

🤖 Generated with Claude Code

@MaxGhenis
MaxGhenis changed the base branch from lwi-county-facts-pin to main November 16, 2026 20:54
A full county run with --county-column county_fips_2023 refused at
prepare_run: the PUMA ladder's county overlaps are 2020 counties, so a
national ladder declares 3,143 county-equivalents (Connecticut's eight
legacy counties) against the 3,144 the County Files need, and the
planning-region facts would not match its Connecticut codes.

Under county_fips_2023 the rung now regroups the ladder's (PUMA, tract)
overlaps by each tract's 2023 county-equivalent
(county_fips_2023_from_tracts) for both the county populations and the
minimum support step's PUMA overlaps. Every PUMA's population is
conserved and every county outside Connecticut keeps its 2020 overlaps.
Any other county column is unchanged. A national ladder built today
gives 3,144 county-equivalents, the nine regions and the same total; an
opt-in test reruns that check against MICROCOSM_US_PUMA_LADDER_ARTIFACT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 16, 2026 22:16
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Hand-off to the Microcosm hub (head c5c7b4697).

Review. Finished. Round 2 approved the PR at 118f96f93 with no findings. Round 3, scoped to c5c7b4697 (2023 county populations from the ladder's tract overlaps, as specified in microcosm#1647), returned APPROVE with no findings. Against a national ladder built 11/16, it confirmed that population is conserved across 2,462 PUMAs and 51 state totals, every non-Connecticut overlap is identical, and each of 876 Connecticut tracts maps to one of the nine planning regions, giving 3,144 counties. With the default --county-column county_fips, population and support-overlap outputs are byte-identical to 118f96f93's tools. The reviews are kept in the LWI build folder (runs/B/review-1630-r2.md, review-1630-r3.md).

Checks. Locally and in round 3, the three county test files pass (200 passed); the three new tests fail against 118f96f93's tools. CI on this head: engine-us, lint, wheels, installed-emitter and select-countries pass, the UK jobs skip, and engine-free is still running. The PR is MERGEABLE.

Depends on. microcosm#1614, now merged (c7a5f9679). Independent of microcosm#1627.

Follow-ups owed. The malformed-geography stale receipt stays tracked in microcosm#1638. A full county run (R2) still waits on a national donor that carries detailed_industry_recode, major_industry_recode and worked_last_year (microcosm#922); that is the LWI lead's to schedule, not a merge blocker.

c5c7b46 passes county_column to ladder_population from
do_materialize. The stub in
test_materialize_refuses_a_geography_label_conflict_before_the_engine_pass
took only the ladder path and geographies, so engine-free CI failed with
a TypeError (1 failed, 29,850 passed). The stub now accepts the column
with the default value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Head moved to 5186faa25: a test-only fix. Engine-free CI failed at c5c7b4697 with one test (1 failed, 29,850 passed): the ladder_population stub in test_us_acs_local_geography_labels.py did not accept the county_column argument that c5c7b4697 passes. The stub now takes it with the default. No source changed, and the file passes locally (5 passed). The hub has been told and the hand-off line updated.

@MaxGhenis
MaxGhenis requested a review from juaristi22 November 17, 2026 00:39
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Why María's review is requested: the gap 1 fix (c5c7b4697) follows her spec in microcosm#1647, she wrote above (19:22Z 11/16) that she reviews it in round 2, and #1647's order of work has her review before #1630 lands. The hub holds the merge until she approves or says to go ahead without her.

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 5186faa25: approve, with notes. One item needs a check before R2.

I read main...5186faa25 and checked the gap-1 commits (c5c7b4697, 5186faa25) against my spec. I found nothing that builds wrong targets on a full county_fips_2023 run. County rows reconcile to their state rows. Holdouts, the Kalawao cell, the Connecticut support limit, the receipts and the resume paths work as the PR says.

1. Before R2: a county whose households all have zero AGI stops the run after the engine pass.

  • Minimum support fills only counties with no household (tools/us_acs_local_county_surface.py:2111-2119). A county with one or two households gets no top-up.
  • A county's SOI row takes the state row's stored entries for its households. The matrix stores no zeros.
  • So if every household in a county has AGI of exactly zero, its AGI row is empty.
  • attach_surface then refuses the run (:2697-2716). It runs after the engine pass (tools/build_us_acs_local_release.py:2331, :2355).
  • Return-count rows count every tax unit, so AGI is the row at risk.
  • This can happen in R2. In the pilot's staging, Borden TX (48033) had households but none with earnings (experiments/us-county-pilot-20261024/README.md, finding 8).
  • I reproduced it with this PR's code. A county with one household whose AGI is zero passes check_support, then attach_surface refuses it.
  • The message says "County cells have no supporting household". That misleads here: the county has a household.
  • Suggestion: after R1, list the counties where no household has nonzero wages or self-employment, and check their other income. It takes minutes on the staging table and tells us whether R2 can hit this.
  • If a county fails, we need a ruling before R2: drop its AGI row with a receipt, or give it a declared SOI cell, as Kalawao has.
  • This does not block the merge.

2. Grades: the bracket error is on a different scale.

  • The income-distribution error is an L1 sum over 16 brackets (:1449-1453). It runs from 0 to 2.
  • The grade holds it to the same 0.10, 0.20 and 0.50 caps as single relative errors (:1699-1711, :1876-1879).
  • Sampling noise alone breaks those caps. I simulated an unbiased county with a US-like bracket mix, leaving out ACS's own sampling error.
    • At ESS 400 the median L1 is 0.15, so A fails 96% of the time.
    • At ESS 100 it is 0.30, so B fails 96% of the time.
    • At ESS 30 it is 0.54, so C fails 63% of the time.
  • So this family will drive most grades, and push most counties down one grade.
  • Half the L1, the total variation distance, fits the caps. An unbiased county then passes its ESS band's cap 95–100% of the time.
  • The thresholds are provisional and the grades are report-only, so this need not block. But County Files users read the grades. I'd change it before the PoC report, or say it there.

3. The gap-1 commits against my spec.

  • The fix matches the spec. It stays in tools/, leaves the seed-kernel modules alone and keeps the county_fips path unchanged. The 3,144 check on the national ladder is what R2 needs.
  • Three spec items are missing. None blocks R2.
    • county_population_basis is in neither run_metadata (:2763-2783) nor county_support.json (:2582-2598). The identity binds county_column and the ladder digest, so the basis can be recovered.
    • population_spec gives every county the 2020 PUMA county-overlap source text (:294). On a county_fips_2023 run that is wrong for the nine Connecticut regions, and it goes into the target registry.
    • test_us_acs_local_county_properties.py has no Hypothesis property. The two-tract fixture and the opt-in national-ladder test cover the cases for now.
  • If the first two land before R2, R2's registry and identity carry them. Otherwise the PoC report should state the basis.
  • Two spec tests are missing too. Nothing checks that do_materialize passes the column on: the smoke's mocks take it and drop it (test_us_acs_local_county_surface.py:879-883, :938). The 876-tract and 3,143-county checks run only in the opt-in test.
  • The doc does not say that an empty Connecticut region cannot get support moves. Lines 55–59 describe only the 2020 overlaps. Line 400 still suggests a full run can use county_fips.

4. A smaller note.

  • On ACS rows, the income approximation adds the transferred engine leaves (:700-727, :1292-1303). They are QRF draws, not reconciled to the ACS amounts.
  • Staging also stores the reported SSP, RETP and INTP (acs_social_security_income, acs_retirement_income, acs_interest_dividend_rental_income). Those are B19025's own items.
  • Using them on ACS rows would make the two income families test the weights, not the transfer. This can follow.

Known issues. This PR does not make microcosm#1638 or microcosm#1637 worse.

Tests. At 5186faa25, with Python 3.13 and two workers, I ran the county surface, properties, production, geography-label, release-tool, no-county regression and solver-profile suites: 557 passed, 2 skipped. The skips are the opt-in national-ladder test and the pinned-feed test. CI's engine-free and engine-us jobs were still running when I checked.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 3b75cf1 into main Nov 17, 2026
11 checks passed
juaristi22 added a commit that referenced this pull request Nov 17, 2026
R2's attach_surface builds each county's IRS SOI rows from the state
rows' stored entries, and the state AGI row stores no zero. A county
whose households all have AGI of exactly 0 therefore gets an empty AGI
row, and attach_surface refuses the run after the materialize engine
pass (#1630 review, finding 1). Minimum support fills only counties
with no household.

tools/check_us_acs_county_agi_support.py runs on the staging H5 before
R2, in minutes and with no engine pass:
- It reads the household table and only the AGI income columns of the
  person and tax-unit tables (fixed-format blocks are streamed in row
  chunks).
- The AGI income columns are the money inputs of policyengine-us's
  irs_gross_income closure, read from the installed engine's source
  without importing it, less Social Security (taxable share only) and
  deductions, expenses and contributions. A null cell counts as 0, the
  default R2 fills. The check errs towards listing a county.
- With --puma-ladder it takes the county list from the ladder and
  applies R2's own minimum-support moves first.
- It writes a JSON report and exits 2 when a county is at risk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 17, 2026
A county whose households all have a zero measure (AGI exactly 0) gets
an empty county SOI row, and attach_surface refuses the run after the
engine pass. The #1630 review leaves the remedy to a ruling: drop the
row with a receipt, or give the county a declared SOI cell. The
declared-cell path (DECLARED_SOI_COMBINED_CELLS) covers only a county
absent from the SOI facts, so it cannot take a county that is present.

--county-drop-empty-soi-rows (off by default) drops such a row instead
of refusing: only an SOI row whose county cell has households. A county
without households, and an empty ACS households row, are refused as
before. county_surface.json records each dropped row (target, county
cell, concept, target value, households) under
zero_measure_soi_rows_dropped; the run identity records the flag and
county_targets_dropped, the reliability report lists the dropped rows,
and the refresh recipe repeats the flag. Without the flag nothing
changes, the identity bytes included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants