Repository navigation
Make the county rung a production county calibration for the County Files - #1630
Conversation
Extend tools/us_acs_local_county_surface.py and its hooks in
tools/build_us_acs_local_release.py so `--geographies state,cd,county
--soi-mode state_county` calibrates every county: ACS county households
(B11001) as trained levels under new state parents, the IRS SOI county split
with Kalawao in a declared {15005, 15009} cell, held-out ACS validation
families scored after calibration, minimum support, and a reliability grade
and CSV for every county. Adds the production county test file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new engine-free file covers ACS fact parsing and refusals, the trained acs_households family, the declared Kalawao/Maui SOI cell, minimum county support, validation scoring and grades for 3,144 counties, the reliability CSV and its packaging check. It copies the SOI fixture helpers instead of importing the older test module, whose imports pull in policyengine_us. The county smoke test now feeds ACS rows in the pinned shape and person earnings, expects the 12-row matrix and scores validation end to end. The refresh recipe now replays --county-min-support 0 for a run that recorded no support receipt, instead of falling back to the default of 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The county rung doc now describes the trained ACS households family and its new state targets, the four held-out ACS validation families and their within-state share scoring, minimum county support, the declared Kalawao/Maui SOI cell, schema-3 reliability grades with the packaged CSV, the updated row and nnz estimates, and --county-min-support in the full-run command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The county smoke test's materialization becomes a fixture helper, and a new test materializes it twice with different public values for the four held-out ACS validation tables. The target registry, roles and matrix are byte-identical, only the validation design digest differs, and the solved weights are equal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ACS households family gives the nine planning regions 09110-09190 one state 09 parent and nine rows that partition it, and a legacy Connecticut county code outside the planning-region inventory is refused by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#922) Review finding: in a capped smoke with Kalawao left empty and Hawaii's SOI block held out, Kalawao took grade D from Maui's error through their combined SOI cell. A county without household support now always grades U; its training and holdout counts stay in the cell for information. The doc now says Kalawao keeps its own ladder population row (not PEP) and ACS rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on minimum county support: - check_support rebuilt a set of every household's county once per county (3,144 times over about 1.6M codes in a full build). It now asks unsupported_counties once. - The donor floor counted ACS group-quarters records, so a move could take a county's only housing unit and leave its acs_households row empty, turning a build that passed into a refusal. Support and the donor floor now count only households in the household universe, so a county with group-quarters records alone is filled like an empty one. The support invariants property now defines support the same way and checks each county's receipt; a new example covers both cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…block_v1 (#922) Review findings: in a state_county run a county whose staged households are all ACS group quarters has an empty acs_households row, which attach_surface refused only after the engine pass. check_support now counts household-universe records as support in a state_county run, so that county is filled by minimum support or refused before any measure is built. Other runs keep counting every household. The doc now says minimum support fills nothing on block_v1 staging, where every household carries block and tract geoids; filling counties there needs a block redrawn inside the PUMA and county, which is not implemented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Materialize writes county_validation.json with allow_nan=False, and the first and last B19001 brackets have infinite bounds, so every state_county materialize refused. The county smoke test caught it; the design's bracket list now writes an open end as null, and the production tests check that the design and the validation scores are strict JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review finding: the earnings and households-with-earnings families omitted partnership_self_employment_net_earnings on donor records, though ACS SEMP (B19061, B19051) counts partnership self-employment income. The pilot left it out because it built income as earnings plus partnership_income, which already holds most of it. Earnings now add it on donor records, and the income approximation starts from the four base leaves, so neither measure counts it twice. The receipt lists the donor earnings column when staging stores it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
- test_validation_values_change_no_loss_input_and_no_weight wrote its fixtures into tmp_path/scale_<n>, which it never created, so it failed with FileNotFoundError before reaching any assertion. It now creates each scale folder. The test then passes: registry, roles and matrix digests and the solved weights are equal across the two validation value sets, and only the validation design digest differs. - test_calibration_outputs_are_bound_to_the_materialization pins the calibration outputs a rematerialization clears. This branch added county_reliability.csv to CALIBRATION_OUTPUT_FILENAMES without updating the pin. County suites (six files, -n 4): 277 passed, 1 failed before this fix; release-tool suites (four files, -n 4): 676 passed, 2 skipped, 1 failed. Both failing tests pass after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad (#922) Round-1 review of #1630 found that the lean checkpoint drops TYPEHUGQ, so after a reload support_universe() counted every ACS record as a household: 40 households and 360 group-quarters records went from ESS 40 and grade C to ESS 400 and grade A, and a county of group quarters alone from U to A. A run with county targets now keeps TYPEHUGQ in its lean checkpoint; a run without them writes the same columns as before. calibration_evidence grades with reliability_universe(), which refuses a frame whose ACS records lack TYPEHUGQ (such as a checkpoint written before this change) instead of counting it whole, and checks the mask against the run's identity-bound validation measures. The regression test round-trips the checkpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s frame (#922) Round-1 review of #1630, P2: the driver fixture recorded no support move, so a driver that dropped check_support's moved frame or skipped the export's replay still passed. The new test puts eight eligible households and one group-quarters record in 01001 and none in 01003, with K=3, and checks the moved counties in the frame the engine pass sees, the support receipt and run identity, the lean checkpoint after a reload, reliability after the reload, and the consumer export, which reloads unmoved staging. _materialize_county_fixture takes a household table and the county-PUMA overlaps for it. The checkpoint-reload test from ddc4957 built its frame with the ids-only group tables after the household table, so the frame kept household_id alone. It now adds the household table last and checks the frame keeps its columns. Mutation check: dropping the moved frame, skipping the export replay, dropping TYPEHUGQ from the lean checkpoint, and counting every record in calibration_evidence each fail these tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An earlier round-2 review run of #1630 found that the completed-resume shortcut in do_calibrate reused the old calibration_summary.json and called write_reports without calibration_evidence or reliability_universe. A checkpoint from before TYPEHUGQ was kept, with a matching old summary, therefore kept its inflated county grades: 40 households and 360 group-quarters records stayed at A instead of C. The shortcut now recomputes calibration_evidence from the checkpoint and the saved weights before it writes anything, and refuses if any key differs from the summary's (compared as canonical JSON). A checkpoint the current code refuses, such as one without TYPEHUGQ, is refused there as it is by a fresh solve. test_a_completed_resume_ships_only_evidence_this_code_computes covers regraded evidence and a refused checkpoint, and checks that nothing is written. Comparing nothing fails the first case; skipping the recomputation fails both. Release tool, county surface and sparse CD suites: 409 passed, 2 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
A full county run on this branch stops in For the County Files PoC (#922), I read what a full run needs with
No new ladder is needed. The PUMA ladder already carries Proposed change:
I'm scoping this today so that it lands with this PR, before R2. I'll say in this thread whether it comes as a commit here or from a branch merged into this one. 🤖 Generated with Claude Code |
|
The county-equivalent fix comes as commits on this branch. Max's agents write it from my spec before review round 2, and I review it in round 2. It changes only the two county tools, their tests and the doc. No pin moves, and no ladder is rebuilt. 🤖 Generated with Claude Code |
A full county run with --county-column county_fips_2023 refused at prepare_run: the PUMA ladder's county overlaps are 2020 counties, so a national ladder declares 3,143 county-equivalents (Connecticut's eight legacy counties) against the 3,144 the County Files need, and the planning-region facts would not match its Connecticut codes. Under county_fips_2023 the rung now regroups the ladder's (PUMA, tract) overlaps by each tract's 2023 county-equivalent (county_fips_2023_from_tracts) for both the county populations and the minimum support step's PUMA overlaps. Every PUMA's population is conserved and every county outside Connecticut keeps its 2020 overlaps. Any other county column is unchanged. A national ladder built today gives 3,144 county-equivalents, the nine regions and the same total; an opt-in test reruns that check against MICROCOSM_US_PUMA_LADDER_ARTIFACT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hand-off to the Microcosm hub (head Review. Finished. Round 2 approved the PR at Checks. Locally and in round 3, the three county test files pass (200 passed); the three new tests fail against Depends on. microcosm#1614, now merged ( Follow-ups owed. The malformed-geography stale receipt stays tracked in microcosm#1638. A full county run (R2) still waits on a national donor that carries |
c5c7b46 passes county_column to ladder_population from do_materialize. The stub in test_materialize_refuses_a_geography_label_conflict_before_the_engine_pass took only the ladder path and geographies, so engine-free CI failed with a TypeError (1 failed, 29,850 passed). The stub now accepts the column with the default value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Head moved to |
juaristi22
left a comment
There was a problem hiding this comment.
Review at 5186faa25: approve, with notes. One item needs a check before R2.
I read main...5186faa25 and checked the gap-1 commits (c5c7b4697, 5186faa25) against my spec. I found nothing that builds wrong targets on a full county_fips_2023 run. County rows reconcile to their state rows. Holdouts, the Kalawao cell, the Connecticut support limit, the receipts and the resume paths work as the PR says.
1. Before R2: a county whose households all have zero AGI stops the run after the engine pass.
- Minimum support fills only counties with no household (
tools/us_acs_local_county_surface.py:2111-2119). A county with one or two households gets no top-up. - A county's SOI row takes the state row's stored entries for its households. The matrix stores no zeros.
- So if every household in a county has AGI of exactly zero, its AGI row is empty.
attach_surfacethen refuses the run (:2697-2716). It runs after the engine pass (tools/build_us_acs_local_release.py:2331,:2355).- Return-count rows count every tax unit, so AGI is the row at risk.
- This can happen in R2. In the pilot's staging, Borden TX (48033) had households but none with earnings (
experiments/us-county-pilot-20261024/README.md, finding 8). - I reproduced it with this PR's code. A county with one household whose AGI is zero passes
check_support, thenattach_surfacerefuses it. - The message says "County cells have no supporting household". That misleads here: the county has a household.
- Suggestion: after R1, list the counties where no household has nonzero wages or self-employment, and check their other income. It takes minutes on the staging table and tells us whether R2 can hit this.
- If a county fails, we need a ruling before R2: drop its AGI row with a receipt, or give it a declared SOI cell, as Kalawao has.
- This does not block the merge.
2. Grades: the bracket error is on a different scale.
- The income-distribution error is an L1 sum over 16 brackets (
:1449-1453). It runs from 0 to 2. - The grade holds it to the same 0.10, 0.20 and 0.50 caps as single relative errors (
:1699-1711,:1876-1879). - Sampling noise alone breaks those caps. I simulated an unbiased county with a US-like bracket mix, leaving out ACS's own sampling error.
- At ESS 400 the median L1 is 0.15, so A fails 96% of the time.
- At ESS 100 it is 0.30, so B fails 96% of the time.
- At ESS 30 it is 0.54, so C fails 63% of the time.
- So this family will drive most grades, and push most counties down one grade.
- Half the L1, the total variation distance, fits the caps. An unbiased county then passes its ESS band's cap 95–100% of the time.
- The thresholds are provisional and the grades are report-only, so this need not block. But County Files users read the grades. I'd change it before the PoC report, or say it there.
3. The gap-1 commits against my spec.
- The fix matches the spec. It stays in
tools/, leaves the seed-kernel modules alone and keeps thecounty_fipspath unchanged. The 3,144 check on the national ladder is what R2 needs. - Three spec items are missing. None blocks R2.
county_population_basisis in neitherrun_metadata(:2763-2783) norcounty_support.json(:2582-2598). The identity bindscounty_columnand the ladder digest, so the basis can be recovered.population_specgives every county the 2020 PUMA county-overlapsourcetext (:294). On acounty_fips_2023run that is wrong for the nine Connecticut regions, and it goes into the target registry.test_us_acs_local_county_properties.pyhas no Hypothesis property. The two-tract fixture and the opt-in national-ladder test cover the cases for now.
- If the first two land before R2, R2's registry and identity carry them. Otherwise the PoC report should state the basis.
- Two spec tests are missing too. Nothing checks that
do_materializepasses the column on: the smoke's mocks take it and drop it (test_us_acs_local_county_surface.py:879-883,:938). The 876-tract and 3,143-county checks run only in the opt-in test. - The doc does not say that an empty Connecticut region cannot get support moves. Lines 55–59 describe only the 2020 overlaps. Line 400 still suggests a full run can use
county_fips.
4. A smaller note.
- On ACS rows, the income approximation adds the transferred engine leaves (
:700-727,:1292-1303). They are QRF draws, not reconciled to the ACS amounts. - Staging also stores the reported SSP, RETP and INTP (
acs_social_security_income,acs_retirement_income,acs_interest_dividend_rental_income). Those are B19025's own items. - Using them on ACS rows would make the two income families test the weights, not the transfer. This can follow.
Known issues. This PR does not make microcosm#1638 or microcosm#1637 worse.
Tests. At 5186faa25, with Python 3.13 and two workers, I ran the county surface, properties, production, geography-label, release-tool, no-county regression and solver-profile suites: 557 passed, 2 skipped. The skips are the opt-in national-ladder test and the pinned-feed test. CI's engine-free and engine-us jobs were still running when I checked.
🤖 Generated with Claude Code
R2's attach_surface builds each county's IRS SOI rows from the state rows' stored entries, and the state AGI row stores no zero. A county whose households all have AGI of exactly 0 therefore gets an empty AGI row, and attach_surface refuses the run after the materialize engine pass (#1630 review, finding 1). Minimum support fills only counties with no household. tools/check_us_acs_county_agi_support.py runs on the staging H5 before R2, in minutes and with no engine pass: - It reads the household table and only the AGI income columns of the person and tax-unit tables (fixed-format blocks are streamed in row chunks). - The AGI income columns are the money inputs of policyengine-us's irs_gross_income closure, read from the installed engine's source without importing it, less Social Security (taxable share only) and deductions, expenses and contributions. A null cell counts as 0, the default R2 fills. The check errs towards listing a county. - With --puma-ladder it takes the county list from the ladder and applies R2's own minimum-support moves first. - It writes a JSON report and exits 2 when a county is at risk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A county whose households all have a zero measure (AGI exactly 0) gets an empty county SOI row, and attach_surface refuses the run after the engine pass. The #1630 review leaves the remedy to a ruling: drop the row with a receipt, or give the county a declared SOI cell. The declared-cell path (DECLARED_SOI_COMBINED_CELLS) covers only a county absent from the SOI facts, so it cannot take a county that is present. --county-drop-empty-soi-rows (off by default) drops such a row instead of refusing: only an SOI row whose county cell has households. A county without households, and an empty ACS households row, are refused as before. county_surface.json records each dropped row (target, county cell, concept, target value, households) under zero_measure_soi_rows_dropped; the run identity records the flag and county_targets_dropped, the reliability report lists the dropped rows, and the refresh recipe repeats the flag. Without the flag nothing changes, the identity bytes included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Draft, stacked on #1614 (the county facts pin); its base is
lwi-county-facts-pin. Part of #922, the Living Wage Institute County Files: the 2024 county calibration that the 2026 and 2027 files are projected from.What this does
Makes the ACS-local county rung (
--geographies state,cd,county --soi-mode state_county) a production county calibration. Logic lives intools/us_acs_local_county_surface.py;tools/build_us_acs_local_release.pyonly calls its hooks.Training targets:
Held out, scored only: four ACS county families:
Each is scored against a pro-rata baseline (the state target times the county's population share). They never enter the loss, the target registry, the matrix or the weights; a test checks that changing them moves no loss input or weight.
Reliability: every county gets an A-D grade (U when it has no household support) in
county_reliability.jsonand a packagedcounty_reliability.csv. A grade comes from supporting rows, distinct source households, Kish ESS, and training and holdout error. ACS group-quarters records count toward neither support nor ESS. The A-D thresholds are provisional and recorded as such.Minimum support:
--county-min-support K(default 3, 0 off, at most 50) fills each county the staging frame leaves empty with exactly K same-PUMA households. Households are ranked by a salted hash, so the result does not depend on row order. Only the county columns are rewritten, and every move is recorded incounty_support.jsonand bound into the run identity. Counties still without support, including group-quarters-only counties, are refused before the engine pass.Kalawao (15005) shares one declared SOI cell with Maui (15009), for the SOI families only.
Connecticut planning regions are the county equivalents. Under
--county-column county_fips_2023, the county populations and the minimum support step's PUMA overlaps come from the ladder's tract overlaps, regrouped by each tract's 2023 county-equivalent (c5c7b4697; see below).docs/us-acs-local-county-rung.mddocuments the surface, the evidence, the grades and the sizes. That adds 12,625 county and ACS state household rows (about 17,492 with population and district rows) and at most 7,944,270 matrix nonzeros (about 63.6 MB) for 1,588,854 households.Round-1 review fixes (
ddc49578b,edc871ab8)The independent review returned REQUEST_CHANGES: one P1, one P2, and one pre-existing P2 that is out of scope.
TYPEHUGQ, sosupport_universe()counted every ACS record. In the reviewer's case, 40 households and 360 group-quarters records went from ESS 40 and grade C to ESS 400 and grade A.TYPEHUGQin its lean checkpoint. A run without them writes the same columns as before, so its checkpoint bytes do not change.calibration_evidencegrades withreliability_universe(). It refuses a frame whose ACS records lackTYPEHUGQ, such as a checkpoint written before this change, and checks the mask against the run's identity-bound validation measures.edc871ab8.test_support_moves_reach_materialize_the_checkpoint_and_the_exportputs eight eligible households and one group-quarters record in 01001, none in 01003, and sets K=3.household_idalone and two of its assertions failed.check_support's returned frame, skipping the export's replay, droppingTYPEHUGQfrom the lean checkpoint, and grading withsupport_universeeach fail these tests.county_surface.jsonsuccess receipt. Filed as County rung: a malformed-geography refusal leaves an old county_surface.json success receipt #1638.Local runs at
edc871ab8(this PR has no CI because its base is notmain):-n 0: 124 passed, 1 deselected (run with the next group);-n 4: 67 passed;ddc49578bin a 156-test run whose only failures were the two P1 test cases fixed above;-n 3: 394 passed, 2 skipped (both need the real facts feed);ruff checkandruff format --checkon the changed files, andgit diff --check, pass.Fix from the round-2 attempt (
118f96f93)The first round-2 run failed three times on a provider transport error before it delivered, so it does not count as a round. Its working notes confirmed one more path for the round-1 P1:
--resumeand every epoch done,do_calibratereused the oldcalibration_summary.jsonand calledwrite_reportswithoutcalibration_evidenceorreliability_universe. A checkpoint from beforeTYPEHUGQwas kept, with a matching old summary, therefore kept its inflated grade (40 households and 360 group-quarters records at A, not C). The shortcut now recomputescalibration_evidencefrom the checkpoint and the saved weights before writing anything, and refuses if any key differs from the summary's; a checkpoint the current code refuses is refused there too.Test:
test_a_completed_resume_ships_only_evidence_this_code_computes(regraded evidence; refused checkpoint; nothing written in either case). Comparing nothing fails the first case and skipping the recomputation fails both. Release tool, county surface and sparse CD suites: 409 passed, 2 skipped. The same notes found nothing in consistency, held-outs or minimum support.2023 county populations (
c5c7b4697)A full county run refused at
prepare_run. The PUMA ladder's county overlaps are on 2020 counties, so a national ladder built on 11/16 withtools/build_us_puma_ladder_artifact.pydeclares 3,143 county-equivalents, with Connecticut's eight legacy counties. The rung requires 3,144, and the county facts use the nine planning regions.ladder_county_overlapsregroups the ladder's(PUMA, tract)overlaps bycounty_fips_2023_from_tracts(tract)when the county column iscounty_fips_2023. It feeds bothwith_county_populations(throughladder_population) andcounty_puma_overlaps.county_fipspath still readscounty_overlap_countyunchanged.MICROCOSM_US_PUMA_LADDER_ARTIFACTis set.-n 4). The three new tests fail against118f96f93's tools.ruff check,ruff format --check,tools/ci_test_plan.py verifyandgit diff --checkpass.Known gaps and open questions
block_geoid/tract_geoid. Under--location-rule block_v1no household is eligible, so a full build refuses when a county is empty. Filling such a county would mean redrawing a block and re-deriving every block-based geography. That is not implemented. The planned R1 staging (build_us_acs_multispine_base.py) uses legacy location, where the step fills the four empty counties.Verification
d05559c16,5c5515b08,88ab158b5,a770ba7f7, written by a Codex lane whose sandbox could not commit, imported from its bundle): it fixes the support fixture inventory, refuses unprepared county facts with a recorded refusal instead of reopening the file, excludes group quarters from reliability, and validates every B19001 bracket label. Its lane completed no pytest run (host load ~147).921d86625merges Pin US county facts beside the main feed for the county rung #1614 at8f71d34a7(main429d7a0deand Pin US county facts beside the main feed for the county rung #1614's round-3 fix), with no conflicts.921d86625: the three spec-engine--checkmodes,tools/ci_test_plan.py verify,ruff check .,ruff format --checkon the four changed Python files andgit diff --checkpass. County suites (production, properties, solver profile, surface, no-county regression, county facts;-n 4): 277 passed, 1 failed. Release-tool suites (release tool, district ESS gate, sparse CD, Modal stage plan;-n 4): 676 passed, 2 skipped, 1 failed. Both failures were tests that could not pass. One validation-values test wrote into a folder it never created. The calibration-outputs pin did not list the newcounty_reliability.csv. Both are fixed in44ddec5a8; each passes on rerun.Plan
Local verification.Done; see above.ddc49578b, P2 inedc871ab8. The first round-2 run did not deliver; the gap its notes confirmed is fixed in118f96f93. Round 2 at that head (8f71d34a7..118f96f93) returned APPROVE with no findings: both round-1 findings closed, the completed-resume fix sound, and the no-county byte differences traced to HDF modification-time messages that also differ base against base. The review has finished. The malformed-geography stale receipt stays tracked in microcosm#1638.Pin US county facts beside the main feed for the county rung #1614 lands first.The microcosm hub landed it at 20:54Z on 11/16 (mergec7a5f9679) and retargeted this PR tomain. It is mergeable, and CI is running atc5c7b4697.Scoped review round 3 ofAPPROVE with no findings: population conserved across 2,462 PUMAs and 51 state totals, non-Connecticut overlaps identical, 876 Connecticut tracts each in one of nine regions, 3,144 counties, and default-column outputs byte-identical to the older tools. Handed to the hub.118f96f93..c5c7b4697(the 2023 county populations).detailed_industry_recode,major_industry_recodeandworked_last_year, which R1 requires and no published national build has (microcosm#922).🤖 Generated with Claude Code