Repository navigation
Pin US county facts beside the main feed for the county rung - #1614
Conversation
The ACS-local county rung needs county-grain Chronicle facts: IRS SOI county TY2022, Census PEP county vintage 2024, and seven ACS 2020-2024 5-year county tables. Folding them into the main US feed would grow it about 6.5x for every fiscal compile while only county builds read them, so they get their own pin. - us/chronicle_county_facts.json and chronicle_county_facts_scope.json pin a bare v5 facts file at Chronicle main 3e9d3a069 (scope us_county_facts, 10 pairs). The main pin, its scope and the parity resources do not change; the two scopes share no pair. - chronicle_feed.py loads both pins through one validator; the main pin's messages are unchanged. - us_runtime/county_facts.py reads the file in one pass: digest and row count against the pin, refusal of any row on another contract, at another geography level or outside the scope, Puerto Rico dropped with a count, rows projected to the fields the rung reads. - The county rung takes --county-facts and --county-facts-sha256 (the pin's digest), requires --feed-sha256, loads the file once in prepare_run and never re-reads --feed for county facts. The run identity and refresh recipe record them; a no-county run's identity keeps its bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Declare the county pin and its scope as legacy resources in
tools/generate_us_bundle_from_constants.py and regenerate
us/country_package.json, so load_country_spec('us') and the spec-engine
load_bundle('us') load again (test_country_spec failed at 0a6f201b), and
repin the US spec-engine contracts for the new manifest bytes.
Loader: hash the file and check its digest and row count before parsing
any row, so a wrong file is named as such; hash again while parsing and
refuse a file that changed between the passes; refuse a row that is not
a JSON object and a county code with non-ASCII digits; keep fewer fields
per row (the pinned file's kept rows take about 0.9 GB instead of about
1.5 GB, loaded in about 5 s).
County rung: record a county facts file that cannot be read in
county_surface.json like any other refusal; the refresh recipe names a
placeholder county facts file and the pinned digest, never the build
machine's path; the error for missing SOI county rows names the county
facts pin.
Tests: lock the pin's literals; refuse a pin naming a consumer artifact
and malformed pins; match the digest refusal; non-object rows, non-ASCII
digits, the real receipt and a mid-read change; the projected rows give
the same SOI surface as full rows; an off-pin county file and a missing
one are refused before any target; a run without county geography is
refused; a run without county facts adds no metadata key.
Docs: correct the per-table county counts (B19013 has 3,143; B01003
reaches 3,144 across two record sets), the size figures (5.2 KB per ACS
row, a 171.7 MB main feed, 6.4x), the feed build command
(--chronicle-command), the c32c0b0 comparison key and its status, the
raw SOI county sums against their state parents (AGI up to 5% short), what
--feed-sha256 checks, and that county materialize runs are local; the
agent guide no longer describes a county-inclusive main feed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main renewed the US bundle generator tombstone (#1596) and added a publisher test (#1612). The four-step spec-engine repin on the merged tree changes nothing: spec_sha256 stays fdf6e8f4 and coverage is 42400/42400 configuration fields, 41/41 inventory checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ther (#922) The 922 MB county facts file is never committed; the repository holds only its pin (digest, size, row count, scope), and an operator builds or copies the file and passes it with --county-facts. A change that re-pins the main US feed at a Chronicle commit also rebuilds and re-pins the county facts at that commit, so a US build reads one Chronicle base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1603) Main re-pinned the US spec-engine files this branch also pins. The six conflicts were all generated counts and digests; each took main's side and the five-step repin recomputed them on the merged tree (repin_us_spec_engine_contracts.py --legacy-only, generate_us_bundle_from_constants.py, repin_us_spec_engine_contracts.py twice per #1530, spec_engine_coverage.py). - Configuration fields: 42,401 (main's 42,395 plus this branch's six pointers for the two declared legacy resources); 41/41 inventory checks. - US bundle spec_sha256: d3a3c6c6fbcf4bd7cae760215b909214e5593d606da74aa8f78e6c347e3b41b5. - The three --check modes and tools/ci_test_plan.py verify exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The loader's second pass now hashes the whole file before it refuses. A row it cannot accept stops parsing, not hashing, so a file swapped between the two passes is named as changed whatever its rows hold, and a malformed row in the pinned bytes is always a ValueError. Before, a swapped row with a text period or a list contract raised AttributeError or TypeError, which prepare_run did not record in county_surface.json. - Tests: county ids with the wrong prefix, four or six digits, or letters; a text period, a list layout, a list contract and a text geography; blank lines are hashed but not counted; every swapped file is named as changed; a property test that any pinned row value loads or raises ValueError; prepare_run records a run with no county facts and a swapped file; and no Modal plan input or option reaches --county-facts. - Docs: the pin holds the digest and row count, not the size; no county step reads --feed and nothing reopens the county facts file after prepare_run; refusals are recorded before the staging frame loads (the state admin targets come from --feed earlier); the Modal refusal comes from its check; the recipe's --feed-sha256 is a placeholder; 6.4x, not 6.5x; ten pairs from nine packages; "exceeded" the folder cap; stale feed wording; the changelog names --feed-sha256. Not changed: the unreachable county facts fallback in attach_surface (the production county rung rewrites that function) and dropping projected rows after the SOI split (the production rung reads them for the ACS families). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1096's exact softmax cap projection in solve moved the shared spec pins, so both conflicts were pin files. us-f0-coverage.json takes main's version and is regenerated by spec_engine_coverage.py. In test_us_multispine_pool_tool.py both pin comments are kept and the tools recompute spec_sha256 (fa75bd68 on main, 1e727e1c here). Repinned with the five-step sequence: repin --legacy-only, bundle generation, repin twice, coverage. Then: - repin, bundle and coverage --check: pass - ci_test_plan.py verify: pass - ruff check .: pass - county focused suite (-n 4): 746 passed, 1 skipped - spec-engine pin suites (-n 2): 116 passed - national pin suites (-n 2): 481 passed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l's parser (#922) Round-3 review P2: test_a_release_plan_cannot_pass_the_county_facts checked only the planned argv, so it still passed with the release tool's parse_function registration removed. The check stage would then run the tool with --help and return 0, skipping the county argument check. The test now runs the check stage's own _parse_check on this checkout with the planned state_county argv and asserts return code 2 and the "--soi-mode state_county needs --county-facts and --county-facts-sha256" refusal. With parse_function set to None the new assertion fails (return code 0), so the registration is pinned. - test_us_modal_stage_plan_tool.py (-n 4): 284 passed - ruff check and format on the file, ci_test_plan.py verify, git diff --check: pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main gained #1619, #1621, #1609, #1540, #1629 and #1594 since 429d7a0. Two files conflicted. docs/evidence/spec-engine/us-f0-coverage.json and test_us_multispine_pool_tool.py took main's side, and the five-step US spec-engine repin then regenerated them: spec_sha256 3fd1a384..., with the field counts and pointer digests from tools/spec_engine_coverage.py. The spec_sha256 comment keeps main's #1387 line and this PR's #922 line. Checks: the three --check modes, ci_test_plan verify, ruff check and git diff --check pass. Tests: county focused 751 (1 skip), pins 116 and national 481, all passing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main Chronicle feed may stay at 41b2fb2, the commit it had when the county facts were pinned on their own. A change that re-pins it anywhere else must re-pin the county facts to the same commit, as docs/us-chronicle-feed-repin.md says, so a US build reads one Chronicle base. Asked for by the feed's owner as a condition of landing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scoped re-review at
|
|
Hand-off to the Microcosm hub (head Review. Finished. Round 3 approved the PR's own change ( The feed owner's four conditions. (1) The scoped re-review passed and all 12 checks pass. (2) The four resources are byte-equal to main. (3) The county pin is unchanged. (4) The tools recomputed the generated values. Main ( Depends on it. microcosm#1630 (the county rung, stacked on this branch; its review is finished) is retargeted to Follow-ups owed. None on this PR. |
Main brought five first-parent merges since 14797c6: #1640 (UK dense bus floor for 21 families), #1642 (Belgium's ten gate entries in the gate contract), #1643 (Belgium's Axiom engine pinned at 753ce7d9), #1608 (promotions notify collector; emitter-interface floor 0.16.0) and #1614 (LWI's US county facts pin). Main changed nothing under packages/microcosm-fit or packages/microcosm-graph. Two pin files conflicted, both through #1614; each block took main's text plus this branch's own comment lines, placed after main's sentence: - docs/evidence/spec-engine/us-f0-coverage.json: two spec_sha256 blocks, main's side (the branch has no comment there). - packages/microcosm-build/tests/engine_workflow/us/test_us_multispine_pool_tool.py: main's #922 county-facts comment, then the branch's #1543 line, then spec_sha256. No code or test file other than these two pins conflicted, and no other file was changed by both sides. The tools then wrote every value on the merged tree: - tools/repin_us_spec_engine_contracts.py: the multispine pool spec_sha256 9ad067de (tools/generate_us_bundle_from_constants.py --check computes the same digest). It left the branch's EXPECTED_HASHES seed_map d5898b93 and seed_protocol 206be396 and the loader golden b7c843a1 as they were, and main's field-usage and coverage-tool pins as main has them. - tools/spec_engine_coverage.py: us-f0-coverage.json's two spec_sha256 lines (42401/42401 configuration fields; 41/41 inventory checks). - fit.qrf/pins.json keeps the branch's values: the merged fit sources equal the branch's and the kernel's implementation hash reproduces, so tools/graph_parity_repin.py was not run. The repinner's quote style was then reformatted with ruff format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WIP (draft). Part of #922, the Living Wage Institute County Files.
What this does
The ACS-local county rung needs county-grain Chronicle facts: IRS SOI county TY2022, Census PEP county vintage 2024, and seven ACS 2020-2024 5-year county tables (B01003, B11001, B19001, B19013, B19025, B19051, B19061). This PR pins them as their own file beside the main US feed pin, instead of growing the main feed about 6.4x for every US build.
us/chronicle_county_facts.jsonandus/chronicle_county_facts_scope.json(scopeus_county_facts, 10 record-set/period pairs). Built at Chronicle main3e9d3a0: 180,326 rows on thechronicle.consumer_fact.v5contract, facts SHA25667b0998a…. The two scopes share no pair, so the main feed, its scope and the target-parity resources do not move.us_runtime/county_facts.load_us_county_factsfirst hashes the file and refuses it unless its digest and row count are the pin's. A second pass hashes it again while parsing and refuses non-object rows, other row contracts, non-county rows, rows outside the scope and a file that changed between the passes. It drops the 78 Puerto Rico municipios with a count and keeps only the fields county targets read: about 0.9 GB of rows, loaded in about 5 s.tools/us_acs_local_county_surface.py: a--soi-mode state_countymaterialize now needs--county-factsand--county-facts-sha256(it must equal the pin's), plus--feed-sha256.prepare_runloads the county facts once, and refusals are recorded incounty_surface.json. The run identity records the county facts receipt; a run without county facts keeps its pre-county bytes. The refresh recipe names a placeholder file and the pinned digest.chronicle_feed.py: one validator (_load_pin) now loads both the main pin and the county pin, and the county pin must be a bare facts file. The main pin's behaviour and messages are unchanged.LEGACY_RESOURCE_PATHSintools/generate_us_bundle_from_constants.pyandcountry_package.jsonis regenerated. That moves the US bundle'sspec_sha256, so the US spec-engine contracts are repinned withtools/repin_us_spec_engine_contracts.pyandtools/spec_engine_coverage.py.docs/us-acs-local-county-rung.md(county facts input, Kalawao),docs/us-chronicle-feed-repin.md(the county facts pin, where its file lives, and that it moves with the main pin),docs/agent-guide.md.41b2fb2and the county pin at3e9d3a0; the next main re-pin starts at or after3e9d3a0.County materialize runs are local: the Modal stage plan cannot pass
--county-facts, and its preflight refuses astate_countymaterialize.Known gaps, handled in follow-up work on this branch's base
state_countyrun still refuses Hawaii until the rung declares a combined SOI cell {15005, 15009}.3e9d3a0, from a clean checkout with the documented command, matched the pin byte for byte: SHA25667b0998a…, 180,326 rows, 922,268,386 bytes. It took 8 minutes; the output folder peaked at 2.89 GB.Plan
Independent hard review of the whole change.Round 1 (20261114-092140, ate09c1828) ran out of wall time with no findings. Round 2 (20261114-232757-lwi-pin-review-r2, at5825f5b7, merge included): APPROVE, no P0/P1; four P2s and nine P3s, fixed inca394849as listed below.A scoped re-review ofRound 3 (ca394849.20261115-094413-lwi-pin-review-r3, atca394849): APPROVE, with one nonblocking P2, fixed in8f71d34a. Round-2 P3-8 (scope completeness) and P3-9 (error order when both resources are invalid) are unchanged; the reviewer found no regression in either. Mark ready once CI is green.us/chronicle_feed.json,us/chronicle_feed_scope.jsonand both target-parity resources are byte-equal to main's; the county pin is unchanged (3e9d3a0, facts67b0998a…, 180,326 rows); and the generated counts and digests are recomputed by the tools. It also asked for a test that the county pin moves with the next main pin (added inc8403535). The Microcosm hub lands it. The scoped re-review of07a39c95..c8403535is queued.Verification so far
An in-session review of the first commit (four dimensions, an adversarial verifier each and a completeness critic): 30 findings, 29 confirmed, plus two critic findings, all addressed in the second commit.
Before the rebase onto
68bbff774: focused county tests 583 passed;test_country_spec.py144 passed;ruff check .clean;tools/ci_test_plan.py verifyok.After the rebase, the four-step repin was rerun (
repin_us_spec_engine_contracts.py --legacy-only,generate_us_bundle_from_constants.py,repin_us_spec_engine_contracts.py,spec_engine_coverage.py), all exit 0. The US bundlespec_sha256moves from main's98d58365…tofdf6e8f4…. The coverage evidence gains exactly six pointers (two legacy rows, three fields each): 42,400/42,400 configuration fields, 41/41 inventory checks.Full local verification on
d2e2767f: the three--checkmodes (repin_us_spec_engine_contracts.py,generate_us_bundle_from_constants.py,spec_engine_coverage.py) exit 0;tools/ci_test_plan.py verifyok;ruff check .clean;pytest tests/engine_free/us tests/engine_free/shared -n 4: 17,149 passed, 156 skipped, 0 failed;test_us_spec_bundle.py,test_spec_engine_coverage_tool.py,test_us_multispine_pool_tool.py: 60 passed. CI ond2e2767f: 12/12 pass.e09c1828merges main at52aea1477(Renew the US bundle generator's migration tombstone to 16 February 2027 (#1585) #1596's tombstone renewal and Test that the build-folder publisher publishes the SCF education-debt fit tables #1612) and adds the doc paragraphs above. The five-step repin on the merged tree (--legacy-only, generate, repin twice per Repinner needs two passes to pin covered_item_count #1530, coverage) changed no file:spec_sha256staysfdf6e8f4…, coverage 42,400/42,400 and 41/41. The tests were not rerun locally one09c1828; CI runs them on this head.5825f5b7merges main at085672fa9(UK stock cells: homes and the land under them in England and Wales (#1524) #1597, US: one copy of each shared helper (#1585 items 4, 5 and 8.3) #1601, Admit a UK local-area release to build folders only once the registry names it #1603), which re-pinned the same US spec-engine files. All six conflicts were generated counts and digests; each took main's side and the five-step repin recomputed them on the merged tree. The US bundlespec_sha256is nowd3a3c6c6…; coverage is 42,401/42,401 configuration fields (main's 42,395 plus this branch's six) and 41/41 inventory checks. The three--checkmodes,tools/ci_test_plan.py verifyandruff check .exit 0. Focused tests (coverage tool, field usage, multispine pool tool, both spec bundle suites, county facts, county surface, Chronicle feed,test_country_spec.py) at-n 2: 333 passed, 0 failed.ca394849fixes the round-2 findings. P2-1: the loader's second pass now hashes the whole file before it refuses. A file swapped between the passes is named as changed whatever its rows hold, and a malformed pinned row is always aValueError, soprepare_runrecords it incounty_surface.json. Before, a swapped row with a text period or a list contract raised an unrecordedAttributeError/TypeError. P2-2, P2-3 and the P3 doc nits: the docs no longer claim the pin holds the file's size, or that nothing reads--feed; they say when refusals are recorded and where the Modal refusal comes from. P2-4: new tests cover the id checks, blank lines, swapped files, any pinned row value (a property test: it loads or raisesValueError), theNoneguard and the Modal plan. The swap and property tests fail on the old loader. Not changed: the unreachable fallback inattach_surface(the production county rung rewrites that function and takes the fix) and dropping projected rows after the SOI split (the production rung reads them). Focused tests on the fix (county facts, county surface, Chronicle feed, Modal stage plan, county properties) at-n 4: 432 passed, 0 failed;ruff check .,ruff format --check,tools/ci_test_plan.py verifyandgit diff --checkclean.county_facts.pyis not byte-attested, so no spec-engine pin moved.d2efc978merges main at429d7a0de(Project the softmax parametrization exactly onto the capped simplex each step #1096, the exact softmax cap projection in solve), which moved the shared spec pins. Both conflicts were pin files:us-f0-coverage.jsontook main's version and was regenerated. Intest_us_multispine_pool_tool.pyboth pin comments are kept and the tools recomputespec_sha256(fa75bd68…on main,1e727e1c…here). The five-step repin, the three--checkmodes,tools/ci_test_plan.py verifyandruff check .exit 0. County focused suite (county facts, county surface, Chronicle feed, Modal stage plan, county properties, release tool) at-n 4: 746 passed, 1 skipped. Spec-engine pin suites at-n 2: 116 passed. National pin suites at-n 2: 481 passed. A repo-wideruff format --check .flags 53 files, all also on main and none touched here; CI does not run it.8f71d34afixes round-3 P2: the county-facts Modal test now runs the check stage's_parse_checkon this checkout with the plannedstate_countyargv, and asserts return code 2 and the missing--county-facts/--county-facts-sha256refusal. With the release tool'sparse_functionset toNonethe test fails (return code 0, from--help).test_us_modal_stage_plan_tool.pyat-n 4: 284 passed.07a39c95merges main at99814fe43(Keep ACS partnership_income at zero where SEMP already holds it (#1387) #1619, Add the County Files household measures sidecar and schema manifest tools (#922) #1621, Add the shared gate-battery pieces for US adoption (#1585 item 11, PR 1 of 4) #1609, Align DESIGN.md and US builder help with how releases calibrate #1540, Drop the wall-clock assert from the composed stage-scan test #1629, Add microcosm-withdraw-release-build and refuse withdrawn parent builds #1594). Both conflicts (us-f0-coverage.json,test_us_multispine_pool_tool.py) took main's side and the five-step repin regenerated them:spec_sha256is3fd1a384…. The hand-written comment above that pin keeps main's ACS rows carry a donor partnership_income draw ($337.7B weighted) that can stack on SEMP #1387 line and this PR's County-calibrated County Files (3,144 counties, 2026 + 2027): build epic #922 line. The three--checkmodes,tools/ci_test_plan.py verify,ruff check .andgit diff --checkexit 0. County focused suite at-n 4: 751 passed, 1 skipped. Spec-engine pin suites at-n 2: 116 passed. National pin suites at-n 2: 481 passed. The PR's own change against main is the same as before the merge apart fromspec_sha256and the new test.c8403535addstest_the_county_pin_moves_with_the_next_main_pin: the main feed pin may stay at41b2fb2; at any other commit the county pin must name the same one. A probe with the main pin moved and the county pin left behind fails it; both moved together, or the main pin moved onto3e9d3a0, pass. County facts suite: 48 passed.At
c8403535: the four feed and parity resources are byte-equal to main at99814fe43and at14797c665(Rename Populace to Microcosm in prose and code #1541); the county pin and its scope file are unchanged from8f71d34a;git merge-treewith14797c665is clean.tools/build_us_fiscal_refresh_release.pyimports the same three names fromchronicle_feedas at8f71d34aand reads neither the county pin norspec_sha256.🤖 Generated with Claude Code