Repository navigation
US support-mix bake-off: CPS years vs ACS rows vs location clones at fixed budgets - #1067
Merged
Merged
Conversation
MaxGhenis
force-pushed
the
support-mix-bakeoff
branch
from
September 30, 2026 11:11
2eb9031 to
40d72cc
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… national_state loss weights, GQ-consistent district population, part-overlap and cache guards Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om household structure, score every holdout level for both products, rescore saved weights Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-readings, receipts for every claim - Verdict rests on the release's loss-weighted held-out error and three-draw means; program receipt is a wash nationally, state tax items a cost; the local file's losses to clones and the national 50/50 alternative are stated. - Rescore keeps solve-time cost; --rescore is explicit; receipts carry input and row digests so stale receipts re-solve; restored 22 solve peaks. - The SPM zero-adult flag is computed in shard/spm-flags (spm_flags.json). - SPM report rates exclude group quarters; the headline ACS-native average uses household-universe measures only. - ASEC quality table rebuilt from a committed script (asec_quality.py). - Hash mirror tested against the port branch's own draws. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ree-year readings - Arm receipts record only the inputs their arm reads (ACS-only arms skip the CPS rows and concepts), concept digests hash contents, and run-grid and report take --newest-year, so a 2023-2025 rerun keeps the ACS-only arms and re-solves the rest. - materialize refuses concept parts cut from another source h5 unless given --replace-stale; the concept store refuses them too. - Re-scoring saved weights requires the same rows digest and inputs; an explicit --rescore otherwise refuses, an implicit one re-solves. - run-grid exits non-zero when an arm fails; report refuses missing receipts unless --allow-missing; spm-flags leaves an unchanged row index alone; the SPM adult rule refuses a person table without role columns. - 13 driver tests; the weight test now checks within-year shape directly. - Memo: at 1.2M one and three CPS years are not separated on CPS-native measures and one year is better on ACS-native ones; more years cost the national file ACS-native accuracy; the 2020-21 allocation shares are the highest of seven years; loss drop 5.2%; rounding and clone-ratio fixes; receipt inputs disclosed as added after the solves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…port reads only current receipts - Concept parts record the concept specs and null-fill summary they were built under; materialize and the concept store treat a mismatch as stale (measures.json names are positional, so a recompile would otherwise relabel old columns). do_arm refuses CPS and ACS stores with different columns. - partcheck: re-materializes a sample of each stored part's households with the current specs and compares with the stored values. - report reads only receipts of its own grid, refuses stale ones unless --allow-stale, and takes --truth and --epochs to check them. - Arm options (--exclude-gq) are part of an arm's inputs. - Tests: recompile staleness, provenance fields, report grid and stale filtering, arm options, spm-flags no-rewrite via a refusing to_parquet, and linearity of the starting weights in the requested mass. - Memo: state error at 600k is slightly worse, not flat; reproduce steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
partcheck re-materialized 200 households from each of the 8 CPS parts and 8 of the 62 ACS parts with the final concept specs and this branch's code: maximum scaled difference 5.8e-8 (float32 rounding). Three CPS parts predate the last compile, which added loss weights only; this confirms their columns. Re-running report on the 60 receipts reproduces every committed table byte for byte. The memo discloses the provenance fields added to the part receipts afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
force-pushed
the
support-mix-bakeoff
branch
from
September 30, 2026 15:05
7463cfa to
8472fba
Compare
…laims Review r5b (Subfleet job 20260930-114531-bakeoff-review-r5b) found four gaps that affect reruns and leave the recorded results alone: - Receipt digests now cover the concept specs and each source's measures.json, so a recompile that leaves the parts in place (new report specs, say) makes every receipt that reads them stale. - partcheck fails a column with a nonfinite value on either side, where a NaN difference used to pass every tolerance. - partcheck refuses an unknown or empty --parts selection, a --per-part under 1 and a negative or nonfinite --tolerance. - report --allow-stale alone leaves stale receipts out instead of also counting them as missing. Nine new tests cover them (test_bakeoff_us_support_mix_tool.py goes from 15 to 24). The memo and README stop pointing at the run's work directory, which no longer exists; the 60 arm receipts and the saved weights went with it. The memo now dates the byte-for-byte report check to 2026-09-30, says the committed materialize/ receipts predate the provenance digests, and restates the rerun follow-up for a fresh work directory. It also gets an active-voice pass and drops the review-round line, which the PR body keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second voice pass over docs/us-support-mix-bakeoff.md: be-verb
identities and comparisons become verbs with an actor ("ACS-native error
runs 44% below the clones'", "the mean capped error ... rises from 0.066 to
0.168", "Budgets count source households"), and one long sentence on larger
ACS fills splits in two. No number or claim changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo counts the 35 national-level held-out targets the arms score, drops its fix history and its self-referential clauses, and cuts four figures no receipt backs. It corrects one range and two roundings, writes September's local file where it said today's, and adds a follow-up on what a rerun on current main meets. No result moves. spm_zero_adult_households reads a stored is_spm_independent_minor_role that holds a value on every row, as the engine does, and refuses a partly null one. support_mix.replicate_block takes the replicate slice from the driver, with a property test that blocks of one budget stay disjoint and replicate 0's blocks nest. New tests drive do_partcheck end to end and check hash_uniform against main's hash_holdout_uniform. The pause file moves under --work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipts README says the per-arm receipts and weights stayed in the run's work directory, which no longer exists. The memo dates its rerun follow-up to main at 3f46d59, ties the age column's ranking to finding why calibration raises age error, and shortens the note on hand-added digests. SPM_ROLE_COLUMN and _spm_role_formula move above the function that reads them, and the hash formula test takes a name that says what it checks. No number moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo and the receipts README no longer say we added the `*_note` digests by hand, which no record shows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
November 11, 2026 04:54
Contributor
Author
|
Handing this PR to the Microcosm hub.
|
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At a fixed budget of source households, which mix of CPS ASEC years, ACS households and CPS location clones gives the national compact file and the CD and county local file the best held-out accuracy? We calibrated 30 arms for each product and scored all 60 calibrations on held-out targets. The memo,
docs/us-support-mix-bakeoff.md, gives the design, verdict and caveats, andexperiments/us-support-mix-bakeoff-20260930/holds the receipts.Merging adds the results and the tooling and changes no build default. A PR that adopts a mix changes how we build the dataset, so it needs its own from-scratch build and certification receipt under the US release build rule. We have chosen one US pool that includes ACS rows, and the national file will take the national mix below from that pool. We have not yet chosen the local mix.
Verdict
We rank arms by the release's own objective on held-out targets: the loss-weighted capped relative error over every held-out CPS-native target (income components, tax items, program receipt) at a level.
For the national compact file, we recommend the newest three CPS years plus ACS to 300,000 source households. With 2022–2024 that comes to 484,080 physical rows, 37% more than the three-year CPS file. Against the CPS-only file, as means over three ACS draws:
Larger ACS fills keep lowering district error (0.266 at 600k, 0.244 at 1.2M), edge state tax items up and do not help state program receipt. The 600k 50/50 fill scores best at state level (0.111) with 2.1 times the rows, but we stop at 300k to hold down build and serving cost.
For the CD and county local file, we recommend the newest three CPS years plus ACS households to 1.2M source households, with no CPS location clones. That comes to 1,384,080 physical rows, 13% fewer than September's 1,588,854-row ACS local file. At 1.2M:
Calibration worsens held-out Census PEP state age cells in every arm: across the 60 calibrations, their mean capped error rises from 0.066 at the starting weights to 0.168. We report the age column, rank no arm on it, and leave finding the cause to a follow-up. We report SPM poverty but neither calibrate to it nor gate on it.
We ran no five-year arm and no arm on the 2023–2025 CPS years. Income years 2020 and 2021 pass the response-quality check but lack Route A enrichment, and we pinned the 2025 ASEC on main (#1044) after we built the Route A export these CPS rows come from. The memo's follow-ups cover both.
What this adds
tools/bakeoff_us_support_mix.pyruns the bake-off throughcompile,shard,truth,materialize,diffcheck,partcheck,run-gridandreport.armsolves a single arm,spm-flagswrites the SPM flag receipt andgridlists the arms. Each run step writes a receipt with wall time and peak RSS, and each arm receipt records digests of the inputs its arm reads.packages/microcosm-build/src/microcosm/build/us_runtime/support_mix.pyholds pure primitives: arm accounting, the nested ACS order and its replicate blocks, clone allocation, starting weights, sparse target-matrix assembly, a mirror of the pending US holdout split and distinct-household ESS.test_us_spine_blindness.pylists it among the runtime modules outside the population-treatment registry.test_us_support_mix.pyadds 25 property, differential and fixture tests, andtest_bakeoff_us_support_mix_tool.pyadds 27 tests of the SPM flag, the staleness guards,partcheckand its wiring, the grid and the report.docs/us-support-mix-bakeoff.md, the memo, includes a CPS year response-quality check thatasec_quality.pycomputes.experiments/us-support-mix-bakeoff-20260930/holds the report tables and the compile, diffcheck, partcheck, SPM-flag, ACS-truth, ASEC-quality and per-part materialization receipts. The per-arm receipts and saved weights stayed in the run's work directory, which no longer exists, so a rerun takes every step fromcompile.We run the engine once per household and reuse its output across arms:
compilegroups the 32,842 registry targets into 419 geography-free concepts;materializecomputes those concepts on household shards with the release tool's own_materialize_target_frame;solve._optimizewith the release's settings.Invariants
Property tests in
test_us_support_mix.py:replicate_blockrefuses a block that runs past the end of the order;An example test checks that district children take their state parent's key.
Differential tests and checks:
hash_uniformequals main'smicrocosm.build.holdout.hash_holdout_uniformfor every drawn key and salt, and reproduces draws from the port branch's own implementation;diffcheckcompared production per-target materialization with concept × mask on 4,000 real households from each source, and across 349 targets the maximum scaled difference came to 0.0;partcheckre-materialized 200 households from each of the 8 CPS parts and 8 of the 62 ACS parts under the final concept specs, and across every column the largest scaled difference from the stored values came to 5.8e-8, float32 rounding.Driver rules in
test_bakeoff_us_support_mix_tool.py:is_spm_independent_minor_rolethat holds a value on every row and uses the role's formula when that column is absent or all null, and it refuses a partly null one;materializerefuses a stored concept part whose receipt names another source h5, other concept specs or another null-fill summary, and rebuilds it only under--replace-stale;arm --rescorere-scores saved weights only when the rows digest and the inputs match;reportreads only its own grid's receipts, refuses missing ones unless run with--allow-missing, and refuses stale ones unless run with--allow-stale, which leaves them out;partcheckfails a column when either side holds a nonfinite value or the scaled difference exceeds the tolerance, refuses an unknown or empty part selection, and refuses a negative or nonfinite tolerance or a sample size under 1;do_partcheckapplies those rules to the parts it re-materializes, checks only the parts it names, and writes no receipt when the selection names no stored part.Review record
--rescoreand keep the original solve cost, computed the SPM flag in code and left group quarters out of SPM rates.reportnow reads only current receipts of its own grid, and concept parts record the specs and null-fill summary they came from; round 5 found receipt reuse still skipping that check, which 4d45da8 closes. Three CPS concept parts predated the lastcompile, so after that round we addedpartcheck.Changes since round 5
e0b80ea merges main at 1d174fc.
4d45da8 fixes round 5's four tool findings, each with a test:
partcheckfails nonfinite columns;partcheckrefuses unknown or empty part selections and a bad tolerance or sample size;report --allow-stalealone writes a report and leaves stale receipts out of the missing count.It discloses the fifth finding in the memo and the README: the
materialize/receipts predate the provenance digests we added to the run's own copies, and on 2026-09-30 their numerical fields matched those copies'. The run's work directory, which held the per-arm receipts and saved weights, no longer exists. The same commit therefore dates the memo's byte-for-byte report reproduction to 2026-09-30 and says a rerun takes every step. We also rewrote the cost notes to name thearms.csvfields that came from the grid log, and cut the memo's review history.fb91442 rewrites the memo's be-verb sentences with active verbs. No number moves.
Changes since round 6
e452e44 answers round 6:
spm_zero_adult_householdsreads a storedis_spm_independent_minor_rolewith a value on every row, as the engine does, and refuses a partly null one, with a test;support_mix.replicate_block, with property tests;do_partcheck, with the engine and the shard loader stubbed, on a nonfinite part and on a narrowed selection, and checkhash_uniformagainst main'shash_holdout_uniform;--work, the--replace-stalehelp names every input a stale part can come from, and the docstrings drop "pending" for the merged hash primitive.No result moves. Both changed test modules pass when a script calls each test directly (25 and 27 tests).
e0bf3ac merges main at 3f46d59; main's commits since 1d174fc touch none of this PR's files.
Changes since round 7
SPM_ROLE_COLUMNand_spm_role_formulamove above the function that reads them, and the hash formula test takes the nametest_hash_uniform_follows_its_formula_and_refuses_empty_inputs. No number moves.Changes since round 8
*_notedigests afterwards, in the memo and the receipts README, where e477748 said "by hand". No number moves.🤖 Generated with Claude Code