Skip to content

US support-mix bake-off: CPS years vs ACS rows vs location clones at fixed budgets - #1067

Merged
MaxGhenis merged 20 commits into
mainfrom
support-mix-bakeoff
Nov 11, 2026
Merged

MaxGhenis merged 20 commits into
mainfrom
support-mix-bakeoff

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

At a fixed budget of source households, which mix of CPS ASEC years, ACS households and CPS location clones gives the national compact file and the CD and county local file the best held-out accuracy? We calibrated 30 arms for each product and scored all 60 calibrations on held-out targets. The memo, docs/us-support-mix-bakeoff.md, gives the design, verdict and caveats, and experiments/us-support-mix-bakeoff-20260930/ holds the receipts.

Merging adds the results and the tooling and changes no build default. A PR that adopts a mix changes how we build the dataset, so it needs its own from-scratch build and certification receipt under the US release build rule. We have chosen one US pool that includes ACS rows, and the national file will take the national mix below from that pool. We have not yet chosen the local mix.

Verdict

We rank arms by the release's own objective on held-out targets: the loss-weighted capped relative error over every held-out CPS-native target (income components, tax items, program receipt) at a level.

For the national compact file, we recommend the newest three CPS years plus ACS to 300,000 source households. With 2022–2024 that comes to 484,080 physical rows, 37% more than the three-year CPS file. Against the CPS-only file, as means over three ACS draws:

  • held-out state CPS-native error falls from 0.119 to 0.113, and district error from 0.353 to 0.291;
  • state income components improve (0.250 to 0.224), state tax items worsen (0.103 to 0.113), and state program receipt does not measurably change (0.208 against 0.198, draw SD 0.014);
  • every ACS-native level improves (district 0.236 to 0.188, county 0.307 to 0.259).

Larger ACS fills keep lowering district error (0.266 at 600k, 0.244 at 1.2M), edge state tax items up and do not help state program receipt. The 600k 50/50 fill scores best at state level (0.111) with 2.1 times the rows, but we stop at 300k to hold down build and serving cost.

For the CD and county local file, we recommend the newest three CPS years plus ACS households to 1.2M source households, with no CPS location clones. That comes to 1,384,080 physical rows, 13% fewer than September's 1,588,854-row ACS local file. At 1.2M:

  • held-out district CPS-native error ties the clones (0.169 against 0.170) with 55% of their rows, and state error falls below theirs (0.112 against 0.114);
  • ACS-native error runs 44% below the clones' in districts and 31% below in counties;
  • the clones and the CPS-only file beat the recommendation on state program receipt (0.218 and 0.220 against 0.256) and state tax items (0.096 and 0.101 against 0.117).

Calibration worsens held-out Census PEP state age cells in every arm: across the 60 calibrations, their mean capped error rises from 0.066 at the starting weights to 0.168. We report the age column, rank no arm on it, and leave finding the cause to a follow-up. We report SPM poverty but neither calibrate to it nor gate on it.

We ran no five-year arm and no arm on the 2023–2025 CPS years. Income years 2020 and 2021 pass the response-quality check but lack Route A enrichment, and we pinned the 2025 ASEC on main (#1044) after we built the Route A export these CPS rows come from. The memo's follow-ups cover both.

What this adds

  • tools/bakeoff_us_support_mix.py runs the bake-off through compile, shard, truth, materialize, diffcheck, partcheck, run-grid and report. arm solves a single arm, spm-flags writes the SPM flag receipt and grid lists the arms. Each run step writes a receipt with wall time and peak RSS, and each arm receipt records digests of the inputs its arm reads.
  • packages/microcosm-build/src/microcosm/build/us_runtime/support_mix.py holds pure primitives: arm accounting, the nested ACS order and its replicate blocks, clone allocation, starting weights, sparse target-matrix assembly, a mirror of the pending US holdout split and distinct-household ESS. test_us_spine_blindness.py lists it among the runtime modules outside the population-treatment registry.
  • test_us_support_mix.py adds 25 property, differential and fixture tests, and test_bakeoff_us_support_mix_tool.py adds 27 tests of the SPM flag, the staleness guards, partcheck and its wiring, the grid and the report.
  • docs/us-support-mix-bakeoff.md, the memo, includes a CPS year response-quality check that asec_quality.py computes.
  • experiments/us-support-mix-bakeoff-20260930/ holds the report tables and the compile, diffcheck, partcheck, SPM-flag, ACS-truth, ASEC-quality and per-part materialization receipts. The per-arm receipts and saved weights stayed in the run's work directory, which no longer exists, so a rerun takes every step from compile.

We run the engine once per household and reuse its output across arms:

  • compile groups the 32,842 registry targets into 419 geography-free concepts;
  • materialize computes those concepts on household shards with the release tool's own _materialize_target_frame;
  • each arm assembles concept × state or district mask as a sparse matrix and calls the production solve._optimize with the release's settings.

Invariants

Property tests in test_us_support_mix.py:

  • an arm's CPS, ACS and clone source households sum to its budget, and the primitives refuse an infeasible arm instead of truncating it;
  • ACS selections nest across budgets and don't depend on input order;
  • replicate blocks of one budget never share a household, replicate 0's blocks nest by budget, and replicate_block refuses a block that runs past the end of the order;
  • clone copies sum to the requested count and differ by at most one across households;
  • starting weights stay positive and sum to the requested mass, each CPS year gets an equal share, CPS and ACS split in proportion to source households, and a row's copies sum back to its weight;
  • distinct-household ESS never exceeds the row ESS or the number of distinct units;
  • the holdout role depends only on the group key, and vintage tokens never change a role.

An example test checks that district children take their state parent's key.

Differential tests and checks:

  • the sparse target matrix equals a naive dense construction, including cells that carry several targets;
  • hash_uniform equals main's microcosm.build.holdout.hash_holdout_uniform for every drawn key and salt, and reproduces draws from the port branch's own implementation;
  • diffcheck compared production per-target materialization with concept × mask on 4,000 real households from each source, and across 349 targets the maximum scaled difference came to 0.0;
  • partcheck re-materialized 200 households from each of the 8 CPS parts and 8 of the 62 ACS parts under the final concept specs, and across every column the largest scaled difference from the stored values came to 5.8e-8, float32 rounding.

Driver rules in test_bakeoff_us_support_mix_tool.py:

  • the tool flags a household for SPM exclusion if and only if one of its SPM units has no member the pinned engine counts as an adult; like the engine, it reads a stored is_spm_independent_minor_role that holds a value on every row and uses the role's formula when that column is absent or all null, and it refuses a partly null one;
  • the tool refuses to flag from a person table without household-role columns instead of falling back to age;
  • an arm's receipt records digests of the inputs its arm reads and of no others;
  • a concept digest moves with the parts' contents and with what their columns mean, the compiled concept specs and the source's measure list;
  • materialize refuses a stored concept part whose receipt names another source h5, other concept specs or another null-fill summary, and rebuilds it only under --replace-stale;
  • arm --rescore re-scores saved weights only when the rows digest and the inputs match;
  • moving the grid to newer CPS years keeps the ACS-only arms' labels;
  • replicate tables cover only groups with more than one draw;
  • report reads only its own grid's receipts, refuses missing ones unless run with --allow-missing, and refuses stale ones unless run with --allow-stale, which leaves them out;
  • partcheck fails a column when either side holds a nonfinite value or the scaled difference exceeds the tolerance, refuses an unknown or empty part selection, and refuses a negative or nonfinite tolerance or a sample size under 1;
  • do_partcheck applies those rules to the parts it re-materializes, checks only the parts it names, and writes no receipt when the selection names no stored part.

Review record

  • Round 1 found three High and six Medium findings, and we fixed all of them: collapsed multi-target cells, dropped unsupported holdout targets, overlapping concept parts, national loss weights, group-quarters handling and CPS key checks.
  • Round 2 found two High and seven Medium findings, and we fixed them or stated them in the memo. It found the national gain resting on seed 0 alone, so we moved the verdict to three-draw means and the loss-weighted headline. We limited a "more CPS years" claim to the national product and added the counter-readings and the age-cell finding. We also made re-scoring require --rescore and keep the original solve cost, computed the SPM flag in code and left group quarters out of SPM rates.
  • Round 3 found no Highs, judged the PR merge-ready and raised four Mediums, which we fixed:
    • the one-year against three-year readings, which we re-read under the noise rule;
    • a false claim about 2020–21 item allocation;
    • staleness bugs that would have re-solved the ACS-only arms and reused stale CPS concept parts in a 2023–2025 rerun;
    • a rescore path that could score weights against changed inputs.
  • Round 4 found no Highs, judged the PR merge-ready and raised two rerun-path gaps and six Lows, which we fixed. report now reads only current receipts of its own grid, and concept parts record the specs and null-fill summary they came from; round 5 found receipt reuse still skipping that check, which 4d45da8 closes. Three CPS concept parts predated the last compile, so after that round we added partcheck.
  • Round 5 (GPT-6.1 Sol) reviewed 8dc5e7d and 8472fba, judged the PR merge-ready and raised two Medium and three Low findings for reruns, which 4d45da8 addresses (below).
  • Round 6 reviewed the whole PR at fb91442 and found the code, the tests and the main merge sound. It raised three blocking findings in the memo's text (a wrong count, fix history and self-referential clauses), 13 Lows and five groups of nits, which e452e44 addresses (below).
  • Round 7 checked e452e44 and the merge, found every round-6 finding resolved and the code, tests, merge and numbers sound, flagged one wrong README sentence, and raised two Lows and eleven nits, all answered below.
  • Round 8 checked e477748, found every round-7 finding resolved, and flagged one phrase no record supports, which 3653f69 replaces.

Changes since round 5

  • e0b80ea merges main at 1d174fc.

  • 4d45da8 fixes round 5's four tool findings, each with a test:

    • receipt digests now cover the concept specs and the measure list, so a specs-only recompile makes old receipts stale;
    • partcheck fails nonfinite columns;
    • partcheck refuses unknown or empty part selections and a bad tolerance or sample size;
    • report --allow-stale alone writes a report and leaves stale receipts out of the missing count.

    It discloses the fifth finding in the memo and the README: the materialize/ receipts predate the provenance digests we added to the run's own copies, and on 2026-09-30 their numerical fields matched those copies'. The run's work directory, which held the per-arm receipts and saved weights, no longer exists. The same commit therefore dates the memo's byte-for-byte report reproduction to 2026-09-30 and says a rerun takes every step. We also rewrote the cost notes to name the arms.csv fields that came from the grid log, and cut the memo's review history.

  • fb91442 rewrites the memo's be-verb sentences with active verbs. No number moves.

Changes since round 6

  • e452e44 answers round 6:

    • the memo counts the 35 national-level held-out targets the arms score, where it said 36, cuts its last fix history and its self-referential clauses, and drops five claims no receipt backs: two memory peaks, the Route A run's smoke gate, the person-input counts and a Census caveat;
    • it corrects the age-error range (0.151 to 0.184 across the national arms), two roundings (2.52M rows, 1.7 points) and the 50/50 fill's comparison, writes September's local file where it said today's, and adds a follow-up on what a rerun on main as of 3f46d59 must handle;
    • spm_zero_adult_households reads a stored is_spm_independent_minor_role with a value on every row, as the engine does, and refuses a partly null one, with a test;
    • the driver's replicate slice moves into support_mix.replicate_block, with property tests;
    • new tests drive do_partcheck, with the engine and the shard loader stubbed, on a nonfinite part and on a narrowed selection, and check hash_uniform against main's hash_holdout_uniform;
    • the pause file moves under --work, the --replace-stale help names every input a stale part can come from, and the docstrings drop "pending" for the merged hash primitive.

    No result moves. Both changed test modules pass when a script calls each test directly (25 and 27 tests).

  • e0bf3ac merges main at 3f46d59; main's commits since 1d174fc touch none of this PR's files.

Changes since round 7

  • e477748 makes the receipts README say that the per-arm receipts and weights stayed in the run's work directory. In the memo it dates the rerun follow-up to main at 3f46d59, rewrites three passages and rewraps a fourth. SPM_ROLE_COLUMN and _spm_role_formula move above the function that reads them, and the hash formula test takes the name test_hash_uniform_follows_its_formula_and_refuses_empty_inputs. No number moves.

Changes since round 8

  • 3653f69 says we added the *_note digests afterwards, in the memo and the receipts README, where e477748 said "by hand". No number moves.

🤖 Generated with Claude Code

MaxGhenis and others added 13 commits September 30, 2026 11:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… national_state loss weights, GQ-consistent district population, part-overlap and cache guards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om household structure, score every holdout level for both products, rescore saved weights

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-readings, receipts for every claim

- Verdict rests on the release's loss-weighted held-out error and three-draw
  means; program receipt is a wash nationally, state tax items a cost; the
  local file's losses to clones and the national 50/50 alternative are stated.
- Rescore keeps solve-time cost; --rescore is explicit; receipts carry input
  and row digests so stale receipts re-solve; restored 22 solve peaks.
- The SPM zero-adult flag is computed in shard/spm-flags (spm_flags.json).
- SPM report rates exclude group quarters; the headline ACS-native average
  uses household-universe measures only.
- ASEC quality table rebuilt from a committed script (asec_quality.py).
- Hash mirror tested against the port branch's own draws.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ree-year readings

- Arm receipts record only the inputs their arm reads (ACS-only arms skip the
  CPS rows and concepts), concept digests hash contents, and run-grid and
  report take --newest-year, so a 2023-2025 rerun keeps the ACS-only arms and
  re-solves the rest.
- materialize refuses concept parts cut from another source h5 unless given
  --replace-stale; the concept store refuses them too.
- Re-scoring saved weights requires the same rows digest and inputs; an
  explicit --rescore otherwise refuses, an implicit one re-solves.
- run-grid exits non-zero when an arm fails; report refuses missing receipts
  unless --allow-missing; spm-flags leaves an unchanged row index alone; the
  SPM adult rule refuses a person table without role columns.
- 13 driver tests; the weight test now checks within-year shape directly.
- Memo: at 1.2M one and three CPS years are not separated on CPS-native
  measures and one year is better on ACS-native ones; more years cost the
  national file ACS-native accuracy; the 2020-21 allocation shares are the
  highest of seven years; loss drop 5.2%; rounding and clone-ratio fixes;
  receipt inputs disclosed as added after the solves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…port reads only current receipts

- Concept parts record the concept specs and null-fill summary they were
  built under; materialize and the concept store treat a mismatch as stale
  (measures.json names are positional, so a recompile would otherwise
  relabel old columns). do_arm refuses CPS and ACS stores with different
  columns.
- partcheck: re-materializes a sample of each stored part's households with
  the current specs and compares with the stored values.
- report reads only receipts of its own grid, refuses stale ones unless
  --allow-stale, and takes --truth and --epochs to check them.
- Arm options (--exclude-gq) are part of an arm's inputs.
- Tests: recompile staleness, provenance fields, report grid and stale
  filtering, arm options, spm-flags no-rewrite via a refusing to_parquet,
  and linearity of the starting weights in the requested mass.
- Memo: state error at 600k is slightly worse, not flat; reproduce steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
partcheck re-materialized 200 households from each of the 8 CPS parts and
8 of the 62 ACS parts with the final concept specs and this branch's code:
maximum scaled difference 5.8e-8 (float32 rounding). Three CPS parts predate
the last compile, which added loss weights only; this confirms their
columns. Re-running report on the 60 receipts reproduces every committed
table byte for byte. The memo discloses the provenance fields added to the
part receipts afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 7 commits November 9, 2026 12:36
…laims

Review r5b (Subfleet job 20260930-114531-bakeoff-review-r5b) found four gaps
that affect reruns and leave the recorded results alone:

- Receipt digests now cover the concept specs and each source's
  measures.json, so a recompile that leaves the parts in place (new report
  specs, say) makes every receipt that reads them stale.
- partcheck fails a column with a nonfinite value on either side, where a
  NaN difference used to pass every tolerance.
- partcheck refuses an unknown or empty --parts selection, a --per-part
  under 1 and a negative or nonfinite --tolerance.
- report --allow-stale alone leaves stale receipts out instead of also
  counting them as missing.

Nine new tests cover them (test_bakeoff_us_support_mix_tool.py goes from 15
to 24).

The memo and README stop pointing at the run's work directory, which no
longer exists; the 60 arm receipts and the saved weights went with it. The
memo now dates the byte-for-byte report check to 2026-09-30, says the
committed materialize/ receipts predate the provenance digests, and restates
the rerun follow-up for a fresh work directory. It also gets an active-voice
pass and drops the review-round line, which the PR body keeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second voice pass over docs/us-support-mix-bakeoff.md: be-verb
identities and comparisons become verbs with an actor ("ACS-native error
runs 44% below the clones'", "the mean capped error ... rises from 0.066 to
0.168", "Budgets count source households"), and one long sentence on larger
ACS fills splits in two. No number or claim changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo counts the 35 national-level held-out targets the arms score,
drops its fix history and its self-referential clauses, and cuts four
figures no receipt backs. It corrects one range and two roundings, writes
September's local file where it said today's, and adds a follow-up on
what a rerun on current main meets. No result moves.

spm_zero_adult_households reads a stored is_spm_independent_minor_role
that holds a value on every row, as the engine does, and refuses a partly
null one. support_mix.replicate_block takes the replicate slice from the
driver, with a property test that blocks of one budget stay disjoint and
replicate 0's blocks nest. New tests drive do_partcheck end to end and
check hash_uniform against main's hash_holdout_uniform. The pause file
moves under --work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipts README says the per-arm receipts and weights stayed in the
run's work directory, which no longer exists. The memo dates its rerun
follow-up to main at 3f46d59, ties the age column's ranking to finding
why calibration raises age error, and shortens the note on hand-added
digests. SPM_ROLE_COLUMN and _spm_role_formula move above the function
that reads them, and the hash formula test takes a name that says what
it checks. No number moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo and the receipts README no longer say we added the `*_note`
digests by hand, which no record shows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 11, 2026 04:54
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Handing this PR to the Microcosm hub.

  • Head: 3653f69.
  • Review: an independent Opus 5.5 reviewer approved this head in round 9 (~/PolicyEngine/_reviews/microcosm-1067-final/review-r9/review_r9.md). Rounds 6 to 8 asked for changes, and the commits in the body's change lists answer them. Both changed test modules pass when a script calls each test directly at e477748 (25 and 27 tests); 3653f69 changes two prose lines only.
  • CI: lint, wheels and select-countries pass on this head, and the engine, installed-emitter and integration-uk jobs are still running. The merge waits for every check to pass on this head.
  • Depends on: nothing. Merging changes no build default.
  • Merge message: if this lands as a squash, its message should follow the body's change lists. The pushed messages of e452e44 and e477748 say "four figures", "current main" and "hand-added digests", and the body corrects all three.
  • Follow-ups: the memo's Follow-ups section lists a 2023–2025 rerun, five-year arms, a re-solve once target_split merges, and the cause of the age-cell loss. Adopting a mix needs its own from-scratch build PR under the US release build rule.

@MaxGhenis
MaxGhenis merged commit 89660a3 into main Nov 11, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

I merged this at 3653f69. All 12 checks passed on that head, and an independent Opus 5.5 review approved it. Its CI ran 18 seconds before #1210 reached main, and both add an entry to test_us_spine_blindness.py, so I ran that test and both PRs' own tests on the merged tree first: 576 passed.

@MaxGhenis
MaxGhenis deleted the support-mix-bakeoff branch November 11, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant