Repository navigation
Conversation
…1 state cells (#297) The fiscal-refresh build can join each person's CPS ASEC A_HGA from the pinned Census person member (exact income year and PERIDNUM, with the Census household, line and age checked) and classify it into the seven ACS S1501 population-25-and-over classes as cps_educational_attainment. The stored-input register flags the column as auxiliary: carried for conditioning and calibration, not consumed by rules. The US target compiler binds the S1501 state 25+ detail cells as person counts of the column, and the tool's materializer counts them. The pinned Chronicle feed has no S1501 facts, so nothing binds until PolicyEngine/chronicle#347 merges and the feed is re-pinned. experiments/modal_297.py runs the tests and the receipt on Modal; only it reads the draft package, from the fork. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…Z-e2621e231) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vahid-ahmadi
left a comment
There was a problem hiding this comment.
The new runtime module isn't classified, so engine-free fails on both Pythons (packages/microcosm-build/tests/engine_free/us/test_us_spine_blindness.py:3317). test_us_runtime_module_classification_is_complete requires every file in us_runtime/ to be either a registered population operator (_SPINE_BLIND_OPERATOR_MODULES) or a reviewed non-registry module (_OTHER_US_RUNTIME_MODULES). The PR adds packages/microcosm-build/src/microcosm/build/us_runtime/educational_attainment.py but doesn't touch test_us_spine_blindness.py. In CI run 39824121242 both engine-free lanes fail with unclassified: ('educational_attainment.py',). Each lane stops at that first failure ("stopping after 1 failures", 2,408 passed on 3.13), so most of the engine-free suite has not run at this head.
Suggested fix: add educational_attainment.py to whichever classification fits, with that list's spine-access checks. A green run after that is the first full engine-free result for this branch.
Review drafted with AI assistance and checked against the code at 7d38af9.
|
David, Vahid found that |
Six files conflicted, each where main's federal student loan work (microcosm#1073, #1074) and this branch's attainment column touched the same lines. Both sides are kept: - tools/build_us_fiscal_refresh_release.py: both stages, both flag groups and one restated-age-band set that names both materializers. - fiscal_targets.py: both import blocks. - stored_inputs.py, its test support and test_us_stored_input_register.py: the register holds the four loan inputs and the auxiliary attainment column (30 + 6 + 6 + 6 + 1 + 2 + 1 + 4 + 1 entries). - docs/us-fact-to-target.md: both sections. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
test_us_runtime_module_classification_is_complete refused the new us_runtime module, and both engine-free lanes stopped at that first failure. The module joins the auxiliary attainment class from the pinned ASEC person member and reads no spine or support-channel column, so it goes in _OTHER_US_RUNTIME_MODULES beside spm_independence_role.py. The changelog now says what the S1501 targets wait for: the feed pin took only the Federal Student Aid packages of chronicle#347 (microcosm#1506), so the S1501 record sets still have to join the feed's scope. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
David, with |
us_runtime.fiscal_targets imports the attainment target helpers, so the gate battery's binding identity reaches one more module than main's. The count test in test_transport_evidence_kernels.py lists each such module by name (as it does the FSA student loan targets from #1074). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Part of #297 (the student debt pilot, Project 2: PolicyEngine/us-student-debt#1, ROI task us-student-debt#5).
Summary
The US fiscal-refresh build can now carry educational attainment as an auxiliary person column,
cps_educational_attainment, beside the raw CensusA_HGAit is classified from. The US target compiler binds the ACS 1-year S1501 state cells for the population 25 and over as person counts of that column, and the tool's materializer counts them. Nothing binds in a release yet: the pinned Chronicle feed has no S1501 facts (see "What the S1501 targets wait on").--cps-educational-attainment(opt-in) joins each person to the pinned official CPS ASEC person member on (income year,PERIDNUM) and checks the Census household (source_household_id=PH_SEQ),P_SEQ,A_LINENOandA_AGE. It never predicts or defaults a value, and a PUF support clone takes its source person's code.--asec-attainment-source INCOME_YEAR=PATHsupplies local archives; unmapped years are fetched and verified against the same pins (spm_role_source.ASEC_SPM_ROLE_SOURCES). A signal gate refuses missing or unknown values, a universe mismatch, and implausible below-diploma or bachelor's-or-higher shares.PERIDNUM).Classes and code mapping
Seven classes, the S1501 population-25-and-over categories (Chronicle value ids reused), plus
not_in_universe. Merging the first two gives the six classes the issue asks for (US_EDUCATIONAL_ATTAINMENT_SIX_CLASSES).A_HGASCHLless_than_9th_grade9th_to_12th_grade_no_diplomahigh_school_graduate_includes_equivalencysome_college_no_degreeassociates_degree(occupational 41 + academic 42)bachelors_degreegraduate_or_professional_degreenot_in_universeSources checked code for code:
A_HGA, "Item 18h - Educational attainment": the ASEC public-use data dictionaries for survey years 2023-2026,https://www2.census.gov/programs-surveys/cps/datasets/<YYYY>/march/asec<YYYY>_ddl_pub_full.pdf(PDF pages 22, 23, 24, 28). All four list 0 and 31-46 identically. In the three pinned members,A_HGA == 0exactly whenA_AGE < 15.SCHL:https://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMS_Data_Dictionary_2024.txt.S1501_C01_006E-013E(25 and over) and001E-005E(18 to 24).The
SCHLmap is defined and tested, but the ACS-local lane does not readSCHLfrom PUMS yet (out of scope here).ROI classes (
US_DEGREE_ROI_CLASSES): two-year degree =associates_degree, four-year degree =bachelors_degree, each against thehigh_school_graduate_includes_equivalencybaseline.Column choice
A Microcosm auxiliary column, not a policyengine-us input: the analysis joins it on
person_id, and no policyengine-us change is needed. Recorded on #297: #297 (comment).For the published default
populace-us-2024-spm-20260915, the receipt run also wrote aperson_id, A_HGA, cps_educational_attainmentsidecar (166,321 rows, sha25634994413e255524f16ab9d985bd8ca8470deab1149b3ee395d2ee307c9d7e62b). It is in the run's Modal volume (microcosm-297-cache,results/20261022T171517Z-e2621e231/) and is not committed, so the ROI work can start before a new build exists.Provenance flag
microcosm.data.stored_inputs.US_AUXILIARY_STORED_COLUMNSis a new register of auxiliary columns, carried for conditioning and calibration and not consumed by rules. It is merged intoUS_STORED_NON_VARIABLE_COLUMNS, so the stored-input gates accept the column. Its reason starts with "Auxiliary:" and names its producer, and the register tests bind it toeducational_attainment.US_CPS_EDUCATIONAL_ATTAINMENT_COLUMN. The stage's receipt (us_cps_educational_attainment.jsonin the build artifacts) recordscolumn_role: auxiliary.Targets and compile path
fiscal_targets._acs_attainment_reference_from_factadmitscensus_acs.acs1_<year>.s1501.state_attainment_25_plus.<st>detail rows for the 50 states and DC. The family iscensus_educational_attainment, the materializerpopulation_attainment, and the person count is taken over the cell's half-open age band. It leaves the universe totals unbound, because PEP binds population by state and age. It also leaves the 18-to-24 universe unbound (the ACS counts dormitory residents where they study), and Puerto Rico. A changed category, age band or measure raises instead of binding._population_attainment_household_valuescounts the cell, andpopulation_attainmentjoins the materializers that accept a restated age bound. A frame without the column refuses to materialize these targets rather than binding zeros.fiscal_target_references.json's CD-to-state reconciliation rule matches oneducational_attainment_categorytoo, since one state's seven rows differ only in class.census_acs.educational_attainment.State totals vs S1501 (receipt)
experiments/297-cps-educational-attainment-receipt.md(JSON:experiments/receipts/297-cps-educational-attainment-s1501.json). The stage ran on the published default at its calibrated weights; the S1501 facts were compiled through the real compiler (357 targets, 0 mismatches against the raw Census response) and materialized with the tool's own code.National, 25 and over:
less_than_9th_gradein small states. A standalone per-state entropy calibration to the cells, which is not the release solve, fits exactly. It keeps at least 82.6% of each state's Kish ESS, but its largest weight ratio is 131 (LA). Merging the two below-diploma classes thickens those cells.What the S1501 targets wait on
PolicyEngine/chronicle#347 merged on 2026-10-22 with the S1501 package
census/acs_s1501_state_2024. The US feed pin has since moved to Chronicle41b2fb2b(#1506), which took that pull request's Federal Student Aid packages only:chronicle_feed_scope.jsonlists no S1501 record set, so the pinned feed still carries none.us/chronicle_feed.json,chronicle_feed_scope.json) adds the S1501 pairs, and the parity manifest regenerates and declarescensus_acs.acs1_2024compiled. A release that compiles those rows must run with--cps-educational-attainment.Everything else here works without it.
experiments/modal_297.pystill reads the package from the fork's draft (DTrim99/chronicle@81e3f986, labelled TEMPORARY in the file), as the receipt run did. The facts there are content-identical to a local build at4448a7cand at81e3f98. No lockfile, feed pin or package resource points at the fork.Merged with main on 2026-11-19
fiscal_targets.py, the stored-input register with its test support and test, anddocs/us-fact-to-target.md. Both sides are kept. The register holds the four loan inputs and the attainment column.educational_attainment.pyis in_OTHER_US_RUNTIME_MODULESintest_us_spine_blindness.py, besidespm_independence_role.py. It reads no spine or support-channel column.fiscal_targets.pyimports this module, so the gate battery's binding identity reaches one more module than main's.test_transport_evidence_kernels.pylists it inBINDING_MODULES_ADDED_AFTER_BASE, beside the FSA targets module from Calibrate US federal student loan balances to Federal Student Aid portfolio targets and show them on the dashboard #1074. The first full engine-free run (on8446eb2ee) failed only that count.us/gates.jsonand does not run in shadow, unlike the gates main declared in US: use the shared Microcosm modules where no output changes #1585. Declaring it adds aus_nationalentry and changes that scope's pinned digests.SCF education-debt imputation (#1073 / PR #1246), note only
#1073's hold-out already tried SCF
educas a predictor.build_plus_educationraised household rank correlation from 0.168 to 0.207 but worsened the share-with-debt error from -0.30 pp to -0.92 pp. This column supplies the host side: map SCFeducand the host reference person's class to the same seven classes. The share error argues for adding it alongside the FSA borrower-count calibration (#1074), not as a bare predictor. #1246 is unchanged.Modal runs (
experiments/modal_297.py, appmicrocosm-297-experiments, volumemicrocosm-297-cache)20261022T171517Z-e2621e231(fc-01M5J9R996YS9K51QVQQAFTVZ1), at this branch's head: tests (the new module, fiscal targets, the stored-input register, the Chronicle feed, labels, the data shard's stored-input tests, the builder's attainment/age/ledger-filter subset,ci_test_plan.py verify), then the engine-contract register, spec-bundle and bundle-contract tests, then the receipt. All three passed (tests 420 s, contract 489 s, receipt 123 s; peak 4.6 GB). The receipt is committed in the follow-up commit.20261022T160713Z-d25305d52(fc-01M5J5VMMRPHA6206FSPZBFQPC): the probe of the host and the Census members that set the identity checks. The host'sPH_SEQis globalized, andsource_household_idcarries Census's. Every hostPERIDNUMis found in its year's member.Notes
🤖 Generated with Claude Code