Repository navigation
Declare list edits on reform-coverage probes instead of pinning whole lists - #1197
Merged
Merged
Conversation
… (WIP core) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The apply-time guard accepted only the baseline, but PolicyEngine-US applies a reform set twice while building CountryTaxBenefitSystem (and again in Simulation), so every list-edit reform raised on the release scorer's path. The guard now accepts the reform's own resolution and reads the target system's list over the whole edit period. - list_parameter_baseline refuses a non-list value inside the period with a ValueError (a null breakpoint used to raise TypeError past callers) and names every change instant in date order. - Baseline reads in _build_reform run inside the probe-id error prefix; the gate copies resolved lists into its evidence. - ReformCoverageProbe refuses a non-mapping list_edits, empty paths, and a list pinned at any depth of parameter_changes; periods must be strict YYYY-MM-DD (fromisoformat also takes ISO week dates). - Tests: shipped-manifest assertions on declared edits, loader/schema and gate evidence tests, Hypothesis properties for resolution, validation, loader round-trip and the period-constancy read, engine consumers moved to _build_reform, and real-engine drift, guard, loud-failure and differential tests plus a live manifest-current check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's NIIT pass-through probe (microcosm#1239) pinned the whole gov.irs.investment.income.sources list, which schema_version 2 refuses. It now declares the edit: remove passive_partnership_s_corp_income over 2024. On PolicyEngine-US 2.2.1 that resolves to exactly the list main pinned. The manifest regenerates; against main it differs only in schema_version, the description and the ten list-edit probes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1. The apply-time guard checks each stretch of the edit period on its own
(check_resolved_list_edit_target): every stretch must hold the baseline
or the reform's own resolution. An earlier application of the same edit
over an overlapping period is accepted, and the two cover the union.
2. CountryTaxBenefitSystem applies reforms before it backdates each
parameter's first value, so an edit starting on or before a list's first
breakpoint at that first pass would be backdated outside its period
(QBI's 2018 edit held from 2015 through 2018 on PolicyEngine-US 2.2.1).
Every reader now refuses such an edit, and installed resolution reads
the first-pass breakpoints, so resolution and both constructors agree.
3. The post-export plan's dry run flags its simulate RECORDS_REQUESTS_ONLY;
the smoke gate then hands it probes and builds no reform, so recording
resolves no list edit and needs no engine.
4. The loader refuses "list_edits": null instead of reading it as {}.
5. The docstrings and comments state the length and inverse guarantees
for baselines without repeated items only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
The Microcosm hub now owns this PR and lands it.
|
Contributor
Author
|
Landed by the Microcosm hub as 90e4d94 (merge of 8721c84 into main 57a32f8; main's tree equals the tested merge).
|
MaxGhenis
added a commit
that referenced
this pull request
Nov 12, 2026
Main 90e4d94 is #1197 alone: reform-coverage probes that change a list-valued parameter declare list_edits (the items they remove or add) instead of pinning the whole list, and the manifest's schema_version 2 refuses a list pinned in parameter_changes. #779 pins three whole immigration-status lists, so they are converted here, and three files conflicted. Conflicts, each resolved by keeping both sides: - packages/microcosm-build/src/microcosm/build/us_runtime/release_input_coverage.py (4 hunks): the branch's REFORM_COVERAGE_MEASURE_UNITS beside main's ListParameterEdit block; ReformCoverageProbe keeps the branch's measure_unit field and main's list_edits field; the assert_release_input_coverage_manifest_current docstring keeps the branch's measure-unit bullet and adds main's list-edit bullet; the per-probe loop runs the branch's _probe_measure_drift check, then main's list-edit resolution. - packages/microcosm-build/tests/engine_contract/us/test_release_input_coverage.py (2 hunks): main's import block (a superset of the branch's); the branch's TestProbeMeasureUnits and main's TestShippedListEditsOnTheInstalledEngine both kept. - packages/microcosm-build/tests/engine_scenario/us/test_us_reform_coverage_probe_engine_drift.py (3 hunks): main's list-edit versions, extended with the branch's H.R.1 restoration probes. The docstring bullet names them, the list-edit probe set includes _RESTORED_ITEMS, and the resolve test expects each restoration probe's restored status set. In main's auto-merged lines, the declares-loop compares a restoration probe's (remove, add) as sets with _RESTORED_ITEMS, and _LIST_PARAMETERS gains the three status lists the shipped probes now edit. Two semantic conflicts in packages/microcosm-build/tests/engine_free/shared/test_release_input_coverage.py, which git merged without markers: - The branch's test_people_count_probe_states_effect_and_floor_in_people built a synthetic probe that pinned a whole list, which #1197's ReformCoverageProbe refuses. It now declares list_edits {gov.example.statuses: add REFUGEE over 2027-01-01.2100-12-31}: still a status-list restoration scored in persons. - Main's _GraphEngine fake offered only variables(), but the merged check also runs the branch's _probe_measure_drift, which reads variable_metadata and variable_unit. Eight of main's TestManifestListEditResolution tests failed with AttributeError. The fake now answers both for each shipped probe's budget_measure, in the unit the probe declares, so those tests again see only the list-edit findings they set up. Production code is unchanged. Converted probes (tools/build_us_release_input_coverage_manifest.py), each now parameter_changes {} with one list edit over its old pinned period: - hr1_medicaid_humanitarian_eligibility_restoration: gov.hhs.medicaid.eligibility.eligible_immigration_statuses, 2026-10-01.2100-12-31, add REFUGEE, ASYLEE, DEPORTATION_WITHHELD, CONDITIONAL_ENTRANT, PAROLED_ONE_YEAR. - hr1_aca_lawful_presence_restoration: gov.aca.ineligible_immigration_statuses, 2027-01-01.2100-12-31, remove TPS, REFUGEE, ASYLEE, DEPORTATION_WITHHELD, PAROLED_ONE_YEAR. - hr1_snap_humanitarian_eligibility_restoration: gov.usda.snap.eligibility.eligible_immigration_statuses, 2025-07-01.2100-12-31, add the same five as Medicaid. hr1_aca_below_fpl_exception_restoration pins a boolean and is unchanged. On policyengine-us 2.2.1 and policyengine-core 3.32.5 (the uv.lock pins), each edit's start is after its list's first breakpoint (2018-01-01). Each resolves to the old pinned list's items, and the reformed system holds the same set as the old pinned reform at every instant checked from 2018-01-01 to 2101-01-01. ACA keeps the old order exactly. Medicaid and SNAP put CUBAN_HAITIAN_ENTRANT third, from the baseline, instead of sixth; every 2.2.1 reader of the three lists uses np.isin, so order cannot change a score. Tests and documents that named the old count: - packages/microcosm-build/tests/engine_free/shared/test_release_input_coverage.py: _SHIPPED_LIST_EDIT_PROBE_IDS gains the three probes through a new _HR1_RESTORATION_EDITS table, and the shipped-manifest test is now ..._exactly_thirteen_list_edit_probes. It asserts thirteen, keeps the one-item rule for the ten source and deduction probes, and pins each restoration probe's path, remove and add. - packages/microcosm-build/tests/engine_free/shared/test_reform_probe_list_edits.py: the docstring says thirteen shipped probes. - changelog.d/us-reform-probe-list-edits.changed.md (#1197's): all thirteen such probes, the three H.R.1 restorations among them. Regenerated: packages/microcosm-build/src/microcosm/build/us/release_input_coverage_manifest.json, by tools/build_us_release_input_coverage_manifest.py (171 required, 6 reviewed exclusions, 47 probes, 13 with list_edits). A second run left it byte-identical, and its diff from git's auto-merge touches only the three converted probes. #1197 changes no attested input: tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py both pass --check, and in write mode they change nothing. Local runs: the 28 pin-test files plus test_reform_probe_list_edits.py and test_us_post_export_scoring.py, in one process, found the two semantic conflicts above (9 failed, 926 passed, 1 skipped, every failure in the one test file). After the fix that file passes on its own (155 passed). On policyengine-us 2.2.1, the engine_contract release-input-coverage test, the probe engine-drift test and the H.R.1 channel tests pass (126). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reform-coverage smoke probes that change a list-valued parameter now declare the edit instead of pinning the whole list:
(
"add"for reactivation probes.) The list resolves against the installed PolicyEngine-US baseline when_build_reformbuilds the reform, and resolution fails loudly when the edit no longer fits.On 2026-09-28 the US release failed its post-export smoke because four probes pinned lists copied from an older PolicyEngine-US. After 2.2.1 moved
tanftounearned_spm_unit, the pinned SNAP source lists re-added TANF and scored wrong-signed. #1046 re-derived the lists by hand. This PR removes the failure class: a declared edit tracks the engine's own list, and a stale one raises instead of silently scoring a different reform.What changed
release_input_coverage.pyListParameterEditholds a boundedYYYY-MM-DD.YYYY-MM-DDperiod plus theremoveandadditems.resolve_list_parameter_editresolves an edit against a baseline list without the engine.list_parameter_piecesreturns the value a parameter holds over a period as constant stretches: the value at the start, then each breakpoint in(start, stop]where it changes. It refuses an edit that starts on or before the parameter's first breakpoint (see First breakpoint below).list_parameter_baselinereads a policyengine-core parameter tree through those stretches and refuses a list that changes inside the period, naming every change instant in date order. It also refuses non-list values.check_resolved_list_edit_targetchecks the target system at apply time (see_build_reformbelow).installed_list_parameter_baselinebuilds one PolicyEngine-US system per process, with a reform that records each list's first breakpoint at the system's first reform pass. It refuses an edit that starts on or before that breakpoint, then reads the baseline from the finished system's tree.resolve_probe_parameter_changesreturns the fullReform.from_dictpayload.ReformCoverageProbelist_edits.parameter_changes.ListParameterEditvalue.schema_version1 with a regeneration hint and rejects any other version. It parseslist_editsstrictly: unknown keys, non-arrays,nulland malformed edits raise, naming the resource, probe and path, and an omittedlist_editsloads as no edits. The PR bumps the schema instead of keeping v1 loadable, because a v1 manifest pins lists that v2 refuses.reform_coverage_smoke._build_reformcheck_resolved_list_edit_target). A stretch passes when it holds the baseline the edit resolved against or the edit's own resolution, so a period can mix the two. So a resolved list replaces only the baseline it came from, and when an earlier application of the same edit covers part of the period, this one extends the resolution over the union. Any other stretch fails, and the message names the first one.CountryTaxBenefitSystem.__init__, andSimulationapplies it again.CountryTaxBenefitSystemapplies reforms before it backdates each parameter's first value to 2015 (backdate_parameters). An edit starting on or before a list's first breakpoint at that first pass becomes the first value, and the backdating copies it outside its period. On 2.2.1 a QBI edit over 2018 held from 2015 through 2018. Every function that reads a list from a parameter tree refuses such an edit (list_parameter_pieces), and installed resolution checks against the first-pass breakpoints, so resolution, the country constructor andSimulationall refuse the same edits. No shipped edit comes near a first breakpoint: QBI's list starts 2018-01-01 and the others 2008–2013, while the shipped edits start in 2024 or 2026._record_post_export_requestsflags its recordingsimulateasRECORDS_REQUESTS_ONLY. The smoke gate then hands it each probe and builds no reform, so recording resolves no list edit and needs no engine (before this, recording a list-edit probe built aCountryTaxBenefitSystem).list_editswithperiod,remove,addand theresolvedlist, soreform_coverage_smoke.jsonnames the lists the gate scored.assert_release_input_coverage_manifest_current()resolves every list edit against the live engine. The release tool calls it before calibration, so a stale edit fails in seconds instead of after the export. A caller that injects an engine skips this unless it also passeslist_baseline.tools/build_us_release_input_coverage_manifest.py: the original nine, plus main's NIIT pass-through probe (microcosm#1239). That probe reached this branch in the merge with main, pinninggov.irs.investment.income.sources, and now removespassive_partnership_s_corp_incomeover 2024. The regenerated manifest differs from main's only inschema_version, the description and those ten probes. On the locked PolicyEngine-US 2.2.1, rechecked at the round-2 head, every resolved list equals the list main pinned. Seven match item for item, and the three SNAP unearned-source lists hold the same items in another order, which leaves anaddssum unchanged. The probes score what they scored before.release_gate_preflight.pydoesn't build reforms (it only readsbinding_inputs), so it needed no change. Main's newtools/probe_us_post_export.pyscores through the same gate and_build_reform.docs/us-fact-to-target.mdgains probe-authoring guidance ("declare list edits; never pin a list"; start each edit after the list's first breakpoint), and achangelog.dfragment records the change.Invariants
Hypothesis properties test invariants 1–9 and the engine-free halves of 11 and 12, and examples test 10, 13 and the engine halves of 11 and 12.
resolve(baseline, edit) == [b for b in baseline if b not in remove] + list(add). Kept items keep baseline order and multiplicity, every occurrence of a removed item goes, andaddappends its items once each, in declared order. On a baseline without repeated items the set identity holds, the length islen(baseline) - len(remove) + len(add), and the result has no duplicates; with repeats the length formula fails (removing"a"from["a", "a", "b"]leaves["b"]).ValueErrornaming the path, period, offending items and installed baseline. It never returns a list.resolve_probe_parameter_changesequals the scalarparameter_changesunchanged, plus one{period: resolved}per edit. It reads the baseline once per edit and prefixes a misfit with the probe id.list_parameter_baselinereturns the value at start iff the edit starts after the list's first breakpoint and the parameter holds one list from start through stop. An edit starting on or before the first breakpoint raisesValueErrornaming that breakpoint and why: PolicyEngine-US would backdate the edited first value outside the period. A list that changes inside the period raises naming every change instant in date order, and a non-list value anywhere in the period raisesValueError, neverTypeError. The stretcheslist_parameter_piecesreturns reproduce the value on every day of the period.ListParameterEditaccepts bounded, ordered, strictYYYY-MM-DDranges and nothing else (date.fromisoformatalone would also accept ISO week dates like2024-W01-1). Items must be unique, non-empty and disjoint betweenremoveandadd, with at least one item. A probe refuses any pinned list.nulllist_edits.CountryTaxBenefitSystem(reform=…)(the release scorer's path) andSimulation(reform=…)build each edited list with its resolution inside the period and the baseline outside it.check_resolved_list_edit_targetpasses iff every stretch of the period holds the baseline or the resolution; otherwise it names the first stretch that holds anything else. On the engine, two applications of one edit over overlapping periods (in either order, or as a reform tuple) hold the resolution over the union and the baseline outside it.simulateflaggedRECORDS_REQUESTS_ONLYreceives each probe in place of its reform, and nothing calls_build_reform; the release tool's recorder records the real smoke consumer's plan with the engine packages unimportable.Tests
engine_free/shared/test_reform_probe_list_edits.py(new): Hypothesis properties 1–8, plus a stub of the policyengine-core parameter tree.engine_free/shared/test_release_input_coverage.py:nullrefusal), the gate'slist_editsevidence, recording mode, and preflight resolution with an injected engine or baseline. A property test checks that the preflight fails the edits that don't fit and no others.engine_free/us/test_us_workers_compensation.py: the declared edit, plus a property that the probe resolves to any baseline minus its leaf (TANF only if the baseline has it).engine_scenario/us/test_us_reform_coverage_probe_engine_drift.py(reworked from Re-derive four reform-coverage smoke probes from PolicyEngine-US 2.2.1 #1046):CountryTaxBenefitSystembuilds (regression tests for the double-apply bug below).engine_free/us/test_us_post_export_scoring.py: the release recorder records the real smoke plan with the engine packages blocked and builds no reform (invariant 13).engine_contract/us/test_release_input_coverage.py: the liveassert_release_input_coverage_manifest_current()passes on the shipped manifest, and fails, naming the probe, for a stale removal, an already-present addition, or a period spanning a breakpoint.engine_scenario/us/tests for workers' compensation, disability benefits, educator expenses and domestic production: moved fromReform.from_dict(probe.parameter_changes), which is now a no-op, to_build_reform. They check the reformed list in and out of the period and keep their household effects. Two new properties check that the educator and domestic-production reforms move above-the-line deductions by the leaf amount and never move income tax the wrong way.Review round 2 (GPT-6.1 Sol, REQUEST CHANGES at 391b906)
check_resolved_list_edit_targetCountryTaxBenefitSystemRECORDS_REQUESTS_ONLYon the recorder'ssimulate_build_reformspied, not replaced"list_edits": nullloaded as{}NonecaseBug caught during development
The first apply-time check accepted only the baseline. PolicyEngine-US applies a reform set twice while building
CountryTaxBenefitSystem, so every list-edit reform raised on the release scorer's path. The new engine tests reproduced it: 8 failures, all in the paths that buildCountryTaxBenefitSystem. The check now also accepts the reform's own resolution, which those tests pin.Verification
Round 2, locally on the loaded shared host:
ci_test_plan.py verifyOK; ruff check and format clean on all 16 changed Python files;test_reform_probe_list_edits.py,engine_free/shared/test_release_input_coverage.pyandtest_us_workers_compensation.py203 passed;test_us_post_export_scoring.py47 passed, 1 skipped; the resolved-versus-pinned comparison above ran on PolicyEngine-US 2.2.1. The engine-scenario suite didn't finish locally; CI's engine-us jobs run it.At 8721c84 all 12 CI checks pass, including
ci, engine-us and engine-free on 3.13 and 3.14.On its own, the merge commit 14ba3a7 fails one engine-free test: the shipped-probe test still expected nine ids, and the merged manifest has ten. 8721c84 fixes it, so only a bisect that stops on the merge commit would see the failure.
axiom: n/a: Microcosm release-gate infrastructure; this PR encodes no policy.
🤖 Generated with Claude Code