Skip to content

Declare list edits on reform-coverage probes instead of pinning whole lists - #1197

Merged
MaxGhenis merged 5 commits into
mainfrom
list-edit-reform-probes
Nov 12, 2026
Merged

MaxGhenis merged 5 commits into
mainfrom
list-edit-reform-probes

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Reform-coverage smoke probes that change a list-valued parameter now declare the edit instead of pinning the whole list:

"parameter_changes": {},
"list_edits": {
  "gov.usda.snap.income.sources.unearned": {
    "period": "2024-01-01.2024-12-31",
    "remove": ["workers_compensation"]
  }
}

("add" for reactivation probes.) The list resolves against the installed PolicyEngine-US baseline when _build_reform builds the reform, and resolution fails loudly when the edit no longer fits.

On 2026-09-28 the US release failed its post-export smoke because four probes pinned lists copied from an older PolicyEngine-US. After 2.2.1 moved tanf to unearned_spm_unit, the pinned SNAP source lists re-added TANF and scored wrong-signed. #1046 re-derived the lists by hand. This PR removes the failure class: a declared edit tracks the engine's own list, and a stale one raises instead of silently scoring a different reform.

What changed

  • release_input_coverage.py
    • ListParameterEdit holds a bounded YYYY-MM-DD.YYYY-MM-DD period plus the remove and add items.
    • resolve_list_parameter_edit resolves an edit against a baseline list without the engine.
    • list_parameter_pieces returns the value a parameter holds over a period as constant stretches: the value at the start, then each breakpoint in (start, stop] where it changes. It refuses an edit that starts on or before the parameter's first breakpoint (see First breakpoint below).
    • list_parameter_baseline reads a policyengine-core parameter tree through those stretches and refuses a list that changes inside the period, naming every change instant in date order. It also refuses non-list values.
    • check_resolved_list_edit_target checks the target system at apply time (see _build_reform below).
    • installed_list_parameter_baseline builds one PolicyEngine-US system per process, with a reform that records each list's first breakpoint at the system's first reform pass. It refuses an edit that starts on or before that breakpoint, then reads the baseline from the finished system's tree.
    • resolve_probe_parameter_changes returns the full Reform.from_dict payload.
  • ReformCoverageProbe
    • Gains list_edits.
    • Refuses a list pinned at any depth of parameter_changes.
    • Refuses a path that appears in both fields.
    • Refuses a non-ListParameterEdit value.
  • Manifest schema → 2. The loader refuses schema_version 1 with a regeneration hint and rejects any other version. It parses list_edits strictly: unknown keys, non-arrays, null and malformed edits raise, naming the resource, probe and path, and an omitted list_edits loads as no edits. The PR bumps the schema instead of keeping v1 loadable, because a v1 manifest pins lists that v2 refuses.
  • reform_coverage_smoke._build_reform
    • Resolves list edits on the installed engine. A misfit raises, with the probe id at the front of the message, before the gate scores anything.
    • Returns a reform that, at apply time, re-reads each edited list from the target system and checks each constant stretch of the period on its own (check_resolved_list_edit_target). A stretch passes when it holds the baseline the edit resolved against or the edit's own resolution, so a period can mix the two. So a resolved list replaces only the baseline it came from, and when an earlier application of the same edit covers part of the period, this one extends the resolution over the union. Any other stretch fails, and the message names the first one.
    • The check must accept the edit's own resolution: PolicyEngine-US applies a reform set twice inside CountryTaxBenefitSystem.__init__, and Simulation applies it again.
    • First breakpoint. CountryTaxBenefitSystem applies reforms before it backdates each parameter's first value to 2015 (backdate_parameters). An edit starting on or before a list's first breakpoint at that first pass becomes the first value, and the backdating copies it outside its period. On 2.2.1 a QBI edit over 2018 held from 2015 through 2018. Every function that reads a list from a parameter tree refuses such an edit (list_parameter_pieces), and installed resolution checks against the first-pass breakpoints, so resolution, the country constructor and Simulation all refuse the same edits. No shipped edit comes near a first breakpoint: QBI's list starts 2018-01-01 and the others 2008–2013, while the shipped edits start in 2024 or 2026.
  • Plan dry run. _record_post_export_requests flags its recording simulate as RECORDS_REQUESTS_ONLY. The smoke gate then hands it each probe and builds no reform, so recording resolves no list edit and needs no engine (before this, recording a list-edit probe built a CountryTaxBenefitSystem).
  • Smoke evidence. Each list-edit probe's result records list_edits with period, remove, add and the resolved list, so reform_coverage_smoke.json names the lists the gate scored.
  • Release preflight. assert_release_input_coverage_manifest_current() resolves every list edit against the live engine. The release tool calls it before calibration, so a stale edit fails in seconds instead of after the export. A caller that injects an engine skips this unless it also passes list_baseline.
  • Probes. The PR converts ten list-valued probes in tools/build_us_release_input_coverage_manifest.py: the original nine, plus main's NIIT pass-through probe (microcosm#1239). That probe reached this branch in the merge with main, pinning gov.irs.investment.income.sources, and now removes passive_partnership_s_corp_income over 2024. The regenerated manifest differs from main's only in schema_version, the description and those ten probes. On the locked PolicyEngine-US 2.2.1, rechecked at the round-2 head, every resolved list equals the list main pinned. Seven match item for item, and the three SNAP unearned-source lists hold the same items in another order, which leaves an adds sum unchanged. The probes score what they scored before.
  • release_gate_preflight.py doesn't build reforms (it only reads binding_inputs), so it needed no change. Main's new tools/probe_us_post_export.py scores through the same gate and _build_reform.
  • Docs and changelog. docs/us-fact-to-target.md gains probe-authoring guidance ("declare list edits; never pin a list"; start each edit after the list's first breakpoint), and a changelog.d fragment records the change.

Invariants

Hypothesis properties test invariants 1–9 and the engine-free halves of 11 and 12, and examples test 10, 13 and the engine halves of 11 and 12.

  1. Resolution identity. For any baseline and any edit that fits it, resolve(baseline, edit) == [b for b in baseline if b not in remove] + list(add). Kept items keep baseline order and multiplicity, every occurrence of a removed item goes, and add appends its items once each, in declared order. On a baseline without repeated items the set identity holds, the length is len(baseline) - len(remove) + len(add), and the result has no duplicates; with repeats the length formula fails (removing "a" from ["a", "a", "b"] leaves ["b"]).
  2. Loud failure. If a removed item is absent or an added item is already present, resolution raises ValueError naming the path, period, offending items and installed baseline. It never returns a list.
  3. Round trip. The inverse edit always fits the result. On a baseline without repeated items it restores the baseline up to order (removed items return at the end); with repeats it restores the item set, each removed item once.
  4. No silent double application. Re-applying an edit to its own result always raises.
  5. Payload merge. resolve_probe_parameter_changes equals the scalar parameter_changes unchanged, plus one {period: resolved} per edit. It reads the baseline once per edit and prefixes a misfit with the probe id.
  6. One baseline per period. list_parameter_baseline returns the value at start iff the edit starts after the list's first breakpoint and the parameter holds one list from start through stop. An edit starting on or before the first breakpoint raises ValueError naming that breakpoint and why: PolicyEngine-US would backdate the edited first value outside the period. A list that changes inside the period raises naming every change instant in date order, and a non-list value anywhere in the period raises ValueError, never TypeError. The stretches list_parameter_pieces returns reproduce the value on every day of the period.
  7. Validation. ListParameterEdit accepts bounded, ordered, strict YYYY-MM-DD ranges and nothing else (date.fromisoformat alone would also accept ISO week dates like 2024-W01-1). Items must be unique, non-empty and disjoint between remove and add, with at least one item. A probe refuses any pinned list.
  8. Loader round trip. Any valid edit written into a v2 manifest loads back unchanged, and the loader refuses every schema version other than 2 and a null list_edits.
  9. Differential (engine). For random edits of the six real list parameters, the pure resolver equals the list the engine-applied reform holds on a random day in the period. Outside the period the baseline holds.
  10. Apply idempotence (engine). Applying a list-edit reform twice equals applying it once, and CountryTaxBenefitSystem(reform=…) (the release scorer's path) and Simulation(reform=…) build each edited list with its resolution inside the period and the baseline outside it.
  11. Stretch-wise apply check. check_resolved_list_edit_target passes iff every stretch of the period holds the baseline or the resolution; otherwise it names the first stretch that holds anything else. On the engine, two applications of one edit over overlapping periods (in either order, or as a reform tuple) hold the resolution over the union and the baseline outside it.
  12. First breakpoint. Every function that reads a list from a parameter tree refuses an edit that starts on or before the list's first breakpoint, naming it; installed resolution uses the breakpoint at the engine's first reform pass. On the engine, QBI edits over 2015, 2017 and 2018 fail at resolution. A 2017 edit resolved on the finished tree alone applies to the finished system, but both constructors refuse it. A 2019 edit leaves 2015–2018 at the baseline through both constructors.
  13. Recording builds no reform. A simulate flagged RECORDS_REQUESTS_ONLY receives each probe in place of its reform, and nothing calls _build_reform; the release tool's recorder records the real smoke consumer's plan with the engine packages unimportable.

Tests

  • engine_free/shared/test_reform_probe_list_edits.py (new): Hypothesis properties 1–8, plus a stub of the policyengine-core parameter tree.
  • engine_free/shared/test_release_input_coverage.py:
    • The shipped-manifest tests assert the declared edits.
    • Tests cover probe validation, the schema-2 loader (including the null refusal), the gate's list_edits evidence, recording mode, and preflight resolution with an injected engine or baseline. A property test checks that the preflight fails the edits that don't fit and no others.
  • engine_free/us/test_us_workers_compensation.py: the declared edit, plus a property that the probe resolves to any baseline minus its leaf (TANF only if the baseline has it).
  • engine_scenario/us/test_us_reform_coverage_probe_engine_drift.py (reworked from Re-derive four reform-coverage smoke probes from PolicyEngine-US 2.2.1 #1046):
    • Each list-edit probe declares one item.
    • Each resolves on the installed engine to the baseline with only that item changed.
    • Each built reform holds its list only inside its period.
    • Idempotence and CountryTaxBenefitSystem builds (regression tests for the double-apply bug below).
    • The apply-time check refuses a list that changed under it.
    • Loud failures on the real engine: a missing item, an item already present, a breakpoint inside the period, a non-list parameter, an unknown path, a node.
    • The differential property (invariant 9).
    • Overlapping applications, first-breakpoint refusals and the 2019 QBI edit (invariants 11 and 12).
    • The SNAP household and alimony scenarios and their properties stay.
  • engine_free/us/test_us_post_export_scoring.py: the release recorder records the real smoke plan with the engine packages blocked and builds no reform (invariant 13).
  • engine_contract/us/test_release_input_coverage.py: the live assert_release_input_coverage_manifest_current() passes on the shipped manifest, and fails, naming the probe, for a stale removal, an already-present addition, or a period spanning a breakpoint.
  • engine_scenario/us/ tests for workers' compensation, disability benefits, educator expenses and domestic production: moved from Reform.from_dict(probe.parameter_changes), which is now a no-op, to _build_reform. They check the reformed list in and out of the period and keep their household effects. Two new properties check that the educator and domestic-production reforms move above-the-line deductions by the leaf amount and never move income tax the wrong way.

Review round 2 (GPT-6.1 Sol, REQUEST CHANGES at 391b906)

# Finding Fix Test
1 (medium) The whole-period check refused a safe overlapping application of the same edit Stretch-wise check_resolved_list_edit_target Stretch property (engine-free); overlapping-union and different-edit refusal (engine)
2 (medium) Resolution read the backdated tree, while the constructor's first pass applies before backdating Refuse edits on or before the first breakpoint; installed resolution reads first-pass breakpoints First-breakpoint property (engine-free); QBI 2015/2017/2018 refusals, constructor refusals, a 2019 edit through both constructors (engine)
3 (medium) Recording the plan built CountryTaxBenefitSystem RECORDS_REQUESTS_ONLY on the recorder's simulate Gate recording/scoring test; recorder end-to-end with engine imports blocked and _build_reform spied, not replaced
4 (low) "list_edits": null loaded as {} Removed the None case Null on a list-only and a mixed probe; omission still loads
5 (low) Length and inverse claims overstated for repeated items Restated docstrings, comments and invariants 1 and 3 Existing repeat properties

Bug caught during development

The first apply-time check accepted only the baseline. PolicyEngine-US applies a reform set twice while building CountryTaxBenefitSystem, so every list-edit reform raised on the release scorer's path. The new engine tests reproduced it: 8 failures, all in the paths that build CountryTaxBenefitSystem. The check now also accepts the reform's own resolution, which those tests pin.

Verification

Round 2, locally on the loaded shared host: ci_test_plan.py verify OK; ruff check and format clean on all 16 changed Python files; test_reform_probe_list_edits.py, engine_free/shared/test_release_input_coverage.py and test_us_workers_compensation.py 203 passed; test_us_post_export_scoring.py 47 passed, 1 skipped; the resolved-versus-pinned comparison above ran on PolicyEngine-US 2.2.1. The engine-scenario suite didn't finish locally; CI's engine-us jobs run it.

At 8721c84 all 12 CI checks pass, including ci, engine-us and engine-free on 3.13 and 3.14.

On its own, the merge commit 14ba3a7 fails one engine-free test: the shipped-probe test still expected nine ids, and the merged manifest has ten. 8721c84 fixes it, so only a bisect that stops on the merge commit would see the failure.

axiom: n/a: Microcosm release-gate infrastructure; this PR encodes no policy.

🤖 Generated with Claude Code

MaxGhenis and others added 5 commits September 28, 2026 19:32
… (WIP core)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The apply-time guard accepted only the baseline, but PolicyEngine-US
  applies a reform set twice while building CountryTaxBenefitSystem (and
  again in Simulation), so every list-edit reform raised on the release
  scorer's path. The guard now accepts the reform's own resolution and reads
  the target system's list over the whole edit period.
- list_parameter_baseline refuses a non-list value inside the period with a
  ValueError (a null breakpoint used to raise TypeError past callers) and
  names every change instant in date order.
- Baseline reads in _build_reform run inside the probe-id error prefix; the
  gate copies resolved lists into its evidence.
- ReformCoverageProbe refuses a non-mapping list_edits, empty paths, and a
  list pinned at any depth of parameter_changes; periods must be strict
  YYYY-MM-DD (fromisoformat also takes ISO week dates).
- Tests: shipped-manifest assertions on declared edits, loader/schema and
  gate evidence tests, Hypothesis properties for resolution, validation,
  loader round-trip and the period-constancy read, engine consumers moved to
  _build_reform, and real-engine drift, guard, loud-failure and differential
  tests plus a live manifest-current check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's NIIT pass-through probe (microcosm#1239) pinned the whole
gov.irs.investment.income.sources list, which schema_version 2 refuses.
It now declares the edit: remove passive_partnership_s_corp_income over
2024. On PolicyEngine-US 2.2.1 that resolves to exactly the list main
pinned. The manifest regenerates; against main it differs only in
schema_version, the description and the ten list-edit probes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1. The apply-time guard checks each stretch of the edit period on its own
   (check_resolved_list_edit_target): every stretch must hold the baseline
   or the reform's own resolution. An earlier application of the same edit
   over an overlapping period is accepted, and the two cover the union.
2. CountryTaxBenefitSystem applies reforms before it backdates each
   parameter's first value, so an edit starting on or before a list's first
   breakpoint at that first pass would be backdated outside its period
   (QBI's 2018 edit held from 2015 through 2018 on PolicyEngine-US 2.2.1).
   Every reader now refuses such an edit, and installed resolution reads
   the first-pass breakpoints, so resolution and both constructors agree.
3. The post-export plan's dry run flags its simulate RECORDS_REQUESTS_ONLY;
   the smoke gate then hands it probes and builds no reform, so recording
   resolves no list edit and needs no engine.
4. The loader refuses "list_edits": null instead of reading it as {}.
5. The docstrings and comments state the length and inverse guarantees
   for baselines without repeated items only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

The Microcosm hub now owns this PR and lands it.

  • The head is 8721c84. Today's body edit left it unchanged.
  • GPT-6.1 Sol requested changes at 391b906 (~/reviews/microcosm-1197/review-sol.md), and 8721c84 answers all five findings. The body's round-2 table maps each finding to its fix and test.
  • The hub's independent Opus review at 8721c84 (~/reviews/microcosm-hub/runs/review-1197-2026-11-12/review.md) killed 8 of 8 mutants and blocked only on stale body sentences, R1–R3, with nits R4 and R5. The review says those fixes turn its verdict to APPROVE at this SHA without a re-run, and today's body edit makes all five.
  • All 12 CI checks pass at 8721c84, including ci, and GitHub reports the PR as mergeable with a clean merge state. Nothing on this PR is pending.
  • microcosm#779 (open, waiting on Max's d2729) adds three probes that pin whole lists: the Medicaid, ACA and SNAP immigration-status lists. A plain merge of the two PRs would leave a schema-2 manifest holding three pinned lists, which the loader refuses. Whichever PR lands second converts those three probes to list edits that start after each list's first breakpoint, regenerates the manifest, and raises the shipped-probe test from ten to thirteen. The hub orders the two.
  • No hold applies to this PR; d2729 is Max's call on Impute US humanitarian immigration statuses to cited stocks (H.R.1 channels) #779.

@MaxGhenis
MaxGhenis merged commit 90e4d94 into main Nov 12, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Landed by the Microcosm hub as 90e4d94 (merge of 8721c84 into main 57a32f8; main's tree equals the tested merge).

  • Checks on 8721c84: 12 passed, ci included.
  • Reviews: the earlier review's five findings are answered in code; an independent review at this head killed 8 of 8 mutants and asked only for description fixes, which the description now carries.
  • Since this head's CI ran, main changed one help-text string in tools/build_us_fiscal_refresh_release.py and no function this pull request edits.
  • Order with Impute US humanitarian immigration statuses to cited stocks (H.R.1 channels) #779: it adds three probes that pin whole lists. Its next merge of main converts them to list edits that start after each list's first breakpoint, regenerates the manifest and raises the shipped-probe test from ten to thirteen.

@MaxGhenis
MaxGhenis deleted the list-edit-reform-probes branch November 12, 2026 16:51
MaxGhenis added a commit that referenced this pull request Nov 12, 2026
Main 90e4d94 is #1197 alone: reform-coverage probes that change a
list-valued parameter declare list_edits (the items they remove or add)
instead of pinning the whole list, and the manifest's schema_version 2
refuses a list pinned in parameter_changes. #779 pins three whole
immigration-status lists, so they are converted here, and three files
conflicted.

Conflicts, each resolved by keeping both sides:

- packages/microcosm-build/src/microcosm/build/us_runtime/release_input_coverage.py
  (4 hunks): the branch's REFORM_COVERAGE_MEASURE_UNITS beside main's
  ListParameterEdit block; ReformCoverageProbe keeps the branch's
  measure_unit field and main's list_edits field; the
  assert_release_input_coverage_manifest_current docstring keeps the
  branch's measure-unit bullet and adds main's list-edit bullet; the
  per-probe loop runs the branch's _probe_measure_drift check, then
  main's list-edit resolution.
- packages/microcosm-build/tests/engine_contract/us/test_release_input_coverage.py
  (2 hunks): main's import block (a superset of the branch's); the
  branch's TestProbeMeasureUnits and main's
  TestShippedListEditsOnTheInstalledEngine both kept.
- packages/microcosm-build/tests/engine_scenario/us/test_us_reform_coverage_probe_engine_drift.py
  (3 hunks): main's list-edit versions, extended with the branch's
  H.R.1 restoration probes. The docstring bullet names them, the
  list-edit probe set includes _RESTORED_ITEMS, and the resolve test
  expects each restoration probe's restored status set. In main's
  auto-merged lines, the declares-loop compares a restoration probe's
  (remove, add) as sets with _RESTORED_ITEMS, and _LIST_PARAMETERS
  gains the three status lists the shipped probes now edit.

Two semantic conflicts in
packages/microcosm-build/tests/engine_free/shared/test_release_input_coverage.py,
which git merged without markers:

- The branch's test_people_count_probe_states_effect_and_floor_in_people
  built a synthetic probe that pinned a whole list, which #1197's
  ReformCoverageProbe refuses. It now declares
  list_edits {gov.example.statuses: add REFUGEE over
  2027-01-01.2100-12-31}: still a status-list restoration scored in
  persons.
- Main's _GraphEngine fake offered only variables(), but the merged
  check also runs the branch's _probe_measure_drift, which reads
  variable_metadata and variable_unit. Eight of main's
  TestManifestListEditResolution tests failed with AttributeError. The
  fake now answers both for each shipped probe's budget_measure, in the
  unit the probe declares, so those tests again see only the list-edit
  findings they set up. Production code is unchanged.

Converted probes (tools/build_us_release_input_coverage_manifest.py),
each now parameter_changes {} with one list edit over its old pinned
period:

- hr1_medicaid_humanitarian_eligibility_restoration:
  gov.hhs.medicaid.eligibility.eligible_immigration_statuses,
  2026-10-01.2100-12-31, add REFUGEE, ASYLEE, DEPORTATION_WITHHELD,
  CONDITIONAL_ENTRANT, PAROLED_ONE_YEAR.
- hr1_aca_lawful_presence_restoration:
  gov.aca.ineligible_immigration_statuses, 2027-01-01.2100-12-31,
  remove TPS, REFUGEE, ASYLEE, DEPORTATION_WITHHELD, PAROLED_ONE_YEAR.
- hr1_snap_humanitarian_eligibility_restoration:
  gov.usda.snap.eligibility.eligible_immigration_statuses,
  2025-07-01.2100-12-31, add the same five as Medicaid.

hr1_aca_below_fpl_exception_restoration pins a boolean and is
unchanged. On policyengine-us 2.2.1 and policyengine-core 3.32.5 (the
uv.lock pins), each edit's start is after its list's first breakpoint
(2018-01-01). Each resolves to the old pinned list's items, and the
reformed system holds the same set as the old pinned reform at every
instant checked from 2018-01-01 to 2101-01-01. ACA keeps the old
order exactly. Medicaid and SNAP put CUBAN_HAITIAN_ENTRANT third, from
the baseline, instead of sixth; every 2.2.1 reader of the three lists
uses np.isin, so order cannot change a score.

Tests and documents that named the old count:

- packages/microcosm-build/tests/engine_free/shared/test_release_input_coverage.py:
  _SHIPPED_LIST_EDIT_PROBE_IDS gains the three probes through a new
  _HR1_RESTORATION_EDITS table, and the shipped-manifest test is now
  ..._exactly_thirteen_list_edit_probes. It asserts thirteen, keeps the
  one-item rule for the ten source and deduction probes, and pins each
  restoration probe's path, remove and add.
- packages/microcosm-build/tests/engine_free/shared/test_reform_probe_list_edits.py:
  the docstring says thirteen shipped probes.
- changelog.d/us-reform-probe-list-edits.changed.md (#1197's): all
  thirteen such probes, the three H.R.1 restorations among them.

Regenerated: packages/microcosm-build/src/microcosm/build/us/release_input_coverage_manifest.json,
by tools/build_us_release_input_coverage_manifest.py (171 required, 6
reviewed exclusions, 47 probes, 13 with list_edits). A second run left
it byte-identical, and its diff from git's auto-merge touches only the
three converted probes. #1197 changes no attested input:
tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py
both pass --check, and in write mode they change nothing.

Local runs: the 28 pin-test files plus test_reform_probe_list_edits.py
and test_us_post_export_scoring.py, in one process, found the two
semantic conflicts above (9 failed, 926 passed, 1 skipped, every
failure in the one test file). After the fix that file passes on its
own (155 passed). On policyengine-us 2.2.1, the engine_contract
release-input-coverage test, the probe engine-drift test and the H.R.1
channel tests pass (126).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant