Skip to content

Test the UK nation filters against policyengine-uk's country formula - #568

Merged
MaxGhenis merged 2 commits into
mainfrom
fix/558-uk-nation-scoping
Nov 11, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
fix/558-uk-nation-scoping

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #569

Update (PR sweep, 2026-11-11): the sweep merged current main (e4ad54c), so the test now runs against policyengine-uk 2.174.1, the version main pins. An independent Opus 5.5 review re-ran the module at that head (10 passed) and three mutants: one on the registry's side and two on policyengine-uk's. Each failed the module. The certified-data table and the "Not fixed here" figures below come from October (main 07bae75, policyengine-uk 2.102.3, Enhanced FRS); the sweep didn't re-run them, and the bundle now loads Microcosm by default.

What this adds

Since #565 (6.2.3), the UK nation regions filter the stored household region column, and countries/uk/regions.py lists which regions make up each nation. policyengine-uk keeps its own mapping in the country formula, which derives country from region. tests/test_uk_regions.py checks the registry against _REGION_COUNTRIES, a hard-coded table in the test. Nothing checks either list against policyengine-uk.

tests/test_uk_nation_model_agreement.py takes the expected nations from policyengine-uk itself. A situation-based policyengine_uk.Simulation calculates country for every value of its Region enum, UNKNOWN included, in about 0.3s.

  • Same nations: the registry's nations are exactly policyengine-uk's (apart from UNKNOWN).
  • Exhaustive over Region: each nation filter keeps exactly the regions policyengine-uk places in that nation, and UNKNOWN falls in none.
  • Property (Hypothesis, 100 examples): region-only data with 1–3 people per household, person, benefit-unit and household rows in separate orders, and str or bytes regions.
    • Each nation filter keeps exactly the households policyengine-uk assigns to that nation, once each, with all their people and their benefit units.
    • A nation with no households raises No households found.
    • The four nations are disjoint and together keep every household whose region policyengine-uk places in a nation.
  • End to end: Simulation(..., scoping_strategy=country/<nation>).run() runs through the UK model for all four nations. The input has region and no country, and includes an UNKNOWN household, which every nation filter drops before the model runs. The output's country is the nation. The existing tests apply the strategies directly.

Scoping code is unchanged; it stays as #565 made it, filtering stored region with no model calculation.

What the tests catch

Mutation check: 14 mutants, each run against a copy.

  • policyengine-uk's side. All four mutants fail this module and pass main's tests/test_uk_regions.py, because main's tests compare the registry with a hard-coded table rather than with the model:
    • a new Region value that the formula defaults to England;
    • Wales folded into England;
    • a new nation;
    • UNKNOWN mapped to England.
  • The registry's side: dropping LONDON from England, or adding UNKNOWN to it, fails this module, as it fails main's tests.
  • The filter cascade. Each of these fails this module:
    • benefit units left unfiltered, filtered by household id, or dropped;
    • a person dropped, or people filtered by position;
    • household rows duplicated;
    • bytes regions not matched.

Dropping the weights passes this module. Main's tests catch it.

Certified-data check of #565 (October 2026, not re-run)

#565 noted that the certified artifact was unvalidated. On enhanced_frs_2024_25_year_2026.h5, with main at 07bae75 and policyengine-uk 2.102.3, each nation filter keeps exactly the households policyengine-uk's country assigns to that nation:

region households weighted households
country/england 37,961 26,354,698.47
country/scotland 5,405 2,729,247.77
country/wales 3,821 1,590,754.53
country/northern_ireland 5,659 783,847.56

The four sets are disjoint and cover all 52,846 households (31,458,548.32 weighted). Simulation.run() for country/scotland returns the same 5,405 households and leaves the caller's tables unchanged.

Not fixed here

Scoped UK runs give the region all £31.73bn of UK business rates and region-relative deciles and poverty lines (#567). On 6.2.3, scoped Scotland household net income is £103.98bn, against £139.15bn for the same households in the national run. This reaches API nation results once PolicyEngine/policyengine-sim-api#733 deploys 6.2.3.

History of this PR

It first fixed #558 (nation regions raising on data without country) by having policyengine-uk calculate country on the unscoped dataset before filtering. #565 fixed the same failure while this was in review, filtering stored region, and its repository guidance rules out a preliminary model calculation. So this PR now carries only the tests. The earlier commits (1f6d02f, b1e138d, f12e77f) remain in this PR's history.

Tests: the new module (10 tests), test_uk_regions, test_scoping_strategy, test_region_group_strategy, test_filtering, test_entity_utils, test_entity_utils_characterization and test_region pass locally on Python 3.13.

Review: an independent Opus review agreed the change follows the repository guidance (the model runs only inside the test, never in scoping). It asked for a stronger property, UNKNOWN kept in the end-to-end input, and an accurate commit message. All three are done in da51cf1.

axiom: n/a: tests of region scoping in the analysis wrapper, no policy encoded

🤖 Generated with Claude Code

Since #565 the nation regions filter the stored household region column,
and policyengine.py lists which regions make up each nation. policyengine-uk
keeps its own mapping in the formula that derives country from region.
These tests take the expected nations from that formula, for every value of
policyengine-uk's Region enum, so the two copies cannot drift apart
unnoticed. A Hypothesis property checks region-only data with several
people per household, separately ordered tables and str or bytes regions,
and each nation runs end to end through the UK model.

Fixes #569

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the fix/558-uk-nation-scoping branch from 1f6d02f to da51cf1 Compare October 9, 2026 04:56
@MaxGhenis MaxGhenis changed the title Scope UK nations on data that stores region but not country Test the UK nation filters against policyengine-uk's country formula Oct 9, 2026
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 11, 2026 13:09
@MaxGhenis
MaxGhenis merged commit d5dd33d into main Nov 11, 2026
14 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge audit (PR sweep of 2026-11-11): the sweep merged main into this branch (e4ad54c), so the test runs against policyengine-uk 2.174.1, the version main pins. An independent Opus 5.5 review approved head e4ad54c with no blocking findings. It ran the module at that head (10 passed), checked that hiding policyengine-uk skips the module cleanly, and ran three mutants: NORTH_EAST moved from England's filter to Scotland's in the registry, Northern Ireland mapped to Scotland in policyengine-uk's country formula, and a new Region value that the formula defaults to England. Each mutant failed the module. All 13 checks passed, including Test (3.11) through (3.14); the PR was MERGEABLE with no changes requested. Non-blocking notes from the review: Region is imported by module path, and the module doesn't check the category dtype or the weights (main's test_uk_regions.py covers both). This closes #569.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check the UK nation filters against policyengine-uk's country formula UK nation regions fail because the household data has no country column

1 participant