diff --git a/changelog.d/c1-23-local-area-reports.added.md b/changelog.d/c1-23-local-area-reports.added.md new file mode 100644 index 000000000..902e9a121 --- /dev/null +++ b/changelog.d/c1-23-local-area-reports.added.md @@ -0,0 +1 @@ +Stage 12 now calculates UK-wide and nation reports on the release's local-area dataset (the dataset UK constituency and local-authority regions use): the UK worker loads that dataset, and such a report's run returns its full results together with its constituency and local-authority results and the codes of the areas disclosure control withheld, which it stores as `withheld_area_codes` rows; such a run fails if a level has no estimated area, published or withheld. The economy output gains optional `constituency_withheld_codes` and `local_authority_withheld_codes`, left out of every other result. The v2 version manifest's UK entry declares the local-area dataset (`local_area_reports`), under a new key that readers released before it never read. diff --git a/docs/migration/stage-12-modal-worker-foundation.md b/docs/migration/stage-12-modal-worker-foundation.md index 5b5900672..e76f89be5 100644 --- a/docs/migration/stage-12-modal-worker-foundation.md +++ b/docs/migration/stage-12-modal-worker-foundation.md @@ -471,8 +471,24 @@ national run leaves them out too: both decide with `uk_area_breakdowns_omitted`, so their results still compare. Production computes those breakdowns in a separate `breakdown: "local_areas"` request on the routed dataset. The entry adapts no such request into a Stage 12 report -(skip reason `unsupported_options`), and Stage 12 has no local-area report -yet, so the shadow comparison does not cover the breakdown numbers. +(skip reason `unsupported_options`), so the shadow comparison does not cover +the breakdown numbers. + +A UK-wide or nation report on that routed dataset, the local-area dataset, is +an ordinary detailed-save report: the simulation service names that dataset in +both simulations' provenance, and no field marks the run. Its simulations load +the local-area dataset in place of the region's own, keeping the region's +scoping, and check the area columns as it loads; the plan copies +`UK_AREA_DATASET_COLUMNS` for them; and the aggregate keeps both area levels, +with each level's withheld codes (`constituency_withheld_codes`, +`local_authority_withheld_codes`) from the same disclosure pass, which the +report run stores as `withheld_area_codes` rows beside its full results. +Disclosure control may withhold every area of a level, but a run whose +disclosure pass estimated no area of a level fails +(`report_coordination_failed`), so a succeeded run always has area or +withheld-code rows for each level. All three decide by +`calculates_on_local_area_dataset` (`stage12_runtime/local_area_datasets.py`), +on the same routing rule as `uk_area_breakdowns_omitted`. Aggregation uses the retained output tables without rerunning either model. The precomputed baseline and reform simulations retain their original policy @@ -673,36 +689,76 @@ skips any country the manifest does not mark as economy-capable, with the reason `unsupported_country`, so the coordinator receives only those countries' reports. -Readers released before a schema change reject fields they do not know, and -the manifest is published before the Cloud Run revision that reads it is -promoted. The publisher therefore writes the whole schema version 2 manifest -under the `active-schema-2` key and, in schema version 1 form (without the -capability and household fields), the same manifest under the -`active-with-region-datasets` key, without each country bundle's -`region_datasets` under the `active-with-cache` key, and without cache releases -either under the `active` key; readers released before each field keep reading -the key that lacks it. A schema version 1 document marks every listed country -economy-capable, with no household function. - -Current readers use the `active-schema-2` document while each older key holds -its schema version 1 form. Schema version 1 cannot describe a country that is -not economy-capable, so a manifest holding one is written only under -`active-schema-2` and the older keys keep the last manifest written there; such -a document is used unless the document under the older keys was validated -later. Otherwise readers use the `active-with-region-datasets` copy only while -it, with its region datasets removed, equals the manifest under -`active-with-cache`; otherwise they read `active-with-cache`, falling back to -`active` before the first publish that writes it. Code released before schema -version 2 or before region datasets republishes only the older keys it knows, so -after a rollback the out-of-date copies stop matching and are no longer read, -and the next publish starts from what the older code wrote. That document has -no household functions, so the manifest names none for versions published -before the rollback until each is published again, and it lacks whatever only -`active-schema-2` held, such as a version with a household-only country. Code -released before cache releases writes only `active`, which current readers do -not read while `active-with-cache` exists; a rollback that far needs the newer -keys deleted from the Modal Dict first. Removing the older copies needs a later -change, once no deployed or rollback revision reads them. +Readers released before a schema change reject fields they do not know, and the +manifest is published before the Cloud Run revision that reads it is promoted. +The publisher therefore writes the whole schema version 2 manifest under the +`active-schema-2-with-local-area-reports` key, the same manifest without the UK +entry's `local_area_reports` under the `active-schema-2` key (both described +below), and, in schema version 1 form (without the capability and household +fields), the same manifest under the `active-with-region-datasets` key, without +each country bundle's `region_datasets` under the `active-with-cache` key, and +without cache releases either under the `active` key; readers released before +each field keep reading the key that lacks it. A schema version 1 document +marks every listed country economy-capable, with no household function. + +Readers released before `local_area_reports` use the `active-schema-2` document +while each older key holds its schema version 1 form; current readers use the +local-area report key's document in its place while `active-schema-2` holds its +copy, and otherwise follow the same rules. Schema version 1 cannot describe a +country that is not economy-capable, so a manifest holding one is written only +under the two schema version 2 keys (`active-schema-2` and +`active-schema-2-with-local-area-reports`) and the older keys keep the last +manifest written there; such a document is used unless the document under the +older keys was validated later. Otherwise readers use the +`active-with-region-datasets` copy only while it, with its region datasets +removed, equals the manifest under `active-with-cache`; otherwise they read +`active-with-cache`, falling back to `active` before the first publish that +writes it. Code released before schema version 2 or before region datasets +republishes only the older keys it knows, so after a rollback the out-of-date +copies stop matching and are no longer read, and the next publish starts from +what the older code wrote. That document has no household functions, so the +manifest names none for versions published before the rollback until each is +published again, and it lacks whatever only the schema version 2 keys held, +such as a version with a household-only country. Code released before cache +releases writes only `active`, which current readers do not read while +`active-with-cache` exists; a rollback that far needs the newer keys deleted +from the Modal Dict first. Removing the older copies needs a later change, once +no deployed or rollback revision reads them. + +The UK entry's `local_area_reports` says that the worker's coordinator runs UK +local-area report runs, the runs of UK-wide and nation reports on the +local-area dataset, which calculate their full results and their constituency +and local-authority results, with withheld codes: the version of the runs it +accepts (`input_version`, `LOCAL_AREA_REPORTS_VERSION`) and the local-area +dataset (`dataset`). That dataset is the one the release routes constituency +and local-authority regions to, or null when it routes them to no dataset of +their own and a UK-wide or nation run on the national dataset calculates both +area levels itself. The publisher decides it by `uk_area_regions_routed` and +`routed_uk_area_dataset_name`, the rule the national aggregation's +`uk_area_breakdowns_omitted` and the UK worker use, so the simulation service, +which can't read the release kind from the catalog, can tell from it whether a +report on the local-area dataset is possible and which dataset it is. + +The field is written only under the `active-schema-2-with-local-area-reports` +key, which current readers read first; `active-schema-2` holds the same +manifest without it, for readers released before it. An entry without it leaves +the field out, so its document and digest are as before. Readers use the new +key's document only while, without the field, it equals the `active-schema-2` +document and that document is current, and the publisher requires both keys. +Code released before the field republishes only the older keys, so after a +rollback the out-of-date copy stops matching and is no longer read; the next +publish starts from what the older code wrote, so versions published before the +rollback lose their `local_area_reports` until each is published again. +Deleting `active-schema-2` also stops readers using the new key, so the +region-dataset rollback step below sends every reader to `active-with-cache`. + +The same window as for region datasets applies: a rollback deploys the older +coordinator before its publish job runs, so until then updated entry services +may start a local-area report run, whose simulations on the local-area dataset +the older UK worker refuses, failing only that run with its error recorded. To +avoid it, delete the `active-schema-2-with-local-area-reports` key from the +`simulation-api-v2-version-manifest` Modal Dict before deploying the older +code. A rollback to code released before region datasets has one window: the older worker is deployed before its publish job runs, and until that publish the @@ -712,7 +768,8 @@ Only Stage 12 comparison runs fail in that window, never production responses; if the older deploy's validation fails, the window lasts until a later publish. To avoid it, delete the `active-schema-2` and `active-with-region-datasets` keys from the `simulation-api-v2-version-manifest` Modal Dict before deploying the -older code. +older code; readers released with `active-schema-2-with-local-area-reports` +then stop reading that key too. `region_datasets` names the dataset each region type loads. The entry service records that dataset as a state or district run's dataset provenance, and the diff --git a/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/macro_output.py b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/macro_output.py index 4fbd2aa01..2681faaa5 100644 --- a/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/macro_output.py +++ b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/macro_output.py @@ -325,8 +325,20 @@ class CongressionalDistrictImpactOutput(MacroOutputModel): districts: list[CongressionalDistrictImpactRecord] +def _is_none(value: object) -> bool: + return value is None + + class SingleYearMacroOutput(MacroOutputModel): - """Completed response returned by a single-year macro simulation.""" + """Completed response returned by a single-year macro simulation. + + ``constituency_withheld_codes`` and ``local_authority_withheld_codes`` + hold, sorted, the codes of each level's areas the run estimated but + disclosure control did not publish, as ``LocalAreaBreakdownOutput``'s do. + They are listed for a UK-wide or nation report on the local-area dataset; + null means the result does not report them, and the JSON then leaves both + keys out, so every other result is written as before. + """ spm_provenance: SPMComparisonProvenance | None = None @@ -346,13 +358,15 @@ class SingleYearMacroOutput(MacroOutputModel): constituency_impact: GeographicImpactOutput | None local_authority_impact: GeographicImpactOutput | None congressional_district_impact: CongressionalDistrictImpactOutput | None + constituency_withheld_codes: list[str] | None = Field( + default=None, exclude_if=_is_none + ) + local_authority_withheld_codes: list[str] | None = Field( + default=None, exclude_if=_is_none + ) cliff_impact: CliffImpactOutput | None = None -def _is_none(value: object) -> bool: - return value is None - - class LocalAreaBreakdownOutput(MacroOutputModel): """Completed result of a UK local-area breakdown run. diff --git a/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest.py b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest.py index 4e13ce2de..ae45eb1fe 100644 --- a/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest.py +++ b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest.py @@ -5,35 +5,90 @@ households. It is stored under its own key. Whenever schema version 1 can describe the same manifest, it is also written in that form, without those fields, under the keys that readers released before schema version 2 read: -with region datasets, without them, and without them or cache releases. +with region datasets, without them, and without them or cache releases. The +UK entry's ``local_area_reports`` is written only under its own key, which +readers released with it read first. """ from __future__ import annotations import json -import re from collections.abc import Mapping from copy import deepcopy -from datetime import datetime from hashlib import sha256 -from typing import TYPE_CHECKING, Annotated, Literal, Protocol +from typing import TYPE_CHECKING, Protocol from packaging.version import InvalidVersion, Version -from pydantic import BaseModel, ConfigDict, Field, model_validator -from policyengine_simulation_contract.stage12_bundle import ( - CountryId, - Sha256Digest, - Stage12BundleManifest, +from policyengine_simulation_contract.stage12_manifest_models import ( + LOCAL_AREA_REPORTS_VERSION, + V1_APPLICATION_PREFIX, + V2_APPLICATION_PREFIX, + CallableName, + LocalAreaReports, + ManifestText, + Stage12CacheRelease, + StrictManifestModel, + V2CountryWorker, + V2VersionManifest, + V2WorkerValidation, + V2WorkerVersion, + v2_application_name, ) if TYPE_CHECKING: + from datetime import datetime + from pydantic.main import IncEx +__all__ = [ + "ACTIVE_MANIFEST_KEY", + "LOCAL_AREA_REPORTS_MANIFEST_KEY", + "LOCAL_AREA_REPORTS_VERSION", + "PRE_CACHE_MANIFEST_KEY", + "REGION_DATASETS_MANIFEST_KEY", + "SCHEMA_2_MANIFEST_KEY", + "V1_APPLICATION_PREFIX", + "V1_ROUTING_STATE_NAME", + "V2_APPLICATION_PREFIX", + "V2_MANIFEST_KEYS", + "V2_VERSION_MANIFEST_NAME", + "CallableName", + "LocalAreaReports", + "ManifestText", + "Stage12CacheRelease", + "StrictManifestModel", + "V2CountryWorker", + "V2ManifestLoader", + "V2ManifestStore", + "V2VersionManifest", + "V2WorkerValidation", + "V2WorkerVersion", + "add_v2_worker", + "assert_separate_manifest_names", + "build_next_v2_manifest", + "manifest_digest", + "move_v2_default", + "parse_v2_manifest", + "pre_cache_manifest_payload", + "pre_region_datasets_manifest_payload", + "publish_v2_manifest", + "read_v2_manifest", + "schema_1_manifest_payload", + "schema_2_manifest_payload", + "v2_application_name", + "v2_manifest_copies", + "validate_manifest_has_no_credentials", + "write_v2_manifest", +] + V1_ROUTING_STATE_NAME = "simulation-api-routing-state" V2_VERSION_MANIFEST_NAME = "simulation-api-v2-version-manifest" -# The schema version 2 manifest. Readers released with schema version 2 read -# this key first. +# The schema version 2 manifest with the UK entry's ``local_area_reports``. +# Readers released with it read this key first. +LOCAL_AREA_REPORTS_MANIFEST_KEY = "active-schema-2-with-local-area-reports" +# The schema version 2 manifest without ``local_area_reports``, which readers +# released before it forbid. They read this key first. SCHEMA_2_MANIFEST_KEY = "active-schema-2" # Readers released before schema version 2 forbid unknown fields, so they read # the keys below, which hold the same manifest in schema version 1 form. This @@ -50,229 +105,12 @@ # A write leaves the schema version 1 keys alone when schema version 1 can't # describe the manifest. V2_MANIFEST_KEYS = ( + LOCAL_AREA_REPORTS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY, REGION_DATASETS_MANIFEST_KEY, ACTIVE_MANIFEST_KEY, PRE_CACHE_MANIFEST_KEY, ) -V2_APPLICATION_PREFIX = "policyengine-simulation-v2-py" -V1_APPLICATION_PREFIX = "policyengine-simulation-py" -_MAX_MODAL_APPLICATION_NAME_LENGTH = 64 -_CALLABLE_PATTERN = re.compile(r"^[a-z][a-z0-9_]{0,127}$") - -ManifestText = Annotated[str, Field(min_length=1, max_length=255)] -CallableName = Annotated[str, Field(pattern=_CALLABLE_PATTERN.pattern)] - - -class StrictManifestModel(BaseModel): - """Base for v2 version manifest models; unknown fields are rejected.""" - - model_config = ConfigDict(extra="forbid", frozen=True) - - -class V2CountryWorker(StrictManifestModel): - """A country served by a v2 worker application, and what it can calculate. - - An economy-capable country names the callable that runs one of its report - simulations; only such a country's reports are given to the report - coordinator. ``household_callable`` names the function that calculates - its households, and is None when the application has none or it was not - validated. - """ - - country: CountryId - economy_capable: bool - single_simulation_callable: CallableName | None = None - household_callable: CallableName | None = None - - @model_validator(mode="after") - def validate_capabilities(self) -> V2CountryWorker: - """Require a simulation callable exactly when the country is economy-capable. - - A country must also have at least one capability. - """ - if self.economy_capable != (self.single_simulation_callable is not None): - msg = ( - "an economy-capable country, and only one, names a simulation callable" - ) - raise ValueError(msg) - if not self.economy_capable and self.household_callable is None: - msg = "a country needs at least one capability" - raise ValueError(msg) - return self - - -class Stage12CacheRelease(StrictManifestModel): - """The US cache a worker image holds: its manifest digest and coverage. - - The cache covers 3 datasets and 60 baselines, one per region-group segment - and year. - """ - - manifest_sha256: Sha256Digest - partition_sha256: Sha256Digest - years: tuple[int, ...] - segment_count: Literal[20] = 20 - dataset_count: Literal[3] = 3 - baseline_count: Literal[60] = 60 - - -class V2WorkerValidation(StrictManifestModel): - """Receipt for a successful validation of a deployed v2 worker application. - - Names the application and bundle digest that were validated, when, and the - validation invocation IDs: overall, for each economy-capable country's - simulation worker, and for each country's household function. - """ - - validated: Literal[True] - application_name: ManifestText - bundle_manifest_sha256: Sha256Digest - validated_at: datetime - validation_invocation_id: ManifestText - country_validation_invocation_ids: dict[CountryId, ManifestText] - household_validation_invocation_ids: dict[CountryId, ManifestText] = Field( - default_factory=dict - ) - cache: Stage12CacheRelease | None = None - - -def _require_function_receipts( - countries: tuple[V2CountryWorker, ...], validation: V2WorkerValidation -) -> None: - """Require a validation invocation for exactly the functions countries name.""" - economy_countries = { - worker.country for worker in countries if worker.economy_capable - } - if set(validation.country_validation_invocation_ids) != economy_countries: - msg = "worker validation does not cover exactly the economy-capable countries" - raise ValueError(msg) - household_countries = { - worker.country for worker in countries if worker.household_callable is not None - } - if set(validation.household_validation_invocation_ids) != household_countries: - msg = "worker validation does not cover exactly the household functions" - raise ValueError(msg) - - -class V2WorkerVersion(StrictManifestModel): - """One deployed v2 worker application and the bundle it runs. - - Lists its report coordinator callable, its per-country workers and its - validation receipt; the countries must match the bundle's, in order, and - every function a country names must have been validated. - """ - - application_name: ManifestText - report_coordinator_callable: CallableName - countries: tuple[V2CountryWorker, ...] - bundle: Stage12BundleManifest - bundle_manifest_sha256: Sha256Digest - validation: V2WorkerValidation - cache: Stage12CacheRelease | None = None - - @model_validator(mode="after") - def validate_worker(self) -> V2WorkerVersion: - """Check the worker's name, validation receipt and countries match its bundle. - - The application name must be the v2 name derived from the bundle's - PolicyEngine.py version, the validation must name this application and - bundle digest, and the countries must equal the bundle's, in order, - each named function with a validation invocation. - """ - if not self.application_name.startswith(V2_APPLICATION_PREFIX): - msg = "v2 application uses an invalid name prefix" - raise ValueError(msg) - if self.application_name.startswith(V1_APPLICATION_PREFIX): - msg = "v2 application name collides with the v1 prefix" - raise ValueError(msg) - expected_name = v2_application_name(self.bundle.policyengine_version) - if self.application_name != expected_name: - msg = "v2 application name does not match its bundle version" - raise ValueError(msg) - if self.validation.application_name != self.application_name: - msg = "worker validation names another application" - raise ValueError(msg) - if self.validation.bundle_manifest_sha256 != self.bundle_manifest_sha256: - msg = "worker validation names another bundle digest" - raise ValueError(msg) - if self.validation.cache != self.cache: - msg = "worker cache metadata differs from worker validation" - raise ValueError(msg) - countries = tuple(worker.country for worker in self.countries) - bundle_countries = tuple(country.country for country in self.bundle.countries) - if len(countries) != len(set(countries)): - msg = "v2 worker countries must be unique" - raise ValueError(msg) - if countries != bundle_countries: - msg = "v2 worker countries do not match the normalized bundle" - raise ValueError(msg) - _require_function_receipts(self.countries, self.validation) - return self - - def economy_worker(self, country: str) -> V2CountryWorker | None: - """Return ``country``'s worker when it can run economy reports.""" - return next( - ( - worker - for worker in self.countries - if worker.country == country and worker.economy_capable - ), - None, - ) - - def economy_simulation_callable(self, country: str) -> str: - """Return the callable that runs one of ``country``'s report simulations. - - Raises LookupError when the manifest does not mark ``country`` as - economy-capable. - """ - worker = self.economy_worker(country) - if worker is None or worker.single_simulation_callable is None: - msg = ( - f"country {country!r} is not economy-capable in {self.application_name}" - ) - raise LookupError(msg) - return worker.single_simulation_callable - - def household_worker(self, country: str) -> V2CountryWorker | None: - """Return ``country``'s worker when it has a household function.""" - return next( - ( - worker - for worker in self.countries - if worker.country == country and worker.household_callable is not None - ), - None, - ) - - -class V2VersionManifest(StrictManifestModel): - """Registry of deployed v2 worker versions, keyed by PolicyEngine.py version. - - ``default_version`` is the version used when a request does not pin one, and - ``generation`` identifies the publish that produced this manifest. - """ - - schema_version: Literal[2] = 2 - generation: ManifestText - default_version: ManifestText - versions: dict[str, V2WorkerVersion] - - @model_validator(mode="after") - def validate_versions(self) -> V2VersionManifest: - """Check the default has an entry and each key is its bundle's version.""" - if not self.versions: - msg = "v2 manifest must contain a worker version" - raise ValueError(msg) - if self.default_version not in self.versions: - msg = "v2 default version has no worker entry" - raise ValueError(msg) - for version, worker in self.versions.items(): - if version != worker.bundle.policyengine_version: - msg = "v2 manifest version key does not match its bundle" - raise ValueError(msg) - return self class V2ManifestStore(Protocol): @@ -287,29 +125,6 @@ def __setitem__(self, key: str, value: object) -> None: ... -def v2_application_name(policyengine_version: str) -> str: - """Derive the v2 Modal application name from a PolicyEngine.py version. - - Dots and plus signs become hyphens. Raises ValueError for a version with - characters a Modal name cannot hold, or a name over 64 characters. - """ - if not policyengine_version or not re.fullmatch( - r"[A-Za-z0-9][A-Za-z0-9._+-]*", policyengine_version - ): - msg = "PolicyEngine.py version cannot form a Modal application name" - raise ValueError(msg) - name = V2_APPLICATION_PREFIX + policyengine_version.replace(".", "-").replace( - "+", "-" - ) - if len(name) > _MAX_MODAL_APPLICATION_NAME_LENGTH: - msg = ( - "v2 Modal application name exceeds " - f"{_MAX_MODAL_APPLICATION_NAME_LENGTH} characters" - ) - raise ValueError(msg) - return name - - def manifest_digest(manifest: V2VersionManifest) -> str: """Return the SHA-256 of the manifest's canonical JSON (sorted, compact).""" payload = json.dumps( @@ -353,7 +168,9 @@ def schema_1_manifest_payload( return None validation_exclude = {"household_validation_invocation_ids"} worker_exclude: dict[str, IncEx | bool] = { - "countries": {"__all__": {"economy_capable", "household_callable"}}, + "countries": { + "__all__": {"economy_capable", "household_callable", "local_area_reports"} + }, "validation": validation_exclude, } if not include_region_datasets: @@ -369,6 +186,16 @@ def schema_1_manifest_payload( return payload +def schema_2_manifest_payload(manifest: V2VersionManifest) -> dict[str, object]: + """Return the manifest as readers released before local-area reports accept it.""" + return manifest.model_dump( + mode="json", + exclude={ + "versions": {"__all__": {"countries": {"__all__": {"local_area_reports"}}}} + }, + ) + + def pre_region_datasets_manifest_payload( manifest: V2VersionManifest, ) -> dict[str, object] | None: @@ -501,7 +328,34 @@ def _is_current_schema_2_document(store: V2ManifestStore, document: object) -> b return _newest_validation(older_manifest) <= _newest_validation(manifest) +def _is_current_local_area_copy(store: V2ManifestStore, document: object) -> bool: + """Return whether ``document``, under the local-area report key, is the latest. + + Code released before ``LocalAreaReports`` publishes only the older keys, + for example after a rollback, which leaves an out-of-date copy behind. A + current copy, without the field, is exactly the manifest under + ``active-schema-2``, which is itself current. Without an + ``active-schema-2`` copy it is not current: deleting that key is how an + operator sends every reader back to the older keys before a rollback. + An unreadable document counts as current, so readers and publishers + report it. + """ + try: + manifest = parse_v2_manifest(document) + except (TypeError, ValueError): + return True + without_field = schema_2_manifest_payload(manifest) + if store.get(SCHEMA_2_MANIFEST_KEY) != without_field: + return False + return _is_current_schema_2_document(store, without_field) + + def _current_manifest(store: V2ManifestStore) -> object: + local_area_document = store.get(LOCAL_AREA_REPORTS_MANIFEST_KEY) + if local_area_document is not None and _is_current_local_area_copy( + store, local_area_document + ): + return local_area_document # Until the first schema version 2 publish, only the older keys exist. schema_2_document = store.get(SCHEMA_2_MANIFEST_KEY) if schema_2_document is not None and _is_current_schema_2_document( @@ -618,7 +472,8 @@ def v2_manifest_copies(manifest: V2VersionManifest) -> dict[str, object]: keep the last manifest written there. """ copies: dict[str, object] = { - SCHEMA_2_MANIFEST_KEY: manifest.model_dump(mode="json") + LOCAL_AREA_REPORTS_MANIFEST_KEY: manifest.model_dump(mode="json"), + SCHEMA_2_MANIFEST_KEY: schema_2_manifest_payload(manifest), } schema_1_copies = _schema_1_copies(manifest) if schema_1_copies is not None: diff --git a/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest_models.py b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest_models.py new file mode 100644 index 000000000..3657bf97f --- /dev/null +++ b/libs/policyengine-simulation-contract/src/policyengine_simulation_contract/stage12_manifest_models.py @@ -0,0 +1,320 @@ +"""The v2 worker version manifest's models, and the v2 application name. + +A manifest (``V2VersionManifest``) lists each deployed v2 worker application +(``V2WorkerVersion``): the countries it serves and what each can calculate +(``V2CountryWorker``, with ``LocalAreaReports`` for the UK), the bundle it +runs, and its validation receipt. ``stage12_manifest`` stores and reads it, +and re-exports every name here. +""" + +from __future__ import annotations + +import re +from datetime import datetime +from typing import Annotated, Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from policyengine_simulation_contract.stage12_bundle import ( + CountryId, + Sha256Digest, + Stage12BundleManifest, +) + +V2_APPLICATION_PREFIX = "policyengine-simulation-v2-py" +V1_APPLICATION_PREFIX = "policyengine-simulation-py" +_MAX_MODAL_APPLICATION_NAME_LENGTH = 64 +_CALLABLE_PATTERN = re.compile(r"^[a-z][a-z0-9_]{0,127}$") +# The local-area report runs a coordinator runs: a run of a UK-wide or nation +# report on the local-area dataset, given as any report run is, calculates its +# full results and its constituency and local-authority results, with the +# codes disclosure control withholds. A later change to what such a run +# accepts or writes raises it. +LOCAL_AREA_REPORTS_VERSION = 1 + +ManifestText = Annotated[str, Field(min_length=1, max_length=255)] +CallableName = Annotated[str, Field(pattern=_CALLABLE_PATTERN.pattern)] + + +class StrictManifestModel(BaseModel): + """Base for v2 version manifest models; unknown fields are rejected.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + +class LocalAreaReports(StrictManifestModel): + """The UK local-area report runs a worker application's coordinator runs. + + ``input_version`` is the ``LOCAL_AREA_REPORTS_VERSION`` of the runs it + accepts. ``dataset`` is the local-area dataset, which the release routes + constituency and local-authority regions to: a run of a UK-wide or nation + report on it also calculates the area results. It is None when the + release routes them to no dataset of their own, so a UK-wide or nation + run on the national dataset calculates both area levels itself. The + worker decides both by the rule its national aggregation uses + (``uk_area_regions_routed``). + """ + + input_version: Annotated[int, Field(ge=1)] + dataset: ManifestText | None + + +class V2CountryWorker(StrictManifestModel): + """A country served by a v2 worker application, and what it can calculate. + + An economy-capable country names the callable that runs one of its report + simulations; only such a country's reports are given to the report + coordinator. ``household_callable`` names the function that calculates + its households, and is None when the application has none or it was not + validated. ``local_area_reports`` is set only on an economy-capable UK + entry whose coordinator runs local-area report runs, and left out of the + JSON otherwise, so other entries are written, and digested, as before. + """ + + country: CountryId + economy_capable: bool + single_simulation_callable: CallableName | None = None + household_callable: CallableName | None = None + local_area_reports: LocalAreaReports | None = Field( + default=None, exclude_if=lambda value: value is None + ) + + @model_validator(mode="after") + def validate_capabilities(self) -> V2CountryWorker: + """Require a simulation callable exactly when the country is economy-capable. + + A country must also have at least one capability. + """ + if self.economy_capable != (self.single_simulation_callable is not None): + msg = ( + "an economy-capable country, and only one, names a simulation callable" + ) + raise ValueError(msg) + if not self.economy_capable and self.household_callable is None: + msg = "a country needs at least one capability" + raise ValueError(msg) + if self.local_area_reports is not None and ( + self.country != "uk" or not self.economy_capable + ): + msg = "only an economy-capable UK entry runs local-area report runs" + raise ValueError(msg) + return self + + +class Stage12CacheRelease(StrictManifestModel): + """The US cache a worker image holds: its manifest digest and coverage. + + The cache covers 3 datasets and 60 baselines, one per region-group segment + and year. + """ + + manifest_sha256: Sha256Digest + partition_sha256: Sha256Digest + years: tuple[int, ...] + segment_count: Literal[20] = 20 + dataset_count: Literal[3] = 3 + baseline_count: Literal[60] = 60 + + +class V2WorkerValidation(StrictManifestModel): + """Receipt for a successful validation of a deployed v2 worker application. + + Names the application and bundle digest that were validated, when, and the + validation invocation IDs: overall, for each economy-capable country's + simulation worker, and for each country's household function. + """ + + validated: Literal[True] + application_name: ManifestText + bundle_manifest_sha256: Sha256Digest + validated_at: datetime + validation_invocation_id: ManifestText + country_validation_invocation_ids: dict[CountryId, ManifestText] + household_validation_invocation_ids: dict[CountryId, ManifestText] = Field( + default_factory=dict + ) + cache: Stage12CacheRelease | None = None + + +def _require_local_area_datasets( + countries: tuple[V2CountryWorker, ...], bundle: Stage12BundleManifest +) -> None: + """Require each declared local-area dataset to be a dataset of the country.""" + for worker in countries: + declared = worker.local_area_reports + if declared is None or declared.dataset is None: + continue + identities = { + dataset.identity + for country in bundle.countries + if country.country == worker.country + for dataset in country.datasets + } + if declared.dataset not in identities: + msg = ( + f"the {worker.country} local-area dataset {declared.dataset!r} is " + "not a dataset of its bundle" + ) + raise ValueError(msg) + + +def _require_function_receipts( + countries: tuple[V2CountryWorker, ...], validation: V2WorkerValidation +) -> None: + """Require a validation invocation for exactly the functions countries name.""" + economy_countries = { + worker.country for worker in countries if worker.economy_capable + } + if set(validation.country_validation_invocation_ids) != economy_countries: + msg = "worker validation does not cover exactly the economy-capable countries" + raise ValueError(msg) + household_countries = { + worker.country for worker in countries if worker.household_callable is not None + } + if set(validation.household_validation_invocation_ids) != household_countries: + msg = "worker validation does not cover exactly the household functions" + raise ValueError(msg) + + +class V2WorkerVersion(StrictManifestModel): + """One deployed v2 worker application and the bundle it runs. + + Lists its report coordinator callable, its per-country workers and its + validation receipt; the countries must match the bundle's, in order, and + every function a country names must have been validated. + """ + + application_name: ManifestText + report_coordinator_callable: CallableName + countries: tuple[V2CountryWorker, ...] + bundle: Stage12BundleManifest + bundle_manifest_sha256: Sha256Digest + validation: V2WorkerValidation + cache: Stage12CacheRelease | None = None + + @model_validator(mode="after") + def validate_worker(self) -> V2WorkerVersion: + """Check the worker's name, validation receipt and countries match its bundle. + + The application name must be the v2 name derived from the bundle's + PolicyEngine.py version, the validation must name this application and + bundle digest, and the countries must equal the bundle's, in order, + each named function with a validation invocation. + """ + if not self.application_name.startswith(V2_APPLICATION_PREFIX): + msg = "v2 application uses an invalid name prefix" + raise ValueError(msg) + if self.application_name.startswith(V1_APPLICATION_PREFIX): + msg = "v2 application name collides with the v1 prefix" + raise ValueError(msg) + expected_name = v2_application_name(self.bundle.policyengine_version) + if self.application_name != expected_name: + msg = "v2 application name does not match its bundle version" + raise ValueError(msg) + if self.validation.application_name != self.application_name: + msg = "worker validation names another application" + raise ValueError(msg) + if self.validation.bundle_manifest_sha256 != self.bundle_manifest_sha256: + msg = "worker validation names another bundle digest" + raise ValueError(msg) + if self.validation.cache != self.cache: + msg = "worker cache metadata differs from worker validation" + raise ValueError(msg) + countries = tuple(worker.country for worker in self.countries) + bundle_countries = tuple(country.country for country in self.bundle.countries) + if len(countries) != len(set(countries)): + msg = "v2 worker countries must be unique" + raise ValueError(msg) + if countries != bundle_countries: + msg = "v2 worker countries do not match the normalized bundle" + raise ValueError(msg) + _require_function_receipts(self.countries, self.validation) + _require_local_area_datasets(self.countries, self.bundle) + return self + + def economy_worker(self, country: str) -> V2CountryWorker | None: + """Return ``country``'s worker when it can run economy reports.""" + return next( + ( + worker + for worker in self.countries + if worker.country == country and worker.economy_capable + ), + None, + ) + + def economy_simulation_callable(self, country: str) -> str: + """Return the callable that runs one of ``country``'s report simulations. + + Raises LookupError when the manifest does not mark ``country`` as + economy-capable. + """ + worker = self.economy_worker(country) + if worker is None or worker.single_simulation_callable is None: + msg = ( + f"country {country!r} is not economy-capable in {self.application_name}" + ) + raise LookupError(msg) + return worker.single_simulation_callable + + def household_worker(self, country: str) -> V2CountryWorker | None: + """Return ``country``'s worker when it has a household function.""" + return next( + ( + worker + for worker in self.countries + if worker.country == country and worker.household_callable is not None + ), + None, + ) + + +class V2VersionManifest(StrictManifestModel): + """Registry of deployed v2 worker versions, keyed by PolicyEngine.py version. + + ``default_version`` is the version used when a request does not pin one, and + ``generation`` identifies the publish that produced this manifest. + """ + + schema_version: Literal[2] = 2 + generation: ManifestText + default_version: ManifestText + versions: dict[str, V2WorkerVersion] + + @model_validator(mode="after") + def validate_versions(self) -> V2VersionManifest: + """Check the default has an entry and each key is its bundle's version.""" + if not self.versions: + msg = "v2 manifest must contain a worker version" + raise ValueError(msg) + if self.default_version not in self.versions: + msg = "v2 default version has no worker entry" + raise ValueError(msg) + for version, worker in self.versions.items(): + if version != worker.bundle.policyengine_version: + msg = "v2 manifest version key does not match its bundle" + raise ValueError(msg) + return self + + +def v2_application_name(policyengine_version: str) -> str: + """Derive the v2 Modal application name from a PolicyEngine.py version. + + Dots and plus signs become hyphens. Raises ValueError for a version with + characters a Modal name cannot hold, or a name over 64 characters. + """ + if not policyengine_version or not re.fullmatch( + r"[A-Za-z0-9][A-Za-z0-9._+-]*", policyengine_version + ): + msg = "PolicyEngine.py version cannot form a Modal application name" + raise ValueError(msg) + name = V2_APPLICATION_PREFIX + policyengine_version.replace(".", "-").replace( + "+", "-" + ) + if len(name) > _MAX_MODAL_APPLICATION_NAME_LENGTH: + msg = ( + "v2 Modal application name exceeds " + f"{_MAX_MODAL_APPLICATION_NAME_LENGTH} characters" + ) + raise ValueError(msg) + return name diff --git a/libs/policyengine-simulation-contract/tests/test_stage12_manifest.py b/libs/policyengine-simulation-contract/tests/test_stage12_manifest.py index eb3b464af..7d7640c80 100644 --- a/libs/policyengine-simulation-contract/tests/test_stage12_manifest.py +++ b/libs/policyengine-simulation-contract/tests/test_stage12_manifest.py @@ -15,27 +15,36 @@ ) from policyengine_simulation_contract.stage12_manifest import ( ACTIVE_MANIFEST_KEY, + LOCAL_AREA_REPORTS_MANIFEST_KEY, + LOCAL_AREA_REPORTS_VERSION, PRE_CACHE_MANIFEST_KEY, REGION_DATASETS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY, V1_ROUTING_STATE_NAME, V2_VERSION_MANIFEST_NAME, + LocalAreaReports, Stage12CacheRelease, V2CountryWorker, V2ManifestLoader, V2VersionManifest, V2WorkerValidation, V2WorkerVersion, + add_v2_worker, assert_separate_manifest_names, manifest_digest, + move_v2_default, parse_v2_manifest, pre_cache_manifest_payload, pre_region_datasets_manifest_payload, publish_v2_manifest, + read_v2_manifest, schema_1_manifest_payload, + schema_2_manifest_payload, v2_application_name, + v2_manifest_copies, validate_manifest_has_no_credentials, ) +from pydantic import ValidationError class FakeStore: @@ -227,7 +236,8 @@ def test_publish_writes_one_complete_v2_document_per_key() -> None: ) assert store.writes == [ - (SCHEMA_2_MANIFEST_KEY, manifest.model_dump(mode="json")), + (LOCAL_AREA_REPORTS_MANIFEST_KEY, manifest.model_dump(mode="json")), + (SCHEMA_2_MANIFEST_KEY, schema_2_manifest_payload(manifest)), (REGION_DATASETS_MANIFEST_KEY, schema_1_manifest_payload(manifest)), (ACTIVE_MANIFEST_KEY, pre_region_datasets_manifest_payload(manifest)), (PRE_CACHE_MANIFEST_KEY, pre_cache_manifest_payload(manifest)), @@ -469,8 +479,11 @@ def test_a_household_only_country_leaves_schema_1_readers_on_their_manifest() -> manifest = publish_v2_manifest(store=store, worker=_worker(household_only=("uk",))) # Schema version 1 cannot describe a country without a single-simulation - # function, so only the schema version 2 document changes. - assert [key for key, _ in store.writes] == [SCHEMA_2_MANIFEST_KEY] + # function, so only the schema version 2 documents change. + assert [key for key, _ in store.writes] == [ + LOCAL_AREA_REPORTS_MANIFEST_KEY, + SCHEMA_2_MANIFEST_KEY, + ] assert store.values[ACTIVE_MANIFEST_KEY] == earlier assert schema_1_manifest_payload(manifest, include_cache=True) is None worker = manifest.versions["5.2.0"] @@ -835,3 +848,199 @@ def test_a_household_only_manifest_is_read_until_an_older_publish_follows() -> N store.values[ACTIVE_MANIFEST_KEY] = pre_region_datasets_manifest_payload(older) assert V2ManifestLoader(store).load().default_version == "5.3.0" + + +# The fields of a country entry before local-area report runs. Readers +# released before them forbid any other field, so adding one would stop them +# loading. +_COUNTRY_FIELDS_BEFORE_LOCAL_AREA_REPORTS = { + "country", + "economy_capable", + "household_callable", + "single_simulation_callable", +} +_UK_LOCAL_AREA_DATASET = "populace_uk_2023_local" + + +def _local_area_worker( + version: str = "5.2.0", *, dataset: str | None = _UK_LOCAL_AREA_DATASET +) -> V2WorkerVersion: + """Return a worker whose UK coordinator runs local-area report runs. + + Its UK bundle also holds the local-area dataset ``dataset`` names. + """ + worker = _regional_worker(version, with_household_functions=True).model_dump( + mode="json" + ) + for country in worker["bundle"]["countries"]: + if country["country"] == "uk": + local_area = dict(country["datasets"][0]) + local_area["identity"] = _UK_LOCAL_AREA_DATASET + country["datasets"].append(local_area) + for country in worker["countries"]: + if country["country"] == "uk": + country["local_area_reports"] = { + "input_version": LOCAL_AREA_REPORTS_VERSION, + "dataset": dataset, + } + return V2WorkerVersion.model_validate(worker) + + +def _uk(worker: V2WorkerVersion) -> V2CountryWorker: + return next(country for country in worker.countries if country.country == "uk") + + +def _stored_countries(document: dict, version: str = "5.2.0") -> list[dict]: + return document["versions"][version]["countries"] + + +def test_a_publish_writes_the_local_area_declaration_under_its_own_key() -> None: + store = FakeStore() + + manifest = publish_v2_manifest(store=store, worker=_local_area_worker()) + + stored = store.values[LOCAL_AREA_REPORTS_MANIFEST_KEY] + assert stored == manifest.model_dump(mode="json") + (uk,) = [ + country for country in _stored_countries(stored) if country["country"] == "uk" + ] + assert uk["local_area_reports"] == { + "input_version": 1, + "dataset": _UK_LOCAL_AREA_DATASET, + } + loaded = _uk(V2ManifestLoader(store).load().versions["5.2.0"]) + assert loaded.local_area_reports == LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, dataset=_UK_LOCAL_AREA_DATASET + ) + + +def test_a_uk_worker_whose_national_runs_keep_the_areas_declares_no_dataset() -> None: + worker = _local_area_worker(dataset=None) + + assert _uk(worker).local_area_reports == LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, dataset=None + ) + + +@pytest.mark.parametrize( + "key", + [ + SCHEMA_2_MANIFEST_KEY, + REGION_DATASETS_MANIFEST_KEY, + ACTIVE_MANIFEST_KEY, + PRE_CACHE_MANIFEST_KEY, + ], +) +def test_readers_released_before_local_area_reports_never_see_the_field( + key: str, +) -> None: + store = FakeStore() + publish_v2_manifest(store=store, worker=_local_area_worker()) + + for country in _stored_countries(store.values[key]): + assert set(country) <= _COUNTRY_FIELDS_BEFORE_LOCAL_AREA_REPORTS + + +def test_the_schema_2_key_holds_the_manifest_as_its_readers_accept_it() -> None: + store = FakeStore() + manifest = publish_v2_manifest(store=store, worker=_local_area_worker()) + + schema_2 = store.values[SCHEMA_2_MANIFEST_KEY] + for country in _stored_countries(schema_2): + assert set(country) == _COUNTRY_FIELDS_BEFORE_LOCAL_AREA_REPORTS + worker = manifest.versions["5.2.0"] + without_declaration = worker.model_copy( + update={ + "countries": tuple( + country.model_copy(update={"local_area_reports": None}) + for country in worker.countries + ) + } + ) + assert parse_v2_manifest(schema_2) == manifest.model_copy( + update={"versions": {"5.2.0": without_declaration}} + ) + + +def test_a_worker_without_the_declaration_is_written_as_before() -> None: + worker = _regional_worker(with_household_functions=True) + manifest = add_v2_worker(current=None, worker=worker) + + for country in worker.model_dump(mode="json")["countries"]: + assert "local_area_reports" not in country + copies = v2_manifest_copies(manifest) + assert copies[LOCAL_AREA_REPORTS_MANIFEST_KEY] == copies[SCHEMA_2_MANIFEST_KEY] + + +def test_a_local_area_copy_left_behind_by_an_older_publish_is_not_read() -> None: + """After a rollback, code released before the field writes only older keys.""" + store = FakeStore() + publish_v2_manifest(store=store, worker=_local_area_worker()) + older = add_v2_worker( + current=read_v2_manifest(store), + worker=_regional_worker("5.3.0", with_household_functions=True), + ) + older = move_v2_default(current=older, version="5.3.0") + for key, copy in v2_manifest_copies(older).items(): + if key != LOCAL_AREA_REPORTS_MANIFEST_KEY: + store.values[key] = copy + + loaded = V2ManifestLoader(store).load() + + assert loaded.default_version == "5.3.0" + assert _uk(loaded.versions["5.2.0"]).local_area_reports is None + + +def test_the_declaration_names_an_input_version_of_one_or_more() -> None: + with pytest.raises(ValidationError, match="input_version"): + LocalAreaReports(input_version=0, dataset=None) + + +def test_only_an_economy_capable_uk_entry_declares_local_area_reports() -> None: + declaration = LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, dataset=None + ) + + with pytest.raises(ValidationError, match="UK"): + V2CountryWorker.model_validate( + { + **_country_worker("us").model_dump(mode="json"), + "local_area_reports": declaration.model_dump(mode="json"), + } + ) + with pytest.raises(ValidationError, match="economy-capable"): + V2CountryWorker.model_validate( + { + **_country_worker("uk", economy=False, household=True).model_dump( + mode="json" + ), + "local_area_reports": declaration.model_dump(mode="json"), + } + ) + + +def test_the_declared_dataset_is_a_dataset_of_the_uk_bundle() -> None: + with pytest.raises(ValidationError, match="populace_uk_2023_missing"): + _local_area_worker(dataset="populace_uk_2023_missing") + + +def test_the_region_dataset_rollback_step_sends_every_reader_to_the_cache_copy() -> ( + None +): + """The Stage 12 doc's step before deploying code released before region datasets. + + Deleting ``active-schema-2`` and ``active-with-region-datasets`` stops + readers released with the local-area key from reading it, as it stops + older readers reading those two keys. + """ + store = FakeStore() + publish_v2_manifest(store=store, worker=_local_area_worker()) + del store.values[SCHEMA_2_MANIFEST_KEY] + del store.values[REGION_DATASETS_MANIFEST_KEY] + + loaded = V2ManifestLoader(store).load() + + worker = loaded.versions["5.2.0"] + us = next(country for country in worker.bundle.countries if country.country == "us") + assert us.region_datasets is None + assert _uk(worker).local_area_reports is None diff --git a/libs/policyengine-simulation-contract/tests/test_stage12_manifest_default.py b/libs/policyengine-simulation-contract/tests/test_stage12_manifest_default.py index 8a85a9696..c9eb6e307 100644 --- a/libs/policyengine-simulation-contract/tests/test_stage12_manifest_default.py +++ b/libs/policyengine-simulation-contract/tests/test_stage12_manifest_default.py @@ -9,6 +9,7 @@ import pytest from policyengine_simulation_contract.stage12_manifest import ( ACTIVE_MANIFEST_KEY, + LOCAL_AREA_REPORTS_MANIFEST_KEY, PRE_CACHE_MANIFEST_KEY, REGION_DATASETS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY, @@ -21,6 +22,7 @@ publish_v2_manifest, read_v2_manifest, schema_1_manifest_payload, + schema_2_manifest_payload, v2_manifest_copies, write_v2_manifest, ) @@ -93,7 +95,8 @@ def test_writing_a_manifest_writes_each_readers_key_and_reads_back(): write_v2_manifest(store=store, manifest=manifest) assert store.writes == [ - (SCHEMA_2_MANIFEST_KEY, manifest.model_dump(mode="json")), + (LOCAL_AREA_REPORTS_MANIFEST_KEY, manifest.model_dump(mode="json")), + (SCHEMA_2_MANIFEST_KEY, schema_2_manifest_payload(manifest)), (REGION_DATASETS_MANIFEST_KEY, schema_1_manifest_payload(manifest)), (ACTIVE_MANIFEST_KEY, pre_region_datasets_manifest_payload(manifest)), (PRE_CACHE_MANIFEST_KEY, pre_cache_manifest_payload(manifest)), diff --git a/libs/policyengine-simulation-persistence/README.md b/libs/policyengine-simulation-persistence/README.md index 7a7bab9ac..b3ca22046 100644 --- a/libs/policyengine-simulation-persistence/README.md +++ b/libs/policyengine-simulation-persistence/README.md @@ -270,7 +270,10 @@ altered a modelled table, or, while that change is still a pull request, its head commit. Until that pull request merges, the job's last step fails and says so. A merge commit keeps the pull request's head on `master`; if the Public API squashes or rebases the pull request instead, its head never -reaches `master`, so move the pin to the commit that landed. +reaches `master`, so move the pin to the commit that landed. The pin may +also move to a later merged commit that alters no modelled table, such as +one adding a table another package writes; the job still checks only this +package's tables, so moving the pin is no check of that other table. **Keeping it current.** A sim pull request that changes the models moves the pin in the same change. The `Public API schema drift` workflow runs the same diff --git a/libs/policyengine-simulation-persistence/public-api-commit b/libs/policyengine-simulation-persistence/public-api-commit index 90f10c4b5..0954d35d2 100644 --- a/libs/policyengine-simulation-persistence/public-api-commit +++ b/libs/policyengine-simulation-persistence/public-api-commit @@ -1 +1 @@ -429d348aeff84d1d345799fc93ee7c02f7e2014a +ca4d396ebb13d655e139eb1c6b07e8c4fe20e224 diff --git a/libs/policyengine-stage12-persistence/README.md b/libs/policyengine-stage12-persistence/README.md index 9dd737d62..3e99f1305 100644 --- a/libs/policyengine-stage12-persistence/README.md +++ b/libs/policyengine-stage12-persistence/README.md @@ -60,3 +60,8 @@ unless both URLs are set: - `REPORT_RUN_TEST_DATABASE_URL`: logs in as `policyengine_v2_modal_worker`; - `REPORT_RUN_TEST_SEED_DATABASE_URL`: logs in as a role that may insert the rows the simulation service would create. + +The tests of a run's withheld area codes need the chain at the revision that +adds `withheld_area_codes` (`1e1e3f079f2c`) or later. On an older database +they fail at the completion's insert, which names the missing table; the +other tests don't read that table. diff --git a/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py b/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py index fa077ac80..78ff3b86e 100644 --- a/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py +++ b/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py @@ -38,6 +38,9 @@ from policyengine_simulation_contract.stage12_execution import ReportRunIds decile_type_type = ENUM("income", "wealth", name="v2_decile_type", create_type=False) +uk_area_type_type = ENUM( + "constituency", "local_authority", name="v2_uk_area_type", create_type=False +) def _report_run_keys() -> tuple[Column[Any], ...]: @@ -150,6 +153,15 @@ def _disclosure_values() -> tuple[Column[Any], ...]: Column("local_authority_code", String(64), nullable=False), Column("local_authority_name", String(255), nullable=False), ) +# The codes of the areas a local-area run estimated but disclosure control did +# not publish, one row per area: none means none were withheld. +withheld_area_codes = Table( + "withheld_area_codes", + metadata, + *_report_run_keys(), + Column("area_type", uk_area_type_type, nullable=False), + Column("code", String(64), nullable=False), +) budget_summary = Table( "budget_summary", metadata, @@ -379,6 +391,11 @@ def report_run_result( ), shared_by_reports=False, ), + ResultRows( + withheld_area_codes, + _withheld_rows(economy, report_keys), + shared_by_reports=False, + ), ), ) @@ -554,6 +571,21 @@ def _area_rows( ) +def _withheld_rows( + economy: SingleYearMacroOutput, keys: Mapping[str, Any] +) -> tuple[dict[str, Any], ...]: + """Return a row per area disclosure control withheld, for the levels it lists.""" + levels = ( + ("constituency", economy.constituency_withheld_codes), + ("local_authority", economy.local_authority_withheld_codes), + ) + return tuple( + {"id": uuid4(), **keys, "area_type": area_type, "code": code} + for area_type, codes in levels + for code in codes or () + ) + + def _budget_rows( economy: SingleYearMacroOutput, keys: Mapping[str, Any], country: str ) -> tuple[dict[str, Any], ...]: diff --git a/libs/policyengine-stage12-persistence/tests/economy_output.py b/libs/policyengine-stage12-persistence/tests/economy_output.py index f2e4aee80..5a55cc4d6 100644 --- a/libs/policyengine-stage12-persistence/tests/economy_output.py +++ b/libs/policyengine-stage12-persistence/tests/economy_output.py @@ -237,3 +237,15 @@ def uk_output_with_areas() -> dict[str, Any]: } ] return output + + +def uk_output_with_withheld_codes() -> dict[str, Any]: + """Return the UK output with areas, and the codes of three withheld areas. + + A UK-wide or nation report on the local-area dataset lists them. + """ + return { + **uk_output_with_areas(), + "constituency_withheld_codes": ["S14000045"], + "local_authority_withheld_codes": ["E06000064", "W06000015"], + } diff --git a/libs/policyengine-stage12-persistence/tests/test_report_run_results.py b/libs/policyengine-stage12-persistence/tests/test_report_run_results.py index 47bcdf68b..1cd058005 100644 --- a/libs/policyengine-stage12-persistence/tests/test_report_run_results.py +++ b/libs/policyengine-stage12-persistence/tests/test_report_run_results.py @@ -14,6 +14,7 @@ US_STATE_FIPS, uk_economy_output, uk_output_with_areas, + uk_output_with_withheld_codes, us_output_with_every_part, ) from policyengine_stage12_persistence import ( @@ -25,7 +26,7 @@ ) from policyengine_stage12_persistence.report_run_results import ResultRows from report_run_database import NOW, SeededReportRun, worker_database_url -from sqlalchemy import insert, select +from sqlalchemy import MetaData, Table, insert, select from sqlalchemy.exc import IntegrityError LATER = NOW + timedelta(minutes=5) @@ -595,3 +596,113 @@ def test_a_district_in_an_unknown_state_is_refused(seeded_run: SeededReportRun) country="us", district_codes=DISTRICT_CODES, ) + + +def _withheld_area_codes(seeder, report_run_id) -> list[dict[str, object]]: + """Return a report run's withheld area code rows, as the seed role reads them. + + The table is reflected here, not with the seeded tables, so the other tests + run on a database built before the Public API added it. + """ + table = Table( + "withheld_area_codes", MetaData(schema="public"), autoload_with=seeder.engine + ) + with seeder.engine.connect() as connection: + return [ + dict(row._mapping) + for row in connection.execute( + select(table).where(table.c.report_run_id == report_run_id) + ) + ] + + +def test_a_local_area_result_stores_its_withheld_codes_with_the_completion( + store, seeder, seeded_run: SeededReportRun +) -> None: + run_ids = seeded_run.run_ids + + written = _complete( + store, seeded_run, output=uk_output_with_withheld_codes(), country="uk" + ) + + assert written == RunWrite(changed=True, status=RunStatus.SUCCEEDED) + stored = _withheld_area_codes(seeder, run_ids.report_run_id) + assert sorted( + ( + row["area_type"], + row["code"], + row["baseline_simulation_run_id"], + row["reform_simulation_run_id"], + ) + for row in stored + ) == [ + ( + "constituency", + "S14000045", + run_ids.baseline_simulation_run_id, + run_ids.reform_simulation_run_id, + ), + ( + "local_authority", + "E06000064", + run_ids.baseline_simulation_run_id, + run_ids.reform_simulation_run_id, + ), + ( + "local_authority", + "W06000015", + run_ids.baseline_simulation_run_id, + run_ids.reform_simulation_run_id, + ), + ] + # The published areas are stored beside them, as for any UK result. + for table_name in ("constituency_impacts", "local_authority_impacts"): + assert seeder.rows(table_name, report_run_id=run_ids.report_run_id) + + +def test_a_repeated_local_area_completion_stores_no_second_set_of_codes( + store, seeder, seeded_run: SeededReportRun +) -> None: + run_ids = seeded_run.run_ids + _complete(store, seeded_run, output=uk_output_with_withheld_codes(), country="uk") + + repeated = store.complete_report_run( + run_ids, + modal_call_id="fc-coordinator", + completed_at=LATER, + result=report_run_result( + uk_output_with_withheld_codes(), run_ids=run_ids, country="uk" + ), + ) + + assert repeated == RunWrite(changed=False, status=RunStatus.SUCCEEDED) + assert len(_withheld_area_codes(seeder, run_ids.report_run_id)) == 3 + + +def test_a_withheld_code_that_cannot_be_stored_rolls_back_the_completion( + store, seeder, seeded_run: SeededReportRun +) -> None: + # One code listed twice breaks the table's unique key in its insert. + run_ids = seeded_run.run_ids + store.claim_report_run( + run_ids.report_run_id, modal_call_id="fc-coordinator", claimed_at=NOW + ) + output = { + **uk_output_with_withheld_codes(), + "constituency_withheld_codes": ["S14000045", "S14000045"], + } + + with pytest.raises(IntegrityError, match="uq_withheld_area_codes_run_type_code"): + store.complete_report_run( + run_ids, + modal_call_id="fc-coordinator", + completed_at=LATER, + result=report_run_result(output, run_ids=run_ids, country="uk"), + ) + + report_run = seeder.row("report_runs", run_ids.report_run_id) + assert report_run["status"] == "running" + assert report_run["data_version"] is None + for table_name in ("constituency_impacts", "local_authority_impacts"): + assert seeder.rows(table_name, report_run_id=run_ids.report_run_id) == [] + assert _withheld_area_codes(seeder, run_ids.report_run_id) == [] diff --git a/libs/policyengine-stage12-persistence/tests/test_withheld_area_codes.py b/libs/policyengine-stage12-persistence/tests/test_withheld_area_codes.py new file mode 100644 index 000000000..01eae40ba --- /dev/null +++ b/libs/policyengine-stage12-persistence/tests/test_withheld_area_codes.py @@ -0,0 +1,87 @@ +"""Tests for the withheld area codes of a UK result, as result-table rows.""" + +from __future__ import annotations + +from typing import Any +from uuid import UUID + +from economy_output import ( + uk_economy_output, + uk_output_with_areas, + uk_output_with_withheld_codes, +) +from policyengine_simulation_contract.macro_output import SingleYearMacroOutput +from policyengine_simulation_contract.stage12_execution import ReportRunIds +from policyengine_stage12_persistence.report_run_results import ( + report_run_result, + withheld_area_codes, +) + +RUN_IDS = ReportRunIds( + report_run_id=UUID("00000000-0000-0000-0000-00000000000a"), + baseline_simulation_run_id=UUID("00000000-0000-0000-0000-00000000000b"), + reform_simulation_run_id=UUID("00000000-0000-0000-0000-00000000000c"), +) +RUN_KEYS = { + "report_run_id": RUN_IDS.report_run_id, + "baseline_simulation_run_id": RUN_IDS.baseline_simulation_run_id, + "reform_simulation_run_id": RUN_IDS.reform_simulation_run_id, +} + + +def _withheld_rows(output: dict[str, Any]) -> list[dict[str, Any]]: + result = report_run_result(output, run_ids=RUN_IDS, country="uk") + (rows,) = [rows for rows in result.rows if rows.table is withheld_area_codes] + assert rows.shared_by_reports is False + return [ + {name: value for name, value in row.items() if name != "id"} + for row in rows.rows + ] + + +def test_a_uk_result_stores_one_row_per_withheld_area() -> None: + assert _withheld_rows(uk_output_with_withheld_codes()) == [ + {**RUN_KEYS, "area_type": "constituency", "code": "S14000045"}, + {**RUN_KEYS, "area_type": "local_authority", "code": "E06000064"}, + {**RUN_KEYS, "area_type": "local_authority", "code": "W06000015"}, + ] + + +def test_withheld_rows_have_their_own_ids() -> None: + output = { + **uk_output_with_areas(), + "constituency_withheld_codes": ["S14000045", "S14000046"], + "local_authority_withheld_codes": [], + } + result = report_run_result(output, run_ids=RUN_IDS, country="uk") + + (rows,) = [rows for rows in result.rows if rows.table is withheld_area_codes] + assert len({row["id"] for row in rows.rows}) == 2 + + +def test_a_result_that_lists_no_withheld_codes_stores_none() -> None: + assert _withheld_rows(uk_output_with_areas()) == [] + + +def test_a_result_without_withheld_codes_dumps_as_before() -> None: + # National results, and every result calculated before the field, are + # written and compared exactly as they were. + dumped = SingleYearMacroOutput.model_validate(uk_economy_output()).model_dump( + mode="json" + ) + + assert "constituency_withheld_codes" not in dumped + assert "local_authority_withheld_codes" not in dumped + + +def test_a_result_with_withheld_codes_keeps_them() -> None: + output = { + **uk_output_with_areas(), + "constituency_withheld_codes": [], + "local_authority_withheld_codes": ["E06000064"], + } + + dumped = SingleYearMacroOutput.model_validate(output).model_dump(mode="json") + + assert dumped["constituency_withheld_codes"] == [] + assert dumped["local_authority_withheld_codes"] == ["E06000064"] diff --git a/projects/policyengine-simulation-executor/fixtures/macro_output.py b/projects/policyengine-simulation-executor/fixtures/macro_output.py index 404296b5b..84dee9808 100644 --- a/projects/policyengine-simulation-executor/fixtures/macro_output.py +++ b/projects/policyengine-simulation-executor/fixtures/macro_output.py @@ -26,6 +26,13 @@ "cliff_impact", } +# Listed only by a UK-wide or nation report on the local-area dataset, and +# left out of every other result, so the result keys above are unchanged. +LOCAL_AREA_WITHHELD_CODE_KEYS = { + "constituency_withheld_codes", + "local_authority_withheld_codes", +} + CURRENT_REQUIRED_BUDGET_KEYS = { "budgetary_impact", "tax_revenue_impact", diff --git a/projects/policyengine-simulation-executor/src/modal/utils/update_v2_version_manifest.py b/projects/policyengine-simulation-executor/src/modal/utils/update_v2_version_manifest.py index 1272e84e4..b5b3b94c9 100644 --- a/projects/policyengine-simulation-executor/src/modal/utils/update_v2_version_manifest.py +++ b/projects/policyengine-simulation-executor/src/modal/utils/update_v2_version_manifest.py @@ -20,10 +20,13 @@ household_callable_name, ) from policyengine_simulation_contract.stage12_manifest import ( + LOCAL_AREA_REPORTS_MANIFEST_KEY, + LOCAL_AREA_REPORTS_VERSION, SCHEMA_2_MANIFEST_KEY, V1_ROUTING_STATE_NAME, V2_MANIFEST_KEYS, V2_VERSION_MANIFEST_NAME, + LocalAreaReports, V2CountryWorker, V2ManifestLoader, V2ManifestStore, @@ -42,6 +45,23 @@ write_v2_manifest, ) from policyengine_simulation_executor.stage12_bundle import load_stage12_bundle +from policyengine_simulation_executor.uk_local_authority_metadata import ( + routed_uk_area_dataset_name, + uk_area_regions_routed, +) + + +def uk_local_area_reports() -> LocalAreaReports: + """Return the UK local-area report runs this code's coordinator runs. + + The dataset is the one the installed release routes UK area regions to, + or None when it routes them to none, by the rule the worker and Stage + 12's national aggregation use; it reads only the packaged release. + """ + return LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, + dataset=routed_uk_area_dataset_name() if uk_area_regions_routed() else None, + ) def build_worker(validation_payload: object) -> V2WorkerVersion: @@ -49,7 +69,9 @@ def build_worker(validation_payload: object) -> V2WorkerVersion: Every bundle country has a single-simulation worker, so each is recorded as economy-capable. A country's household function is recorded when the - record shows that it calculated the validation household. Raises + record shows that it calculated the validation household. The UK entry + records that this code's coordinator runs local-area report runs, and on + which dataset (``uk_local_area_reports``). Raises ValueError if the record is invalid or the entry carries a credential-like key. """ @@ -68,6 +90,9 @@ def build_worker(validation_payload: object) -> V2WorkerVersion: if country.country in validation.household_validation_invocation_ids else None ), + local_area_reports=( + uk_local_area_reports() if country.country == "uk" else None + ), ) for country in resolved.bundle.countries ), @@ -184,18 +209,23 @@ def _refresh(store: V2ManifestStore) -> V2VersionManifest: return manifest +# Current readers read the local-area report key first, and use it only +# while the schema version 2 key holds its copy. +REQUIRED_KEYS = frozenset({LOCAL_AREA_REPORTS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY}) + + def _key_holds(store: V2ManifestStore, key: str, copies: tuple[object, ...]) -> bool: """Return whether ``key`` holds one of ``copies``, or is an older key that expired. Modal expires a Dict entry after 7 days without a read or write, and - current readers read the schema version 1 keys only when the schema - version 2 key is missing or out of date, so they can expire between - deployments. A write never deletes a key, so a missing older key is never - one a part-way write left behind, and the next write restores it. The - schema version 2 key, which current readers read first, is required. + current readers read the older keys only when the newest key is missing + or out of date, so they can expire between deployments. A write never + deletes a key, so a missing older key is never one a part-way write left + behind, and the next write restores it. The two keys current readers + need are required (``REQUIRED_KEYS``). """ value = store.get(key) - if value is None and key != SCHEMA_2_MANIFEST_KEY: + if value is None and key not in REQUIRED_KEYS: return True return value in copies diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_builder.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_builder.py index 7afbd7252..e467ca83e 100644 --- a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_builder.py +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_builder.py @@ -61,8 +61,10 @@ class SimulationOutputBuilder: ``runtime`` is given. ``omit_uk_area_breakdowns`` leaves the UK constituency and local-authority breakdowns out, for a UK-wide or nation-level run on a bundle that routes area regions to a local-area - dataset (``uk_area_breakdowns_omitted``); those results come from a - separate local-area breakdown run (``LocalAreaBreakdownBuilder``). + dataset (``uk_area_breakdowns_omitted``); in the runtime those results + come from a separate local-area breakdown run + (``LocalAreaBreakdownBuilder``), and Stage 12 keeps them for a report on + the local-area dataset itself, with ``uk_withheld_area_codes``. """ country: str @@ -291,6 +293,20 @@ def _uk_geography_disclosure_output( ) return self._uk_geography_disclosure + def uk_withheld_area_codes(self) -> dict[str, list[str]] | None: + """Return each UK level's withheld area codes, sorted; None outside the UK. + + They are the areas the disclosure pass behind the area results + estimated but did not publish, from that same pass. + """ + disclosure = self._uk_geography_disclosure_output() + if disclosure is None: + return None + return { + level: disclosure.withheld_codes(level) + for level in ("constituency", "local_authority") + } + def _simulation_year(self) -> int: """Return the simulated year, which labels the disclosure release.""" for value in ( diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/aggregation.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/aggregation.py index 24bde0c91..4b732f85a 100644 --- a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/aggregation.py +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/aggregation.py @@ -15,6 +15,8 @@ stage12_output_plan_sha256, ) +from .local_area_datasets import is_local_area_report + if TYPE_CHECKING: from collections.abc import Mapping @@ -175,7 +177,11 @@ def build_aggregate_report( the stored frames, so no model is recomputed. A UK national or nation-level report whose bundle routes area regions to another dataset has no area codes in its frames (see ``output_planning``), so its constituency and - local-authority breakdowns are left out. + local-authority breakdowns are left out, unless the report is on that + dataset (``is_local_area_report``): it then gets both levels and their + withheld codes, from the same disclosure pass, and raises ``ValueError`` + when a level has neither published nor withheld areas, so a succeeded run + always has rows for each level. """ from policyengine_simulation_executor.simulation_output_builder import ( # noqa: PLC0415 # resolved per call so tests can patch simulation_output_builder.SimulationOutputBuilder SimulationOutputBuilder, @@ -228,7 +234,8 @@ def stand_in( "region": report.baseline.geography.region, **report.baseline.options, } - output = SimulationOutputBuilder( + local_area_report = is_local_area_report(report) + builder = SimulationOutputBuilder( country=country, simulation_params=params, country_module=country_module, @@ -238,10 +245,15 @@ def stand_in( resolved_data_version=report.baseline.bundle.dataset.artifact_revision, resolved_region_code=report.baseline.geography.region, uk_local_authority_metadata=uk_local_authority_metadata, - omit_uk_area_breakdowns=uk_area_breakdowns_omitted( - country, report.baseline.geography.region - ), - ).serialize() + omit_uk_area_breakdowns=not local_area_report + and uk_area_breakdowns_omitted(country, report.baseline.geography.region), + ) + output = builder.serialize() + withheld = builder.uk_withheld_area_codes() if local_area_report else None + if withheld is not None: + _require_estimated_areas(output, withheld) + output["constituency_withheld_codes"] = withheld["constituency"] + output["local_authority_withheld_codes"] = withheld["local_authority"] output.update( build_spm_result( report=report, @@ -259,6 +271,22 @@ def stand_in( } +def _require_estimated_areas( + output: Mapping[str, Any], withheld: Mapping[str, list[str]] +) -> None: + """Raise unless each UK area level has an area, published or withheld. + + Disclosure control may withhold every area of a level, but a UK-wide or + nation report on the local-area dataset always estimates some, so a level + with neither means its area results weren't calculated. A succeeded run + of such a report then always has rows for each level. + """ + for level in ("constituency", "local_authority"): + if not (output.get(f"{level}_impact") or withheld[level]): + msg = f"the local-area report estimated no {level} area" + raise ValueError(msg) + + def build_spm_result( *, report: ReportExecutionInput, diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/local_area_datasets.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/local_area_datasets.py new file mode 100644 index 000000000..18d853890 --- /dev/null +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/local_area_datasets.py @@ -0,0 +1,115 @@ +"""The local-area dataset a UK-wide or nation simulation may be calculated on. + +A UK local-area report run calculates its UK-wide or nation region on the +dataset the release routes constituency and local-authority regions to +(``routed_uk_area_dataset_name``), not on the dataset the region loads +otherwise, so its area results come only from that dataset's own weights. It +does so only on a release that routes them (``uk_area_regions_routed``), +whose UK-wide and nation runs leave the area results out: the rule Stage 12's +national aggregation uses (``uk_area_breakdowns_omitted``), and the one the v2 +manifest's ``LocalAreaReports`` declares to the simulation service. The +service names that dataset in each simulation's provenance, as it names any +dataset. The UK worker accepts it, selects it in place of the region's own +dataset, keeps the region's scoping (a nation's filter), and checks the +columns the area outputs read as it loads. +""" + +from __future__ import annotations + +from dataclasses import replace +from typing import TYPE_CHECKING + +from policyengine_simulation_contract.stage12_bundle import region_type_of + +from policyengine_simulation_executor.stage12_bundle import load_stage12_bundle +from policyengine_simulation_executor.uk_local_authority_metadata import ( + routed_uk_area_dataset_name, + uk_area_regions_routed, +) + +if TYPE_CHECKING: + from policyengine_simulation_contract.stage12_bundle import ( + Stage12CountryBundle, + Stage12Dataset, + ) + from policyengine_simulation_contract.stage12_execution import ( + ReportExecutionInput, + SimulationExecutionInput, + ) + + from policyengine_simulation_executor.region_resolution import RegionResolution + +# The region types a local-area report run covers: the UK and its nations. +LOCAL_AREA_RUN_REGION_TYPES = frozenset({"national", "country"}) + + +def calculates_on_local_area_dataset(simulation: SimulationExecutionInput) -> bool: + """Whether ``simulation`` is of the UK or a nation, on the local-area dataset. + + That is the dataset the release routes UK area regions to, on a release + that routes them to one. + """ + if simulation.geography.country != "uk": + return False + region_type = region_type_of("uk", simulation.geography.region) + if region_type not in LOCAL_AREA_RUN_REGION_TYPES or not uk_area_regions_routed(): + return False + return simulation.bundle.dataset.identity == routed_uk_area_dataset_name() + + +def is_local_area_report(report: ReportExecutionInput) -> bool: + """Whether a run of ``report`` also calculates the UK area results. + + It does for a report of the UK or a nation whose simulations are on the + local-area dataset, an ordinary report whose simulations the simulation + service names on that dataset: it returns its full results and its + constituency and local-authority results, with their withheld codes. + Both of a report's simulations share its dataset and region. + """ + return calculates_on_local_area_dataset(report.baseline) + + +def named_local_area_dataset( + country: Stage12CountryBundle, simulation: SimulationExecutionInput +) -> Stage12Dataset | None: + """Return the local-area dataset ``simulation`` names, or None. + + It is the UK's local-area dataset when the release routes UK area regions + to it, the simulation is of the UK or one of its nations, and its + provenance names that dataset. The caller still checks the provenance's + URI and revision against it. + """ + if country.country != "uk" or not calculates_on_local_area_dataset(simulation): + return None + identity = simulation.bundle.dataset.identity + return next((item for item in country.datasets if item.identity == identity), None) + + +def local_area_dataset_name(simulation: SimulationExecutionInput) -> str | None: + """Return the local-area dataset ``simulation`` names in the installed bundle.""" + country = next( + ( + item + for item in load_stage12_bundle().bundle.countries + if item.country == simulation.geography.country + ), + None, + ) + if country is None: + return None + dataset = named_local_area_dataset(country, simulation) + return None if dataset is None else dataset.identity + + +def dataset_region( + region: RegionResolution, local_area_dataset: str | None +) -> RegionResolution: + """Return the resolution the simulation's dataset is selected by. + + A local-area run selects the local-area dataset; every other run selects + the dataset its region names. Only the dataset differs: the simulation + is still scoped by ``region`` itself. + """ + if local_area_dataset is None: + return region + return replace(region, dataset_reference=local_area_dataset) diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/output_planning.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/output_planning.py index f619053fe..0281e8d24 100644 --- a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/output_planning.py +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/output_planning.py @@ -28,6 +28,8 @@ uk_area_codes_required, ) +from .local_area_datasets import calculates_on_local_area_dataset + if TYPE_CHECKING: import pandas as pd from policyengine_simulation_contract.stage12_bundle import CountryId @@ -110,9 +112,8 @@ def _include_cliff_impacts(simulation: SimulationExecutionInput) -> bool: def _required_dataset_variables( + simulation: SimulationExecutionInput, *, - country: CountryId, - region: str, aggregates: tuple[ReportAggregate, ...], ) -> dict[str, tuple[str, ...]]: """Return input-dataset columns copied into the materialized output. @@ -121,11 +122,21 @@ def _required_dataset_variables( simulations, except in a national or nation-level report whose bundle routes UK area regions to another dataset. Its national dataset may then carry no area codes, so none are planned, and the aggregate leaves the - area breakdowns out (``build_aggregate_report``). + area breakdowns out (``build_aggregate_report``). A UK-wide or nation + simulation on the dataset the release routes area regions to + (``calculates_on_local_area_dataset``) has them, and a local-area report + reads them, so they are planned there too, also when the simulation is + planned alone: its run's file then holds what a local-area report reads. """ - if country != "uk" or ReportAggregate.GEOGRAPHIC not in aggregates: + if ( + simulation.geography.country != "uk" + or ReportAggregate.GEOGRAPHIC not in aggregates + ): return {} - if not uk_area_codes_required(region): + if not ( + uk_area_codes_required(simulation.geography.region) + or calculates_on_local_area_dataset(simulation) + ): return {} return UK_GEOGRAPHIC_DATASET_VARIABLES @@ -223,9 +234,7 @@ def _resolve_output_plan( baseline_variables = model.resolve_entity_variables(baseline) reform_variables = model.resolve_entity_variables(reform) dataset_variables = _required_dataset_variables( - country=country, - region=baseline_input.geography.region, - aggregates=requirements.aggregates, + baseline_input, aggregates=requirements.aggregates ) entities = [] for entity in sorted( diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/simulation.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/simulation.py index 2d542f1ab..c552bdc09 100644 --- a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/simulation.py +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/stage12_runtime/simulation.py @@ -37,6 +37,7 @@ ) from policyengine_simulation_executor.stage12_bundle import load_stage12_bundle from policyengine_simulation_executor.uk_local_authority_metadata import ( + detect_uk_area_dataset_metadata, detect_uk_run_dataset_metadata, ) @@ -56,6 +57,11 @@ Stage12FailureDetail, validate_policy_periods, ) +from .local_area_datasets import ( + dataset_region, + local_area_dataset_name, + named_local_area_dataset, +) from .output_planning import ( apply_output_plan, resolve_simulation_output_plan, @@ -172,12 +178,13 @@ def _require_installed_bundle(simulation: SimulationExecutionInput) -> None: for item in resolved.bundle.countries if item.country == simulation.geography.country ) - # A region loads the dataset the bundle names for its region type. Entry + # A region loads the dataset the bundle names for its region type, and a + # local-area run of the UK or a nation the local-area dataset. Entry # services released before regional provenance name the default dataset # for every region, so the default stays accepted until they are updated. regional = country.dataset_for_region(simulation.geography.region) named = simulation.bundle.dataset.identity - regional_entry = next( + regional_entry = named_local_area_dataset(country, simulation) or next( (item for item in country.datasets if item.identity == regional), None ) if regional_entry is not None and named == regional_entry.identity: @@ -258,6 +265,7 @@ def calculate_simulation_frames( msg = "population artifact differs from bundle dataset provenance" raise ValueError(msg) _require_installed_bundle(simulation) + local_area_dataset = local_area_dataset_name(simulation) country = simulation.geography.country params = simulation_runtime_params(simulation) from policyengine_simulation_executor.simulation_runtime import ( # noqa: PLC0415 # resolved per call so tests can patch simulation_runtime helpers @@ -288,7 +296,7 @@ def calculate_simulation_frames( with stage_span(Stage.DATASET_RESOLUTION): dataset_selection = _resolve_dataset_selection( params, - region_resolution=region, + region_resolution=dataset_region(region, local_area_dataset), ) _require_loaded_dataset_matches_provenance(simulation, dataset_selection) with stage_span(Stage.DATASET_LOAD): @@ -308,11 +316,18 @@ def calculate_simulation_frames( ), dataset_name=dataset_selection.name, ) - uk_local_authority_metadata = detect_uk_run_dataset_metadata( - country, - dataset, - region_code=region.code, - dataset_name=dataset_selection.name, + uk_local_authority_metadata = ( + detect_uk_run_dataset_metadata( + country, + dataset, + region_code=region.code, + dataset_name=dataset_selection.name, + ) + if local_area_dataset is None + # It must carry every column the area outputs read. + else detect_uk_area_dataset_metadata( + dataset, dataset_name=dataset_selection.name + ) ) with stage_span(Stage.POLICY_NORMALIZATION): policy = normalise_policy(simulation.policy) diff --git a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/uk_local_authority_metadata.py b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/uk_local_authority_metadata.py index 70766120a..9fb60190e 100644 --- a/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/uk_local_authority_metadata.py +++ b/projects/policyengine-simulation-executor/src/policyengine_simulation_executor/uk_local_authority_metadata.py @@ -349,7 +349,8 @@ def uk_area_breakdowns_omitted(country: str, region_code: str | None) -> bool: the routed dataset (``breakdown: "local_areas"``). Constituency and local-authority runs, UK runs on a bundle that does not route area regions, and other countries' runs are unaffected. The runtime and Stage - 12 both decide by this. + 12 both decide by this, except that Stage 12 keeps both levels for a + report on the routed dataset itself (``is_local_area_report``). """ return country == "uk" and not uk_area_codes_required(region_code) diff --git a/projects/policyengine-simulation-executor/tests/test_simulation_api_contracts.py b/projects/policyengine-simulation-executor/tests/test_simulation_api_contracts.py index 03d639d93..507143329 100644 --- a/projects/policyengine-simulation-executor/tests/test_simulation_api_contracts.py +++ b/projects/policyengine-simulation-executor/tests/test_simulation_api_contracts.py @@ -4,6 +4,7 @@ CURRENT_REQUIRED_BUDGET_KEYS, CURRENT_SINGLE_YEAR_MACRO_KEYS, CURRENT_SINGLE_YEAR_MACRO_RESULT, + LOCAL_AREA_WITHHELD_CODE_KEYS, ) from policyengine_simulation_contract.gateway_models import ( BudgetWindowAnnualImpact, @@ -33,7 +34,9 @@ def test_job_status_result_preserves_current_single_year_macro_dict_contract(): def test_internal_single_year_macro_schema_matches_current_public_keys(): - assert set(SingleYearMacroOutput.model_fields) == CURRENT_SINGLE_YEAR_MACRO_KEYS + assert set(SingleYearMacroOutput.model_fields) == ( + CURRENT_SINGLE_YEAR_MACRO_KEYS | LOCAL_AREA_WITHHELD_CODE_KEYS + ) def test_internal_single_year_macro_schema_serializes_current_public_contract(): @@ -48,7 +51,7 @@ def test_openapi_publishes_the_single_year_macro_result_schema(): assert "SingleYearMacroOutput" in schemas assert set(schemas["SingleYearMacroOutput"]["properties"]) == ( - CURRENT_SINGLE_YEAR_MACRO_KEYS + CURRENT_SINGLE_YEAR_MACRO_KEYS | LOCAL_AREA_WITHHELD_CODE_KEYS ) result_schema = schemas["JobStatusResponse"]["properties"]["result"] assert result_schema == { diff --git a/projects/policyengine-simulation-executor/tests/test_stage12_local_area_aggregation.py b/projects/policyengine-simulation-executor/tests/test_stage12_local_area_aggregation.py new file mode 100644 index 000000000..a0331f394 --- /dev/null +++ b/projects/policyengine-simulation-executor/tests/test_stage12_local_area_aggregation.py @@ -0,0 +1,369 @@ +"""Tests for the area results of a UK-wide report on the local-area dataset.""" + +from __future__ import annotations + +import copy +from typing import Any + +import pandas as pd +import pytest +from fixtures.macro_output import CURRENT_SINGLE_YEAR_MACRO_RESULT +from policyengine_simulation_contract.stage12_execution import ( + AggregateReportArtifactPayload, + ReportExecutionInput, +) +from policyengine_simulation_executor import ( + simulation_output_builder, + simulation_output_geographic, +) +from policyengine_simulation_executor.simulation_output_builder import ( + SimulationOutputBuilder, +) +from policyengine_simulation_executor.simulation_output_geographic import ( + UKAreaIncomeChange, + UKGeographyDisclosureOutput, +) +from policyengine_simulation_executor.stage12_runtime import ( + RetainedSimulationOutput, + build_aggregate_report, + local_area_datasets, +) +from policyengine_stage12_persistence.report_run_results import ( + report_run_result, + withheld_area_codes, +) +from stage12_fake_report_runs import REPORT_RUN_IDS +from stage12_fake_stores import FakeArtifacts +from stage12_runtime_support import bundle_provenance, report_execution_input + +NATIONAL_DATASET = "microcosm_uk_2024_25" +LOCAL_AREA_DATASET = "microcosm_uk_2024_25_dense" +UK_REFORM = { + "gov.hmrc.income_tax.charges.CB_HITC.phase_out_start": { + "2026-01-01.2100-12-31": 70_000 + } +} +PUBLISHED = UKAreaIncomeChange( + tier="reported", + average=-10.0, + average_lower=-12.0, + average_upper=-8.0, + relative=-0.0003, + relative_lower=-0.0004, + relative_upper=-0.0002, + population=41_000.0, +) +# Two constituencies and two authorities estimated; one of each published. +DISCLOSURE = UKGeographyDisclosureOutput( + areas={ + "constituency": {"E14001063": PUBLISHED}, + "local_authority": {"E06000063": PUBLISHED}, + }, + estimated={ + "constituency": frozenset({"E14001063", "W07000081"}), + "local_authority": frozenset({"E06000063", "W06000015", "E06000064"}), + }, +) + + +def _route_uk_area_regions(monkeypatch) -> None: + """Make the installed release route UK area regions to the local-area dataset.""" + monkeypatch.setattr( + "policyengine_simulation_executor.uk_local_authority_metadata." + "uk_area_regions_routed", + lambda: True, + ) + monkeypatch.setattr(local_area_datasets, "uk_area_regions_routed", lambda: True) + monkeypatch.setattr( + local_area_datasets, "routed_uk_area_dataset_name", lambda: LOCAL_AREA_DATASET + ) + + +def _uk_report(*, region: str, dataset: str) -> ReportExecutionInput: + """Return a UK report whose two simulations are on ``dataset``.""" + provenance = bundle_provenance() + bundle = provenance.model_copy( + update={ + "country_package_name": "policyengine-uk", + "country_package_version": "2.174.1", + "dataset": provenance.dataset.model_copy( + update={ + "identity": dataset, + "uri": f"hf://policyengine/microcosm-uk/{dataset}.h5@revision", + } + ), + } + ) + payload = report_execution_input().model_dump(mode="json") + for role, policy in (("baseline", {}), ("reform", UK_REFORM)): + payload[role].update( + { + "policy": policy, + "geography": {"country": "uk", "region": region}, + "bundle": bundle.model_dump(mode="json"), + "population": { + "kind": "dataset", + "artifact": { + "uri": bundle.dataset.uri, + "media_type": "application/x-hdf5", + "content_sha256": "a" * 64, + }, + }, + } + ) + return ReportExecutionInput.model_validate(payload) + + +class _CapturingBuilder: + """Stands in for the output builder: records its fields, lists codes.""" + + built: dict[str, Any] + + def __init__(self, **values: Any) -> None: + type(self).built = values + + def serialize(self) -> dict[str, Any]: + return {"constituency_impact": [], "local_authority_impact": []} + + def uk_withheld_area_codes(self) -> dict[str, list[str]]: + return { + "constituency": ["W07000081"], + "local_authority": ["E06000064", "W06000015"], + } + + +def _aggregate(monkeypatch, report: ReportExecutionInput) -> dict[str, Any]: + monkeypatch.setattr( + simulation_output_builder, "SimulationOutputBuilder", _CapturingBuilder + ) + frames = { + entity: pd.DataFrame({f"{entity}_id": [1]}) + for entity in ("person", "benunit", "household") + } + artifacts = FakeArtifacts() + return build_aggregate_report( + report=report, + baseline=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.baseline) + ), + reform=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.reform) + ), + )["result"] + + +@pytest.mark.parametrize("region", ["uk", "country/wales"]) +def test_a_uk_wide_or_nation_report_on_the_local_area_dataset_gets_its_areas( + monkeypatch, region: str +) -> None: + _route_uk_area_regions(monkeypatch) + + result = _aggregate( + monkeypatch, _uk_report(region=region, dataset=LOCAL_AREA_DATASET) + ) + + assert _CapturingBuilder.built["omit_uk_area_breakdowns"] is False + assert result["constituency_withheld_codes"] == ["W07000081"] + assert result["local_authority_withheld_codes"] == ["E06000064", "W06000015"] + + +def test_a_uk_wide_report_on_the_national_dataset_still_leaves_its_areas_out( + monkeypatch, +) -> None: + _route_uk_area_regions(monkeypatch) + + result = _aggregate(monkeypatch, _uk_report(region="uk", dataset=NATIONAL_DATASET)) + + assert _CapturingBuilder.built["omit_uk_area_breakdowns"] is True + assert "constituency_withheld_codes" not in result + assert "local_authority_withheld_codes" not in result + + +@pytest.mark.parametrize( + "region", ["constituency/E14001063", "local_authority/E06000063"] +) +def test_an_area_report_keeps_its_area_and_lists_no_withheld_codes( + monkeypatch, region: str +) -> None: + # Its result, and its comparison with production, stay as they are. + _route_uk_area_regions(monkeypatch) + + result = _aggregate( + monkeypatch, _uk_report(region=region, dataset=LOCAL_AREA_DATASET) + ) + + assert _CapturingBuilder.built["omit_uk_area_breakdowns"] is False + assert "constituency_withheld_codes" not in result + + +def test_the_builder_lists_each_levels_withheld_codes_from_one_disclosure_pass( + monkeypatch, +) -> None: + passes = [] + + def disclose(*_args, **_kwargs) -> UKGeographyDisclosureOutput: + passes.append(1) + return DISCLOSURE + + monkeypatch.setattr( + simulation_output_geographic, "compute_uk_geography_disclosure_output", disclose + ) + builder = SimulationOutputBuilder( + country="uk", + simulation_params={"time_period": "2026"}, + country_module=None, + dataset=None, + baseline=None, + reform=None, + ) + + assert builder.uk_withheld_area_codes() == { + "constituency": ["W07000081"], + "local_authority": ["E06000064", "W06000015"], + } + # The codes come from the disclosure pass the area results use. + builder.uk_withheld_area_codes() + assert len(passes) == 1 + + +def test_a_builder_outside_the_uk_lists_no_withheld_codes() -> None: + builder = SimulationOutputBuilder( + country="us", + simulation_params={}, + country_module=None, + dataset=None, + baseline=None, + reform=None, + ) + + assert builder.uk_withheld_area_codes() is None + + +class _FullOutputBuilder(_CapturingBuilder): + """Stands in for the output builder with a complete economy output.""" + + def serialize(self) -> dict[str, Any]: + output = copy.deepcopy(CURRENT_SINGLE_YEAR_MACRO_RESULT) + # A UK result: its areas, and no congressional districts. + output.update( + congressional_district_impact=None, + constituency_impact=[], + local_authority_impact=[], + ) + return output + + +def test_a_local_area_aggregate_is_stored_with_its_withheld_codes(monkeypatch) -> None: + _route_uk_area_regions(monkeypatch) + monkeypatch.setattr( + simulation_output_builder, "SimulationOutputBuilder", _FullOutputBuilder + ) + report = _uk_report(region="uk", dataset=LOCAL_AREA_DATASET) + frames = { + entity: pd.DataFrame({f"{entity}_id": [1]}) + for entity in ("person", "benunit", "household") + } + artifacts = FakeArtifacts() + aggregate = build_aggregate_report( + report=report, + baseline=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.baseline) + ), + reform=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.reform) + ), + ) + + # The coordinator validates the aggregate, then stores its result. + result = AggregateReportArtifactPayload.model_validate(aggregate).result + stored = report_run_result(result, run_ids=REPORT_RUN_IDS, country="uk") + (withheld,) = [rows for rows in stored.rows if rows.table is withheld_area_codes] + assert [(row["area_type"], row["code"]) for row in withheld.rows] == [ + ("constituency", "W07000081"), + ("local_authority", "E06000064"), + ("local_authority", "W06000015"), + ] + + +class _AreaBuilder(_CapturingBuilder): + """Stands in for an output builder with given published and withheld areas.""" + + published: dict[str, list[dict[str, Any]]] + withheld: dict[str, list[str]] + + def serialize(self) -> dict[str, Any]: + return {f"{level}_impact": records for level, records in self.published.items()} + + def uk_withheld_area_codes(self) -> dict[str, list[str]]: + return self.withheld + + +def _aggregate_wales(monkeypatch) -> dict[str, Any]: + monkeypatch.setattr( + simulation_output_builder, "SimulationOutputBuilder", _AreaBuilder + ) + report = _uk_report(region="country/wales", dataset=LOCAL_AREA_DATASET) + frames = { + entity: pd.DataFrame({f"{entity}_id": [1]}) + for entity in ("person", "benunit", "household") + } + artifacts = FakeArtifacts() + return build_aggregate_report( + report=report, + baseline=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.baseline) + ), + reform=RetainedSimulationOutput( + frames, artifacts.add_simulation(report.reform) + ), + )["result"] + + +@pytest.mark.parametrize("level", ["constituency", "local_authority"]) +def test_a_local_area_report_that_estimated_no_area_of_a_level_fails( + monkeypatch, level: str +) -> None: + # A succeeded run then proves each level's area or withheld-code rows + # exist, which a staging check reading only the run's status relies on. + _route_uk_area_regions(monkeypatch) + monkeypatch.setattr( + _AreaBuilder, + "published", + {"constituency": [], "local_authority": []}, + raising=False, + ) + monkeypatch.setattr( + _AreaBuilder, + "withheld", + {"constituency": ["W07000081"], "local_authority": ["W06000015"], level: []}, + raising=False, + ) + + with pytest.raises(ValueError, match=f"estimated no {level} area"): + _aggregate_wales(monkeypatch) + + +def test_a_local_area_report_with_every_area_of_a_level_published_succeeds( + monkeypatch, +) -> None: + # Disclosure control may withhold nothing at a level, likely for a + # nation's few local authorities. + _route_uk_area_regions(monkeypatch) + published = {"local_authority_code": "W06000015", "local_authority_name": "Cardiff"} + monkeypatch.setattr( + _AreaBuilder, + "published", + {"constituency": [], "local_authority": [published]}, + raising=False, + ) + monkeypatch.setattr( + _AreaBuilder, + "withheld", + {"constituency": ["W07000081"], "local_authority": []}, + raising=False, + ) + + result = _aggregate_wales(monkeypatch) + + assert result["local_authority_impact"] == [published] + assert result["local_authority_withheld_codes"] == [] diff --git a/projects/policyengine-simulation-executor/tests/test_stage12_local_area_dataset_load.py b/projects/policyengine-simulation-executor/tests/test_stage12_local_area_dataset_load.py new file mode 100644 index 000000000..a60c6349f --- /dev/null +++ b/projects/policyengine-simulation-executor/tests/test_stage12_local_area_dataset_load.py @@ -0,0 +1,516 @@ +"""Tests for a UK-wide or nation simulation calculated on the local-area dataset.""" + +from __future__ import annotations + +from contextlib import nullcontext +from dataclasses import dataclass, field +from types import SimpleNamespace +from typing import Any + +import pandas as pd +import pytest +from policyengine_simulation_contract.stage12_bundle import ( + ResolvedStage12Bundle, + Stage12BundleManifest, + Stage12CountryBundle, + Stage12Dataset, + Stage12RegionDataset, +) +from policyengine_simulation_contract.stage12_execution import ( + BundleProvenance, + DatasetArtifactMediaType, + DatasetArtifactReference, + DatasetPopulationInput, + DatasetProvenance, + EntityOutputPlan, + GeographySelection, + PlannedSimulationExecutionInput, + ReportAggregate, + ReportExecutionInput, + ReportOutputRequirements, + SimulationExecutionInput, + SimulationRole, + Stage12OutputPlan, +) +from policyengine_simulation_contract.uk_geography import ( + UKLocalAuthorityBoundaryVersion, +) +from policyengine_simulation_executor.simulation_runtime import ( + DatasetSelection, + RegionResolution, +) +from policyengine_simulation_executor.stage12_runtime import ( + local_area_datasets, +) +from policyengine_simulation_executor.stage12_runtime import simulation as worker +from stage12_runtime_support import ( + BASELINE_ID, + EVALUATION_ID, + REFORM_ID, + report_execution_input, +) + +NATIONAL_DATASET = "microcosm_uk_2024_25" +LOCAL_AREA_DATASET = "microcosm_uk_2024_25_dense" +POLICYENGINE_VERSION = "6.17.1" +COUNTRY_PACKAGE_VERSION = "2.174.1" +BUNDLE_MANIFEST_SHA256 = "b" * 64 +# Wales as .py's region registry resolves it on the national dataset: the +# region's own dataset, and the filter that scopes a simulation to it. +WALES = RegionResolution( + code="country/wales", + dataset_reference=NATIONAL_DATASET, + scoping_strategy="households in Wales", +) + + +def _dataset(identity: str) -> Stage12Dataset: + return Stage12Dataset( + identity=identity, + uri=f"hf://policyengine/microcosm-uk/{identity}.h5@{identity}-revision", + artifact_revision=f"{identity}-revision", + sha256="a" * 64, + repo_type="model", + ) + + +def _route_uk_area_regions(monkeypatch, *, routed: bool = True) -> None: + """Make the installed release route UK area regions to the local-area dataset.""" + monkeypatch.setattr(local_area_datasets, "uk_area_regions_routed", lambda: routed) + monkeypatch.setattr( + local_area_datasets, "routed_uk_area_dataset_name", lambda: LOCAL_AREA_DATASET + ) + + +def _installed_uk_bundle() -> ResolvedStage12Bundle: + """Return an installed bundle holding the national and local-area datasets.""" + national = _dataset(NATIONAL_DATASET) + region_datasets = { + # As .py 6.17.1's bundle: nations have no region dataset of their own. + "national": NATIONAL_DATASET, + "constituency": LOCAL_AREA_DATASET, + "local_authority": LOCAL_AREA_DATASET, + } + return ResolvedStage12Bundle( + bundle=Stage12BundleManifest( + policyengine_version=POLICYENGINE_VERSION, + policyengine_requirement=f"policyengine=={POLICYENGINE_VERSION}", + core_package_version="3.33.3", + core_package_requirement="policyengine-core==3.33.3", + countries=( + Stage12CountryBundle( + country="uk", + country_package_name="policyengine-uk", + country_package_version=COUNTRY_PACKAGE_VERSION, + country_package_requirement=( + f"policyengine-uk=={COUNTRY_PACKAGE_VERSION}" + ), + data_package_name="microcosm-uk", + data_package_version="0.1.0", + data_release_version="release", + data_artifact_revision=national.artifact_revision, + default_dataset=NATIONAL_DATASET, + default_dataset_uri=national.uri, + datasets=(national, _dataset(LOCAL_AREA_DATASET)), + region_datasets=tuple( + Stage12RegionDataset(region_type=region_type, dataset=dataset) + for region_type, dataset in region_datasets.items() + ), + ), + ), + ), + bundle_manifest_sha256=BUNDLE_MANIFEST_SHA256, + ) + + +def _provenance(identity: str) -> BundleProvenance: + dataset = _dataset(identity) + return BundleProvenance( + policyengine_version=POLICYENGINE_VERSION, + country_package_name="policyengine-uk", + country_package_version=COUNTRY_PACKAGE_VERSION, + dataset=DatasetProvenance( + identity=identity, + uri=dataset.uri, + artifact_revision=dataset.artifact_revision, + data_package_name="microcosm-uk", + data_package_version="0.1.0", + ), + bundle_manifest_sha256=BUNDLE_MANIFEST_SHA256, + ) + + +def _uk_simulation( + region: str, identity: str, *, provenance: BundleProvenance | None = None +) -> PlannedSimulationExecutionInput: + bundle = provenance or _provenance(identity) + return PlannedSimulationExecutionInput( + evaluation_id=EVALUATION_ID, + simulation_execution_id=BASELINE_ID, + role=SimulationRole.BASELINE, + policy={}, + population=DatasetPopulationInput( + artifact=DatasetArtifactReference( + uri=bundle.dataset.uri, + media_type=DatasetArtifactMediaType.HDF5, + content_sha256="a" * 64, + ) + ), + year=2026, + geography=GeographySelection(country="uk", region=region), + bundle=bundle, + output_plan=Stage12OutputPlan( + country="uk", + requirements=ReportOutputRequirements( + aggregates=tuple(ReportAggregate), + include_cliff_impacts=False, + labor_supply_response_active=False, + ), + entities=( + EntityOutputPlan( + entity="household", + materialized_variables=("household_id", "household_net_income"), + ), + ), + ), + ) + + +@pytest.mark.parametrize("region", ["uk", "country/wales"]) +def test_the_installed_bundle_accepts_the_local_area_dataset_for_uk_and_a_nation( + monkeypatch, region: str +) -> None: + _route_uk_area_regions(monkeypatch) + monkeypatch.setattr(worker, "load_stage12_bundle", _installed_uk_bundle) + + worker._require_installed_bundle(_uk_simulation(region, LOCAL_AREA_DATASET)) + + +def test_the_installed_bundle_refuses_the_dataset_when_the_release_does_not_route( + monkeypatch, +) -> None: + # On such a release a UK-wide run calculates the area results itself, as + # Stage 12's national aggregation decides (uk_area_breakdowns_omitted). + _route_uk_area_regions(monkeypatch, routed=False) + monkeypatch.setattr(worker, "load_stage12_bundle", _installed_uk_bundle) + + with pytest.raises(RuntimeError, match="differs from the installed bundle"): + worker._require_installed_bundle(_uk_simulation("uk", LOCAL_AREA_DATASET)) + + +def test_the_installed_bundle_refuses_the_local_area_dataset_at_another_revision( + monkeypatch, +) -> None: + _route_uk_area_regions(monkeypatch) + monkeypatch.setattr(worker, "load_stage12_bundle", _installed_uk_bundle) + provenance = _provenance(LOCAL_AREA_DATASET) + provenance = provenance.model_copy( + update={ + "dataset": provenance.dataset.model_copy( + update={"artifact_revision": "an-older-revision"} + ) + } + ) + + with pytest.raises(RuntimeError, match="differs from the installed bundle"): + worker._require_installed_bundle( + _uk_simulation("uk", LOCAL_AREA_DATASET, provenance=provenance) + ) + + +class _StoppedBeforeSimulatingError(Exception): + """Raised by a stubbed simulation build once the dataset has loaded.""" + + +@dataclass +class _Calculation: + """What a calculation stopped before simulating did.""" + + selected_by: list[RegionResolution] = field(default_factory=list) + built_with: list[Any] = field(default_factory=list) + detected_on: list[str] = field(default_factory=list) + + +def _area_household() -> pd.DataFrame: + return pd.DataFrame( + { + "household_id": [1, 2], + "constituency_code_oa": ["W07000081", "W07000082"], + # Cumberland exists only in the LAD23 boundaries. + "la_code_oa": ["E06000063", "W06000015"], + "region_code_oa": ["W92000004", "W92000004"], + "source_household_id": [10, 11], + } + ) + + +def _run_until_built( + monkeypatch, + simulation: PlannedSimulationExecutionInput, + *, + household: pd.DataFrame, +) -> _Calculation: + """Run a UK calculation of Wales up to its simulation build.""" + from policyengine_simulation_executor import simulation_runtime + from policyengine_simulation_executor.stage12_cache import state_datasets + + calculation = _Calculation() + dataset = SimpleNamespace( + data=SimpleNamespace(entity_data={"household": household}) + ) + uris = { + item.identity: item.uri + for item in _installed_uk_bundle().bundle.countries[0].datasets + } + + def select(_params, *, region_resolution: RegionResolution) -> DatasetSelection: + calculation.selected_by.append(region_resolution) + name = region_resolution.dataset_reference or NATIONAL_DATASET + return DatasetSelection( + name=name, uri=uris[name], is_default=name == NATIONAL_DATASET + ) + + def detect_on_the_national_dataset(_country, _dataset, **kwargs): + calculation.detected_on.append(kwargs["dataset_name"]) + + def stop_before_simulating(_params, *, resolved, **_kwargs): + calculation.built_with.append(resolved) + raise _StoppedBeforeSimulatingError + + monkeypatch.setattr(worker, "_require_installed_bundle", lambda _: None) + _route_uk_area_regions(monkeypatch) + monkeypatch.setattr( + local_area_datasets, "load_stage12_bundle", _installed_uk_bundle + ) + monkeypatch.setattr(simulation_runtime, "setup_gcp_credentials", nullcontext) + monkeypatch.setattr( + simulation_runtime, "_country_module", lambda _country: object() + ) + monkeypatch.setattr(simulation_runtime, "resolve_region", lambda **_: WALES) + monkeypatch.setattr(simulation_runtime, "_resolve_dataset_selection", select) + monkeypatch.setattr( + simulation_runtime, "_load_dataset", lambda *_args, **_kwargs: dataset + ) + monkeypatch.setattr( + state_datasets, "baked_state_data_folder", lambda **_kwargs: None + ) + monkeypatch.setattr( + worker, "detect_uk_run_dataset_metadata", detect_on_the_national_dataset + ) + monkeypatch.setattr(worker, "build_stage12_simulation", stop_before_simulating) + + with pytest.raises(_StoppedBeforeSimulatingError): + worker.calculate_simulation_frames(simulation) + return calculation + + +def test_a_local_area_run_loads_the_local_area_dataset_scoped_to_its_nation( + monkeypatch, +) -> None: + calculation = _run_until_built( + monkeypatch, + _uk_simulation("country/wales", LOCAL_AREA_DATASET), + household=_area_household(), + ) + + (selected_by,) = calculation.selected_by + assert selected_by.code == "country/wales" + assert selected_by.dataset_reference == LOCAL_AREA_DATASET + (built_with,) = calculation.built_with + assert built_with.dataset_selection.name == LOCAL_AREA_DATASET + # The simulation keeps Wales's own scoping on the local-area dataset. + assert built_with.region == WALES + # The area columns were checked on the local-area dataset as it loaded. + assert calculation.detected_on == [] + + +def test_a_national_run_of_a_nation_loads_the_dataset_its_region_names( + monkeypatch, +) -> None: + calculation = _run_until_built( + monkeypatch, + _uk_simulation("country/wales", NATIONAL_DATASET), + household=_area_household(), + ) + + assert calculation.selected_by == [WALES] + assert calculation.detected_on == [NATIONAL_DATASET] + + +def test_a_local_area_run_detects_the_authority_boundaries_of_its_dataset( + monkeypatch, +) -> None: + detected = [] + original = worker.detect_uk_area_dataset_metadata + + def record_detection(dataset, *, dataset_name): + metadata = original(dataset, dataset_name=dataset_name) + detected.append((dataset_name, metadata.boundary_version)) + return metadata + + monkeypatch.setattr(worker, "detect_uk_area_dataset_metadata", record_detection) + + _run_until_built( + monkeypatch, + _uk_simulation("uk", LOCAL_AREA_DATASET), + household=_area_household(), + ) + + assert detected == [(LOCAL_AREA_DATASET, UKLocalAuthorityBoundaryVersion.LAD23)] + + +@pytest.mark.parametrize( + "missing", + ["constituency_code_oa", "la_code_oa", "region_code_oa", "source_household_id"], +) +def test_a_local_area_dataset_without_an_area_column_fails_before_simulating( + monkeypatch, missing: str +) -> None: + with pytest.raises(ValueError, match=f"has no {missing} column"): + _run_until_built( + monkeypatch, + _uk_simulation("uk", LOCAL_AREA_DATASET), + household=_area_household().drop(columns=[missing]), + ) + + +def _uk_report(region: str, identity: str) -> ReportExecutionInput: + """Return a UK report whose two simulations are on the dataset ``identity``.""" + baseline = _uk_simulation(region, identity).model_dump( + mode="json", exclude={"output_plan"} + ) + reform = { + **baseline, + "simulation_execution_id": str(REFORM_ID), + "role": SimulationRole.REFORM.value, + "policy": {"gov.hmrc.income_tax.rates.uk[0].rate": {"2026-01-01": 0.21}}, + } + return ReportExecutionInput( + evaluation_id=EVALUATION_ID, + baseline=SimulationExecutionInput.model_validate(baseline), + reform=SimulationExecutionInput.model_validate(reform), + requested_aggregates=tuple(ReportAggregate), + ) + + +@pytest.mark.parametrize("region", ["uk", "country/wales"]) +def test_a_uk_wide_or_nation_report_on_the_local_area_dataset_is_recognised( + monkeypatch, region: str +) -> None: + _route_uk_area_regions(monkeypatch) + + assert local_area_datasets.is_local_area_report( + _uk_report(region, LOCAL_AREA_DATASET) + ) + + +def test_a_report_on_the_national_dataset_is_no_local_area_report( + monkeypatch, +) -> None: + _route_uk_area_regions(monkeypatch) + + assert not local_area_datasets.is_local_area_report( + _uk_report("uk", NATIONAL_DATASET) + ) + + +def test_a_constituency_report_on_the_local_area_dataset_is_an_ordinary_report( + monkeypatch, +) -> None: + # Its national aggregation already calculates its one area. + _route_uk_area_regions(monkeypatch) + + assert not local_area_datasets.is_local_area_report( + _uk_report("constituency/W07000081", LOCAL_AREA_DATASET) + ) + + +def test_no_report_is_a_local_area_report_where_the_release_does_not_route( + monkeypatch, +) -> None: + _route_uk_area_regions(monkeypatch, routed=False) + + assert not local_area_datasets.is_local_area_report( + _uk_report("uk", LOCAL_AREA_DATASET) + ) + + +def test_a_us_report_is_no_local_area_report(monkeypatch) -> None: + _route_uk_area_regions(monkeypatch) + + assert not local_area_datasets.is_local_area_report(report_execution_input()) + + +@pytest.mark.parametrize( + ("region", "identity", "on_local_area_dataset"), + [ + ("uk", LOCAL_AREA_DATASET, True), + ("country/wales", LOCAL_AREA_DATASET, True), + ("uk", NATIONAL_DATASET, False), + ("constituency/W07000081", LOCAL_AREA_DATASET, False), + ], +) +def test_a_simulation_is_on_the_local_area_dataset_by_its_region_and_dataset( + monkeypatch, region: str, identity: str, *, on_local_area_dataset: bool +) -> None: + # The rule a simulation's stored output plan uses for its area columns. + _route_uk_area_regions(monkeypatch) + + assert ( + local_area_datasets.calculates_on_local_area_dataset( + _uk_simulation(region, identity) + ) + is on_local_area_dataset + ) + + +def test_the_installed_release_accepts_and_selects_its_own_local_area_dataset() -> None: + # Over the installed .py bundle and its routing, unpatched: a release + # upgrade that renames or moves the dataset fails here, not in staging. + from policyengine_simulation_executor.simulation_runtime import ( + _resolve_dataset_selection, + ) + from policyengine_simulation_executor.stage12_bundle import load_stage12_bundle + from policyengine_simulation_executor.stage12_runtime.worker_inputs import ( + simulation_runtime_params, + ) + from policyengine_simulation_executor.uk_local_authority_metadata import ( + routed_uk_area_dataset_name, + uk_area_regions_routed, + ) + + assert uk_area_regions_routed() + resolved = load_stage12_bundle() + uk = next(item for item in resolved.bundle.countries if item.country == "uk") + dataset = next( + item for item in uk.datasets if item.identity == routed_uk_area_dataset_name() + ) + simulation = _uk_simulation( + "uk", + dataset.identity, + provenance=BundleProvenance( + policyengine_version=resolved.bundle.policyengine_version, + country_package_name=uk.country_package_name, + country_package_version=uk.country_package_version, + dataset=DatasetProvenance( + identity=dataset.identity, + uri=dataset.uri, + artifact_revision=dataset.artifact_revision, + data_package_name=uk.data_package_name, + data_package_version=uk.data_package_version, + ), + bundle_manifest_sha256=resolved.bundle_manifest_sha256, + ), + ) + + worker._require_installed_bundle(simulation) + assert local_area_datasets.local_area_dataset_name(simulation) == dataset.identity + selection = _resolve_dataset_selection( + simulation_runtime_params(simulation), + region_resolution=local_area_datasets.dataset_region( + RegionResolution(code="uk"), dataset.identity + ), + ) + # The worker refuses a selection that differs from the provenance. + worker._require_loaded_dataset_matches_provenance(simulation, selection) + assert selection.name == dataset.identity diff --git a/projects/policyengine-simulation-executor/tests/test_stage12_output_planning.py b/projects/policyengine-simulation-executor/tests/test_stage12_output_planning.py index 2717f67a1..a7baa3259 100644 --- a/projects/policyengine-simulation-executor/tests/test_stage12_output_planning.py +++ b/projects/policyengine-simulation-executor/tests/test_stage12_output_planning.py @@ -397,3 +397,92 @@ def test_a_simulation_planned_alone_gets_its_self_comparisons_plan( assert plan == resolve_report_output_plan(_self_comparison(standalone)) assert plan.requirements.include_cliff_impacts is bool(options) + + +# The dataset a routing release assigns to UK constituency and local-authority +# regions, which a UK-wide or nation simulation may also be calculated on. +UK_LOCAL_AREA_DATASET = "microcosm_uk_2024_25_dense" + + +def _route_uk_area_regions_to_the_local_area_dataset(monkeypatch) -> None: + """Make the installed release route UK area regions to ``UK_LOCAL_AREA_DATASET``.""" + from policyengine_simulation_executor.stage12_runtime import local_area_datasets + + _route_uk_area_regions(monkeypatch, routed=True) + monkeypatch.setattr(local_area_datasets, "uk_area_regions_routed", lambda: True) + monkeypatch.setattr( + local_area_datasets, + "routed_uk_area_dataset_name", + lambda: UK_LOCAL_AREA_DATASET, + ) + + +def _on_dataset( + simulation: SimulationExecutionInput, dataset_name: str +) -> SimulationExecutionInput: + """Return ``simulation`` with its provenance naming the dataset ``dataset_name``.""" + dataset = simulation.bundle.dataset.model_copy( + update={ + "identity": dataset_name, + "uri": f"hf://policyengine/data/{dataset_name}.h5@revision", + } + ) + return simulation.model_copy( + update={"bundle": simulation.bundle.model_copy(update={"dataset": dataset})} + ) + + +@pytest.mark.parametrize( + ("region", "dataset_name", "copies_area_columns"), + [ + ("uk", UK_LOCAL_AREA_DATASET, True), + ("country/wales", UK_LOCAL_AREA_DATASET, True), + ("uk", "populace_uk_2023", False), + ("country/wales", "populace_uk_2023", False), + ], + ids=["uk-local-area", "wales-local-area", "uk-national", "wales-national"], +) +def test_a_uk_simulation_planned_alone_copies_area_columns_on_the_local_area_dataset( + monkeypatch, region: str, dataset_name: str, *, copies_area_columns: bool +) -> None: + from policyengine_simulation_executor import uk_local_authority_metadata + + _route_uk_area_regions_to_the_local_area_dataset(monkeypatch) + standalone = _on_dataset( + _simulation("uk", SimulationRole.STANDALONE, region=region), dataset_name + ) + + plan = resolve_simulation_output_plan(standalone) + + # A local-area report reading this run's file needs the area columns. + household = next(item for item in plan.entities if item.entity == "household") + assert household.dataset_variables == ( + uk_local_authority_metadata.UK_AREA_DATASET_COLUMNS + if copies_area_columns + else () + ) + + +def test_a_uk_wide_report_on_the_local_area_dataset_plans_its_full_results( + monkeypatch, +) -> None: + # It returns its full results, cliffs included when asked, and its areas. + from policyengine_simulation_executor import uk_local_authority_metadata + + _route_uk_area_regions_to_the_local_area_dataset(monkeypatch) + national = _report("uk", region="uk", include_cliffs=True) + report = national.model_copy( + update={ + "baseline": _on_dataset(national.baseline, UK_LOCAL_AREA_DATASET), + "reform": _on_dataset(national.reform, UK_LOCAL_AREA_DATASET), + } + ) + + plan = resolve_report_output_plan(report) + + assert plan.requirements.aggregates == tuple(ReportAggregate) + assert plan.requirements.include_cliff_impacts is True + assert {"cliff_gap", "is_on_cliff"}.issubset(_additional_variables(plan, "person")) + assert _dataset_variables(plan, "household") == set( + uk_local_authority_metadata.UK_AREA_DATASET_COLUMNS + ) diff --git a/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest.py b/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest.py index 739ded97a..f30ba6dbf 100644 --- a/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest.py +++ b/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest.py @@ -6,7 +6,11 @@ from datetime import UTC, datetime import pytest -from policyengine_simulation_contract.stage12_manifest import V2WorkerValidation +from policyengine_simulation_contract.stage12_manifest import ( + LOCAL_AREA_REPORTS_VERSION, + LocalAreaReports, + V2WorkerValidation, +) from src.modal.utils import update_v2_version_manifest as publisher @@ -56,6 +60,40 @@ def test_publisher_records_every_bundle_country_as_economy_capable() -> None: ] +def _local_area_reports(worker) -> dict[str, LocalAreaReports | None]: + return {country.country: country.local_area_reports for country in worker.countries} + + +def test_publisher_records_the_uk_local_area_dataset_its_coordinator_runs_on() -> None: + from policyengine_simulation_executor.uk_local_authority_metadata import ( + routed_uk_area_dataset_name, + uk_area_regions_routed, + ) + + assert uk_area_regions_routed() + worker = publisher.build_worker(_validation().model_dump(mode="json")) + + assert _local_area_reports(worker) == { + "us": None, + "uk": LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, + dataset=routed_uk_area_dataset_name(), + ), + } + + +def test_publisher_records_no_dataset_when_uk_national_runs_keep_the_areas( + monkeypatch, +) -> None: + monkeypatch.setattr(publisher, "uk_area_regions_routed", lambda: False) + + worker = publisher.build_worker(_validation().model_dump(mode="json")) + + assert _local_area_reports(worker)["uk"] == LocalAreaReports( + input_version=LOCAL_AREA_REPORTS_VERSION, dataset=None + ) + + def test_publisher_records_each_validated_household_function() -> None: payload = _validation().model_dump(mode="json") payload["household_validation_invocation_ids"] = {"uk": "household-uk-1"} diff --git a/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest_default.py b/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest_default.py index 3b81a8d80..f550f8164 100644 --- a/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest_default.py +++ b/projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest_default.py @@ -15,6 +15,7 @@ import pytest from policyengine_simulation_contract.stage12_manifest import ( ACTIVE_MANIFEST_KEY, + LOCAL_AREA_REPORTS_MANIFEST_KEY, PRE_CACHE_MANIFEST_KEY, REGION_DATASETS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY, @@ -243,6 +244,19 @@ def test_verify_refuses_a_manifest_changed_since_the_move(monkeypatch, tmp_path, _run(monkeypatch, "--verify-plan", str(tmp_path / "plan.json")) +def test_verify_refuses_a_manifest_whose_schema_2_key_was_deleted( + monkeypatch, tmp_path, store +): + # Current readers then stop using the local-area report key and read an + # older copy, without household functions, so the move isn't live. + _with_both_workers(store) + _move(monkeypatch, tmp_path, _installed_version()) + del store.values[SCHEMA_2_MANIFEST_KEY] + + with pytest.raises(ValueError, match="is not what this default move wrote"): + _run(monkeypatch, "--verify-plan", str(tmp_path / "plan.json")) + + def test_restore_puts_the_default_back_only_while_the_move_is_live( monkeypatch, tmp_path, store, capsys ): @@ -431,10 +445,18 @@ def test_verify_accepts_an_expired_older_key( assert f"v2 default version: {_installed_version()}" in capsys.readouterr().out -def test_move_refuses_a_manifest_without_its_newest_key(monkeypatch, tmp_path, store): - # Current readers read the schema version 2 key first, so it is required. +@pytest.mark.parametrize( + "required_key", + [LOCAL_AREA_REPORTS_MANIFEST_KEY, SCHEMA_2_MANIFEST_KEY], + ids=["active-schema-2-with-local-area-reports", "active-schema-2"], +) +def test_move_refuses_a_manifest_without_its_newest_key( + monkeypatch, tmp_path, store, required_key +): + # Current readers read the local-area report key first, and use it only + # while the schema version 2 key holds its copy, so both are required. _with_both_workers(store) - del store.values[SCHEMA_2_MANIFEST_KEY] + del store.values[required_key] writes = store.writes with pytest.raises(ValueError, match="copies of the v2 manifest differ"): diff --git a/projects/policyengine-simulation-gateway/tests/golden/openapi.json b/projects/policyengine-simulation-gateway/tests/golden/openapi.json index e1397025e..653d81f5a 100644 --- a/projects/policyengine-simulation-gateway/tests/golden/openapi.json +++ b/projects/policyengine-simulation-gateway/tests/golden/openapi.json @@ -2657,6 +2657,34 @@ } ] }, + "constituency_withheld_codes": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Constituency Withheld Codes" + }, + "local_authority_withheld_codes": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Local Authority Withheld Codes" + }, "cliff_impact": { "anyOf": [ { @@ -2689,7 +2717,7 @@ "congressional_district_impact" ], "title": "SingleYearMacroOutput", - "description": "Completed response returned by a single-year macro simulation." + "description": "Completed response returned by a single-year macro simulation.\n\n``constituency_withheld_codes`` and ``local_authority_withheld_codes``\nhold, sorted, the codes of each level's areas the run estimated but\ndisclosure control did not publish, as ``LocalAreaBreakdownOutput``'s do.\nThey are listed for a UK-wide or nation report on the local-area dataset;\nnull means the result does not report them, and the JSON then leaves both\nkeys out, so every other result is written as before." }, "TelemetryEnvelope": { "properties": { diff --git a/ruff.toml b/ruff.toml index 325e559de..16acea826 100644 --- a/ruff.toml +++ b/ruff.toml @@ -117,7 +117,7 @@ runtime-evaluated-base-classes = [ "policyengine_simulation_contract.stage12_bundle.StrictBundleModel", "policyengine_simulation_contract.stage12_execution.SimulationExecutionInput", "policyengine_simulation_contract.stage12_execution.StrictContractModel", - "policyengine_simulation_contract.stage12_manifest.StrictManifestModel", + "policyengine_simulation_contract.stage12_manifest_models.StrictManifestModel", "policyengine_simulation_executor.precompute_models._StrictModel", "policyengine_simulation_executor.stage_reporting_simulation.StageReportingSimulation", "pydantic.BaseModel",