Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/microcosm-dataset-selectors.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add Microcosm selectors for the certified US national and local-area datasets and accept `--data-producer microcosm` in the release certification CLI. Existing selectors and certification metadata remain supported.
6 changes: 3 additions & 3 deletions docs/bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ package scaffold:
```python
from policyengine.provenance import materialize_dataset

result = materialize_dataset("us", "populace_us_2024")
result = materialize_dataset("us", "microcosm_us_2024")
print(result.path)
print(result.bundle_dataset.sha256)
```
Expand Down Expand Up @@ -202,7 +202,7 @@ To certify a new data release from a data-producer manifest, run:
```bash
python scripts/bundle.py certify-data \
--country uk \
--data-producer populace \
--data-producer microcosm \
--manifest-uri hf://dataset/policyengine/populace-uk-private@<release>/releases/<release>/release_manifest.json
```

Expand All @@ -211,7 +211,7 @@ For US Microcosm releases, certify the Microcosm release manifest directly:
```bash
python scripts/bundle.py certify-data \
--country us \
--data-producer populace \
--data-producer microcosm \
--manifest-uri hf://dataset/policyengine/populace-us@<release>/releases/<release>/release_manifest.json \
--regional-manifest-uri hf://dataset/policyengine/populace-us@<local-area-release>/releases/<local-area-release>/release_manifest.json \
--model-version <policyengine-us-version>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Set values from the two published release manifests rather than copying an
older certification:

```
NATIONAL_RELEASE_ID = populace-us-2024-<national-build>-<sha>-<timestamp>Z
LOCAL_AREA_RELEASE_ID = populace-us-2024-<local-build>-<sha>-<timestamp>Z
NATIONAL_RELEASE_ID = <published national release ID>
LOCAL_AREA_RELEASE_ID = <published local-area release ID>
MODEL_VERSION = <policyengine-us version declared by the national release>
```

Expand All @@ -47,7 +47,7 @@ artifact.
```bash
python scripts/bundle.py certify-data \
--country us \
--data-producer populace \
--data-producer microcosm \
--model-version "$MODEL_VERSION" \
--manifest-uri "hf://dataset/policyengine/populace-us@$NATIONAL_RELEASE_ID/releases/$NATIONAL_RELEASE_ID/release_manifest.json" \
--regional-manifest-uri "hf://dataset/policyengine/populace-us@$LOCAL_AREA_RELEASE_ID/releases/$LOCAL_AREA_RELEASE_ID/release_manifest.json"
Expand Down
2 changes: 1 addition & 1 deletion docs/engineering/runbooks/canonical-spm-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ Ordinary `check` passes for the staged draft; this is metadata consistency only.
coordinator; do not invent future versions to break that dependency.
Refresh country and wrapper locks against real registry bytes.
5. Certify the published data release through `scripts/bundle.py certify-data
--country us --data-producer populace --model-version
--country us --data-producer microcosm --model-version
"$PUBLISHED_COUNTRY_VERSION" --manifest-uri "$PUBLISHED_RELEASE_MANIFEST_URI"`.
The URI must identify the actual immutable producer release. This command
updates the country pin and data metadata from real compatibility evidence.
Expand Down
14 changes: 7 additions & 7 deletions docs/engineering/skills/data-certification.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Open the work on a fresh branch from current `main` (use a clean worktree if
the checkout is dirty).

```bash
python scripts/bundle.py certify-data --country uk --data-producer populace \
python scripts/bundle.py certify-data --country uk --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<cut-tag>/releases/<release-id>/release_manifest.json"
```

Expand All @@ -46,7 +46,7 @@ its `release` and `build` as strings, for example
its own folder, at the commit that published it:

```bash
python scripts/bundle.py certify-data --country uk --data-producer populace \
python scripts/bundle.py certify-data --country uk --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<commit>/releases/<release>/<build>/release_manifest.json"
```

Expand Down Expand Up @@ -104,7 +104,7 @@ Microcosm national release with one of:
`dataset_role: non_default_local_area`, as for the US below.

```bash
python scripts/bundle.py certify-data --country uk --data-producer populace \
python scripts/bundle.py certify-data --country uk --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
--regional-manifest-uri "hf://dataset/policyengine/populace-uk-private@<local-cut-tag>/releases/<local-release-id>/release_manifest.json" \
--model-version "<policyengine-uk-version>"
Expand All @@ -118,7 +118,7 @@ Microcosm national release with one of:
release whose households carry area codes:

```bash
python scripts/bundle.py certify-data --country uk --data-producer populace \
python scripts/bundle.py certify-data --country uk --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
--regional-manifest-uri "hf://model/policyengine/policyengine-uk-data-private@<revision>/releases/<version>/release_manifest.json" \
--regional-release-kind area_coded_national \
Expand Down Expand Up @@ -152,7 +152,7 @@ Microcosm national release with one of:
--limitation "the dense cut's VAT predates the policyengine-uk VAT factor fix (#2187, 2.135.7); the dashboard holds a VAT limit" \
--output regional-input-surface.json

.venv/bin/python scripts/bundle.py certify-data --country uk --data-producer populace \
.venv/bin/python scripts/bundle.py certify-data --country uk --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
--regional-manifest-uri "<the routed local-area release_manifest.json URI>" \
--accept-regional-model-mismatch "<who accepted it and why>" \
Expand Down Expand Up @@ -320,7 +320,7 @@ folder (it has `releases.json`), so give each manifest's folder at the commit
that published it:

```bash
python scripts/bundle.py certify-data --country us --data-producer populace \
python scripts/bundle.py certify-data --country us --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-national/<build>/release_manifest.json" \
--regional-manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-local/<build>/release_manifest.json" \
--model-version "<policyengine-us-version>"
Expand Down Expand Up @@ -362,7 +362,7 @@ exact target installed, then certify with the reason and the evidence.
--limitation "<what the build model and the target differ in>" \
--output regional-input-surface.json

.venv/bin/python scripts/bundle.py certify-data --country us --data-producer populace \
.venv/bin/python scripts/bundle.py certify-data --country us --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-national/<build>/release_manifest.json" \
--regional-manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-local/<build>/release_manifest.json" \
--accept-regional-model-mismatch "<who accepted it and why>" \
Expand Down
15 changes: 9 additions & 6 deletions docs/microsim.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,10 +128,11 @@ Microdata is stored as HDF5 on Hugging Face. `ensure_datasets` downloads, caches

```python
datasets = pe.us.ensure_datasets(
datasets=["microcosm_us_2024"],
years=[2024, 2026],
data_folder="./data", # local cache directory
)
dataset = datasets["populace_us_2024_2026"]
dataset = datasets["microcosm_us_2024_2026"]
```

The default US dataset is **Microcosm US 2024** — a Microcosm-built dataset
Expand All @@ -144,7 +145,9 @@ remain available as named, non-default bundle datasets.

PolicyEngine.py obtains the repository type, immutable revision, and SHA-256
from the installed release bundle. An existing file in the configured data
directory is reused only after hash verification.
directory is reused only after hash verification. The original US selectors
`populace_us_2024` and `populace_us_2024_acs_local` remain supported and select
the same certified artifacts.

List datasets already known to the country:

Expand All @@ -155,7 +158,7 @@ pe.us.load_datasets() # or pe.uk.load_datasets()
### US local-area dataset

Alongside the certified national default, the bundle registers a US dataset for
finer geographic work: `populace_us_2024_acs_local`. It is a
finer geographic work: `microcosm_us_2024_acs_local`. It is a
Microcosm US 2024 build of roughly **1.6 million households** on an **ACS 2024
multispine**, with each household **PUMA-assigned** to a 119th-Congress
congressional district, county, and state, and calibrated to **state
Expand All @@ -170,14 +173,14 @@ Two-line load:
```python
import policyengine as pe

sim = pe.us.managed_microsimulation(dataset="populace_us_2024_acs_local")
sim = pe.us.managed_microsimulation(dataset="microcosm_us_2024_acs_local")
```

Or materialize it as a `PolicyEngineUSDataset` for `Simulation`:

```python
datasets = pe.us.ensure_datasets(datasets=["populace_us_2024_acs_local"], years=[2024])
dataset = datasets["populace_us_2024_acs_local_2024"]
datasets = pe.us.ensure_datasets(datasets=["microcosm_us_2024_acs_local"], years=[2024])
dataset = datasets["microcosm_us_2024_acs_local_2024"]
```

Because this file carries PUMA-assigned district, county, and state identifiers
Expand Down
2 changes: 1 addition & 1 deletion docs/release-bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ python scripts/bundle.py certify-data --country us \
US Microcosm certification uses the Microcosm release manifest directly:

```bash
python scripts/bundle.py certify-data --country us --data-producer populace \
python scripts/bundle.py certify-data --country us --data-producer microcosm \
--manifest-uri "hf://dataset/policyengine/populace-us@<tag>/releases/<tag>/release_manifest.json" \
--regional-manifest-uri "hf://dataset/policyengine/populace-us@<local-area-tag>/releases/<local-area-tag>/release_manifest.json" \
--model-version "<policyengine-us-version>"
Expand Down
2 changes: 1 addition & 1 deletion scripts/bundle.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ def _parser() -> argparse.ArgumentParser:
certify.add_argument("--country", required=True, choices=["us", "uk"])
certify.add_argument(
"--data-producer",
choices=["legacy", "populace"],
choices=["legacy", "microcosm", "populace"],
help="Data-producer strategy. Defaults to the certification script default.",
)
certify.add_argument(
Expand Down
11 changes: 7 additions & 4 deletions scripts/certify_data_release.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

Usage::

python scripts/bundle.py certify-data --country uk --data-producer populace \\
python scripts/bundle.py certify-data --country uk --data-producer microcosm \\
--manifest-uri "hf://dataset/policyengine/populace-uk-private@<cut-tag>/releases/<release-id>/release_manifest.json"

Pin the cut tag, never a branch: each cut rewrites ``releases/<release-id>/``.
Expand Down Expand Up @@ -47,8 +47,11 @@ def main(argv=None) -> int:
parser.add_argument(
"--data-producer",
default=None,
choices=["legacy", "populace"],
help="Data-producer strategy. Defaults to populace for UK, legacy otherwise.",
choices=["legacy", "microcosm", "populace"],
help=(
"Data-producer strategy. Defaults to microcosm for UK, legacy otherwise. "
"The previous name populace remains accepted."
),
)
parser.add_argument(
"--manifest-uri",
Expand Down Expand Up @@ -189,7 +192,7 @@ def main(argv=None) -> int:
/ f"certify-{args.country}-{result.build_id or 'data'}.changed.md"
)
fragment.write_text(
f"Certify the {args.country.upper()} {result.data_producer} "
f"Certify the {args.country.upper()} {result.display_data_producer} "
f"data release `{result.build_id}` ({result.default_dataset}, "
f"{result.model_package} {result.model_version}) into the "
"PolicyEngine bundle manifest.\n"
Expand Down
12 changes: 8 additions & 4 deletions src/policyengine/provenance/certification.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

result = certify_data_release(
country="uk",
data_producer="populace",
data_producer="microcosm",
manifest_uri=(
"hf://dataset/policyengine/populace-uk-private"
"@microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68"
Expand Down Expand Up @@ -216,14 +216,18 @@ class CertificationResult:
model_version: str
warnings: list[str] = field(default_factory=list)

@property
def display_data_producer(self) -> str:
return "microcosm" if self.data_producer == "populace" else self.data_producer

def summary(self) -> str:
lines = [
f"certified {self.country}: {self.default_dataset} "
f"({self.dataset_count} datasets, build {self.build_id})",
f" manifest: {self.manifest_uri}",
f" manifest sha256: {self.manifest_sha256}",
f" model: {self.model_package}=={self.model_version}",
f" data-producer: {self.data_producer}",
f" data-producer: {self.display_data_producer}",
f" wrote: {self.bundle_path}",
]
for warning in self.warnings:
Expand Down Expand Up @@ -1769,8 +1773,8 @@ def certify(
def certification_strategy(
country: str, data_producer: Optional[str] = None
) -> DataProducerCertificationStrategy:
producer = data_producer or ("populace" if country == "uk" else "legacy")
if producer == "populace":
producer = data_producer or ("microcosm" if country == "uk" else "legacy")
if producer in ("microcosm", "populace"):
return MicrocosmDataProducerCertificationStrategy()
if producer == "legacy":
return LegacyDataProducerCertificationStrategy()
Expand Down
6 changes: 4 additions & 2 deletions src/policyengine/provenance/dataset_materialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,9 @@ def _resolve_bundle_dataset(
manifest: Optional[CountryReleaseManifest] = None,
) -> _BundleDatasetSpec:
country_manifest = manifest or get_release_manifest(country_id)
dataset_name = dataset or country_manifest.default_dataset
dataset_name = country_manifest.bundled_dataset_key(
dataset or country_manifest.default_dataset
)
reference = country_manifest.datasets.get(dataset_name)
if reference is None:
raise DatasetMaterializationError(
Expand Down Expand Up @@ -169,7 +171,7 @@ def materialize_dataset(
data_dir=data_dir,
manifest=manifest,
)
if dataset in manifest.datasets:
if manifest.bundled_dataset_key(dataset) in manifest.datasets:
return _use_bundle_dataset(
country_id,
dataset,
Expand Down
22 changes: 19 additions & 3 deletions src/policyengine/provenance/manifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@
PYPI_REQUEST_TIMEOUT_SECONDS = 30
# Producer paths for these artifacts are relative to the release directory.
RELEASE_SCOPED_ARTIFACT_KINDS = frozenset({"diagnostics", "evidence"})
# Public selectors and their counterpart keys in earlier and newer bundles.
# This index is for lookup only. The manifest records the actual artifact key.
US_MICROCOSM_DATASET_ALIASES = {
"microcosm_us_2024": "populace_us_2024",
"populace_us_2024": "microcosm_us_2024",
"microcosm_us_2024_acs_local": "populace_us_2024_acs_local",
"populace_us_2024_acs_local": "microcosm_us_2024_acs_local",
}

#: Microcosm publishes each build once, in its own folder
#: ``releases/<release>/<build>/``, and never changes it afterwards; a root
Expand Down Expand Up @@ -460,6 +468,14 @@ class CountryReleaseManifest(BaseModel):
source_sha256: Optional[str] = Field(default=None, exclude=True)
"""Byte sha256 of the bundled manifest before Pydantic normalization."""

def bundled_dataset_key(self, dataset: str) -> str:
"""Resolve a public selector to the key recorded in this bundle."""
if self.country_id == "us" and dataset not in self.datasets:
alias = US_MICROCOSM_DATASET_ALIASES.get(dataset)
if alias is not None and alias in self.datasets:
return alias
return dataset

@property
def default_dataset_uri(self) -> str:
if (
Expand Down Expand Up @@ -1330,7 +1346,7 @@ def resolve_dataset_reference(country_id: str, dataset: str) -> str:
return dataset

manifest = get_release_manifest(country_id)
path_reference = manifest.datasets.get(dataset)
path_reference = manifest.datasets.get(manifest.bundled_dataset_key(dataset))
if path_reference is not None:
return build_hf_uri(
repo_id=path_reference.repo_id or manifest.data_package.repo_id,
Expand Down Expand Up @@ -1366,7 +1382,7 @@ def resolve_managed_dataset_reference(
release manifest. Callers can:

- omit `dataset` to use the certified default dataset for the bundle
- pass a logical dataset name present in the bundled/data-release manifests
- pass a logical dataset name in the bundle or its supported alias

Direct URLs or raw Hugging Face references are treated as unmanaged unless
`allow_unmanaged=True` is set explicitly.
Expand All @@ -1376,7 +1392,7 @@ def resolve_managed_dataset_reference(
if dataset is None:
return manifest.default_dataset_uri

if dataset in manifest.datasets:
if manifest.bundled_dataset_key(dataset) in manifest.datasets:
return resolve_dataset_reference(country_id, dataset)

local_path = Path(dataset).expanduser()
Expand Down
Loading
Loading