Repository navigation
Description
Summary
When a Microcosm build is certified into policyengine.py, some checks repeat what Microcosm already established when it published the build. Other checks depend on the model and core versions policyengine.py pins, and those are manual, slow or missing. A certification can hold a release for hours: #720 took 12 h from a ready head to the release.
This issue asks for an evaluation, not for more checks. Which checks must policyengine.py run, given:
- what Microcosm already checks when it publishes a build;
- the policyengine-uk or policyengine-us version and the policyengine-core version that policyengine.py pins;
- the trade-off between what each check catches and how long a release waits for it?
Then automate the chosen set for both countries, and stop running the rest at the policyengine.py boundary. This builds on #722 (the UK hub's steps, their order and their timing) and #462 (a pre-registered promotion scorecard).
Facts below are from main at 5b6d7901 (11 November), Microcosm main at a575057bb, and GitHub runs up to 12 November.
What Microcosm already checks when it publishes
- UK. Three signed gate batteries (spine, calibration seam and release cut) cover the 94 gates that
uk/gates.jsondeclares.release_certification.jsonmust sayshippable: true. The local-area build adds a signed seven-gate report.microcosm-publish-release-buildvalidates the release contract and the signatures again before it publishes. A failing gate passes only through a dated sign-off register. - US. The release builders run their release gates. Route A runs the gate preflight before and after the release, and the publisher preflight.
- Both. Each release manifest records the engine the build used (
built_with_model_package,built_with_core_package) and the model versions it claims to be compatible with.
These results hold for the build's own engine. They do not need to run again when policyengine.py pins that same engine.
What policyengine.py runs today
| Check | Automated? | Time | Repeats a Microcosm check? | Depends on the pinned model or core? |
|---|---|---|---|---|
| Bundle verification (manifests, hashes, TROs) | yes, every PR | about 0.5 min | partly | no |
| Test on 4 Python versions (synthetic or mocked data) | yes, every PR | 68–96 min (#744) | no | yes |
| The same Test again after the merge | yes | 65–111 min (#744) | no, but it repeats the PR's run | yes |
| Hub: producer gates, national hash check | no | not posted separately | likely (to confirm) | no |
| Hub: budget inputs | no | not posted separately | no | yes |
| Hub: reform contract | no | 3 h 22 min (#715); over 4 h 29 min (#720) | no | yes |
| Hub: reform responsiveness, 10 reforms, 2025–2031 | no | 2 h 04 min (#715) | no | yes |
| Hub: 2×2 attribution | no | 9 min (#715) | no | yes |
UC floor (scripts/check_uk_uc_floor.py) |
no | 2–3.5 min | partly: Microcosm calibrates UC targets on the build's engine | yes |
| Regional input-surface audit | no | about 1 min | no | yes |
| Scorecard replay of the released bundle | yes, after PyPI | about 4.5 h (11 November) | no | yes |
Qualification (certify-uk-release), 11 required gates |
yes, eFRS line only | — | — | — |
US certification runbook (docs/engineering/runbooks/build-m-us-populace-certification.md) |
no | — | — | — |
Gaps
- No rule says what a Microcosm build that was certified at publication can skip. The hub runs its whole suite on each cut. The evidence binds to the PR's head commit, so a push means running it all again.
- The checks that do depend on the pinned engine are manual for the Microcosm UK line and missing for the US.
- Since Certify the Microcosm UK national and local-area pair with policyengine-uk 2.132.6 #685 merged (21 October),
maincertifies Microcosm data. So every run ofcertify-uk-releasestops at resolve withother_line(for example run 43218734650 on 12 November). It still runs every two hours. - All 39 candidate PRs it opened (16–24 October) are closed without merging.
- The US has no automated qualification or replay. The scorecard's replay workflows and its
results/folder cover the UK only.
- Since Certify the Microcosm UK national and local-area pair with policyengine-uk 2.132.6 #685 merged (21 October),
- The wait is long, and part of it repeats work.
- From merge to PyPI took 1 h 52 min for Certify the Microcosm UK pair from microcosm#1535 on policyengine-uk 2.174.1 (national eeb6093e, local area 38605444) #744. Most of that was the post-merge Test, the same suite the PR had already passed.
- The hub suite for Certify the Microcosm UK pair on policyengine-uk 2.151.0 (a6299504 national, c9df4b86 local area) #720 took about 8.5 h on one host.
- Dataset PRs record different evidence.
- Certify the Microcosm UK pair on policyengine-uk 2.151.0 (a6299504 national, c9df4b86 local area) #720 carried the hub's full run: budget inputs, producer gates, the hash check, the reform contract, reform responsiveness, the attribution and the replay.
- Certify the Microcosm UK pair from microcosm#1535 on policyengine-uk 2.174.1 (national eeb6093e, local area 38605444) #744 records the UC floor, bundle checks, snapshots and tests.
- The released-bundle replay fails without an alert.
NotifyConsumerssends the request withcontinue-on-error, and no job waits for the result. The latest receipt (policyengine 6.17.2, 11 November) failed integrity with 11construction_inertfailures. - Where the data checks live is not settled.
docs/engineering/skills/data-certification.md:412-419says that the UC floor's benchmark definitions "are to be settled when the UK data checks move to the data pipelines". No issue tracks that move.
Asks
-
An inventory. List every check that runs between a Microcosm build and a policyengine.py release: Microcosm's publication gates, policyengine.py CI, the qualification gates, the hub suite, the UC floor, the regional audit and the scorecard replay. For each check, give:
- what it establishes;
- its run time;
- what it has caught. For example, Qualify each new policyengine-uk or uk-data release automatically before promotion #573 records that manual review caught a default dataset with about half the UC caseload (Bump to policyengine-uk 2.90.2 and revert the UK certified default to enhanced FRS 2024-25 #491) and 34 missing model inputs (Bump policyengine-uk to 2.102.3 for the FYE 2025 absolute poverty line #536).
-
A classification. Sort each check into one of four groups:
- (a) Microcosm already establishes it on the build's engine. Do not run it again when policyengine.py pins that engine.
- (b) It depends on the pinned model or core. Run it only when the pin differs from the build's engine, and choose the smallest set that covers the difference. Candidates are the input-surface audit, budget inputs, the reform contract and reform responsiveness.
- (c) It is about policyengine.py's own layer: the bundle, the manifest, the routes and the wrapper code. Keep it in CI.
- (d) It is useful but not worth the wait. Run it after the release or on a schedule, as a report.
-
A time budget. Agree how long a certification may take for each kind of change, then fit the required checks into that budget. The kinds of change are:
- a pair built on the pinned engine;
- a model or core re-pin with the same data;
- a switch of data line.
Weigh each check's value against its run time. Prefer a quick release with a later report over a long wait for a check that rarely finds anything.
-
Automate the chosen set for the Microcosm UK line and for the US:
- run it without a person starting it;
- tie its evidence to the pair (dataset SHA-256, model wheel, core version), so a push that changes no calculation path does not restart it (UK data certification: which checks must come before a release, and can they run in parallel? #722, question 2);
- run independent steps in parallel (UK data certification: which checks must come before a release, and can they run in parallel? #722, question 3);
- write one evidence summary into the PR.
-
Clean up what remains.
- Re-scope or stop the two-hourly qualification runs that end at
other_line. - If the released-bundle replay stays, make a failed replay visible.
- Settle which checks move to the Microcosm data pipelines, and settle the UC floor's definitions with them.
- Re-scope or stop the two-hourly qualification runs that end at
Open questions
- policyengine.py has one core pin for both countries. When the core moves, which checks must run for each country?
- Should the post-merge Test run the full suite again when the PR's Test passed on the same tree, for example under a merge queue?
- Should any check block the merge, or only the release (the Versioning job)?
- For the US, which external comparisons are stable enough to gate, and which stay as reports?
Not in scope
- Adding checks for their own sake.
- Changes to Microcosm's own build gates (Holdout-masked builds + population-view scorecards as release gates (incl. pre/post-calibration delta) microcosm#302).
Related
#722, #462, #573, #665, #662, #721, PolicyEngine/microcosm#302, PolicyEngine/policyengine-scorecard#156
🤖 Generated with Claude Code
Activity