Skip to content

Decide which dataset checks policyengine.py must run, given what Microcosm already checks and the engine versions it pins #748

Open

Description

@juaristi22

Summary

When a Microcosm build is certified into policyengine.py, some checks repeat what Microcosm already established when it published the build. Other checks depend on the model and core versions policyengine.py pins, and those are manual, slow or missing. A certification can hold a release for hours: #720 took 12 h from a ready head to the release.

This issue asks for an evaluation, not for more checks. Which checks must policyengine.py run, given:

  • what Microcosm already checks when it publishes a build;
  • the policyengine-uk or policyengine-us version and the policyengine-core version that policyengine.py pins;
  • the trade-off between what each check catches and how long a release waits for it?

Then automate the chosen set for both countries, and stop running the rest at the policyengine.py boundary. This builds on #722 (the UK hub's steps, their order and their timing) and #462 (a pre-registered promotion scorecard).

Facts below are from main at 5b6d7901 (11 November), Microcosm main at a575057bb, and GitHub runs up to 12 November.

What Microcosm already checks when it publishes

  • UK. Three signed gate batteries (spine, calibration seam and release cut) cover the 94 gates that uk/gates.json declares. release_certification.json must say shippable: true. The local-area build adds a signed seven-gate report. microcosm-publish-release-build validates the release contract and the signatures again before it publishes. A failing gate passes only through a dated sign-off register.
  • US. The release builders run their release gates. Route A runs the gate preflight before and after the release, and the publisher preflight.
  • Both. Each release manifest records the engine the build used (built_with_model_package, built_with_core_package) and the model versions it claims to be compatible with.

These results hold for the build's own engine. They do not need to run again when policyengine.py pins that same engine.

What policyengine.py runs today

Check Automated? Time Repeats a Microcosm check? Depends on the pinned model or core?
Bundle verification (manifests, hashes, TROs) yes, every PR about 0.5 min partly no
Test on 4 Python versions (synthetic or mocked data) yes, every PR 68–96 min (#744) no yes
The same Test again after the merge yes 65–111 min (#744) no, but it repeats the PR's run yes
Hub: producer gates, national hash check no not posted separately likely (to confirm) no
Hub: budget inputs no not posted separately no yes
Hub: reform contract no 3 h 22 min (#715); over 4 h 29 min (#720) no yes
Hub: reform responsiveness, 10 reforms, 2025–2031 no 2 h 04 min (#715) no yes
Hub: 2×2 attribution no 9 min (#715) no yes
UC floor (scripts/check_uk_uc_floor.py) no 2–3.5 min partly: Microcosm calibrates UC targets on the build's engine yes
Regional input-surface audit no about 1 min no yes
Scorecard replay of the released bundle yes, after PyPI about 4.5 h (11 November) no yes
Qualification (certify-uk-release), 11 required gates yes, eFRS line only — — —
US certification runbook (docs/engineering/runbooks/build-m-us-populace-certification.md) no — — —

Gaps

  1. No rule says what a Microcosm build that was certified at publication can skip. The hub runs its whole suite on each cut. The evidence binds to the PR's head commit, so a push means running it all again.
  2. The checks that do depend on the pinned engine are manual for the Microcosm UK line and missing for the US.
    • Since Certify the Microcosm UK national and local-area pair with policyengine-uk 2.132.6 #685 merged (21 October), main certifies Microcosm data. So every run of certify-uk-release stops at resolve with other_line (for example run 43218734650 on 12 November). It still runs every two hours.
    • All 39 candidate PRs it opened (16–24 October) are closed without merging.
    • The US has no automated qualification or replay. The scorecard's replay workflows and its results/ folder cover the UK only.
  3. The wait is long, and part of it repeats work.
  4. Dataset PRs record different evidence.
  5. The released-bundle replay fails without an alert. NotifyConsumers sends the request with continue-on-error, and no job waits for the result. The latest receipt (policyengine 6.17.2, 11 November) failed integrity with 11 construction_inert failures.
  6. Where the data checks live is not settled. docs/engineering/skills/data-certification.md:412-419 says that the UC floor's benchmark definitions "are to be settled when the UK data checks move to the data pipelines". No issue tracks that move.

Asks

  1. An inventory. List every check that runs between a Microcosm build and a policyengine.py release: Microcosm's publication gates, policyengine.py CI, the qualification gates, the hub suite, the UC floor, the regional audit and the scorecard replay. For each check, give:

  2. A classification. Sort each check into one of four groups:

    • (a) Microcosm already establishes it on the build's engine. Do not run it again when policyengine.py pins that engine.
    • (b) It depends on the pinned model or core. Run it only when the pin differs from the build's engine, and choose the smallest set that covers the difference. Candidates are the input-surface audit, budget inputs, the reform contract and reform responsiveness.
    • (c) It is about policyengine.py's own layer: the bundle, the manifest, the routes and the wrapper code. Keep it in CI.
    • (d) It is useful but not worth the wait. Run it after the release or on a schedule, as a report.
  3. A time budget. Agree how long a certification may take for each kind of change, then fit the required checks into that budget. The kinds of change are:

    • a pair built on the pinned engine;
    • a model or core re-pin with the same data;
    • a switch of data line.

    Weigh each check's value against its run time. Prefer a quick release with a later report over a long wait for a check that rarely finds anything.

  4. Automate the chosen set for the Microcosm UK line and for the US:

  5. Clean up what remains.

    • Re-scope or stop the two-hourly qualification runs that end at other_line.
    • If the released-bundle replay stays, make a failed replay visible.
    • Settle which checks move to the Microcosm data pipelines, and settle the UC floor's definitions with them.

Open questions

  1. policyengine.py has one core pin for both countries. When the core moves, which checks must run for each country?
  2. Should the post-merge Test run the full suite again when the PR's Test passed on the same tree, for example under a merge queue?
  3. Should any check block the merge, or only the release (the Versioning job)?
  4. For the US, which external comparisons are stable enough to gate, and which stay as reports?

Not in scope

Related

#722, #462, #573, #665, #662, #721, PolicyEngine/microcosm#302, PolicyEngine/policyengine-scorecard#156

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions