Skip to content

Build a US national release and stop; certify it as a separate step (#1585 item 11, PR 4a) - #1672

Merged
juaristi22 merged 10 commits into
mainfrom
us-build-then-certify-1585
Nov 18, 2026
Merged

juaristi22 merged 10 commits into
mainfrom
us-build-then-certify-1585

Conversation

@juaristi22

@juaristi22 juaristi22 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #1585 (item 11). This is PR 4a of the re-planned PR 4. A US national build now builds and stops, the way a UK national build does. Certification is a separate command, run on a finished build only when it is to be published. A build made to test a change no longer pays for the post-export checks. A build that turns out well is certified as it stands, without a rebuild.

Why

On the two Route A builds from raw sources, the work after calibration took most of the run (timing.elapsed_through_calibration_seconds against timing.total_build_seconds, read from each build manifest on the Hub):

  • 30 September (populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z): 28.3 h of 33.0 h;
  • 24 October (populace-us-2024-b5db384-8b0c1950b65f-20261024T083103Z): 52.8 h of 56.5 h.

Building the dataset, through calibration, took 4.7 h and 3.7 h. After calibration, the post-export scoring is the work that scales: 41 reform-coverage smoke passes and 55 reform-validation passes, each over 177 batches of up to 2,000 households (352,932 households in all), plus demographics and the holdouts. Today a test build can skip those checks only with flags chosen at launch, and a build that skipped them must be rebuilt to publish.

What changes

The release tool builds, and that is all it does (tools/build_us_fiscal_refresh_release.py):

  • It runs everything up to and including the H5 write and the written-file stored-input check: the input stages and their gates, the pre-solve gates, target compilation, the solve and the terminal batch.
  • It then writes the calibration NPZ and runs the optional export audit, both of which read the calibration result.
  • It writes <out>/build_record.json, completes its staging run and prints the command that certifies the build. Nothing it writes is publishable: the release directory holds the diagnostics and the gate evidence, but no manifest.
  • It refuses an --out that holds a build awaiting certification, before any work. A failed build's --out can still be built into again.

The build record holds what certification reads that is not a file:

  • the build's provenance: its commit, the time it finished and the packages that built it, which the manifests carry as the build's;
  • the eight options the post-export checks read;
  • the 13 gate verdicts the manifests carry;
  • the release gate summary the terminal batch computed;
  • the calibration losses and in-sample fit;
  • the post-export scoring plan;
  • the source-coverage and take-up receipts;
  • the build's staging block, where it staged, and its timing.

It pins by sha256 every file of the build that certification reads: the H5, the calibration weights, the diagnostics, the SSI take-up measurement, the gate evidence the build wrote and, for an --scf-education-debt build, its fit tables. Its paths are relative to itself, so a build directory moved to another machine certifies in place. It carries a digest of itself (record_sha256), taken over the record as JSON reads it back. That catches an edit after the build, not a forgery, until certification signs (PR 4b).

tools/certify_us_national_build.py <out> certifies a finished build:

  • It runs the release's post-export stages from the record: the smoke, reform validation, demographics, the three holdouts, source coverage and take-up participation.
  • It writes both manifests and prints the publication commands. The post-export code is today's, unchanged. It now sits in _certify_us_build, which binds every name it reads from the record.
  • It opens a staging run of its own where the build staged: the build's run id with -certify appended, to the build's staging repository and prefix. The post-export verdicts and the manifests reach the staging dashboard, and a failed certification marks the run failed. The build manifest's staging block, which the publisher checks, is certification's, with the build's inside it under build. A build that did not stage (--no-staging, which --exact-k requires) is certified without staging, and --no-staging opts certification out, as a build may opt out. Where the build staged, the block's reason says so: --no-staging at certification; the build staged (run <id>).
  • It rebuilds the scoring plan and refuses one that differs from the recorded plan.
  • It refuses:
    • a record of another schema, naming the commit that built it, whose code can certify it;
    • a record that does not match its own digest;
    • a record whose files changed, or a file it would read that appeared after the build;
    • a checkout at another commit than the build's, or with changes outside the build directory;
    • packages other than the ones the build was built with, Python included;
    • a --staging-run-id that names the build or its staging run, whose files a reopened run would start afresh;
    • a release directory that already carries a release manifest;
    • a build whose release gates failed (below).
  • A rerun after a failed certification first removes the earlier attempt's outputs.

A failed release gate no longer withholds the H5. This is its own commit (7b5f0ac).

  • A certified-tier build whose terminal batch fails still writes its weight-evidence sidecar (CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568), its H5 and its record, so it can be evaluated.
  • It then exits with the same Release gates failed: ... error, raised once the record is written.
  • A written-file stored-input failure or an export-audit error after the write joins the failures instead of replacing the refusal. The record is still written.
  • Certification refuses the build on its record's failures, and also on its CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 weight evidence, which only a failed build writes.
  • No gate judges against it. A later build refuses its files as either input-mass reference, and the post-export probe refuses its release directory as the reference. Its files may still start a later build, as its base, selection source or warm start, and the probe can score its H5.
  • An evidence-tier refusal, an unowned failure or a register mismatch, still raises before the write.
  • Input gates still stop the build where they fail. On a green build, the written-file premise still aborts right after the write.

Callers:

  • Route A stops after the build, with ROUTE_A_BUILT.json. A run whose base preflight is blocked exits 4 there, as a certified run does.
    • route_a.sh --certify (or ROUTE_A_CERTIFY=1) on the same RUN_ROOT skips the accepted stages and certifies the build in its own supervised stage. It does not hash the raw inputs again.
    • The certify stage gets the release stage's staging credential. With ROUTE_A_STAGING=0, it runs as the release stage does, online without the HF tokens, and with --no-staging.
    • It then runs the release-gate preflight and the publisher preflight. A failed certification reruns alone, never the build.
  • The exact-k ladder launcher's receipt is marked certified: false and carries the certify command before the publication commands.
  • The post-export probe accepts an uncertified build's release directory as its reference, reading the target surface from the build record.

Docs:

  • the release build rule's new "Build, then certify" section, and its opening, which now says a receipt and the standing check need both halves;
  • the Route A runbook, the agent guide and the README;
  • a changelog fragment.

Compatibility

  • No published build changes, and the contract, loader, publisher and emitter are untouched. Builds already published validate and resolve as before.
  • The manifests of a certified build carry the same blocks as today's.
    • Their provenance (built_at, runtime, the built-with packages and the commit) is the build's, as on main, carried by the record.
    • Their timing gains build_seconds and certify_seconds, and total_build_seconds is their sum.
    • The build manifest's staging block is certification's, with the build's under build. The publisher's staging check, unchanged, reads certification's.
  • _build_manifests takes the recorded gate summary and the build's provenance through new optional arguments, so the contract golden is unchanged.
  • Scripts outside the repository that ran the release tool and expected a release directory now get a build and a printed certify command. They need to run tools/certify_us_national_build.py next.
  • The gate battery's shadow (US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635, merged in) runs in the build: every function that holds a us_national shadow site is reached from the build, and none from certification. Its files sit under <out>/gate_battery_shadow/, outside the release directory, so the record and the manifests bind none of them. A rerun refused for reusing a certified release directory or an --out awaiting certification leaves the earlier build's shadow files as they were: the --out refusal runs before the shadow is armed, and the shadow holds its files until the certified-directory refusal has passed.

Tested

  • Engine-free, on this tree, with no country engine installed (as CI):
    • At the build-only commit: the whole touched surface, 4,671 tests. That is every engine-free file that reaches the release tool, the Route A driver, the launcher and the probe, the files that scan tools/ and the files that read these docs.
    • At the failed-gate commit: the 43 files that reach the release tool, 2,373 tests, and the doc readers, 1,025 tests.
    • At the reordering commit: the six source-shape files, 529 tests.
    • At the first review fix (a9e1ca8): the 43 files that reach the release tool, 2,379 tests, and the files that read these docs, 3,409 tests.
    • At the second review fix (4a1923a): the 43 files that reach the release tool, 2,388 tests, and the files that read these docs, 3,409 tests.
    • At the third review fix (7299987): the 43 files that reach the release tool, 2,389 tests, and the files that read these docs, 3,409 tests.
    • On the merge of main (90d45e7): the 64 files that now reach the release tool, the Route A driver, the launcher or the probe, US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635's shadow tests among them, 4,025 tests, and the other files that read these docs, 2,434 tests.
    • At the fourth review fix (47e6b9c, narrowed in 95279a4): the 64 files that reach the release tool, the Route A driver, the launcher or the probe, 4,035 tests, and the other files that read these docs, 2,434 tests.
    • None failing. tools/ci_test_plan.py verify is ok.
  • The record round trip was proven first. In commit 7a0a199 the build still certified itself in the same process, from the written record, and the end-to-end harness's green and failure runs produced today's outputs.
  • New tests:
    • the record carries what certification reads, in strict JSON, with the build's provenance;
    • a moved build directory reads;
    • each pinned file's change is refused, gate evidence that appeared after the build is refused, and so is a record edited by hand;
    • a record of an earlier schema is refused as such, not as edited, and so is a build into its --out, both naming the code that built it;
    • a changed scoring plan is refused;
    • a wrong commit, a dirty checkout, other packages and an already-certified directory are refused;
    • the dirty check leaves out a build directory inside the work tree, on a real repository;
    • a build with failed gates is refused, and so is any build holding the CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 weight evidence; its evidence-tier counterpart certifies;
    • certification stages where its build staged, in a run of its own, and marks that run failed, or the emitter's, when it does not finish;
    • a build refuses an --out awaiting certification, but not a failed build's, and main refuses one before the base download and leaves it as it was;
    • certification's --no-staging opt-out says when the build staged, and a --staging-run-id naming the build or its staging run is refused before certification touches the build;
    • a record with integer keys keeps its digest;
    • both input-mass references and the probe refuse a failed build as their reference, while a later build's base, selection source and warm start take its files;
    • the green harness builds, finds no manifest, then certifies:
      • its manifests carry the record's provenance and certification's staging block;
      • every artifact the release manifest binds is certification's own output or pinned by the record at the same digest;
    • the failed-gate harness's merge run fails its written-file premise and its export audit after the H5 write, and both join the record's failures and the refusal;
    • Route A's order, its modes, the certify stage's launcher (the release stage's), a blocked build-only run's exit 4, and certification without hashing the inputs again;
    • the probe reads an uncertified build.
  • Source-shape pins that read _main for calls now in certification are re-anchored on both halves: the build's _us_build_record(...) call and certification's _build_manifests(...). The #437 dropped the release H5 write; smoke gate scored a stale artifact (attempts 13/14) #443/CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 order pins now follow the recorded refusal, restated once the written file's checks have run. US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635's pin on the shadow's release is main's: the --out refusal runs with main's first refusals instead.
  • Not run here: any build, and the engine test jobs. The first real build as build-then-certify is owed: a Route A run, then --certify on its RUN_ROOT, on a build machine. It also gives the first measured split between the two.

Order

For review, @MaxGhenis

  1. Which commit certify may run at. This PR requires the build's own commit. Should a descendant whose diff touches only registers, fragments and tests be allowed, as the UK recertify guard does? The manifests already take the build's commit and packages from the record, and certify refuses other packages.
  2. The skip flags. --skip-reform-coverage-smoke, --skip-reform-validation and --skip-out-of-sample-reforms are still accepted and still recorded. A test build no longer needs them. Should certification refuse a record that carries them?
  3. Where the tier is chosen. The tier, certified or evidence, is still chosen at build time. Moving it to certify (PR 4b) needs the build id minted at certify.
  4. The release gate summary is the one the terminal batch computed, recorded, where _build_manifests used to compute it again at the end from the same inputs.
  5. A failed build now leaves an H5. It is the one change in kind here, as its own commit, and can be dropped on its own. Certification refuses such a build on its record and on its CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 weight evidence.

🤖 Generated with Claude Code

juaristi22 and others added 4 commits November 17, 2026 20:03
The US national release tool now ends its build where the export H5 is
written: it records what its post-export checks and manifests need in
<out>/build_record.json, and _certify_us_build runs those checks from the
record. In this commit the build still certifies itself in the same
process, so every output is today's; the next commit makes certification a
separate step.

- The record holds the eight options the checks read, the 13 gate verdicts
  the manifests carry, the release gate summary computed at the end of the
  build, the calibration losses and in-sample fit, the post-export scoring
  plan, the source-coverage and take-up receipts, the staging block and the
  build's timing. It pins the H5, the calibration weights and the
  diagnostics by digest, with paths relative to the record.
- _certify_us_build rebuilds the scoring plan and refuses one that differs
  from the plan the build recorded, then runs the moved post-export code
  unchanged.
- _build_manifests takes the recorded gate summary (release_gate_failures);
  without it, it computes the summary as before, so the contract golden is
  unchanged.
- The optional export audit, the calibration NPZ and the local copy of the
  base H5 read the calibration result or the base, so they run before the
  build ends.
- The manifests' timing gains build_seconds and certify_seconds;
  total_build_seconds is their sum.
- Source-shape tests that read _main for calls now in certification are
  re-anchored on both halves: the build's record and certification's
  _build_manifests call.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1585)

The release tool now builds and stops, as a UK national build does: once
the H5, the calibration weights, the diagnostics and the build's own gate
evidence are written, it writes build_record.json, completes its staging
run and prints the command that certifies the build. Nothing it writes is
publishable. A build made to test a change no longer pays for the
post-export checks, which took 28.3 of the 33.0 hours of the 30 September
Route A build and 52.8 of the 56.5 hours of the 24 October one; a build that
proves itself is certified as it stands, without a rebuild.

- tools/certify_us_national_build.py <out> runs the release's post-export
  checks from the record (the reform-coverage smoke, reform validation,
  demographics, the holdouts, source coverage and take-up participation)
  and writes both manifests and the publication commands. It refuses a
  checkout at another commit than the build's or with changes, a release
  directory that already carries a release manifest, and a record whose
  files changed. A rerun after a failed certification removes the earlier
  attempt's outputs first.
- Certification always writes the build's own staging block, from the
  record, since the publisher checks that the build's staging arrived.
- Route A stops after the build with ROUTE_A_BUILT.json; route_a.sh
  --certify (or ROUTE_A_CERTIFY=1) on the same RUN_ROOT skips the accepted
  stages, certifies the build without credentials, then runs the release
  and publisher preflights.
- The exact-k ladder launcher's receipt is marked uncertified and carries
  the certify command before the publication commands.
- The post-export probe accepts an uncertified build's release directory as
  its reference, reading the target surface from its build record.
- Docs: the US release build rule's "Build, then certify" section, the
  Route A runbook, the agent guide and the README; a changelog fragment.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s it (#1585)

A build is evaluated before anyone certifies it, so a failed terminal gate
no longer withholds the H5 (decided by María, 17 November). A
certified-tier build whose batched pre-export gates fail now:

- writes its weight-evidence sidecar as before (microcosm#568), then its H5,
  calibration weights and build record, which carries the failures;
- leaves its staging run for main() to mark failed, prints that it cannot
  be certified, and exits with the same "Release gates failed: ..." error,
  raised once the record is written.

Certification refuses a record that carries release-gate failures, unless
the build is evidence-tier, whose failures are owned and ride its known
failures as before. An evidence-tier refusal (an unowned failure or a
register mismatch) still raises before the H5 write. Input gates still stop
the build where they fail, and the written-H5 stored-input premise still
aborts right after the write, before the record.

The build records the release gate summary it computed for the terminal
batch, instead of computing it again at the boundary; a summary that
crashed in degraded mode is recorded as missing, and certification refuses
it, as _build_manifests would have raised re-evaluating it.

The #443/#568 source-shape pins now follow the recorded refusal: the
batched refusal precedes the H5 write, which precedes the build record,
which precedes the refusal's raise. The corridor test's failure modes assert
the written H5, the record's failures (equal to the raised message) and the
refusal to certify.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_certify_us_build now follows _main at the end of the file, where the
post-export stages sat as _main's tail, so the whole change shows those
lines in place rather than moved. A pure reordering: the file holds the
same lines.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful split; the record round trip reads cleanly. A few things were checked at b0a5c20:

  • Name binding. Static passes over _certify_us_build find no unbound names, and no former _main local that now resolves to a module-level name. Everything is bound from the record at tools/build_us_fiscal_refresh_release.py:17572-17636.
  • Rerun cleanup. It unlinks only the fixed basenames in US_CERTIFICATION_OUTPUT_FILES (:13597-13609). The H5, NPZ, record, diagnostics, gate evidence and weight sidecar are never touched.
  • Failed-gate path. The evidence-tier refusals still raise before the H5 write (:17360-17393). The certified-tier failure is refused at :13644-13649.
  • Route A. bash -n is clean, a failed certify stage reruns alone, and no network fetch was found on the offline certify path.
  • Tests. The touched engine-free test files pass locally.

Three points:

  1. Built-with versions and built_at now come from the certify process. _build_manifests takes built_at = datetime.now(UTC) (:11188) and _runtime_versions() (:11189, reading installed packages at :2399-2414). It writes them as build.built_at, built_with_core_package and built_with_model_package (:11460-11470), and build_record.json carries neither.
    • On main these came from the process that built the H5.
    • Now a build moved to another machine and certified there (certify checks only the commit, :13652-13657, and --no-sync does not check the venv) would record the certify environment's policyengine-us/policyengine-core as the build's.
    • built_at moves from "late by the post-export time" to "the certify time". The contract describes it as the time the build was made (packages/microcosm-data/src/microcosm/data/contract.py:1671-1675).
    • Would it make sense to record _runtime_versions() and a build timestamp in the record, and pass them through? Certify could also refuse a version mismatch. This also bears on question 1: a descendant commit would make that pin more important.
  2. Some bound files are not pinned (minor). The record pins the H5, the weights and the diagnostics (:13355-13359). The manifests also hash, at certify time, us_ssi_take_up.json (:11634-11638) and whichever gate-evidence files are present (:10605-10616), all written by the build. Pinning those too would match the docstring at :17567-17568.
  3. In-repo --out (minor). The build checks for a dirty worktree before writing (:14022), but certify checks after (:13650). An unignored in-repo --out (anything but out/) therefore makes certify refuse its own build as a "dirty git worktree". Excluding the build directory, or naming it in the message, would help.

Review drafted with AI assistance and checked against the code at b0a5c20.

)

Addresses the review on #1672:

- The manifests' built_at, runtime block and built-with packages came from
  the certifying process. The build record now carries the time the build
  finished and the build's _runtime_versions(), and certification passes
  them, with the build's commit, to _build_manifests. certify_main refuses
  packages other than the ones the build was built with, Python included,
  and names each that differs: the manifests name those versions as the
  build's, and the post-export checks would otherwise score the H5 on
  others.
- The record pinned the H5, the calibration weights and the diagnostics,
  but certification also hashes or reads us_ssi_take_up.json, the gate
  evidence the build wrote and, for --scf-education-debt, the fit tables.
  The record now pins each of those present when the build finishes, under
  its artifact key in the manifests, and certification refuses one that
  appeared after the build. The smoke's evidence is certification's own
  and stays unpinned.
- certify_main checked for a dirty worktree after the build had written
  its --out, so a build whose --out sat unignored inside the work tree made
  its own certification refuse it. The check now leaves out the build
  directory when it lies strictly inside the work tree, and lists the
  paths it found.

The green end-to-end run now asserts that the manifests' provenance is the
record's, and that every artifact the release manifest binds is either
certification's own output or pinned by the record at the same digest. The
release build rule's record and refusals bullets are updated.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to a9e1ca8c6. No path we read publishes an uncertified directory or changes a certified build's verdicts: the publisher needs manifests that only certification writes, and certification refuses a failed record before it writes anything.

certify_main calls _certify_us_build(record) with no telemetry (tools/build_us_fiscal_refresh_release.py:13773, :17670-17672), so certification skips every staged stage and attachment, and the build has already completed its staging run (:17657). The manifests carry the build's staging block, so the publisher's staging check passes (packages/microcosm-data/src/microcosm/data/publish_cli.py:40-64, :182) for a build whose smoke, reform validation, holdouts and manifests never reached staging. A failed certification marks nothing failed there. Could certification open its own staging run, or attach to the build's?

A default Route A run now exits 0 at ROUTE_A_BUILT.json, before the post-export checks and both preflights (tools/route_a/route_a.sh:675-699). The build rule still opens with main passing "its preflight and certification gates" (docs/us-release-build-rule.md:5-6) and still asks a PR for a from-scratch build and certification receipt (:12-15). The new section could say which half a receipt and the standing check need.

Smaller points:

  • build_record.json carries no digest of itself, and a failed certified-tier build now leaves its H5. certify_main refuses that build on the record's own terminal_gate_failures (:13737-13742).
  • route_a.sh --certify runs verify_inputs on every raw input twice before the certify stage (:382, :460), though certification reads none of them.
  • _refuse_certified_release_dir_reuse checks only release_manifest.json (:2589-2608), so a rerun into the same --out overwrites a build that awaits certification.
  • On a failed build, an error from the written-file check or the export audit (:17553-17556 on) replaces "Release gates failed" and leaves an H5 with no record.

Addresses the second review on #1672:

- Certification ran with no telemetry, so its post-export verdicts and
  manifests never reached staging, while the manifests carried the build's
  staging block and passed the publisher's staging check. certify_main now
  starts an emitter run of its own (run kind "certification") and, where
  the build staged, a staging run of its own: the build's run id with
  -certify appended, to the build's repository and prefix, which the record
  now keeps. A failed certification marks that run failed. The build
  manifest's staging block is certification's, with the build's nested
  under "build". A build that did not stage (--exact-k requires
  --no-staging) is certified without staging; certify takes --no-staging,
  --staging-run-id and --staging-dir. Route A's certify stage gets the
  release stage's staging credential, or runs offline with --no-staging
  when ROUTE_A_STAGING=0.
- The build rule's opening now says a receipt and the standing check need
  both halves, and the Route A runbook says a run without --certify shows
  the build half only.
- The record carries a digest of itself, which certification checks. That
  catches an edit, not a forgery, until certification signs (PR 4b).
  Certification also refuses a build whose release directory holds the
  #568 weight evidence, which only a build whose release gates failed
  writes.
- route_a.sh --certify no longer hashes the raw inputs once the build's
  release stage is accepted.
- A build refuses an --out that holds a build awaiting certification; a
  failed build's --out may still be built into again.
- On a failed build, a written-file stored-input failure or an export-audit
  error joins the release gate failures instead of replacing the refusal,
  and the record is still written.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, all three are fixed in a9e1ca8.

  1. Provenance. The build record now carries the time the build finished (built_at, taken when the record is written) and the build's _runtime_versions(). Certification passes them to _build_manifests with the build's commit, so built_at, created_at, runtime and the built-with packages are the build's, as on main. certify_main refuses an environment whose versions differ from the record's, Python included, and names each one that differs. On question 1: if a descendant commit is allowed later, the manifest's git_commit already comes from the record.
  2. Pins. The record now pins every file of the build that certification reads, under its artifact key in the manifests: us_ssi_take_up.json, the gate evidence the build wrote and, for --scf-education-debt, the two fit tables. Certification also refuses one that appeared after the build finished. The smoke's evidence is certification's own, so it stays out. The green end-to-end run now checks that every artifact the release manifest binds is either certification's output or pinned by the record at the same digest.
  3. In-repo --out. The dirty check now leaves out the build directory when it lies strictly inside the work tree (an :(exclude,literal) pathspec), and the refusal lists the paths it found. A test runs the check against a real repository.

The release build rule's record and refusals bullets are updated to match.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks. All of these are addressed in 4a1923a.

  1. Staging. Certification now opens a staging run of its own where its build staged: the build's run id with -certify appended, to the build's staging repository and prefix, which build_record.json now keeps.
    • Its stages are the post-export checks and the manifests, it attaches their files, and a failed certification marks the run failed.
    • The build manifest's staging block, which the publisher checks, is now certification's, with the build's inside it under build. So the publisher's check reads the run that scored what it publishes.
    • Attaching to the build's run would not work: a staging run's files start afresh when it opens, so it would reset the build's progress and its calibration curve.
    • A build that did not stage (--no-staging, which --exact-k requires) is certified without staging, and --no-staging opts certification out.
    • Route A's certify stage now gets the release stage's staging credential when ROUTE_A_STAGING=1. When it is 0, the stage runs offline, without secrets and with --no-staging.
  2. Receipts. The build rule now says a receipt and the standing check need both halves: the build from raw sources, its certification, and the release-gate and publisher preflights. A Route A run without --certify stops at ROUTE_A_BUILT.json and shows the build half only. The Route A runbook says the same.

Smaller points:

  • The record's own digest. The record now carries record_sha256, a digest of its canonical JSON, and certification refuses a record that does not match it. That catches an edit, not a forgery: anyone who recomputes it can still change the record until certification signs (PR 4b). Separately, certification now refuses any build whose release directory holds the CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 weight evidence. Only a build whose release gates failed writes it, and a green build removes any it finds. So clearing a failed record's terminal_gate_failures is no longer enough to certify it.
  • route_a.sh --certify no longer hashes the raw inputs, at the start or after the wait, once the build's release stage is accepted.
  • Overwriting a build awaiting certification. A build now refuses an --out whose build_record.json records a build certification would accept. A failed build's --out can still be built into again, as before builds wrote a record.
  • Errors after the H5 write on a failed build. A written-file stored-input failure or an export-audit error now joins the failed build's release gate failures. The record is still written, and the refusal names every failure. On a green build both still raise as before. An I/O error writing the NPZ or the record itself still ends the build without a record.

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. At 4a1923a4d all three earlier points read as resolved:

  1. Provenance. The build writes built_at and _runtime_versions() into the record (tools/build_us_fiscal_refresh_release.py:17806-17807), and certification passes them with the commit into _build_manifests (:18328-18330, used at :11249-11254). The version check at :13913-13926 includes the Python patch version. Route A builds and certifies from the same .venv, so it should only fire for a build certified on another machine, which is the case it exists for.
  2. Pins. _us_build_files_certification_reads (:13360-13383) covers the SSI measurement, the build's gate evidence and the SCF fit tables. The record pins them (:13472-13478), and a file that appears later is refused (:13728-13739). Certification writes none of these files, so rerunning certify does not trip its own pins.
  3. In-repo --out. The :(exclude,literal) pathspec (:3425-3447) also covers certification's default staging directory under the build root.

The staging run (:13805-13855, :13930-13961) and the nested staging.build block (:18309) read correctly against _staging_undelivered in publish_cli.py:40-64. ROUTE_A_STAGING=0 still certifies offline through NOSECRETS (tools/route_a/route_a.sh:584-585).

One small point: US_BUILD_RECORD_SCHEMA_VERSION is still 1 (:13246), but the record now requires built_at, runtime, staging_settings and record_sha256. A record from an earlier commit of this branch passes the schema check (:13704). It is then refused at :13709-13713 as "changed after its build wrote it", which reads as tampering, and the new guard at :2611-2639 also refuses rebuilding into that --out. Bumping the version to 2 would give the clearer schema message. This only matters for records written before these commits.

Review drafted with AI assistance and checked against the code at 4a1923a4d.

juaristi22 and others added 2 commits November 18, 2026 11:41
Addresses the third review on #1672. The record gained built_at, runtime,
staging_settings and record_sha256 while its schema stayed 1, so a record
from an earlier commit of this branch passed the schema check and was then
refused as edited, for want of a digest.

- US_BUILD_RECORD_SCHEMA_VERSION is 2. Certification refuses a record of
  another schema before it checks the digest, and the refusal names the
  commit that built it, whose code can certify it.
- A build into an --out holding such a record is refused for that reason,
  not with a certify command that would refuse the record. A failed build of
  any schema may still be built over.
- The build rule's refusals bullet lists the schema refusal.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1635 (the US gate battery shadow), #1559 (Route A's default ASEC
pool) and #1664. Two conflicts, both where the two sides added code at the
same place:

- Above main(): this branch's build record and certification section and
  #1635's gate shadow section. Both are kept.
- In _main, after the certified-directory refusal: this branch's refusal of
  an --out awaiting certification and #1635's release of the shadow's files.
  Both are kept, the refusal first, so a rerun that either refusal stops
  leaves the shadow files of the build it protects as they were. #1635's
  order pin now requires both refusals before the release, and the two
  shadow docstrings name both.

Every us_national shadow site sits in the build, which still runs under
main()'s shadow; certification runs none of them. The shadow writes under
<out>/gate_battery_shadow/, outside the release directory, so the record and
the manifests bind none of its files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks. Bumped in 7299987.

  • US_BUILD_RECORD_SCHEMA_VERSION is now 2. Certification checks the schema before the digest. So a record from an earlier commit of this branch is refused as a record of another schema ("has build record schema 1; this tool reads schema 2"), not as edited. The refusal names the commit that built it, whose code can certify it.
  • A build into that --out is refused for the same reason and names that commit, instead of offering a certify command that would refuse the record. A failed build's --out, of any schema, can still be built into again.
  • A test covers both refusals, and the build rule's refusals bullet now lists the schema one.

Separately, main is now merged in (90d45e7), with #1635's gate shadow. Both conflicts keep both sides. The shadow now writes its files only after both reuse refusals, so a refused rerun leaves an earlier build's shadow files as they were, and #1635's order test checks that.

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up at 90d45e7, covering 7299987bb and the merge of main bcb5cba9d.

  • The schema bump reads right. US_BUILD_RECORD_SCHEMA_VERSION is 2 (tools/build_us_fiscal_refresh_release.py:13266), and _read_us_build_record checks the schema before the digest (:13733-13740). A record from an earlier commit is therefore refused as another schema, naming its commit through _certify_with_its_own_code, not as edited. _refuse_overwriting_a_build_awaiting_certification (:2632) gives the same message for a rerun into that --out, and still lets a failed build be built over, schema 1 included. An unreadable record falls through to the existing "awaiting certification" refusal. The new test covers all three paths.
  • The merge keeps the order the description gives. _start_national_gate_shadow arms the shadow at :14739 with its files held. _refuse_certified_release_dir_reuse and _refuse_overwriting_a_build_awaiting_certification run at :14980-14982, and only then does _release_national_gate_shadow_files() let them be written. So a refused rerun leaves an earlier build's shadow files as they were. Dry runs are not held, as the docstring says.

Nothing further. CI was still running at this head at the time of writing.

Review drafted with AI assistance and checked against the code at 90d45e7.

@juaristi22
juaristi22 marked this pull request as ready for review November 18, 2026 12:38
@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to 4a1923a4d. We read the merge of main at 90d45e76a only where you resolved it by hand, and we have not read 7299987bb. The staging run, the receipt paragraph, the input re-hash and the --out refusal answer the earlier points, and we found no path that publishes an uncertified build.

Two points remain open:

  1. A staged build certified with --no-staging passes the publisher's staging check. _certify_progress opts out on the flag (tools/build_us_fiscal_refresh_release.py:13824-13837). The manifest's staging block then reads enabled: false with a reason and nests the build's block under build (:13074-13080, :18309), and the publisher reads the top level and accepts a stated opt-out (packages/microcosm-data/src/microcosm/data/publish_cli.py:62-63). Route A takes that path when the build runs with ROUTE_A_STAGING=1 and --certify runs with ROUTE_A_STAGING=0 (tools/route_a/route_a.sh:341-358). Could certification refuse --no-staging for a build that staged?

  2. If 7b5f0aca3 stays, a failed build leaves its H5 and NPZ at the usual paths. Several readers take them with no look at the record: the probe (tools/probe_us_post_export.py:3062-3143), --selection-source-h5, both input-mass reference options, --base-h5 and --warm-start-calibration-npz. Could they refuse a build directory whose record carries terminal_gate_failures? The digest and the CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568 refusal stop an accidental edit, as the commit says. Removing the three weight files, clearing the failures and recomputing the digest still certifies the build until PR 4b signs.

On the Route A driver:

  • ROUTE_A_CERTIFY defaults to 0 (route_a.sh:51) and route_a.env.example never names it. A default run exits 0 at ROUTE_A_BUILT.json, even with the base preflight blocked (:707-733), where main exits 4. Could certify become the default, with build-only runs setting ROUTE_A_CERTIFY=0?
  • With ROUTE_A_STAGING=0 the certify stage runs under NOSECRETS, which sets HF_HUB_OFFLINE=1 (:584-585). Main's release stage dropped the tokens and stayed online. Could certify do the same?
  • "both preflights" names the release-gate and publisher preflights in the new rule paragraph (docs/us-release-build-rule.md:21-22), and the base and release preflights in the driver and README (route_a.sh:250, README.md:160). Naming the two in the rule would settle it.

Smaller points:

  • A record mapping with integer keys would change its digest across the JSON round trip (:13314-13315). We found no such mapping.
  • --staging-run-id may equal the build's run id (:13930-13932).
  • No test covers the --out refusal's call in _main (:14634-14635).
  • The description still says certification runs without credentials.

The plan for the next certified US build leaves this change out, and we have not yet answered your five questions or the receipt wording. Could the merge wait for those answers and for that build's commit? If it needs to land sooner, please say so here first.

juaristi22 and others added 2 commits November 18, 2026 14:31
Addresses Max's review on #1672 (at 4a1923a):

- A staged build certified with --no-staging passed the publisher's staging
  check: the manifest's top-level staging block, certification's, read as a
  stated opt-out. Certification now refuses --no-staging for a build that
  staged, and a --staging-run-id that names the build or its staging run,
  before it touches the build.
- A failed build keeps its H5 and calibration weights to be evaluated, and
  later builds read such files with no look at the record. --base-h5,
  --selection-source-h5, both input-mass reference options and
  --warm-start-calibration-npz now refuse any file under the --out of a
  build whose record carries release-gate failures that refuse it, or whose
  release directory holds the #568 weight evidence. The post-export probe
  refuses such a build's release directory as its reference and may still
  score its H5. Like the record's digest, this stops a mistake, not an
  edit, until certification signs (PR 4b).
- Route A: with ROUTE_A_STAGING=0 the certify stage runs as the release
  stage does, without the HF tokens but online, where it ran offline. A
  run without --certify whose base preflight is blocked now exits 4, as a
  certified run does, and route_a.env.example names ROUTE_A_CERTIFY.
- The build rule's opening names the two preflights a receipt needs: the
  release-gate preflight on the release manifest and the publisher's
  --preflight-only check.
- The build record is digested as certification reads it back, so a
  mapping with integer keys keeps its digest across the JSON round trip.
- The refusal of an --out awaiting certification now runs with main's
  first refusals, before the base download, any output directory and the
  gate shadow, so a rerun stops in seconds. A test drives main into such an
  --out and finds it unchanged. #1635's shadow docstrings and order test are
  main's again.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eference (#1585)

Narrows two parts of 47e6b9c:

- --no-staging opts certification out of staging again for any build, as
  a build may opt out: the publisher accepts a stated opt-out, and the
  manifest records its reason, which now says when the build staged
  ("--no-staging at certification; the build staged (run <id>)"), beside
  the build's own block. Refusing it meant a staged build could not be
  certified without the staging credential, for example after its
  directory moved to another machine. What an opt-out costs is the
  certification half on the staging dashboard, not a check. A
  --staging-run-id naming the build or its staging run is still refused.
- A failed build's files are refused only where a gate or comparison
  judges against them: the two input-mass references and the probe's
  reference release directory. As a later build's base, selection source
  or warm start they are allowed again, which can cut a rerun's time while
  its data is fixed; that build's own gates judge its result, and its
  manifests record each input's digest.

Part of #1585 (item 11, PR 4a).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

juaristi22 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks. Addressed in 47e6b9c and 95279a4. Two answers differ from what you suggested (points 1 and 2), and Route A's default stays as it is.

  1. --no-staging on a staged build. We kept it as an opt-out rather than refusing it.
    • The publisher's staging check refuses staging that was meant to happen and never arrived. It accepts an explicit opt-out, as it does from a build that never staged.
    • Certification's opt-out is recorded the same way: the manifest's top-level block now says --no-staging at certification; the build staged (run <id>), beside the build's own block under build.
    • What it costs is the certification half on the staging dashboard, not a check: certification still runs every post-export check, and the manifests carry every verdict.
    • Refusing it would stop a staged build from being certified without the staging credential, for example after its directory moves to another machine.
    • Certification does now refuse a --staging-run-id that names the build or its staging run, which would start the build's run files afresh.
  2. A failed build's files. While 7b5f0ac stays, a failed build is refused wherever a gate or comparison judges against it.
    • Both input-mass reference options refuse any file under its --out.
    • The probe refuses its release directory as --reference-release-dir, though it can still score its H5 as --export.
    • As a later build's base, selection source or warm start it stays allowed. Warm-starting from a failed attempt cuts a rerun's time while its data is fixed, that build's own gates judge its result, and its manifests record each input's digest.
    • On the edit path, agreed: removing the three weight files, clearing the failures and recomputing the digest still certifies the build until PR 4b signs. Like the digest, these refusals stop a mistake, not an edit.

On the Route A driver:

  • Certify as the default. Route A still stops after the build by default. That was decided when this PR was scoped, so a run made to test a change stops before the post-export checks. Separately, a run without --certify whose base preflight is blocked now exits 4, as a certified run does, and route_a.env.example names ROUTE_A_CERTIFY.
  • Offline certification. With ROUTE_A_STAGING=0, the certify stage now runs as the release stage does: without the HF tokens, but online.
  • "Both preflights." The rule now names them: the release-gate preflight on the release manifest and the publisher's --preflight-only check.

Smaller points:

  • Integer keys. The record is now digested as certification reads it back, after a JSON round trip, so such a mapping keeps its digest. A test covers integer and mixed keys.
  • --staging-run-id. Refused when it names the build or its staging run, as above.
  • The --out refusal's call. It now runs with main's first refusals, before the base download, any output directory and the gate shadow, so a rerun into an --out awaiting certification stops in seconds instead of after the input stages. A test drives main into such an --out and checks that the build is left as it was. With the move, US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635's shadow docstrings and order test are main's again.
  • The description. Since this morning's update, it has said that the certify stage gets the release stage's staging credential. Its ROUTE_A_STAGING=0 clause still said "offline", which 47e6b9c changes; the description will be updated to match.

On the merge: it needs to land sooner. We'd like to merge it as soon as possible, so that #1676, which prepares the next Route A national build, picks it up, and this PR does not block that build.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Rulings on the five questions in the description:

  1. Which commit certification may run at: the same behaviour as the UK, so certification may run at a later commit than the build's. This comes in a follow-up PR.
  2. The skip flags: accepted and recorded at certification, as now.
  3. Where the tier is chosen: a build is just a build, with no tier, and certification chooses the tier after it. This comes in the same follow-up PR.
  4. The release gate summary: computed once is enough, as now.
  5. A failed build's H5: kept, as now (7b5f0ac stays).

@juaristi22
juaristi22 merged commit 087bcc6 into main Nov 18, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to 95279a47d, which merged as 087bcc6df. Each fix you listed holds there, the merge of main keeps both sides, and every us_national shadow site still sits in the build. We read the code and ran nothing. Agreed on the Route A default and on --no-staging staying an opt-out.

Three points for a follow-up:

  1. The publisher reads the top-level staging block (packages/microcosm-data/src/microcosm/data/publish_cli.py:55-64), which now belongs to certification (tools/build_us_fiscal_refresh_release.py:19132). Before this change that block belonged to the build, so the publisher refused a build whose own staging uploads never arrived. Now such a build passes the check once certification stages or opts out. Could the publisher, or certification, also read staging.build? This one changes what the publisher refuses on main today, so we'd fix it first.

  2. "No gate judges against it" (docs/us-release-build-rule.md:126, the changelog fragment, the docstring at :2724-2725 and the description) covers more than the code refuses. _REFERENCE_H5_OPTIONS names the two input-mass options (:2713-2716). Two references still take a failed build's files: --incumbent-diagnostics (:1368-1374), and the release-gate preflight's --export-input-mass-reference-h5 (tools/preflight_us_release_gates.py:365-372), which Route A passes to both preflights (tools/route_a/route_a.sh:746). Could the refusal cover both, or the sentence name what it covers?

  3. A green build that raises at the written-input check or the export audit leaves its H5 with no record (:18568-18570, :18593-18595), so _failed_build_holding accepts that file as a reference.

Smaller points:

  • A failed build now reaches us_national_stored_inputs_written before it raises (:18553). The shadow then records the terminal phase as complete, where it used to record that phase as blocked with one entry unreached (packages/microcosm-build/src/microcosm/build/us_runtime/gate_shadow.py:1077-1122). The description's line on the shadow could say so for PR 3.
  • _failed_build_holding skips a record it cannot read and walks on (:2739-2742). The --out refusal treats the same record as a build awaiting certification (:2655-2678), and certification cannot read it, so that --out stays stuck.
  • ROUTE_A_BUILT.json stays beside ROUTE_A_DONE.json after --certify (route_a.sh:796).
  • When --base-h5, --selection-source-h5 or --warm-start-calibration-npz belongs to a failed build, the manifest could name that build beside the input's digest, as an optional addition. _failed_build_holding already finds it.

juaristi22 added a commit that referenced this pull request Nov 18, 2026
Two conflicts. The release tool keeps #1672's end of the build and writes
the base's location attributes right after the written file's stored-input
check, before the build record binds the H5's sha256. Certification reads
those attributes back from the written H5 for its source coverage (the
auto-merged line used a name only the build defines). The Route A driver
tests keep both sides: Linux sizing and STOP_BEFORE_RELEASE beside the
certify flow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 18, 2026
#1672 split the release into a build that stops at ROUTE_A_BUILT.json and a
certification (route_a.sh --certify). Step 4 now builds, a new step 5
certifies and runs both preflights, the host table names the certify stage,
and section 7 states María's 18 November rulings on #1672, which a follow-up
brings before this build: certification may run at a later commit than the
build, and the build has no tier (certification chooses it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Rulings 1 and 3 (certification chooses the tier; certification at a later commit, as the UK allows) are in #1681. It also takes up Max's review.

juaristi22 added a commit that referenced this pull request Nov 18, 2026
The stored-input pin (test_us_stored_input_register) holds the written
file's check right after the H5 write and one batched raise in _main. The
integrity list now starts before the write, the refusal after the record
tests the build's refusal and that list instead of terminal_gate_failures,
and the pin expects the premise failure recorded, not raised, on a green
build too (review of #1672).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 18, 2026
Brings in #1672 (build-then-certify) and #1677 (hand-off copy). Clean merge; the only files both sides touched are README.md and docs/agent-guide.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 19, 2026
…o the engine refresh

main brings #1681 (rulings 1 and 3 on #1672: a build has no tier and
certification chooses it; certification may run at a descendant of the
build's commit that changes only tests, docs, changelog fragments and the
standing evidence owners register) and #1677 (the Modal base hand-off copies
the gate shadow directory). No textual conflicts.

Checked where the two sides meet in tools/build_us_fiscal_refresh_release.py:
the release still writes the base's household-location attributes after the
written-H5 stored-input check and before the build record binds the H5's
sha256, and certification still reads them back from the written H5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 19, 2026
Brings #1672, #1677 and #1681. None of them touches a file this branch changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants