Skip to content

Move the US engine to policyengine-us 2.145.3 and prepare the Route A national build - #1676

Draft
juaristi22 wants to merge 43 commits into
mainfrom
us-engine-refresh-route-a
Draft

juaristi22 wants to merge 43 commits into
mainfrom
us-engine-refresh-route-a

Conversation

@juaristi22

@juaristi22 juaristi22 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Moves the US engine to policyengine-us 2.145.3 (core 3.36.1, spm-calculator 1.1.0) and makes main ready for the next US national build through Route A, with the operator runbook in docs/us-national-build-runbook.md.

Base: #1559, #1635, #1664, #1672, #1677 and #1681 have landed on main, and this branch has merged main at 7ff5a26bc (with #1663, #1666 and #1682).

Data improvements this build picks up

Before the build

  • Done: A US build has no tier: certification chooses it, at the build's commit or a later one (#1585 item 11) #1681, María's follow-up to Build a US national release and stop; certify it as a separate step (#1585 item 11, PR 4a) #1672 (rulings 1 and 3), is merged here. The build has no tier, and certification chooses it; certification may run at a later commit only when that commit changes tests, docs, changelog fragments or the standing owners register, so a fix to code needs a new build (runbook section 7).
  • The build comes from this PR's own tree, before it merges: the release build rule asks a change this size for a from-scratch build and certification receipt from its own tree before it merges. The operator builds at this PR's head, pinned by its full SHA, with REQUIRE_ON_MAIN=0 and from a checkout of that commit, and certifies at the same commit, so the PR stays open for fixes. A fix to code after the build means a new build at the new head. The PR then merges with a merge commit, so the built commit lands in main's history.
  • Done: Max's review of A US build has no tier: certification chooses it, at the build's commit or a later one (#1585 item 11) #1681, its four points and its smaller ones, is addressed here (commits 21 and 22), and so are his reviews of this PR at d596fcb60 and bd4ff48a6 (commits 27 and 28). Later review notes that change nothing a build does when it follows the runbook go into the PR that stacks on this one for the local-area path, so the build commit is ba410438e, this PR's last commit.
  • The build is assessed before it is certified, one command per step (runbook section 4):
    • The quick check first (about 2 hours): the base, its preflight and the release-gate dry run, with the gate battery shadow on, then a stop. One command reads it with US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635's pass rules. When it is clean, the operator goes straight on to the build; when it finds something, the evidence comes to María first.
    • Then the build (route_a.sh, about 7 to 9 hours) and its probe (route_a.sh --probe, about 10 hours, against about 48 for certification). The operator sends the run's evidence with the runbook's step 6 command, which collects the logs and records and leaves every data file on the laptop.
    • María assesses them, and the operator certifies with route_a.sh --certify only when she asks.
  • Done: María's rulings of 18 November on hosts, staging and publication (also in the runbook):
    • M1: the base runs locally for now.
    • M2: the build and its certification run on a laptop with enough memory, with the PUF inputs on it.
    • M3: the build and its certification upload progress and diagnostics to staging (ROUTE_A_STAGING=1).
    • M4: Lock the US engine to policyengine-us 2.60.4 #1365 is closed as superseded by this PR.
    • M5: María decides publication and promotion, and coordinates with Max.
  • Only for county files on 2.145.3: ACS local lane: re-review the reviewed-fill consumer register against policyengine-us 2.145.3 #1675.

Commits, in order

  1. Lock. The three US extras pin policyengine-us>=2.145.3,<3 and policyengine-core==3.36.1. uv then resolves one core for both forks, so the UK side moves from 3.34.10 to 3.36.1 too (ruled 17 November). The worker identity pin follows the new uv.lock.
  2. Route A. The driver runs on Linux (stat, df -P), the README has a Linux section, and STOP_BEFORE_RELEASE=1 stops before the release so its gates can be dry-run first.
  3. Take-up. 2.145.3 adds four take-up inputs, left at the engine default (rate_unsourced; everyone eligible takes up, as on 2.2.1). The ABI count moves from 17 to 21.
  4. Engine surface.
    • Social Security: 2.145.3 migrates a stored social_security_<type> to its _reported source, so levers apply (tested in the baseline and under a +10% reform). The adapter treats those names as inputs; columns keep their names.
    • The tax-unit first/second_home_mortgage_interest legs are gone from the engine (pe-us#9605). The PUF chain stops producing them, and the eCPS parity gate records the retired layers.
    • Seven stored-input register entries are now engine inputs and leave the register.
    • The ASEC channel follows the engine default estate_income_would_be_qualified = False.
    • The release copies the base's populace_location_rule onto its H5 (County Files: a proof-of-concept county-calibrated dataset by 23 November #1647).
  5. QRF tail (R7). The tail gate decides at base (pre-calibration) weights and reports the share at final weights in qrf_tail_concentration.json (schema 2) without failing on it. Collectibles gets a q0.999 donor bound.
  6. Frame adapter. The static consumer index reads the 2.145.3 formulas (Lock the US engine to policyengine-us 2.60.4 #1365's index extensions and helper reviews, ported and extended).
  7. SCF 2022 summary extract (The Fed replaced scfp2022s.zip on 2026-10-09, so the pinned SCF 2022 summary extract no longer downloads #1368). The build pins the Fed's republished file and divides its eight dollar columns by the CPI-U-RS ratio 4789/4376 on read; the original file is still read as it is.
  8. Tail dry run. The release's dry run grades the tail register on the base frame, with no calibration shift.
  9. Location attributes after the stored-input check (root attributes only).
  10. Re-pins of the pool engine contracts, engine ABI lock, generated bundle, F0 pins, release input-coverage manifest, year-window tripwires and reform probes, with the tests that pin them.
  11. Docs. The runbook, agent guide, release build rule and changelog fragments.
  12. More re-pins that the retired mortgage leg moves (late-producer counts, pool fixtures, the H3 parity fixture).
  13. ACS local register. Its engine-tier walk tests run only on the reviewed engine (2.2.1); the lane's release gate refuses the installed engine meanwhile. Re-review: ACS local lane: re-review the reviewed-fill consumer register against policyengine-us 2.145.3 #1675.
  14. Core 3.36.1 branch hook. Core 3.36.1's clone binds a copied method to the copy, so the release's batched target and reform materialization stopped with a KeyError. The hook now takes the parent from the call that made the branch, and refuses a simulation copied outside get_branch.
  15. Scenario and workflow tests on 2.145.3. Fifteen scenario tests move their pins with their contracts re-checked. Where the engine changed, the tests follow it: the ACA PTC, SNAP child support in California, the SALT refund probe (19 smoke baselines), the capped educator deduction, HUD gross rent with the utility allowance, and the self-employed health insurance limit.
  16. CI and review follow-ups. The UK spine parity graph re-pinned on core 3.36.1; the ACS stand-in engine; the battery's binding identity; engine-free re-pins the first CI rounds could not reach; the voluntary-filing probe's reason, which now says it measures the pre-take-up credit.
  17. Build a US national release and stop; certify it as a separate step (#1585 item 11, PR 4a) #1672. main merged with the location attributes written before the build record binds the H5, and certification reading them back from the written H5. The runbook now builds, then certifies.
  18. Rulings. The runbook records María's 18 November rulings on hosts, staging and publication (M1 to M5).
  19. A US build has no tier: certification chooses it, at the build's commit or a later one (#1585 item 11) #1681. main merged at cfd5a2a5e, with no conflicts. The location attributes are still written after the written-H5 stored-input check and before the build record binds the H5. A Route A test follows the release stage, which now also accepts a build that failed its gates (exit 3).
  20. Probe, then assess. The runbook probes the build on a 5% household sample (tools/sample_us_export_households.py, tools/probe_us_post_export.py) before María's assessment (certify, build again, or certify as evidence), and certifies only when she asks. The probe leaves out its QRF tail stage, which grades at final weights while the build decides that gate at base weights. S6 stops on a probe failure.
  21. Max's review of A US build has no tier: certification chooses it, at the build's commit or a later one (#1585 item 11) #1681, in the release tool (ea810f35f).
    • The certified tier refuses a failed source-coverage gate before it writes either manifest, as it refuses a failed smoke, so such a build can still be certified as evidence.
    • A failed build's H5 is a --base-h5 only beside an --export-input-mass-reference-h5: without one, the export input-mass gate judges the export against its base.
    • Certification adds every gate failure it finds to certification_failures.json beside the build record, at either tier, and a build that passed its own gates and then failed certification's is no reference until a certification ships it.
    • The build writes build_incomplete.json before its H5 and removes it once its record is written, so an H5 that no record pins is no reference either.
    • A standing evidence owner pattern owns a failure line only from the line's start, so input signal failed: no longer owns every gate ending in those words.
    • The summaries of what a later certifying commit may change name the root conftest.py, LICENSE and Markdown outside the packages' sources, and a publisher test covers version 2 staging blocks in a separately certified build.
  22. One command per step (18667a600).
    • route_a.sh dry-runs the release's gates on the fresh base as a stage of its own before the release starts (ROUTE_A_DRY_RUN=1, the default). A certain failure stops the run in minutes, and a dry run holds only for the release settings it checked.
    • route_a.sh --probe probes an accepted build as two supervised stages under the run directory, and stops.
    • From Max's review: the commit check requires the certification tool and parser on every run, whatever the tier. A rerun to build refuses release settings other than the accepted build's, which it would otherwise keep without a word, and --rebuild sets that build aside, keeping it on disk, to build the release again on the same base.
    • The runbook has one command per step and one command that collects a run's evidence. The README and the settings example follow.
  23. Bind England's PIP at 16–19 and 20–24, and impute UC limited capability for claims without a linked DWP record #1663. main merged at 6d32375b3 (8d33275f8). Only the UK spine graph-parity fixture conflicted. It is main's, with its 56 policyengine-core entries at 3.36.1, as this branch's generator writes them; nothing else in it differs from main's.
  24. Quick check first (6983bad4e). The runbook's first run stops after the base, its preflight and the release-gate dry run, each with the gate battery shadow on. Step 3 reads them with one command, applying US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635's pass rules, and prints QUICK CHECK CLEAN or what it found. The evidence then goes to María after the build and its probe, after the quick check only when it found something, and at any stop. S3 names the three shadow comparisons.
  25. The receipt from this PR's tree (d596fcb60). The runbook builds at this PR's head, pinned by its full SHA, with REQUIRE_ON_MAIN=0, as the release build rule asks of a change this size before it merges.
  26. Frame: share unchanged tables under copy-on-write, and refuse raw writes into shared buffers #1666 and Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682. main merged at 7ff5a26bc (8a48bf6e2). The five pin files that Fill the QRF leaf array in place and pin quantile-forest 1.4.2 #1682's in-place QRF leaf array and the engine refresh both moved conflicted, as Max listed. Each takes this branch's side, and the pins both moved are recomputed on the merged tree: tools/repin_us_spec_engine_contracts.py rewrote the seed digests, the loader's golden vector and the pool tool's spec_sha256, and tools/spec_engine_coverage.py regenerated the F0 report (both --check pass). The worker's lock pin is the merged uv.lock's digest.
  27. Max's review of this PR (bd4ff48a6): five gaps on later or hand-run paths, none on the runbook's.
    • A probe after a failed certification gets the failed build's arguments, its fit and target surface, since the build is no reference.
    • Each attempt in certification_failures.json names the build and its record's digest, so a later build into the same --out inherits none.
    • build_incomplete.json goes down as the build creates its --out, before any file there, its diagnostics included.
    • The certification test reads structure, not line numbers.
    • Route A's rerun guard also compares the digests of what the release read by path: the base H5, the pinned release inputs and the export input-mass reference.
  28. Max's review of bd4ff48a6 (360357c58).
    • The driver runs from the checkout it is started in, and main's has no STOP_BEFORE_RELEASE and no --probe. The commit check now refuses a driver whose files differ from tools/route_a/ at the build commit, and the runbook says to check that commit out first.
    • The input check reads the export input-mass reference before any stage, so a file openssl cannot read stops the run before the base.
    • The certification register's writer keeps what its readers count, and a test runs the rerun guard's comparison.
  29. A test's run root (1382d2d10): the export reference test writes under its own temporary directory.
  30. Shellcheck (ba410438e): the driver's self-check writes its empty value as '', as shellcheck asks; nothing the driver does changes.

What changes in outputs

  • The engine itself: 33 variables removed, about 1,660 added since 2.2.1; new default values for tenant_pays_utilities and living_arrangements_allow_for_food_preparation (now True); federal AGI now includes state tax refunds, estate income and taxable disability benefits.
  • Tax-unit roles now come from microunit instead of age ordering (pe-us#9617 measured -$0.94B of 2024 federal income tax on the certified build).
  • The QRF tail gate no longer fails a build for concentration that calibration adds; it reports it.
  • SCF wealth and auto-loan donors: the same values as before, to float precision.

Tests

Touched tests ran locally in an engine-free environment and in a 2.145.3 environment. The engine-free and US engine jobs stop at a process's first failure, so the whole engine-free list and every US scenario and workflow module also ran locally, and the UK H2 parity test ran on core 3.36.1. The full suite runs in CI.

After main was merged at cfd5a2a5e, the 44 engine-free test files that reach the release tool, Route A, the probe, the certify tool, the preflights or the publisher, or that read the changed docs, ran locally: 3,184 tests. One Route A test failed on the release stage's new text; it is fixed in 5c37a801c, and its file passes (118 tests). tools/ci_test_plan.py verify is ok. The engine jobs run in CI.

For commits 21 to 24, the engine-free test files they touch, or that read the changed docs, ran locally. Before the merge, the release tool's builder file (465 tests) ran first. Then 1,058 tests ran in two runs, covering Route A, the probe, its design, the release-gate dry run and preflight, the export tool's source, the spec bundle, the CI test plan, the logbook, the CI integration job, the publisher guard and the release contract. One Route A test failed: it pins the one place that removes ROUTE_A_BUILT.json, and --rebuild now sets that file aside instead. It is fixed in commit 22. After the merge with 6d32375b3, none failed: 930 tests across Route A, its ASEC pool, the Modal stage plans, the CI test plan and the graph acceptance files, and the builder file's 465. tools/ci_test_plan.py verify is ok. The UK spine parity test needs the UK engine, and shellcheck is not installed locally, so CI runs both. Commit 25 changes only the runbook.

After the merge with 7ff5a26bc, 4,182 engine-free tests ran locally, none failing: the spec-engine loader, the US stacked spine, spine blindness and pool H5 files, microcosm-fit's and microcosm-frame's engine-free suites, the graph executor and parity pins, Route A and the release tool's builder file. tools/ci_test_plan.py verify is ok. The re-pinned engine-contract and engine-workflow tests and the US bundle check need the US engine, so CI runs them.

For commit 27, 805 engine-free tests ran locally, none failing: Route A, the probe and its design, the release-gate preflight and dry run, and the release tool's builder file. tools/ci_test_plan.py verify is ok.

For commits 28 and 29, the Route A driver file (146 tests) and the release tool's certification tests (60) ran locally, none failing, and tools/ci_test_plan.py verify is ok. For commit 30, the driver file ran again at ba410438e, none failing; its shellcheck test needs shellcheck, so CI runs it.

Follow-ups (not in this PR)

  • weeks_worked is an input on 2.145.3 (policyengine-us#9660 Part A1). The pool still leaves it out; carrying WKSWORK weeks changes pool outputs, so it is its own change.
  • The Social Security component names stay as stored today; a rename to _reported is possible later.
  • unemployment_compensation_months is a new unemployment-insurance consumer of weeks_unemployed on 2.145.3.
  • The voluntary-filing PTC probe now measures the pre-take-up credit; moving it to a measure households receive is María's call.
  • The post-export probe's QRF tail stage still grades the export at final weights, and its fidelity check compares that with the build's qrf_tail_concentration.json, whose verdict is at base weights since 17 November. The runbook leaves the stage out; moving it to base weights is its own change.

Not settled here

Only a real build settles the risks listed in the runbook's section 6.

🤖 Generated with Claude Code

juaristi22 and others added 17 commits November 17, 2026 19:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e refresh

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three US extras move from policyengine-us 2.2.1 / core 3.32.5 to the
newest release at lock time (2.145.3, 17 Nov), which needs core >=3.36.0
and spm-calculator 1.1. uv then resolves one core for both forks, so the UK
side moves from 3.34.10 to 3.36.1 as well (ruled 17 Nov). Core 3.35-3.36.1
adds a situation hook, an opt-in uprating base mode and a calculate_add
crash fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver used BSD stat and a df whose output wraps on Linux, so on a
Linux host every input failed its size check. file_bytes tries GNU stat,
then BSD stat; free_bytes reads df -Pk. The tests no longer shim stat,
so CI's runner exercises the GNU path.

STOP_BEFORE_RELEASE=1 stops after the base, the base preflight and the
release config, so the release's gates can be dry-run on the fresh base
(tools/dry_run_us_release_gates.py, #1042) before the multi-day release
(ruled for the next national build on 17 Nov 2026). The README gains a
Linux section (settings, tmux, systemd) and links the operator runbook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-us 2.145.3 adds takes_up_medicare_part_d_if_eligible,
takes_up_csfp_if_eligible, takes_up_school_meals_if_eligible (SPM unit)
and takes_up_ak_permanent_fund_dividend_if_eligible, all defaulting to
true. The take-up mapping must be total over the engine, so each joins the
spec, the contract and the generator as an engine-owned program left at
its default (rate_unsourced, no locked source), as ruled on 17 Nov 2026.
That reproduces 2.2.1, which paid all four programs to every eligible
person. The ABI program count moves from 17 to 21, and each input gets a
battery metric and a take-up operator boundary entry. The engine ABI lock
is regenerated with the other re-pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mortgage legs (ruled R16): policyengine-us removed the tax-unit
  first/second_home_mortgage_interest legs (policyengine-us#9605). The PUF
  chain stops producing them; the person-level home_mortgage_interest
  carries the amount. The eCPS parity gate records the two retired
  reference layers instead of grading them (ported from #1365). The late
  transfer surface drops to 71 targets and the stacked surface to 139.
- Stored-input register: seven entries are engine inputs on 2.145.3
  (tax_unit_role_input, parent_1_id, parent_2_id and the four federal
  student loan inputs) and leave it. The pool's build-time simulation
  projection keeps microunit's tax-unit roles (#1027, ruled R14).
- QBI (ruled R10): the base ASEC channel keeps the engine's own
  qualification defaults; estate income now defaults to non-qualified.
- Location (ruled R17): the release copies the base's household-location
  root attributes onto its H5 and records them, so a block_v1 national
  file no longer reads as legacy to ACS staging (#1647).
- OBBBA estate row (ruled R13): noted as zero by construction, since
  estate_tax needs is_deceased, which no release stores.
- IRMAA holdout: the engine falls back to the earliest input year
  (policyengine-us#10106), so a single-year release now has payers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruled 17 Nov 2026 (R7), the smallest fix for the tail failures of the
19 Oct and 3 Nov national runs:

1. The tail-concentration gate and its per-run register now measure the
   release's base frame at pre-calibration weights, where the QRF draw
   artifact the gate guards (#462) lives. On the 3 Nov base most columns
   that failed at final weights sat at 0.45-0.63 before calibration: the
   excess came from calibration putting weight on few carriers. The share
   at the export's final weights is still measured and reported in
   qrf_tail_concentration.json (schema 2, final_weight_report), and
   decides nothing. The gate shadow reads the same base frame, so both
   verdicts agree. The release dry run's tail margins default to zero,
   since its staged frame is now the release's own verdict.
2. long_term_capital_gains_on_collectibles, the one column above the
   threshold before calibration, gets the weighted donor q0.999 bound that
   non_sch_d_capital_gains has. Tail bounds now also apply to sparse
   snapped amount outputs, lowered onto the donor's snap grid so the snap
   cannot carry a value past them. #481 stays the root fix.

The route A example no longer passes the d490 register, which was measured
at final weights and would now read as stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Static consumer index (B3): ports #1365's index extensions and helper
  reviews and extends them for 2.145.3. Every helper a formula calls is
  reviewed at a code digest (71 reviews: 30 new and 13 re-reviewed since
  #1365). Six general readers cover the sites the index could not read
  (class attributes through get_variable, dict comprehensions, call
  defaults, getattr on parameter nodes, closures passed as arguments,
  helper results passed as arguments), and the digest follows imports
  made inside a helper.
- Generated-variable audit (B2): refreshed to 2.145.3 and spm-calculator
  1.1.0; the 120 generated variables are unchanged.
- Concept mapping (C6): reviewed against 2.145.3, with take-up bindings
  for the four new programs; the coverage golden is regenerated.
- Social Security (R4, revised): pe-us 2.145.3 migrates a stored
  social_security_<type> to its _reported source, so benefit levers apply
  (tested: the same values as storing _reported, in the baseline and
  under a lever reform). The adapter reads the migrated names from the
  installed engine and treats them as storable inputs with their default,
  and refuses a frame that stores both a name and its _reported input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Federal Reserve replaced scfp2022s.zip on 2026-10-09, when it published
the 2025 SCF. The file it serves now states every dollar value in 2025
dollars, and the pinned original is no longer served, so a fresh build
stopped at the SCF wealth stage on the digest check (#1368).

Ruled 17 Nov 2026 (option 2 of #1368): pin the republished file (zip
d4e139c6..., member 6ff73d52..., downloaded from federalreserve.gov on
17 Nov) and convert on read. read_scf_2022_summary_extract identifies the
file by its digest and divides the eight dollar columns Microcosm reads
(liq, stocks, nmmf, bond, networth, wageinc, intdivinc, ssretinc) by the
Fed's CPI-U-RS ratio 4789/4376; sentinel codes keep their value. The
records and weights are the original's: the weighted shares of nonzero,
positive and negative net worth equal the values recorded for the pinned
original. Values equal the original only to float precision. The original
2022-dollar member is still read as it is, so no file is converted twice,
and the reader records which case applied. Both SCF loaders (wealth and
auto loans, whose targets come from the unchanged full file) use it.
Route A's SCF input row moves to the new member.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release decides the QRF tail gate on its base frame at pre-calibration
weights, so its dry run grades the tail register on that same frame with no
calibration or L0-selection shift (support fixed, zero tail-share margins).
The dry-run module keeps its measured default margins (0.35, 0.05, 1.0) for
other callers. The imputation schema admits the collectibles q0.999 bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release now carries the base's household-location rule onto its H5
after the written file passes the stored-input premise check. The
attributes are root metadata only and move no column, so the verdict holds.
The diagnostics test of the release main path stubs the base-location
reader, since its stub base is not an HDF5 file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Regenerated or re-measured on 2.145.3: the pool engine contracts, the engine
ABI lock, the generated spec bundle and its frozen legacy digests, the F0
coverage pins, the release input-coverage manifest (retired eCPS layers are
no longer required), the primary QRF target-order digest, the year-window
tripwires and the eCPS parity known-gap versions.

Reform probes: the above-the-line deduction list names educator_expense_ald,
and the SALT refund probe scores federal income tax (2.145.3 counts the
refund in gross income; about 29 state lists now read it, both ways).

Test pins follow the engine. Two engine changes shape them:
- 2.145.3 backdates every parameter to 2015 before it applies reforms, so
  the QBI list's first breakpoint at the first reform pass is 2015-01-01.
  The list-edit guard still refuses an edit there; a 2017 edit now holds
  only inside its period.
- Files built on 2.2.1 store the retired tax-unit mortgage-interest legs,
  which the stored-input check now refuses, and tax_unit_role_input is an
  engine input, so it left the register.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/us-national-build-runbook.md gives each stage's commands, environment,
inputs and secrets, time, memory and disk, success check and what to send
back, plus the risks only a real build settles. The agent guide and the
release build rule point to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e refresh

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The retired tax-unit first_home_mortgage_interest leg leaves the post-PUF
transfer (72 to 71 targets, 53 to 52 numeric) and the primary PUF pass
(two outputs and two inputs fewer). The pool fixtures, the late-producer
DAG pins and the operator-ordering doc follow. The H3 post-transfer parity
fixture is regenerated with its tool: its synthetic source gains the four
new take-up columns, and its expected cells do not change. The national
gate-scope digests move with the QRF tail gate's notes. Quote style follows
ruff format in the files the re-pins edited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
acs_local_reviewed_fill_consumers.yaml is reviewed against policyengine-us
2.2.1, and on 2.145.3 the consumer walk differs for 34 of its 59 entries.
The walk tests now run only on the engine the register was reviewed
against. One test checks that the ACS local release gate refuses the
installed engine meanwhile, so no local-area release can use the unreviewed
register. The re-review is microcosm#1675. The national release does not
run this gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #1635's second round a verdict difference is recorded and printed as a
warning, and the release still finishes. S3 now has the operator read the
us_national comparison file after the release and report a difference, a
shadow error or a missing file with the hand-off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed write-up. Three CI causes on 4477f5d, all from the engine move rather than flakiness, plus a note on the conflict.

engine-uk (3.13/3.14): H2 spine fixture still on core 3.34.10. test_h2_uk_spine_parity fails at packages/microcosm-graph/tests/engine/uk/test_acceptance_h_parity.py:45 (graph_from_json(uk_spine.json) == graph, differing nodes). Every node in packages/microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json records ["policyengine-core","3.34.10"] in numerical_dependencies, and the lock now installs 3.36.1 for the UK side too. It is the only non-changelog file at head that still says 3.34.10. Regenerating the fixture with its tool should clear both UK jobs.

engine-free (3.13/3.14): ACS local packaging test still expects tax_unit_role_input in the register. test_package_records_the_stored_input_gate_bound_to_the_packaged_bytes (packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py:5597) now raises StoredInputRefusalError: stored column 'tax_unit_role_input' ... is not a variable in policyengine-us 2.2.1. f247728 removed the entry from US_STORED_NON_VARIABLE_COLUMNS, but the test's stand-in engine (_PACKAGE_ENGINE, :54-57) is still a 2.2.1 surface without that variable. The fixture still stores the column (:1300, :4091), and the expected gate lists it in registered_non_variables (:5622). One option is to give the stand-in engine tax_unit_role_input (and the 2.145.3 label) and drop it from the expectation. Whichever way it goes, the test should state which engine the ACS lane is graded against while #1675 is open.

engine-us (3.13/3.14): scenario tests pinned to 2.2.1. packages/microcosm-build/tests/engine_scenario/us/test_us_adult_care.py:11 fails with assert '2.145.3' == '2.2.1'. The scenario lane stops at the first failure, so this is only the first. At head, 15 scenario files still assert version("policyengine-us") == "2.2.1" unconditionally: adult_care, disability_benefits, educator_expenses, energy_subsidy, farm_business_income, housing_inputs, other_health_insurance, relationship_inputs, retirement_distributions, sipp_head_start, ssi_disability_criteria, ssi_take_up, voluntary_filing, wic_claim and workers_compensation. Each needs either a re-pin to 2.145.3 with its contract re-checked, like test_us_medicare_take_up.py:59, or an explicit skip if it stays 2.2.1-specific.

Conflict with main. The history is criss-cross (both sides merged #1559's head 826c5eb), so there are two merge bases. A plain recursive git merge origin/main comes out clean, with the result equal to this branch plus #1664's files. Against the single base 5499382 the only conflict is packages/microcosm-build/tests/engine_free/us/test_route_a_driver.py: main keeps the BSD stat() shim and this branch drops it for file_bytes. The branch's side is right. Note that the last CI run tested a merge with 4d287f4, before #1559 reached main, so merging main also gets a run against the current base.

Things that look consistent: the pyproject and uv.lock pins, the take-up contract constraint, the mortgage-leg retirement (_retire_ecps_reference_layers refuses it if the engine disagrees), the base-weight tail gate (export_frame is only a reweight of base_frame), the SCF deflator covering every summary dollar column the two loaders read, the location attributes written before the post-export scorer binds the sha256, and the Route A driver. Its only diff vs main after #1559 is the Linux portability, STOP_BEFORE_RELEASE, the SCF pin and the commented-out d490 register.

Review drafted with AI assistance and checked against the code at 4477f5d.

juaristi22 and others added 6 commits November 18, 2026 11:28
The release's target and reform materialization wraps get_branch to retain
every engine branch until a batch is checked. Core 3.32 copied the parent's
bound wrapper into a clone; core 3.36.1 rebinds it to the copy, so the hook
looked the copy up as its own parent and stopped with a KeyError. The hook
now takes the parent from the call that created or holds the branch, which
works with either behavior, and refuses a wrapper it cannot trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Version pins move to 2.145.3. Where the engine changed, the tests follow it:
- ACA PTC: a unit that takes up coverage can claim without filing for
  another reason, so the voluntary-filing test uses a unit that does not.
  The probe's reason says so, and the runbook lists its reach as a risk.
- SNAP child support: California now excludes it from gross income.
- SALT refund: the probe scores federal income tax; South Carolina tax no
  longer moves, so the smoke needs 19 baselines, not 20.
- Educator expenses: the ALD list names the capped educator_expense_ald.
- HUD gross rent adds the utility allowance.
- The self-employed health insurance limit subtracts half of SE tax.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lock moves the UK side to core 3.36.1, which each UK graph node
records among its numerical dependencies. Regenerated with
tools/graph_uk_spine_fixture.py; only the core version changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stored-input register follows the locked engine, which defines the
tax-unit role as an input (policyengine-us#9617), so the stand-in engine of
the ACS local release tests defines it too. Its label stays the
reviewed-fill register's engine until that register's re-review (#1675).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efresh

The only conflict is the us_national gate-scope digests in
test_us_battery_bindings.py: both sides moved them. They are re-measured on
the merged gates.json, which carries #1635's last edits and this branch's
base-weight tail gate note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stand-in engine's note now says it once: the lane's tests model a release
on the reviewed-fill register's engine (2.2.1), and the stored-input register
follows the locked engine (2.145.3), which defines tax_unit_role_input.
Review of #1676 (Vahid).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, Vahid. All four points are handled at 61c8fae6a.

  • UK H2 fixture. Regenerated with tools/graph_uk_spine_fixture.py in a UK environment on core 3.36.1. Only the recorded core version changes, and the test passes (6bb6830d5).
  • ACS stand-in. The stand-in engine defines tax_unit_role_input, and the expectation drops it (a25e54c9f). I kept the 2.2.1 label: an autouse fixture makes every lane test report the reviewed-fill register's engine (2.2.1) to the reviewed-fill gate, so a 2.145.3 label would give one simulated release two engines. The stand-in's note now says which engine the lane's tests grade against while ACS local lane: re-review the reviewed-fill consumer register against policyengine-us 2.145.3 #1675 is open (61c8fae6a).
  • Scenario pins. All 15 files are re-pinned with each contract re-checked on 2.145.3 (be8eb1225). Six needed more than the pin:
    • voluntary filing: a unit that takes up ACA coverage now gets the PTC without another reason to file, so the test uses a unit that does not;
    • child support: California excludes it from SNAP gross income;
    • SALT refund: the probe scores federal income tax, South Carolina tax no longer moves, and the smoke needs 19 baselines, not 20;
    • educator expenses: the ALD list names the capped educator_expense_ald;
    • housing: HUD gross rent adds the utility allowance;
    • self-employed health insurance: the earned-income limit now subtracts half of SE tax.
  • Hidden failures. The scenario lane stops at its first failure, so I ran every US scenario and workflow module locally. That found one real crash: core 3.36.1's clone binds a copied method to the copy, so the hook that retains engine branches in batched target and reform materialization stopped with a KeyError. Fixed in dc992cf7c; the batched-materialization and memory workflow modules pass.
  • Main. Merged bcb5cba9d (Pool main's default ASEC years in Route A (d2603) #1559, US gates in us/gates.json, run in shadow beside today's gates (#1585 item 11, PR 2 of 4) #1635, Add the US one-population selection library #1664). The only conflict was the us_national gate-scope digests, re-measured on the merged gates.json (ce509f268). The Route A driver test keeps this branch's side, as you suggested.

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. All four earlier points are resolved at 61c8fae6a:

  • The H2 fixture records core 3.36.1 on all 56 nodes.
  • The ACS stand-in defines tax_unit_role_input and the expectation drops it. Keeping the 2.2.1 label is consistent with _the_release_runs_the_reviewed_engine reporting the register's reviewed_against, and the note at test_us_acs_local_release_tool.py:56-61 explains the hybrid.
  • All 15 scenario files are re-pinned.
  • The merge is clean.

engine-free (3.13/3.14): one new module in the battery's binding identity. The only failure (1 failed, 9633 passed) is test_the_batterys_binding_identity_adds_only_the_leaf_and_its_imports, assert 247 == ((198 + 25) + 23).

  • Computed the same way as the test (binding_identity.describe over TRANSPORT_GATE_REGISTRY), the set holds 247 modules at this head and 246 at main bcb5cba9d.
  • The one extra is microcosm.frame.adapters._policyengine_us_helper_reviews, added in 0f51437 and imported by frame/adapters/policyengine_us.py:35 and _policyengine_us_source_index.py:19, both already in the closure.
  • Adding it to BINDING_MODULES_ADDED_AFTER_BASE (packages/microcosm-build/tests/engine_free/shared/test_transport_evidence_kernels.py:137) with a comment pointing at #1676 should clear both jobs (25 → 26 entries, 247).

Voluntary-filing probe: what it now measures. With takes_up_aca_if_eligible: False (test_us_voluntary_filing.py:37), the test passes because aca_ptc drops from 11,600 to 0.

  • In 2.145.3 the credit a household receives is assigned_aca_ptc = aca_ptc * takes_up_aca_if_eligible, which feeds premium_tax_credit, spm_unit_benefits and healthcare_benefit_value.
  • files_return_for_aca_ptc is true for every unit that takes up a positive credit. So neutralizing would_file_taxes_voluntarily cannot change assigned_aca_ptc for any unit.
    • On this household it is 0 in both runs with take-up off.
    • With take-up on, it is 11,600 in both runs.
  • The probe (tools/build_us_release_input_coverage_manifest.py:1447) still shows that the filing leaf binds into the graph. But its margin is the pre-take-up credit of units that don't enroll, and the reason text says it "removes the credit from otherwise eligible voluntary filers".
  • The engine rule matches 36B(a) and 1.6011-8(a), so this is about the probe's meaning, not a bug. It would help to say in the reason that the margin is pre-take-up aca_ptc, or to move the probe to a measure the leaf still reaches. The runbook already lists the $100M floor as a risk (docs/us-national-build-runbook.md:208).

Branch hook (dc992cf): small test gap. The fix looks right for both clone behaviors:

  • On 3.32 the wrapper's owner is recorded.
  • On 3.36 it falls back to parent, and parent_original.__func__ binds correctly to the child.

The engine-free fake still clones with copy.copy (test_support/microcosm_build/us_batched_target_materialization.py:118), which is the 3.32 behavior. Neither the rebind-to-copy case nor the new RuntimeError is tested outside the engine-workflow lane. A fake clone that rebinds instance methods, plus one refusal test, would keep this from regressing silently.

Things that look consistent:

  • The other five contract changes match 2.145.3 and the law:
    • CA takes the 273.9(c)(17) exclusion.
    • The SALT refund is in federal gross income and SC subtracts it. No other probe needs the state_income_tax baseline, so 19 keys.
    • educator_expense_ald is the 62(a)(2)(D)-capped amount.
    • HUD gross rent includes the utility allowance (24 CFR 5.603).
    • The 162(l) limit nets the attributable half of SE tax (Form 7206).
  • The re-measured us_national gate-scope digests pass on the merged gates.json, whose only change against main is the tail-gate note.

engine-us was still queued at the time of this review.

Review drafted with AI assistance and checked against the code at 61c8fae.

juaristi22 and others added 4 commits November 18, 2026 12:45
…ntity

The policyengine-us 2.145.3 frame adapter keeps the static consumer index's
helper reviews in microcosm.frame.adapters._policyengine_us_helper_reviews,
which the adapter imports, so the gate bindings reach one more module.
The engine-free job's --maxfail=1 stopped before this test in the first CI
round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine-free fake can now copy a branch as core 3.36's clone does, binding
copied methods to the copy, and the branch tests run under both clone
behaviors. A simulation copied outside get_branch carries the wrapper
without being recorded; its get_branch now refuses with a clear error (a
KeyError before), and one helper serves that refusal and the parent lookup.
Review of #1676 (Vahid, second round).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe still shows the filing leaf binds into the graph, but on 2.145.3 its
margin is the pre-take-up credit (aca_ptc) of voluntary filers who do not take
up coverage: the credit a household receives, assigned_aca_ptc, does not
move. The probe's reason, the runbook risk and the scenario test now say so.
Review of #1676 (Vahid, second round).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine-free job stops at its first failure, so a full local run found
six more files on the 2.145.3 lock:
- the engine ABI fakes now follow US_TAKE_UP_PROGRAM_COUNT (21);
- the primary QRF target order holds 64 targets (the mortgage legs left);
- the SALT refund probe is positive on income_tax, and the smoke needs 19
  baselines;
- the release's location-attribute writer is a registered HDF write site
  (root attributes only);
- the legacy pool publication manifest re-pins its checkpoint identity
  (pool_h5 and agreement unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 3 commits November 18, 2026 17:45
Two conflicts. The release tool keeps #1672's end of the build and writes
the base's location attributes right after the written file's stored-input
check, before the build record binds the H5's sha256. Certification reads
those attributes back from the written H5 for its source coverage (the
auto-merged line used a name only the build defines). The Route A driver
tests keep both sides: Linux sizing and STOP_BEFORE_RELEASE beside the
certify flow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1672 split the release into a build that stops at ROUTE_A_BUILT.json and a
certification (route_a.sh --certify). Step 4 now builds, a new step 5
certifies and runs both preflights, the host table names the certify stage,
and section 7 states María's 18 November rulings on #1672, which a follow-up
brings before this build: certification may run at a later commit than the
build, and the build has no tier (certification chooses it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M1: the base runs locally for now. M2: the build and its certification run on
a laptop with enough memory, with the PUF inputs on it. M3: staging upload
on (ROUTE_A_STAGING=1). M4: #1365 closed as superseded by #1676. M5: María
decides publication and promotion and coordinates with Max.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 4 commits November 18, 2026 23:47
…o the engine refresh

main brings #1681 (rulings 1 and 3 on #1672: a build has no tier and
certification chooses it; certification may run at a descendant of the
build's commit that changes only tests, docs, changelog fragments and the
standing evidence owners register) and #1677 (the Modal base hand-off copies
the gate shadow directory). No textual conflicts.

Checked where the two sides meet in tools/build_us_fiscal_refresh_release.py:
the release still writes the base's household-location attributes after the
written-H5 stored-input check and before the build record binds the H5's
sha256, and certification still reads them back from the written H5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the runbook

Certification takes about two days, so the runbook now runs the build, then
probes it on a 5% household sample with tools/sample_us_export_households.py
and tools/probe_us_post_export.py (about 10 hours on the published export),
then waits for María's assessment. The operator certifies only when she
asks, at the tier she names.

- Step 5 runs certification's post-export checks on the sample. It leaves out
  the probe's QRF tail stage, which grades at final weights while the build
  decides that gate at base weights; certification does not run it.
- Step 6 lists what she reads (the build's verdicts, its fit, the gate
  shadow, the probe's failures) and the three answers: certify, build again,
  or certify as evidence.
- Section 7 describes #1681 as merged: the build refuses --evidence-release,
  and a later commit certifies only when it changes tests, docs, changelog
  fragments or the standing owners register, so a code fix needs a new build.
- S6 stops on a probe failure; S3 names the gate shadow's directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the merge of main, the release stage accepts exit 3 too (a build that
failed its release gates, kept for an evidence certification), so its
run_stage line names "0 $BUILT_FAILED_GATES_RC". The pin follows it; the
order it checks (the release config, then the stop, then the run) is the
same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max's review of #1681 at cfd5a2a, its four points and two of its smaller
ones (the Route A points follow with the driver):

1. The certified tier refuses a failed source-coverage gate before it writes
   either manifest, as it refuses a failed smoke, so the build can still be
   certified as evidence.
2. A failed build's H5 is a --base-h5 only beside an
   --export-input-mass-reference-h5: without one, the export input-mass gate
   judges the export against its base.
3. Certification adds every gate failure it finds to
   certification_failures.json beside the build record, at either tier, and
   a build that passed its own gates and then failed certification's is no
   reference until a certification ships it.
4. The build writes build_incomplete.json before its H5 and removes it once
   its record is written, so an H5 that no record pins is no reference.

A standing evidence owner pattern owns a failure line only from its start,
so "input signal failed:" no longer owns every gate ending in those words.
The allowed-path summaries now name the root conftest.py, LICENSE and
Markdown outside package sources, and a publisher test covers version 2
staging blocks in a separately certified build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor

Six open PRs merge cleanly with main and conflict with 5c37a801c in all four US pin files (us-f0-coverage.json, inventory_coverage.py, test_spec_engine_loader.py and test_us_multispine_pool_tool.py):

Fifteen more edit all four files and already conflict with main: Anthony's #1156 and #1683, David's #1085 and twelve agent PRs under my account, which also wait for this one.

Two more of your UK drafts merge cleanly with main and conflict with this head: #1663 in uk_spine.json, and #1687 in uv.lock, worker_identity.py, test_engine_core_pins.py and uk_spine.json. The qualification tests in #1687 ran on policyengine-core 3.34.10, and the lock here moves the UK side to 3.36.1.

I still owe a decision on #1676's from-scratch receipt under docs/us-release-build-rule.md, including whether the first build on main after the merge can count as it. I'll answer here.

juaristi22 and others added 4 commits November 19, 2026 19:09
The build is one command: route_a.sh dry-runs the release's gates on the
fresh base as a stage of its own before the release starts
(ROUTE_A_DRY_RUN=1, the default). Exit 0 or 2 lets the release start; 1, a
certain failure, stops the run in minutes. A dry run holds for the release
settings it checked, by a digest of the release command without its build
ID, so a restart with other settings dry-runs them again.

--probe probes an accepted build on a 5% household sample with
certification's own post-export checks, as two supervised stages under the
run directory, and stops; a failed build gets its fit and target surface
from its record, since it is no reference. --certify stays a step of its own.

From Max's review of #1681:

- check_commit checks the certification tool and parser on every run,
  whatever the tier, and check_flags.py refuses a source without the named
  parser even when no flag is passed.
- A rerun to build refuses release settings other than the accepted build's,
  which it would otherwise keep without a word, and --rebuild sets the
  accepted build and its probe aside, keeping them on disk, to build the
  release again on the same base. It refuses a certified build.

The runbook has one command per step (resolve, build, probe, certify) and
one command that collects a run's logs and records, without its data files,
to send for diagnosis; the README and the settings example follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UK spine graph-parity fixture conflicted. #1663 regenerated it for its
new UK stage outputs, and this branch for policyengine-core 3.36.1. The
merged fixture is main's, with its 56 policyengine-core entries at 3.36.1,
as tools/graph_uk_spine_fixture.py writes them on this branch; nothing else
in it differs from main's. No other file conflicted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first run stops after the quick check: STOP_BEFORE_RELEASE=1 runs the
base, its preflight and the release-gate dry run, each with the gate battery
shadow on (#1635), and stops before the release. Step 3 is one command that
reads it with #1635's pass rules: the dry run passed (exit 0 or 2), the
base's comparison is a finished run's with nothing flagged, and the dry run's
records preflight and assembled with no difference, shadow error or entry
of those phases unobserved. It prints QUICK CHECK CLEAN, and the operator
goes straight on to the build, or what it found, and the operator sends the
evidence and waits.

The evidence goes to María after the build and its probe, before
certification's two days; after the quick check only when it found
something; and at any stop. S3 names the three shadow comparisons. Both
commands were run as the runbook gives them, on dummy run trees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release build rule asks a pull request that changes how the dataset is
built for a from-scratch build and certification receipt from its own tree
before it merges. The runbook's settings now build at the full SHA María
names, #1676's head, with REQUIRE_ON_MAIN=0, so the pull request stays open
for fixes. Certification runs at that commit too; a later fix to code means
a new build at the new head, and the pull request merges with a merge commit,
so the built commit lands in main's history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

juaristi22 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Hi @PavelMakarchuk, could you run the next US national build on your laptop, from this PR's head? It's the first on policyengine-us 2.145.3, with block_v1 household location. The PR stays open, so the build is its receipt before it merges: please build at ba410438e6a6de4b79a7e19178147cd624a81f04, with COMMIT set to that SHA and REQUIRE_ON_MAIN=0 in route_a.env, and run the commands from a checkout of that commit (git fetch origin && git checkout --detach ba410438e in your MAIN checkout): the driver runs from the checkout it's started in, and main's has no quick-check stop or --probe. Everything is in docs/us-national-build-runbook.md at that commit. Sections 1 to 3 cover the host, inputs and settings: the base needs 80 GiB of free RAM to start, the PUF inputs stay on your laptop, and the SCF extract and the Chronicle feed need fetching once.

The run is one command per step (section 4):

  1. Resolve (minutes): bash tools/route_a/route_a.sh --resolve checks the inputs and the commit.
  2. Quick check (about 2 hours): STOP_BEFORE_RELEASE=1 bash tools/route_a/route_a.sh builds the base, runs its preflight and a dry run of the release's gates, then stops before the release.
  3. Read the quick check (minutes): step 3's command prints QUICK CHECK CLEAN or lists what it found. If it's clean, go straight on to step 4, with nothing to send. If it found something, send us the evidence (step 6) and wait.
  4. Build (about 7 to 9 hours): bash tools/route_a/route_a.sh keeps the base and the dry run and builds the release.
  5. Probe (about 10 hours): bash tools/route_a/route_a.sh --probe runs certification's checks on a 5% household sample.
  6. Send us the evidence: step 6's command packs the run's logs and records into one file and leaves every data file on your laptop.
  7. Wait for us. Certification takes about two days, so we'll read the build and the probe first and tell you whether to go ahead.
  8. Certify, when we say so (about two days): bash tools/route_a/route_a.sh --certify, then send the evidence once more.

If anything in section 5 happens (S1 to S6), stop, keep every file, and send us the evidence. Nothing is published at any step. Thank you!

(Edited on 20 November: the build commit is now ba410438e, this PR's last commit, after main was merged in and Max's reviews were addressed; the commands run from a checkout of it.)

@MaxGhenis

Copy link
Copy Markdown
Contributor

Anthony merged #1682 at 21:20Z (7ff5a26bc), a few minutes after your note to Pavel. Main now conflicts with d596fcb60 in five files:

  • the four US pin files (us-f0-coverage.json, inventory_coverage.py, test_spec_engine_loader.py and test_us_multispine_pool_tool.py)
  • us_runtime/worker_identity.py

Merging #1676 after a build at d596fcb60 would mean resolving those five by hand, and two of them hold source code. #1682 pins quantile-forest==1.4.2 and fills the QRF leaf array in place, so a head that merges main first would build with that change.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to d596fcb60. The four points from the #1681 review hold in ea810f35f, and so do the smaller ones. We read the code and ran nothing. We found nothing that a Route A build at this head reaches when it follows the runbook's order: build, probe, then certify at the certified tier.

  • Point 1. The certified tier records the coverage failures and raises before _build_manifests (tools/build_us_fiscal_refresh_release.py:19995-20013, :20101). The reuse refusal needs both manifests (:2654-2671), so the refused build can still go to the evidence tier.
  • Point 2. _parse_args refuses a failed build's H5 as --base-h5 without an export reference (:2279-2288).
  • Point 3. _failed_build_holding holds a build that has a certification register until the certified tier certifies it (:2948-2956).
  • Point 4. The build writes the marker before write_dataset and removes it after the record (:19455, :19461, :19610).

Gaps on later or hand-run paths:

  • After a failed certification of a build that passed its gates, --probe still passes the build's own directory as the probe's reference (tools/route_a/route_a.sh:593-602), and the probe now refuses it (tools/probe_us_post_export.py:3262-3271).
  • certification_failures.json carries no build id per attempt (:19651-19656) and nothing removes it, so a hand-run rebuild into the same --out inherits the old attempts. Route A takes a new --out for each release (route_a.sh:994-995).
  • The build writes calibration_diagnostics.json at :18914, before the marker. A build that stops in between leaves a file with no record and no marker, which a later --incumbent-diagnostics accepts.
  • test_certification_records_each_gate_failure_before_its_tier_decides compares source line numbers, so it would pass with the coverage raise outside if not args.evidence_release.
  • The rerun guard compares the release command (route_a.sh:567-576, :1041-1045). A QRF tail register edited in place with a new QRF_TAIL_EXCLUSIONS_SHA, or an export reference replaced in place, keeps the accepted build.

juaristi22 and others added 2 commits November 19, 2026 22:20
Five pin files conflicted, as Max listed: #1682's in-place QRF leaf array
and this branch's engine refresh each moved the same pins. Each takes this
branch's side, and the pins both sides moved are recomputed on the merged
tree:

- tools/repin_us_spec_engine_contracts.py (twice, so the inventory count is
  read with the new seed hashes in place) rewrote the seed map and seed
  protocol digests, the spec-engine loader's golden vector and the pool
  tool's spec_sha256; tools/spec_engine_coverage.py regenerated the F0
  coverage report (50,842 of 50,842 fields, 41 of 41 inventory checks), and
  both --check runs pass.
- APPROVED_UV_LOCK_SHA256 is the merged uv.lock's digest. The lock merged
  without a conflict (#1682 pins quantile-forest==1.4.2), and uv lock --check
  resolves it unchanged.

The US bundle generator needs policyengine-us, which the engine-free
environment lacks; #1666 and #1682 change none of its inputs, so the bundle
stands, and CI's engine lane checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max's review at d596fcb (issuecomment-7023661200) found nothing a Route A
build reaches in the runbook's order, and five gaps on later or hand-run
paths:

1. --probe after a failed certification passed the build's own directory as
   the probe's reference, which the probe refuses. A build whose
   certification found gates failing now gets the failed build's arguments,
   its fit and target surface, which the probe accepts of any build.
2. certification_failures.json named no build, and nothing removed it, so a
   hand-run rebuild into the same --out inherited the old attempts. Each
   attempt now names the build and its record's digest; only attempts on the
   current record count, and the next attempt drops the others.
3. The build wrote calibration_diagnostics.json before build_incomplete.json.
   The marker now goes down as the build creates its --out, before any file
   there; a dry run, which writes no --out, writes none.
4. The test of point 1 compared line numbers. It now reads the structure:
   each certified-tier refusal is the whole body of its tier branch, the
   statement before it records the failures, and the owner check follows.
5. The rerun guard compared only the release command. It also compares
   release-inputs.key, the digest of what the release read by path when it
   was launched (the base H5, the release's pinned inputs and the export
   input-mass reference), and the dry run's settings key carries it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

juaristi22 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

@PavelMakarchuk you can go ahead with the build at ba410438e, run from a checkout of that commit. The comment above now points to it, and the steps are the same. The commits since it was first posted (main's #1666 and #1682, merged in, and fixes from Max's reviews) shouldn't affect anything in the run. Thanks!

(Edited on 20 November: the build commit is now ba410438e.)

@MaxGhenis

Copy link
Copy Markdown
Contributor

Lines refer to bd4ff48a6. We read that commit against its parent and ran nothing. The five gaps close, and we found nothing in it that changes a first run of the runbook, apart from one new read of the export reference. A separate point concerns which copy of the driver runs.

Before the release starts, the driver now digests $EXPORT_MASS_REF_H5 with openssl for the rerun key (tools/route_a/route_a.sh:584-593, :1030), at steps 2 and 4. If the file exists and openssl cannot read it, the run stops there with "cannot digest the release's inputs", after the base. The pinned inputs already go through the same sha_of (:254), so only a reference on another volume would stop a run that got that far.

Section 4 says to run each command from the MAIN checkout (docs/us-national-build-runbook.md:140). The driver loads itself and its helpers from the directory it runs in (route_a.sh:38-42), and nothing compares that copy with COMMIT. Main's route_a.sh has no STOP_BEFORE_RELEASE and no --probe. From a checkout at main, step 2's command would therefore have no stop before the release, and step 5's would exit with a usage error. The checkout Pavel launches from needs #1676's tools/route_a/.

Smaller points, off the runbook's path:

  • The register's writer keeps only the attempts whose record_sha256 matches (tools/build_us_fiscal_refresh_release.py:19664-19669), while the reader and the driver also count attempts that name no record (:2856, route_a.sh:616).
  • No test runs the rerun guard's comparison (route_a.sh:1079-1085).

@MaxGhenis

Copy link
Copy Markdown
Contributor

The red ci check on bd4ff48a6 comes from a job limit, and no test failed. engine-us (3.14) passed all 717 tests with exit code 0. Its test step ended 109 minutes 59 seconds after the job started, and GitHub cancelled the job at the 110-minute limit while the closing steps ran. engine-us (3.13) passed in 105 minutes.

The suite's own wall time reads 6,552 seconds here, against 5,275 at 5c37a801c for the same 717 tests and 5,032 on main at 7ff5a26bc. All 32 shards ran slower than at 5c37a801c, by 11% to 57%, so no single test accounts for it. If the pace holds, main sees the same cancellation once this merges.

I re-ran the cancelled job.

juaristi22 and others added 3 commits November 20, 2026 12:19
…1585)

From Max's review of bd4ff48 (issuecomment-7025833584):

- The driver loads itself and its helpers from the checkout it runs in, and
  nothing compared that copy with COMMIT. main's route_a.sh has no
  STOP_BEFORE_RELEASE and no --probe, so the runbook's steps 2 and 5 need
  the build commit's driver. check_commit now refuses a driver whose files
  differ from tools/route_a at the build commit, so --resolve reports it at
  step 1, and the runbook says to check that commit out first.
- The rerun key hashes the export input-mass reference before the release.
  The input check now reads it too, with no pin to match, so a reference
  openssl cannot read stops the run before the base, not after it.
- The certification register's writer keeps what its readers count: this
  record's attempts and any that name no record.
- A test runs the rerun guard's comparison: the command, the inputs' key,
  and a run from before the key existed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test set RUN_ROOT to $(pwd)/root, so a run from the repository root
wrote inputs.resolved.json there. It now passes RUN_ROOT as a setting under
tmp_path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`local driver_differs= driver_file driver_files` declares an empty
driver_differs and two more locals, which is what the driver needs, but
shellcheck warns on the space after `=` (SC1007), so the engine-free
shellcheck test failed. Write the empty value as ''. No change to what the
driver does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@MaxGhenis @PavelMakarchuk thanks for the reviews. None of the notes since bd4ff48a6 change what a build does when it follows the runbook, so this PR takes no more commits: the remaining notes go into the PR that stacks on it for the local-area path.

@PavelMakarchuk, please build at ba410438e6a6de4b79a7e19178147cd624a81f04, this PR's last commit, with COMMIT set to that SHA and REQUIRE_ON_MAIN=0 in route_a.env. Run the commands from a checkout of that commit (git fetch origin && git checkout --detach ba410438e in your MAIN checkout): the driver runs from the checkout it's started in, and main's has no quick-check stop or --probe. The steps in the comment above are otherwise the same.

@MaxGhenis

Copy link
Copy Markdown
Contributor

I read the driver and release-tool changes in bd4ff48a6..ba410438e. Step 1 now refuses a driver that differs from tools/route_a/ at COMMIT, and it reads the export reference there, so an unreadable or missing file shows up before the base. The register's writer keeps the attempts its readers count, and I found nothing else that changes a run of the runbook.

All 12 checks pass at ba410438e, and the re-run of engine-us (3.14) at bd4ff48a6 passed in 97 minutes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants