Skip to content

Name the US calibration arms by builder flag in pinned and generated text - #1641

Merged
MaxGhenis merged 8 commits into
mainfrom
stale-calibration-wording-1540
Nov 17, 2026
Merged

MaxGhenis merged 8 commits into
mainfrom
stale-calibration-wording-1540

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

microcosm#1540 aligned DESIGN.md, the US release build rule and the US builder's help text with the code and the release manifests. It left four pieces of wording because changing them moves hash pins, a test-pinned string or a generated contract. This PR changes them and regenerates every pin and generated file with the repo's tools. It also corrects what docs/us-release-build-rule.md says about where microcosm.build.holdout is used. No flag, default, calibration, draw, RNG rule or dataset changes.

What the code does

  • The US builder (tools/build_us_fiscal_refresh_release.py) has three arms. --exact-k runs calibrate_exact_k_ladder. --dense-default-dataset makes one calibrate() call with no L0 term and records dataset.default.method: dense_no_l0. With neither flag, it runs calibrate_l0_refit and records l0_refit. All three reach the same _enforce_ssi_take_up_delivery call in _main. That call passes US_DENSE_SSI_TAKE_UP_ENFORCEMENT_FENCES only when args.dense_default_dataset is set, so a dense run whose selection source first cuts the pool is fenced too.
  • The fence's evidence came from dense runs on two whole-pool frames, P2 and P3. The P2 receipts in microcosm#507 describe "the dense frame (337,704 households, selection disabled, all retained)". microcosm#567's P3 receipts show the same recompute behaviour and attribute it, on both frames, to the full-pool solve's equilibrium.
  • calibrate_l0_refit returns an L0RefitResult whose frame and weights are the refit stage's; its summary properties delegate to the refit.
  • _fiscal_release_cli_defaults in tools/us_bundle_generation/contracts.py reads the contract's solver defaults (epochs, learning rate, weight cap, L0 and L2 settings, seed) by parsing the builder with only its required arguments, so neither arm flag is set. The contract's solver.kernel: kernel:calibrate_l0_refit and l0_contract.default_mode: fixed_l0_then_refit are set by hand, and they match that parse, which is the L0+refit arm.
  • Only the generator, calibration.schema.json and the generated calibration.yaml name the release modes. No Python reads a renamed mode or refit-contract key; test_us_spec_bundle.py counts refit_contracts' entries, and there are still two. The legacy adapter projects the whole calibration document into calibration_contract, and the repinner and coverage tool hash it, so the rename moves those pins and nothing else.
  • Among the packages and tools, microcosm.build.holdout's rotated folds run only in uk_runtime/local_rowwise.py; the ACS local-area experiment experiments/us-acs-local-l2-basis-20260928/prepare.py also uses them. US package and tool code imports only the module's keyed-hash helpers. tools/us_acs_local_cd_surface.py and tools/us_acs_local_county_surface.py hold district and county SOI target blocks out of the ACS local-area solve, and score_cd_holdout and score_county_holdout mark their results report_only. build_us_acs_local_release.py's cd_holdout gate is always passed: True. us_runtime/support_mix.py uses the helpers to split targets and to order ACS households for the support-mix bake-off. The national builder imports neither helper. Its three *_holdout.json writers turn a failed measurement into an "unavailable" report, and no gate reads the reports.

Changes

  1. calibrate_l0_refit (solve.py) now says what it does: a "generate big, then prune" solve whose result is the refit stage's frame and weights. It no longer calls itself "the production … path". The rest of the module's wording about the refit stage follows: L0RefitResult no longer calls the refit weights "the production artifact", its effective_sample_size and options docstrings say "the returned vector" and "refit options" where they said "the shipped vector" and "production options", the module docstring's L2 note says calibrate_l0_refit returns the refit stage's weights where it said they "ship", calibrate_l0_refit and refit_l0_selection say "returned weights" where they said "shipped weights", and a comment in _optimize_proximal says the returned weights, not "the shipped dataset", agree with the reported n_nonzero. The module's pins move either way. The comments and docstrings in test_solve.py that said the refit weights ship or are published take the same wording.
  2. The us_ssi_take_up_delivery_gate docstring (us_runtime/ssi_take_up.py) names the arms by flag where it said "the sparse certified default" and "the dense full-pool arm". The builder passes fences on --dense-default-dataset only, keyed on the flag alone. The L0+refit arm (neither flag) and --exact-k pass none. It calls P2 and P3 "that arm's two observed full-pool frames" where it said "either observed frame".
  3. _US_DENSE_SSI_FENCE_ADJUDICATION, which builds write into their manifests, now opens "Fenced for the --dense-default-dataset arm", calls P2 and P3 "either observed full-pool frame", and ends "The L0+refit and --exact-k arms pass no fences and keep hard enforcement." Its recompute figures are unchanged. test_us_fiscal_refresh_builder.py now requires the opening and closing phrases and refuses "diagnostic arm" and "sparse certified". The fixture fence strings in test_us_ssi_take_up.py follow the new wording.
  4. The US calibration contract's release modes take the dataset.default.method values the builder writes: default_sparse_l0_refit → l0_refit and dense_diagnostic → dense_no_l0. refit_contracts.default_sparse → refit_contracts.l0_refit, with the matching const reference. exact_k_or_full_pool_refit keeps its name. The edits are in tools/us_bundle_generation/contracts.py and the hand-authored calibration.schema.json, and the generator wrote calibration.yaml. solver.kernel stays kernel:calibrate_l0_refit, and l0_contract.default_mode stays fixed_l0_then_refit. Both are set by hand to match the no-flag parse above, and a comment beside release_modes now says so. It also notes that solver.l0.mode keeps the approved binding literal exact_k_projection, which neither the hand-set fields nor the parse sets.
  5. docs/us-release-build-rule.md no longer says microcosm.build.holdout "is imported by the UK runtime and UK tools only". It says which code uses which part, as above. It keeps the paragraph's point in a narrower form: no held-out test compares a US national candidate with the incumbent.
  6. changelog.d/calibration-arm-wording.fixed.md records the change.

Pins and generated files

The values below are against main at 17fecf5e. While this PR was open, microcosm#1619 (for #1387), #1614, #1543 and #1617 changed attested modules or the same pins, so the seed, loader, multispine and pointer-inventory pins and the coverage report conflicted. Each time I merged main, took main's side in the pin files, kept both PRs' comments beside each pin, and re-ran the whole sequence below on the merged tree after the last edit to an attested module; the tools wrote every value.

Every value was written by its tool, none by hand, on Python 3.14 (arm64/darwin) with numpy 2.4.6, pandas 3.0.3, scipy 1.17.1 and torch 2.12.0, the versions pins.json records. The sequence follows the one tools/repin_us_spec_engine_contracts.py documents, with a second repinner pass, a second coverage pass and the graph-parity repin.

  • tools/repin_us_spec_engine_contracts.py --legacy-only, then tools/generate_us_bundle_from_constants.py (rewrote calibration.yaml; spec_sha256 4e2b6c19 → 372858d0), then the repinner twice:
    • inventory_coverage.py EXPECTED_HASHES: seed_map 6474719c → d756dc67, seed_protocol d3a0baf8 → 2c57cbfb. Both moved because solve.py and ssi_take_up.py are in _DIRECT_KERNEL_MODULES.
    • field_usage.py _PINS["calibration"]: still 320 pointers, digest 49a153b0 → 5336f77a (the renamed keys).
    • tools/spec_engine_coverage.py EXPECTED_POINTER_INVENTORY_SHA256: 7765ab26 → 65c75e48.
    • test_spec_engine_calibration_semantics.py CANONICAL_LEGACY_PROJECTION_SHA256: b289d8a5 → 025ed103.
    • test_spec_engine_loader.py golden: d4fc68aa → cb9742d5.
    • test_us_multispine_pool_tool.py spec_sha256: 4e2b6c19 → 372858d0.
    • test_spec_engine_coverage_tool.py: the first pass moved its count and the second restored it, so it is unchanged.
  • tools/spec_engine_coverage.py, run twice: ten digest lines in docs/evidence/spec-engine/us-f0-coverage.json.
  • tools/graph_parity_repin.py calibrate: pins.json implementation_hash 07e4330f → 3df328c2, node_key 2972c813 → 98df2301 (arm64/darwin/py3.14). direct.csv, graph.json and inputs.csv are unchanged.
  • Final checks: generate_us_bundle_from_constants.py --check, repin_us_spec_engine_contracts.py --check and spec_engine_coverage.py --check all exit 0, and tools/ci_test_plan.py verify reports ok. Coverage reports 42401/42401 configuration fields and 41/41 inventory checks.

The comments that record why each pin moved, beside EXPECTED_HASHES, the loader golden and the multispine spec_sha256, now cite this PR. They are comments, so the three --check runs above still pass with them.

ruff format changed only the quotes the repinner writes. It also rewrapped two unrelated statements in ssi_take_up.py; I reverted those so that the module's bytes match its pins.

What this PR leaves as written

  • changelog.d/dense-takeup-enforcement.changed.md still says "Dense diagnostic arm" and "the sparse certified default", and experiments/build_j_recert/issue368_verdict.md and PROGRESS_BUILDJ.md still say "dense diagnostic arm". They record what was true when they were written, so this PR leaves them.
  • Earlier build manifests keep the fence text they were written with. Only new builds write the new text.
  • Other text still calls the L0+refit arm "the sparse default" or "the certified default". That covers the builder's --selection-source-h5 and --selection-source-manifest help (tools/build_us_fiscal_refresh_release.py, around lines 1668 and 1679), three builder comments (around lines 16514, 16575 and 16882), us_runtime/warm_start_selection.py around line 579, docs/us-esi-employer-premiums.md and docs/informed-l0-warm-start-design.md. microcosm#1540's table shows Builds I, J and M–P ran the dense arm on a selected 57,240-household cut, so the selection-source help is misleading: it implies an L0 arm, while those builds ran --dense-default-dataset on a selected cut. None of it was on Align DESIGN.md and US builder help with how releases calibrate #1540's list, and this PR leaves it for a separate change.

axiom: n/a: wording in docs, docstrings, test strings, the builder's manifest fence text and a generated contract, with regenerated pins

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits November 16, 2026 08:03
…text

microcosm#1540 left five pieces of wording because changing them moves
hash pins, a test-pinned string or a generated contract. This changes
them and regenerates every pin with the repo's tools.

- solve.py: calibrate_l0_refit says what it does instead of calling
  itself the production path; L0RefitResult, calibrate_l0_refit and
  refit_l0_selection drop "production artifact" and "shipped weights".
- ssi_take_up.py: the delivery gate docstring names the arms by flag.
  The builder fences only --dense-default-dataset, keyed on the flag
  alone; the L0+refit arm and --exact-k keep hard enforcement.
- build_us_fiscal_refresh_release.py: the dense SSI fence text that
  builds write into manifests names the arms the same way, and its test
  requires the flag names and refuses the retired labels.
- US calibration contract: release modes take the builder's
  dataset.default.method values (l0_refit, dense_no_l0), and
  refit_contracts.default_sparse becomes l0_refit. Nothing outside the
  generator, the schema and the generated YAML reads the names.
- us-release-build-rule.md: says which code uses microcosm.build.holdout
  and that no held-out test compares a US national candidate with the
  incumbent.

Every pin below was written by its tool, none by hand:
repin_us_spec_engine_contracts.py (two passes, --check rc 0),
generate_us_bundle_from_constants.py (--check rc 0),
spec_engine_coverage.py (two passes, --check rc 0; 42395/42395 fields,
41/41 inventory checks), graph_parity_repin.py calibrate (pins.json
only; direct.csv unchanged). ruff format changed only the quotes the
repinner writes.

No flag, default, calibration, draw, RNG rule or dataset changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comments beside EXPECTED_HASHES, the loader golden and the
multispine spec_sha256 list each change that moved them. This adds
microcosm#1641: the solve and ssi_take_up docstrings and the calibration
contract's renamed modes. Comments only; the repinner, coverage and
generator --check runs pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1387 changed acs_transfer.py, another attested module, so the
seed, loader, multispine and coverage-report pins conflicted. Both PRs'
comments stay beside each pin; the repinner, bundle generator, coverage
tool and graph-parity repin rewrote every value, and the three --check
runs exit 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
solve.py's module docstring, L0RefitResult.effective_sample_size and
L0RefitResult.options still said the refit weights "ship" or called them
"production"; they now say what calibrate_l0_refit returns, and three
test_solve.py comments follow. The build rule's holdout paragraph now
scopes "only the UK runtime runs the rotated folds" to the packages and
tools, names the ACS local-area experiment that also uses them, and says
support_mix also orders ACS households with the hash helpers. The
contract generator's release_modes comment no longer says the kernel
follows from _fiscal_release_cli_defaults: the kernel and default mode
are set by hand to match that parse.

solve.py's bytes moved again, so the repinner, bundle generator,
coverage tool and graph-parity repin rewrote the seed, loader,
multispine, coverage-report and calibrate pins; the three --check runs
exit 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 16, 2026 19:39
microcosm#1614 and #1543 moved the same pins as this PR, so the seed,
loader, multispine, pointer-inventory and coverage-report pins
conflicted. Main's side was taken in each pin file and both PRs'
comments kept; the repinner, bundle generator, coverage tool and
graph-parity repin then rewrote every value. The three --check runs
exit 0 and tools/ci_test_plan.py verify reports ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SSI gate docstring now calls P2 and P3 that arm's two observed
full-pool frames: microcosm#567 records two attempts on each. A comment
in _optimize_proximal and the last test_solve docstring that said the
refit weights ship or are published now say returned weights. The
release_modes comment notes that solver.l0.mode keeps the approved
binding literal exact_k_projection.

solve and ssi_take_up are attested modules, so the repinner, coverage
tool and graph-parity repin rewrote the seed, loader, multispine,
coverage-report and calibrate pins. The three --check runs exit 0 and
tools/ci_test_plan.py verify reports ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1617 moved the seed, loader and multispine pins and the
coverage report that this PR also moves. Main's side was taken in each
pin file and both PRs' comments kept; the repinner, bundle generator,
coverage tool and graph-parity repin then rewrote every value. The three
--check runs exit 0 and tools/ci_test_plan.py verify reports ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1650 edited docs/us-release-build-rule.md in other paragraphs
(a sentence on Route A's builds and a 16 November 2026 block before
section 1); git merged it cleanly around this PR's holdout paragraph.
It touches no attested module or spec input, so no pin moves; the three
--check runs and tools/ci_test_plan.py verify confirm it on the merged
tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

MaxGhenis commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Hand-off to the Microcosm hub to land on green CI.

@MaxGhenis
MaxGhenis merged commit 0036b95 into main Nov 17, 2026
12 checks passed
juaristi22 added a commit that referenced this pull request Nov 17, 2026
main's 124 commits since 9504278 conflicted only in the six F0 pin
files. They were resolved to main's side and re-pinned: repin
--legacy-only (no change), the generator with the US engine (no bundle
file changed), the full repin, spec_engine_coverage.py, then ruff format
on the five Python files it rewrites. Coverage reads 42404/42404
configuration fields and 41/41 inventory checks, and every --check
passes. spec_sha256 is now 0088d3f6; the pool test's comment keeps
main's three lines (#922, #1543, #1641) and this branch's us/gates.json
line.

main's microcosm#922 county work merged cleanly into the ACS local tool:
county support, its re-applied moves, the resume shortcut's evidence
check and do_package's county CSV all sit outside this branch's shadow
sites, which keep their call shapes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant