Skip to content

Publish release builds into their own folders through the emitter, while UK releases stay on the pointer files - #1505

Merged
anth-volk merged 28 commits into
mainfrom
h2c-release-publication-handover
Nov 5, 2026
Merged

anth-volk merged 28 commits into
mainfrom
h2c-release-publication-handover

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1504

Hand-over step H2c, narrowed (tracker 1b · 3). Microcosm can publish a release build into its own folder through the separately installed emitter, while UK releases stay on today's tagged publisher and latest*.json pointer files until their repository's switch (tracker 2c). Each publication path refuses a repository in the other's layout, so the UK national, dense and local-area releases always keep one way to change their current build. That does not yet hold for the UK repository's other releases after its switch; see Open for the switch below.

Why it was narrowed

The first version moved the UK assemblers to the emitter and removed the old UK path. But the emitter refuses every change to a current build while any latest*.json file exists, so from its merge until the pointer files were deleted no UK build could have been made current, and UK Slack alerts would have stopped. Now UK releases keep publishing, promoting and alerting through the pointer files until one switch commit writes releases.json and deletes them (2c · 4); only then do the UK assemblers move to the emitter (2c · 2).

What this PR does

  • Publication into build folders (microcosm.build.release_publication, command microcosm-publish-release-build in microcosm-build).

    • It checks a UK national, dense or local-area build in the build-folder layout with the release contract.
    • It starts a short-lived emitter with only its Hugging Face module and hands every file over with its role, SHA-256 and size through the interface's publish_release call. The emitter's release writer writes releases/<release>/<build>/ and releases.json in one commit.
    • --make-current publishes the build as current, --promote makes a published build current, and --preflight-only checks the build without starting an emitter.
    • The command exits 0 only when the receipt is published and, when current was asked for, names the build as current.
    • Before any emitter starts, it refuses:
      • with HF_HUB_OFFLINE or TRANSFORMERS_OFFLINE set (staging_dataset.hub_offline);
      • while the repository has any latest*.json file (the interface's is_pointer_file rule), or when its files cannot be listed, httpx network errors included. It lists them with huggingface_hub in its own process, passing HF_TOKEN or else HUGGINGFACE_TOKEN, as the emitter reads them.
    • After the emitter starts, it hands nothing over unless the emitter's write-access check passes (staging_dataset.write_access_problem, which now takes the wording of the change it checks).
    • A tagged UK release directory, the only kind today's assemblers write, is refused before the listing (build_folder_identity), and the tagged publisher refuses a build-folder one (it pins no revision); tests cover both.
  • The tagged publisher (microcosm-publish-release, tools/publish_release.sh) refuses a repository that has releases.json. It checks before writing a tag, a commit or a pointer file, reading the file at the main revision its commit builds on. Only the Hub's 404 for the file (microcosm.data.release.missing_on_hub) lets it go on; huggingface_hub's LocalEntryNotFoundError, which it raises when it could not ask the Hub (a 5xx after its retries, a 403), and every other error stop it before anything is written. Otherwise it is as on main: it publishes the UK national, dense and local-area releases, promotes them with line= / --promote-line through latest-<line>.json, and sends their Slack alerts.

  • The release contract checks a UK release in the layout its release manifest uses:

    • the tagged layout today's UK assemblers write, with per-build tags and the dense release's attempt_id;
    • or the build-folder layout: release and build as strings in both manifests; built_at, built_with_model_package and built_with_core_package at the top level; a role on each artifact; and no revision.

    Every other release stays tagged.

  • The loader (microcosm.data.load, download), for a registry entry that names a release (UK 2025 national and dense):

    • It reads releases.json where the repository has one: the current build, or a listed build that build= pins (a withdrawn one loads with a warning, unchanged).
    • Where the repository has none, it follows the entry's pointer file exactly as before (latest-national.json for the national release, with its role check), so load("uk") no longer fails for want of releases.json.
    • Only the Hub's 404 for releases.json falls back to the pointer file (and, in offline mode, a releases.json never cached, as an offline load did before). Any other error propagates, a server error included. build= needs releases.json, and unverified_root=True is refused only once it exists.
  • The UK assemblers are main's: tagged manifests, --cut-tag, and tools/publish_release.sh commands.

  • Kept from review round 1: the up-front offline and write-access checks, and command tests for the pending, refused and unavailable receipts.

  • Undone from review round 1: the dense assembler's separate --promote command and the documentation for the pointer-file period. A POINTER_FILES_PRESENT refusal now points to tools/publish_release.sh, since the up-front refusal covers that case.

The split for the switch (tracker 2c · 2)

These commits take the UK assembler, UK contract, old-path removal and alerts-stop changes out. Tracker 2c · 2, stacked on this PR, re-applies them by reverting them, newest first:

Change taken out Tests commit Code commit
UK contract: tagged layout accepted for UK releases 666dfbd90 660873b3a
Old UK path and its Slack alerts kept: tagged publisher, line=, --promote-line, latest-<line>.json, tools/publish_release.sh, tools/release.env.example fc324c8a8 900a927a2
UK assemblers on tagged manifests 42bb85771 ea2310c13
Documentation of today's UK flow (README, agent guide, both runbooks, microcosm-data README) — 392deaaea

The code and test reverts apply without conflicts (review round 2 tried them). What else 2c · 2 has to do when it reverts them:

  • Keep dataset_role_for_line. 660873b3a brings it back, and the loader's pointer-file fallback (d7bce422f, kept) imports it. Without it import microcosm.data fails, because the package imports the loader, so every module and test of the package fails, not only the fallback. Keep the function, or remove the fallback in the same change.
  • Three tests need changes:
    • test_the_tagged_publisher_refuses_a_uk_promotion_once_releases_json_exists (from ae0e3271a) uses the helpers for UK latest-<line>.json promotion (_as_uk_national_line_release, _uk_hub) that the revert of fc324c8a8 removes; drop or rewrite it.
    • Round 1's test_an_assembled_national_build_publishes_through_the_emitter, which the revert of 42bb85771 restores, calls publish_release_build, which now lists the repository's files. Only test_uk_release_publication's fixture stubs that listing, so without a stub (_repository_files) the test would query the private UK repository: offline it fails with "could not list", and in CI's engine-free jobs it sends an anonymous request and fails on the 401.
    • test_the_tagged_publisher_refuses_a_uk_build_folder_release (from a7a039ec0) only asserts a ValueError and no Hub request, so it holds after the revert of 900a927a2, whose refusal by release name comes first.
  • Documentation. The revert of 392deaaea conflicts with 0592245b2 in the microcosm-data README, whose paragraphs both edit; take round 1's text there. 0592245b2's root README section and changelog fragment, and 651e94d99's note on the other UK releases, describe the time before the switch and outlive the reverts, so 2c · 2 rewrites them.
  • The refusal advice. The POINTER_FILES_PRESENT advice (d7bce422f) sends the operator to tools/publish_release.sh. After the switch that script refuses the repository and, by name, the three releases, so 2c · 2 changes the advice; the remedy then is deleting the stray pointer file. Round 1's separate advice for a promotion does not come back with the reverts.

Release alerts

UK and US release alerts do not stop. Until a repository's switch, its releases publish through tools/publish_release.sh and the tagged publisher sends their Slack alerts as today (SLACK_WEBHOOK_POPULACE_UK, SLACK_WEBHOOK_POPULACE_US). The switch (2c · 4) waits until the dashboard posts alerts from changes to releases.json (tracker 1b · 6). Publications through microcosm-publish-release-build send none of their own, and none can happen before the switch.

Open for the switch (tracker 2c)

After the UK switch, the UK repository's releases other than the three have no publication path, and load("uk", 2023) loses its pointer:

  • is_pointer_file matches every latest*.json, so the switch has to delete latest.json, the June 2023 release's pointer, as well as latest-national.json;
  • the ("uk", 2023) registry entry names no release and reads only latest.json;
  • after the switch the tagged publisher refuses the whole repository, and microcosm-publish-release-build refuses the June 2023 release and populace-uk-<year>-<tier>-k<N> IDs by name.

This needs a decision for 2c, and it is being raised with Anthony; this PR does not change the code for it.

Pending

  • US releases: the proposed stable releases (populace-us-2024-national, role national; populace-us-2024-local, role local_area) are recorded in microcosm.data.contract.PROPOSED_US_RELEASES and read by nothing until they are agreed with the US data builder (tracker 2c · 5).
  • A pinned withdrawn build loads with a warning, as before. Whether it should be refused instead is tracker 1b · 2, not decided.
  • File roles beyond dataset and calibration_diagnostics: calibration_weights for the calibration weights, and the artifact's key otherwise.

Environment variables

Variable Read by Provider How it reaches the reader
HF_TOKEN, then HUGGINGFACE_TOKEN, then the token hf auth login saved the emitter's release writer, started by microcosm-publish-release-build the operator's shell or hf auth login on the build machine the emitter is the command's child process (the interface's launcher, with no env) and inherits its environment
HF_TOKEN, then HUGGINGFACE_TOKEN, then huggingface_hub's own order (HUGGING_FACE_HUB_TOKEN, then the saved token under HF_TOKEN_PATH or HF_HOME) huggingface_hub in microcosm-publish-release-build's own process, listing the repository's files the same read in the command's process; the command passes HF_TOKEN or HUGGINGFACE_TOKEN to HfApi, since huggingface_hub does not read the latter
HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE staging_dataset.hub_offline in the command, which refuses when either is true; huggingface_hub in the command and the emitter the operator's shell, unset normally read in the command's process; the emitter inherits them
HF_ENDPOINT (optional) huggingface_hub in the command and the emitter the operator's shell the same
MICROCOSM_EMITTER_COMMAND (optional) the interface's launcher the operator's shell read in the command's process; otherwise PATH
XDG_CACHE_HOME (optional) the interface's queue-path helper (the emitter's hand-over database) the operator's shell read in the command's process and passed to the emitter
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY, POPULACE_UK_TERMINAL_GATE_SIGNING_KEY the UK release contract, run by the assemblers, both publishers and --preflight-only tools/release.env on the build machine, or the operator's shell tools/publish_release.sh sources tools/release.env; for microcosm-publish-release-build, the operator exports them
SLACK_WEBHOOK_POPULACE_US, SLACK_WEBHOOK_POPULACE_UK notify_release, from the tagged publisher tools/release.env or the operator's shell unchanged from main: the wrapper sources the file and warns when neither is set
HF_TOKEN (loading) huggingface_hub in microcosm.data.load for the private UK repository the consumer's environment unchanged

No workflow, Docker, Terraform or Modal configuration changes, and no secret is added. PR CI receives no credential. Its tests use in-memory fakes for the emitter, the Hub file list and the Hub downloads, which raise huggingface_hub's own errors (test_support/microcosm_data/hub_errors.py), and the shared fixture refuses to start a real emitter. As on main, the live-Hub loader test reaches the Hub anonymously from CI's engine-free jobs; the UK cases skip on the 401.

Tests

Tests first, then the change. Locally I ran only the tests these commits add or modify, each by node ID, from an explicit list that the run script refuses when empty. I ran them with HF_HUB_OFFLINE=1 and without the live-Hub loader test. CI runs the rest.

Commit Run Result
aef137600 merge of main (6874a31cb) — no conflicts
325c0e278 tests for review round 1 (offline and write-access checks, receipt outcomes) 8 test functions 9 failed on assertions, 5 passed (behaviour that already held)
857e7cdde, 4b3f1b36c the review round 1 fixes the same 14 passed
666dfbd90 contract tests (tagged UK layout) 81 node IDs, 179 cases 29 failed (22 ReleaseContractError from the contract refusing the tagged UK layout, 6 assertions, 1 AttributeError for _UK_DENSE_CUT_TAG_RE, which main's mirror test reads), 150 passed
660873b3a the same 179 passed
fc324c8a8 old UK path tests 33 node IDs, 53 cases 16 failed (10 TypeError for the removed line= keyword and 1 SystemExit for the removed --promote-line, the options being restored; 4 assertions; 1 ValueError from the publisher refusing the UK release), 37 passed
900a927a2 the same 53 passed
42bb85771 assembler tests 49 node IDs, 66 cases 4 failed (2 KeyError: 'cut_tag', 2 assertions), 62 passed
ea2310c13 the same 66 passed
ae0e3271a refusals and loader fallback 41 node IDs, 75 cases 25 failed (11 assertions, 4 DID NOT RAISE, 1 DID NOT WARN, 9 FileNotFoundError from the loader reading the missing releases.json without falling back), 50 passed; none on a missing name
d7bce422f the same 75 passed
0592245b2 all 203 node IDs above 372 passed
a06f3d83b empty commit (re-push after GitHub's incident) — —
6ab56bb13 merge of main (0f0e777b2) — no conflicts
a7a039ec0 tests for review round 2 (only the Hub's 404 is absence; the listing's errors and credential; the two layout crossings), with the fake Hubs now raising huggingface_hub's 404 error 149 node IDs, 239 cases: the new tests and every test function of the changed test modules that uses a fake Hub 9 failed (6 DID NOT RAISE for the tagged publisher and the loader, 2 assertions for the credential, 1 httpx.ConnectError raised from the command), 230 passed. Passing already: the offline fallback and both layout crossings, which pin behaviour that held
9f3367cbe the fix the same 239 passed
651e94d99 (head) README note on the other UK releases — —

ruff check and ruff format --check pass on the changed Python files, and python3 tools/ci_test_plan.py verify passes.

🤖 Generated with Claude Code

anth-volk and others added 23 commits November 4, 2026 01:13
Failing tests first for hand-over step H2c. UK national, dense and
local-area release manifests name `release` and `build` as strings,
record `built_at` and the built-with packages at the top level, and give
each artifact a `role` without a `revision`; the build module hands a
build to the emitter's `publish_release` and treats it as current only
when a published receipt names it; the loader reads `releases.json`; the
tagged publisher refuses UK releases and offers no line promotion. The
fake emitter client applies release changes with the interface's shared
checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hand-over step H2c. The UK national, dense and local-area releases hand
each build to the separately installed emitter with the interface's
`publish_release` call (`microcosm-publish-release-build`, in
`microcosm.build.release_publication`). The emitter's release writer
puts it in `releases/<release>/<build>/` and records it in
`releases.json`; Microcosm no longer commits, tags or writes pointer
files for them, and treats a build as current only when the receipt is
`published` and names it as `current`.

Both manifests name `release` and `build` (the calibration attempt) as
strings; the release manifest records `built_at` and the built-with
packages at its top level and a `role` per artifact, with no
`revision`. The contract checks that layout for UK releases, the
assemblers write it and print the new commands, and `microcosm-data`'s
loader reads each UK release's current build from `releases.json`. US
releases keep tags and `latest.json`; the tagged publisher refuses UK
releases and loses line promotion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stable releases proposed for US builds are not agreed with the US
data builder, so the contract should record them, with their
`releases.json` roles, in one place while no US release publishes into
build folders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`PROPOSED_US_RELEASES` names `populace-us-2024-national` (role
`national`, the default) and `populace-us-2024-acs-local` (role
`local_area`), the stable releases proposed for US builds. They are not
agreed with the US data builder, so `publishes_to_build_folder` names no
US release and US releases keep tags and `latest.json`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…names

`RELEASE_ROLES` and `PROPOSED_US_RELEASES` are read-only mappings, as the
shared constants guidance asks, and the contract's `__all__` lists the
build-folder names other modules import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proposal now names one national and one local-area release in the
one `populace-us` repository: `populace-us-2024-national` and
`populace-us-2024-local`, still awaiting the agreement with the US data
builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`PROPOSED_US_RELEASES` follows the adjusted proposal: one
`policyengine/populace-us` repository with `populace-us-2024-national`
(role `national`) and `populace-us-2024-local` (role `local_area`).
Build IDs and the `populace_us_2024_acs_local.h5` file name are
unchanged, and nothing reads the names until the US data builder agrees
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's engine-free jobs failed `test_assembler_stages_a_graph_built_dense_candidate`:
the synthetic graph records engine versions as `unavailable`, and the
build-folder layout's new compatibility-list check refused the dense
release's `==unavailable` specifiers, which the local-area contract
never checked. This test holds the local-area contract to its old scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build-folder layout moved `built_at` and the built-with packages to
the release manifest's top level. The national contract still requires
the compatibility lists to include the built-with versions; the
local-area contract, as before, checks only that the built-with
packages are there, so a graph-built dense candidate whose engines read
`unavailable` assembles again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the pointer files go

Review round 1 of the build-folder publication. These tests fail on
assertions until the fix:

- the dense assembler prints a publication without --make-current and a
  separate --promote command, since the emitter refuses every change to a
  current build while the latest*.json pointer files exist;
- a --promote refused for those pointer files says to promote again once
  they are deleted, not to publish without --make-current;
- with HF_HUB_OFFLINE or TRANSFORMERS_OFFLINE set, the command refuses
  before starting an emitter;
- the command runs the emitter's write-access check and hands nothing over
  when the credential is missing or read-only.

They also add command tests for the pending, refused and unavailable
receipts, and a test that publishes an assembled UK national directory
with the release contract itself; those pin behaviour that already holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t a dense promotion

The publication command now makes the two checks the staged-bundle
hand-over makes before it hands anything to the emitter:

- with HF_HUB_OFFLINE or TRANSFORMERS_OFFLINE set it refuses before
  starting an emitter (staging_dataset.hub_offline), since the emitter
  inherits the variable and the change would stay pending until it
  expired an hour later;
- it runs the emitter's write-access check (staging_dataset's
  write_access_problem, which now takes the wording of the change it
  checks for) and hands nothing over when the emitter is missing or
  outdated, or the credential is missing, read-only or cannot reach the
  repository.

While the repository keeps its latest*.json pointer files the emitter
refuses every change to a current build, so the dense assembler prints a
publication without --make-current and a separate --promote command, as
the national assembler does. A promotion refused for those files says the
build stays published and to promote it again once they are deleted,
instead of advising a publication without --make-current.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…their alerts return

Only the UK national, dense and local-area releases publish into build
folders. The June 2023 UK release and populace-uk-<year>-<tier>-k<N>
releases still publish with tags, latest.json and their Slack alert, so
the README, the agent guide, the data package's README and docstrings,
tools/publish_release.sh and the changelog fragment now say so, and the
wrapper and tools/release.env.example keep SLACK_WEBHOOK_POPULACE_UK for
them. Publications into build folders send no alert until the dashboard
posts one from releases.json.

The runbooks describe the up-front refusals, the unavailable receipt and,
for the dense release, listing each build first and promoting it once the
pointer files are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UK releases keep publishing through the tagged publisher and the pointer
files until their repository switches to releases.json, so the contract
must accept the national, dense and local-area releases in the tagged
layout again. These are main's tests for that layout, beside the
build-folder tests, which now build their fixtures in that layout
explicitly. The test that a UK release refuses the tagged layout moves to
the switch.

Part of the split for the switch: reverting this commit and the next one
re-applies the contract's build-folder-only rule for UK releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The contract checks a UK national, dense or local-area release in the
layout its release manifest uses: the tagged layout today's assemblers
write, with its per-build tags and the dense release's attempt_id, or the
build-folder layout the emitter's publication uses. Every other release
stays tagged. dataset_role_for_line and the dense per-build tag pattern,
which the tagged path reads, come back.

Part of the split for the switch: reverting this commit and the one before
re-applies the build-folder-only rule for UK releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er files

Until the switch, the tagged publisher publishes the UK national, dense
and local-area releases again, and promotes them through latest-<line>.json
with line= and --promote-line, sending the release alert. These are main's
tests for that path, beside the build-folder preparation tests.

Part of the split for the switch: reverting this commit and the next one
re-applies the removal of the old UK path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tagged publisher accepts the UK national, dense and local-area
releases again, and line= / --promote-line promote them through
latest-<line>.json with the release alert, so UK Slack alerts keep being
sent. tools/publish_release.sh and tools/release.env.example are main's
again, with SLACK_WEBHOOK_POPULACE_UK. The build-folder preparation
(prepare_build_folder_release) and the shared publication refusals stay.

Part of the split for the switch: reverting this commit and the one before
re-applies the removal of the old UK path and its alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Until the switch the UK national and dense assemblers keep writing the
tagged manifests (build_id, attempt_id, cut_tag and per-build revisions)
and printing tools/publish_release.sh commands. These are main's assembler
tests again; the build-folder assembler tests, and the test that publishes
an assembled national build through the emitter, move to the switch.

Part of the split for the switch: reverting this commit and the next one
re-applies the assemblers' move to the emitter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UK national and dense assemblers are main's again: they write the
tagged manifests, take --cut-tag, and print tools/publish_release.sh
commands, so a UK build is published and promoted through the pointer
files as today. This also takes out the dense assembler's separate
--promote command added in review round 1.

Part of the split for the switch: reverting this commit and the one before
re-applies the assemblers' move to the emitter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README's release section, the agent guide, the UK national and dense
runbooks and microcosm-data's README are main's again: UK builds are
assembled with tags, published and promoted through latest*.json pointer
files with tools/publish_release.sh, and announced in Slack, until their
repository switches to releases.json. This also takes out review round 1's
wording for that period, which a narrower plan replaces.

Part of the split for the switch: reverting this commit re-applies the
documentation of the UK flow through the emitter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
So that no UK release is ever left without a way to change its current
build, each publication path refuses a repository in the other's layout:

- the tagged publisher refuses a repository that has releases.json, before
  it writes a tag, a commit or a pointer file;
- microcosm-publish-release-build refuses a repository that still has any
  latest*.json file, or whose files it cannot list, before any emitter
  starts.

For a registry entry that names a release, the loader reads releases.json
where the repository has one and otherwise follows its pointer file as
before (main's line-pointer tests come back for that path), so load("uk")
no longer fails for want of releases.json; only the file's absence, not any
other error, falls back. The root-file escape hatch is refused only once
releases.json exists. A POINTER_FILES_PRESENT refusal now points to
tools/publish_release.sh, and review round 1's separate promotion advice
goes, since the up-front refusal covers that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… files

No UK release can now reach a state where no build can be made current:

- the tagged publisher reads releases.json at the main revision it commits
  on, and refuses a repository that has one before it writes anything;
- microcosm-publish-release-build lists the repository's files with
  huggingface_hub and refuses, before the build is checked or any emitter
  starts, while any latest*.json file is there (the interface's
  is_pointer_file rule) or when the files cannot be listed.

The loader reads releases.json for a registry entry that names a release
only where the repository has one; otherwise it follows the entry's
pointer file as before, latest-national.json for the UK national release,
with main's line-pointer reading and role check. Only the file's absence
falls back. build= needs releases.json, and unverified_root is refused only
once it exists. A POINTER_FILES_PRESENT refusal now points to
tools/publish_release.sh, and review round 1's separate promotion advice
is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y's switch

The README and microcosm-data's README describe the build-folder command
beside today's UK flow: it refuses a repository that still has latest*.json
pointer files, the tagged publisher refuses one that has releases.json, and
UK releases keep publishing, promoting and sending Slack alerts through
the pointer files until their repository's switch, which waits until the
dashboard posts alerts from releases.json. The loader reads releases.json
where a repository has one and the pointer files otherwise. The changelog
fragment describes the narrowed change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk changed the title Publish UK release builds into their own folders through the emitter Publish release builds into their own folders through the emitter, while UK releases stay on the pointer files Nov 4, 2026
anth-volk and others added 5 commits November 4, 2026 15:50
…11-04 incident

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 found that the tagged publisher's releases.json check, and
the loader's, treat huggingface_hub's LocalEntryNotFoundError as absence.
huggingface_hub 1.18 raises it, as an EntryNotFoundError and a
FileNotFoundError, when it could not ask the Hub: after a 5xx outlasted
its retries, or on a 403. The fake Hubs now raise the Hub's own 404 error
for a missing file, and these tests fail until the fix:

- the tagged publisher refuses, writing nothing, when the read fails that
  way (no answer, 503, 403);
- the loader does not fall back to the pointer file then, online;
- the new command refuses an httpx network error while listing the
  repository, instead of a traceback;
- its listing passes the credential the emitter reads, HF_TOKEN then
  HUGGINGFACE_TOKEN.

Tests that already pass pin behaviour that holds: an offline load still
falls back when releases.json was never cached, and each publisher refuses
the other's manifest (a UK build-folder build through the tagged publisher,
a tagged UK build through the new command).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tagged publisher's releases.json check and the loader's now share
microcosm.data.release.missing_on_hub: a download failure means the file
is absent only when it is the Hub's EntryNotFoundError for a 404, not
LocalEntryNotFoundError, which huggingface_hub raises when it could not
ask the Hub (a 5xx after its retries, a 403) and nothing is cached. So
during an outage the tagged publisher refuses before writing anything,
instead of possibly writing a tag and a pointer file beside a
releases.json, and the loader does not fall back to the pointer file. In
offline mode, a releases.json that was never cached still reads as absent
for the loader, as an offline load did before.

The new command's listing turns httpx transport errors, such as a refused
connection, into its "could not list" refusal, and passes HF_TOKEN, then
HUGGINGFACE_TOKEN, as the emitter reads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each publisher refusing the other's layout keeps a path for the UK
national, dense and local-area releases. After the UK repository's switch,
its June 2023 release (pointer latest.json) and populace-uk-<year>-<tier>-k<N>
releases have none yet; the README says that is open for the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 5, 2026 13:41
@anth-volk
anth-volk merged commit f44512d into main Nov 5, 2026
11 checks passed
@anth-volk
anth-volk deleted the h2c-release-publication-handover branch November 5, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant