Skip to content

Register raw microdata releases as source artifacts: licence/access fields, hash-only registration, first UK registrations (#221) - #227

Open
MaxGhenis wants to merge 282 commits into
mainfrom
microdata-registration
Open

MaxGhenis wants to merge 282 commits into
mainfrom
microdata-registration

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements chronicle#221: Chronicle registers every raw microdata release its
consumers build from, and stores the content of none of them.

Depends on #222 (the raw-microdata-identity ADR and the governance,
architecture, AGENTS.md, and non-goal amendments). This PR deliberately makes
none of those edits, so the two do not conflict — but until #222 merges, the
governance doc still forbids what this branch does, and the ADR file that four
files cite does not exist on main. Merge #222 first.

Contract is shared verbatim with microcosm#848.

Schema and validation

Manifest file entries gain two fields:

  • licence — the publisher's terms, as an identifier or URL.
  • access — a closed set: public | licensed | restricted.

public is inferred where absent; fetch-artifact now writes the class
explicitly onto every entry it touches. Both fields are required on a
kind: microdata_release manifest, the new manifest-level carve-out.

Only public bytes enter a Chronicle store, so the R2 key
raw/{source_id}/{package_id}/{year}/{sha256}/{filename} exists only for that
class. A licensed or restricted release is registered hash-only: the
manifest carries the checksum, size, vintage, licence, access route, and
verification timestamp, and no bytes are fetched, written, or uploaded.

A registration is identified by {source_id, package_id, year, sha256, filename} — the tuple consumers reference under chronicle_artifact.

New command

chronicle register-artifact registers a licensed or restricted release from
CLI arguments alone. It refuses --access public, a missing or malformed
checksum, a missing licence, vintage, access route, or verification timestamp,
bytes sitting beside the manifest, a publisher_table manifest, a foreign
source_id, and an unrequested reissue. Public releases keep the existing
fetch-artifact path, which gains --access, --licence, and --kind.

First registrations

15 hash-only registrations, every value read verbatim from Microcosm's reviewed
pins and cross-checked field by field:

  • DWP Family Resources Survey 2023-24 — 14 tabs, access: licensed, UK Data
    Service End User Licence, study SN 9367, DOI 10.5255/UKDA-SN-9367-2.
    Microcosm carries 19 licensed_microdata references across five stages;
    they resolve to 14 distinct files (identical SHA-256 where they repeat).
  • HMRC Survey of Personal Incomes Public Use Tape 2022-23 — put2223uk.tab,
    access: restricted, study SN 9422, DOI 10.5255/UKDA-SN-9422-1.

chronicle inventory-artifacts --root db/data is valid with 195 artifacts: 180
with R2 links, 15 hash-only with no local bytes.

BE-SILC 2023 is not registered. Microcosm pins it with no sha256, no
size_bytes, and no per-file filename — the locator names four file roles.
No hash is invented; the blocker and its unblock steps are recorded in
docs/data-sources/be-silc-2023-registration-blocker.md.

Tooling

scripts/register_microdata_releases.py resolves a declarative catalogue
against a read-only microcosm checkout. emit writes the hash-only manifests
byte-stably; plan prints the exact fetch-artifact --upload-r2 commands to
run from a networked machine for the public US releases, copying publisher URLs
verbatim and printing TODO_PUBLISHER_URL where Microcosm records none (CPS
basic monthly 2024, SIPP 2023). The catalogue is the only authored content —
every checksum, size, filename, and vintage is read from Microcosm.

Audit fixes

An adversarial pass over the first implementation found the byte boundary was
escapable. All six are fixed here with regression tests:

  1. Licensed bytes could reach the package tree and R2. The hash-only guard
    only inspected files[year] for the requested year and only entries that
    survived a kind-gated list expansion, while the write target and R2 key are
    directory-scoped. A release registered under one vintage could be fetched
    under another, and a list under a manifest with an absent or misspelled
    kind was invisible to the guard. Both were reproduced writing licensed
    bytes into the tree. The guard now walks every vintage and expands lists
    unconditionally.
  2. A fetch silently deleted an existing registration by replacing the whole
    files[year] value. It now replaces only the entry for its own filename.
  3. emit stopped being re-runnable after a reissue — registration raised on
    the first filename match with a different checksum before reaching the exact
    match. Now two passes.
  4. --skip-hash-only reported a contract violation as valid, disagreeing
    with inventory-artifacts about the same tree. Hash-only entries are now
    always validated; the flag turns off the refusal, not the check.
  5. No way to register a public microdata release as kind: microdata_release, though the ADR puts redistributable public-use files in
    scope. fetch-artifact gains --kind, and a release manifest keeps the list
    shape so several files share one vintage (ACS household + person).
  6. Every generated fetch command died at argparse — plan emitted
    --source-url, which fetch-artifact does not accept. Fixed to --url, and
    a test now parses each generated command against the real CLI parser.

Also hardened _year_mapping, which returned a list and let the caller raise an
uncaught TypeError that validate-package does not catch.

No content, anywhere

No microdata row, cell, or fact is added. No source package parses a release:
validate-package fails with microdata_release_not_parseable, and the FRS and
SPI packages hold nothing but manifest.yaml. Bucket names stay configurable
(ledger-raw default, unchanged).

Chronicle Governance

For any source package, consumer contract, schema, or source-data
boundary change:

  • Approved Chronicle agent role:

    • ledger-source-ingestor — the db/data/** microdata-release manifests
    • ledger-contract-maintainer — the manifest schema (licence, access,
      kind), its validation, and the registration command

    Note both roles' allowed_paths in .github/chronicle-agents.yml predate
    this work and cover none of chronicle/registration.py,
    chronicle/artifacts.py, chronicle/harness.py, chronicle/cli.py,
    db/data/**, scripts/**, docs/**, or README.md. Flagging rather than
    quietly widening the manifest; reviewers may want a follow-up that extends
    the globs.

  • Deterministic checks run:

    • uv run pytest -q — full suite
    • uv run ruff check . — passes
    • uv run ruff format --check . — 14 pre-existing failures, all in files this
      branch does not touch (verified unchanged against origin/main); the
      branch's own files pass
    • uv run chronicle inventory-artifacts --root db/data — valid, 195
      artifacts, 15 hash-only
    • scripts/register_microdata_releases.py emit — re-run is byte-stable
    • every registered checksum cross-checked verbatim against Microcosm's pins
  • LLM judge verdicts:

    • ledger-source-fidelity: PASS — no fact is added. Every registered
      value (sha256, size, filename, vintage) is transcribed verbatim from the
      consumer's reviewed pin, machine-verified against Microcosm, and each entry
      records hash_source plus a note that Chronicle did not recompute the
      checksum. BE-SILC is refused rather than guessed.
    • ledger-contract: PASS — the change adds two manifest fields and one
      manifest kind, preserves provenance, and moves no Microcosm responsibility
      into Chronicle. Registration is identity only; selection, reconciliation,
      aging, and imputation are untouched.
    • ledger-boundary: PASS — no microdata row, cell, or fact enters
      Chronicle; no source package parses a release; no licensed or restricted
      bytes enter any Chronicle store, and the escapes that previously allowed
      that are closed and regression-tested.

Tests

tests/test_chronicle_microdata_registration.py — 69 tests covering the access
and kind vocabularies, every manifest-entry validation code, each command's
refusals, the six audit regressions, public microdata releases, the generated
fetch plan, and the committed FRS/SPI registrations as identity-only.

Two existing assertions in tests/test_chronicle_artifacts.py gain the new
hash_only_count and hash_only_refused_count report keys.

🤖 Generated with Claude Code

Gate round 2 (9/4): rebased onto #226 and #228; all nine round-1 findings closed

Rebased onto ops-rename-slice1 (PR #226 at fb1bc1d, which carries main at 9da0243 = #228). Conflicts in chronicle/artifacts.py were resolved so #226's stray-manifest refusal, files-block check, rename refusal, skipped publish entries and explicit-null files: handling all survive beside this PR's pre-fetch preflight.

  1. Filename aliases — filenames are bare, case-folded names; every alias of a hash-only registration is refused, and manifest-wide path collisions are rejected.
  2. Byte reader — _read_source_artifact_content refuses a hash-only entry before the package tree, the content-addressed cache, or a fetch is touched; SourceArtifactSpec.assert_parseable decides both carve-outs from the manifest alone; validate-package reports hash_only_artifact_not_parseable and manifest_kind_missing; build-suite validates before creating or replacing its output directory.
  3. Reclassifying an archived identity — refused while the recorded R2 object exists.
  4. Strict pre-fetch validation — a --kind that conflicts with the stored kind, a kindless manifest with entries, and every inventory-level manifest error are refused before the publisher is read.
  5. Inference before I/O — the destination basename is inferred from the URL without a read when --filename is omitted, so a gated artifact is never downloaded to be refused.
  6. _upsert_manifest — replaces the matched entry in place under Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226's recorded-identity guard, carries forward fields a fetch does not own, never turns a publisher-table mapping into a list, and always writes kind.
  7. Year keys — the vintage entry is selected by either key spelling; a vintage recorded under both spellings is refused rather than silently preferring one.
  8. Reviewed identity — registrations are consumer_pin (attested_by: PolicyEngine/microcosm, pinned_from path and commit, no verified_at); plan emits --publisher, --vintage, --filename, --expected-sha256/--expected-size-bytes when Microcosm's pin is of the publisher bytes, the allowlisted licence and its evidence; anything unknown is a TODO the CLI refuses. The 15 committed FRS and SPI rows are re-emitted that way with byte-identical SHA-256/size pins (verified against the microcosm blobs at 2fb2e2f8 and de7451bd).
  9. Hermetic tests — tests/fixtures/microcosm holds synthetic consumer manifests in Microcosm's shapes with the reviewed pins verbatim; emit reproduces both committed manifests byte for byte, plan matches a golden command list, every planned command parses against the real CLI, and the committed pins are a golden mapping. The commit-reading tests run against a copy of the fixture in a throwaway consumer repository.

Also implemented from the #222 ADR (which this PR still does not edit): the explicit kind rule with its frozen grandfathered-manifest list (chronicle/grandfathered_manifests.py), the redistributable-licence allowlist with per-term evidence (chronicle/licences.py), artifact-bound licence_evidence, hash_source/attested_by/pinned_from/attestation_evidence, staging outside the repository for public bytes, and the no-tracked-microdata-bytes guard. Docs: docs/agent-source-package-harness.md "Hash-Only Registrations", README quick-start, storage note.

Verification on 7f9bfe6: full suite 1219 passed, 1 skipped; ruff check clean.

Merge order unchanged: #222 first.

Follow-on hardening (9/4, after the round-2 fixes)

  • The byte boundary spans every manifest a package directory keeps (c0d9d74). --manifest (Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226) lets a fetch record into one of several manifests in a publisher directory; the hash-only guard read only that one, so a release registered licensed in manifest.yaml could be fetched, published, parsed, or reclassified through a sibling manifest_<package>.yaml. fetch-artifact now reads every manifest the directory keeps before the publisher is touched and refuses a name or digest any of them registers hash-only, refuses to overwrite a file a sibling records as other bytes, refuses a manifest whose source_id/package_id differ from the fetch's, and refuses an artifact named like a manifest; publish-raw and inventory-artifacts report filename_collision_across_manifests / sha256_collision_across_manifests; register-artifact takes --manifest with the same refusals; the source-package byte reader refuses a file a sibling registers hash-only. 18 tests in tests/test_chronicle_package_directory.py.
  • Manifests are read strictly (7f9bfe6). Duplicate mapping keys are refused (PyYAML would silently keep the last), filename keys are NFC-normalised as well as case-folded, and validate_file_entry reports nameless or malformed entries, misspelled fields, a storage block or chronicle_fetch attestation on a hash-only entry, a consumer attestation signed by Chronicle, and evidence URLs without a host; register-artifact validates every existing entry before recording. tests/test_chronicle_manifest_reading.py.

Gate round 3 (9/4, Sol): twelve findings closed

Entry keys use a closed schema (typos such as acess are refused, never public-by-default); the source-package reader and the fetch validate the selected manifest and every sibling strictly before any boundary decision or byte read; normalized-duplicate manifest names are refused; every no-local-bytes check scans real directory entries by normalized identity (shared helper across fetch, publish, inventory, registration and the public-release tree guards); inventory counts storage.r2 only for a validated content-addressed R2 locator (public microdata releases must also bind source, package and vintage); consumer pins are checked against the recorded commit's blob with git cat-file before emission; registration refuses symlinked output paths and persists under an exclusive lock by atomic same-directory replacement (two-thread regression); default sweeps discover every supported manifest name; only YAML null reads as an empty manifest; blank identity fields are replaced or refused before persistence; FRS selectors compare all stage references. Adversarial follow-ups closed in the same lane, including publish's dry preflight opening a valid artifact before honoring a malformed-sibling error. Full suite on this head: 1,268 passed, 1 skipped; ruff check clean; UK pins and #222-owned files unchanged.

Gate round 4 (9/4): eleven findings closed

Public entries without a sha256 are hashed and collision-checked against gated siblings before any upload, cache, or parse; inventory short-circuits on manifest, sibling, or entry errors before any byte read; source parsing validates storage.r2 locators and package-wide collisions before content access; the cross-bucket key rule is reconciled with #226's historical-route compatibility (a legacy recorded route whose checksum tail matches is preserved history; a key contradicting the entry's own package/year that is not a recorded legacy shape is refused); consumer pins record the repository, commit, and path of the blob actually checked; fetch, publish, and registration share one per-package lock keyed on physical file identity; required textual fields must be non-empty strings; the stray-default guard covers every supported manifest spelling; FRS cross-stage checks compare stable byte identity. Full suite on this head: 1,295 passed, 1 skipped; ruff check clean; UK pins and #222-owned files unchanged.

Integration with PR #226 (cb4c7ef)

Astra gate bbd833a9 round 1 (46560bc)

  • Source readers bypassed shared-file identity checks (medium): the source-package reader now runs the same directory-level owner agreement as inventory and publish (effective digests via the recorded-R2 resolver; an unidentified entry's bytes are hashed against identified siblings) before returning or caching bytes. Red 4f7aea5 (6 cases), fix adb030c.
  • Publish-raw staging path not validated (medium): public microdata publication resolves its staging destination through _validated_microdata_staging_destination (every component, realpath, no symlinked component, outside the repository and every package) before any byte is read or uploaded. Red a164a2b (13 cases incl. staging beneath db/data and a symlinked ancestor), fix 63c6bb9.
  • Registration lacked directory-wide agreement (medium): registration validates the complete proposed package (effective digests, archived collisions) before acquiring the writer lock and again before atomic replacement; an existing sibling collision refuses an unrelated registration with no lock file and no manifest byte changed. Red 2040886 (5 cases incl. revalidation under lock), fix b820b18.
  • Verification on 46560bc: full suite 1,691 passed, 1 skipped (21:20, lane); my spot-check of the artifacts + source-package + consumer-contract + env + manifest-reading + microdata-registration + microdata-staging suites 809 passed; ruff check/format clean on the six changed files.

Astra gate bbd833a9 round 2 (89737a1)

  • Source reader could parse a sibling's public microdata (medium): _assert_package_identities_parseable classifies every sibling's current and archived filename and checksum and refuses before any read, fetch, cache write, or row parse when any sibling registers the selected entry's identity as microdata_release; selected declared and effective digests are checked before I/O. Red 9c8738e (32 cases), control e543cea, fix dfd3b99.
  • Registration omitted effective owners and explicit locator checks (medium): both _prepare_registration_payload calls validate every current and archived R2 locator explicitly (locator errors are refusals), enforce filename/checksum agreement, and run the same owner-agreement validator publication and inventory use (effective digests, owners across vintages within one manifest), before lock creation and again under the lock before replacement, metadata-only. Reproductions: test_registration_refuses_invalid_recorded_sibling_before_mutation (5 cases), test_registration_rechecks_recorded_sibling_under_lock_before_replacement (5 cases), positive control, and a no-artifact-read control; fix 651571b.
  • Verification on 89737a1: full suite 1,737 passed, 1 skipped (24:21, lane); my spot-check of the eight artifact + source-package + consumer-contract + env + manifest-reading + microdata-registration + microdata-staging + package-directory suites 902 passed; ruff check/format clean; scope audit against 46560bc: exactly the five expected files changed.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Review (Fable, main): approving, with the Sol gate running. Cross-checked every registered hash against Microcosm's uk/source_stages.json: 15 of 15 match, none extra, BE-SILC correctly refused rather than guessed. The refusal surface is the right shape: bytes for licensed/restricted can enter no store by any command, and the six escapes the lane found in its own first cut are the kind of thing this boundary exists for.

Sequencing: #222 merges first (its gate is running; merge-on-agreement), then this. No file overlap between the two.

Notes, none blocking:

  • private_microdata → restricted is a safe over-classification (SPI PUT is a UKDS licensed study; both classes are hash-only, so nothing changes byte-wise). Fine as is; say so in the catalogue comment if it is not already.
  • dataset: dwp_dwp-frs-2023-24 is a mechanical id; harmless, could read dwp_frs_2023_24 in a follow-up.
  • The agent-role allowed_paths gap the body flags is real and predates this PR; follow-up to widen the globs rather than silently editing the roles file here.
  • The Microcosm side (microcosm#848, in flight) must reference exactly {source_id: dwp, package_id: dwp-frs-2023-24} and the SPI tuple committed here; I will align that PR to these ids.
  • The public-release fetch commands from plan will be run from a networked session after merge (two need publisher URLs Microcosm does not record: CPS basic monthly 2024, SIPP 2023).

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Gate round 1: changes requested. The 15 registrations match the consumer pins and add no bytes or fact surfaces, but Sol found real escapes around the boundary, all of which stand: filename aliases (./adult.tab vs adult.tab) bypass the hash-only guard; the source-package byte reader checks manifest kind but never the selected entry's access, so a mapping-shaped manifest with access: licensed can still be read, cached, and parsed; register-artifact can reclassify a public identity that already has an R2 object as licensed/restricted, leaving the object in R2 while inventory looks clean; kind conflicts and missing microdata access are not rejected before I/O; with --filename omitted the artifact is downloaded before the inferred filename is refused; _upsert_manifest is destructive and shape-unsafe; year keys are looked up as int only while readers accept '2023'; the generated public-release commands drop publisher/vintage and carry the expected sha256 only as a comment (fetch-artifact needs --expected-sha256 and the metadata params); the catalogue tests hardcode a home-directory checkout and skip on failure.

Two additions from the ADR review (#222 round 2) fold into the same fix: byte custody requires an affirmative, evidence-backed redistribution signal, not merely access: public (an allowlist of redistributable licence terms enforced in code, everything else is licensed); and the prohibition vocabulary is canonicalized to records, rows, columns, row values, cells, and facts.

Fix lane queues behind #226's round-2 fixes because both rewrite chronicle/artifacts.py; it rebases onto that branch first.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Cross-PR convention ruling (chronicle#227 ↔ microcosm#853), so both sides land on one shape:

  1. One Chronicle package per publisher release, files listed under the vintage. Chronicle's kind: microdata_release manifests hold a list under files[year] precisely so several distributed files share one registration (14 FRS tabs = one UKDS study). The consumer reference {source_id, package_id, year, sha256, filename, access} already disambiguates files inside a package by filename + sha256. So the ids are dwp / dwp-frs-2024-25 (year 2024, 14 tabs) and hmrc / hmrc-spi-public-use-tape-2022-23 (year 2022, put2223uk.tab), not one package per tab.
  2. Vintage: FRS 2024-25 (UKDS SN 9563). Microcosm main moved to the 2024-25 tabs today (#832); chronicle#227 registered the 2023-24 hashes from a stale checkout, so only 1 of its 15 hashes still matches main. The Chronicle side regenerates from microcosm origin/main (emit against a fresh checkout) and adds the other licensed/private roots main pins (WAS round 8, LCFS 2023-24 and 2018-20, ETB 1977-2024, SPI PUT): 19 hashes today.
  3. Access class derived from the Microcosm kind as #853 does (licensed_microdata → licensed; private_microdata/restricted_microdata → restricted; public_microdata → public), with byte custody additionally requiring the licence allowlist (ADR round 2). versioned_derived_microdata is never registered.

Chronicle's #227 fix round and a small #853 follow-up apply this.

MaxGhenis and others added 27 commits September 11, 2026 23:50
_recorded_object_identities read a uri with urlsplit, whose path stops at a
'?' or '#'; _validated_recorded_r2 splits provider://bucket/key by partition
and keeps reading. A stray '?' in the authority therefore made a block that
names a release's exact object read as naming nothing, and every archived-alias
check went blind on it. Both now use the same split_r2_uri.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…unit

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It moved modules to be shared; it did not become API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Validating archived objects made _effective_recorded_digest return None for an
entry whose own locator is valid, dropping it out of
validate_package_directory. Red: 1 failed (unreadable history), 2 passed (no
history, valid history).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unreadable history is its own refusal; erasing the digest the current object
records dropped the entry out of validate_package_directory, so inventory and
publish-raw stopped naming the contradiction between two manifests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_a_registered_entry_is_protected_before_it_is_ever_published captured the
manifest before a same-bytes refetch, which rewrites fetched_at, so the final
assertion only held when both fetches landed in the same second. It failed in
a loaded focused run; the same body fails on 83de2e8 with a 1.2s delay
inserted, so the flake is pre-existing, not a regression. The assertion now
reads the bytes the refused fetch must leave alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every case refuses only inside _upsert_manifest, so the publisher read and
the registration lock both precede the refusal: 31 failed, 53 passed, all 31
failures ORDERING VIOLATION.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The preflight read locators only for the selected entry and for entries
naming the same package-local file, so an unreadable storage.r2 or
storage.previous_r2 element in another vintage or a sibling manifest was
refused only by _upsert_manifest -- after the registration lock and after
the publisher was read.

The per-entry loop _upsert_manifest ran over the proposed tree becomes
_assert_recorded_locators_valid, called from three places instead of
copied into a third: the fetch preflight (after every manifest is valid on
its own terms, before the package sweep, release evidence, the lock and
publisher I/O), _upsert_manifest over the proposed tree as the recheck,
and _prepare_registration_payload, whose own copy of the loop -- including
its now-redundant second pass over storage.previous_r2, which
_validated_recorded_storage validates -- is replaced by the same call.
One validator, _validated_recorded_r2, decides in all three.

tests/test_chronicle_artifact_peer6.py: 31 failed before, 84 passed after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hoisting the loop validates the recorded tree, so the entry the fetch
rewrites is checked before the publisher is read rather than only after its
repair. The fetch used to accept a directory whose selected entry's locator
contradicted its declared identity -- silently under --record-revision,
archiving a superseded object the entry never agreed with -- while
inventory-artifacts reported the same tree recorded_r2_identity_mismatch.
The paired control asserts that inventory verdict, so the refusal is the
documented contract and not a new rule.

Whole module against the pre-fix tree 70e128b: 35 failed, 55 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Placing the loop before _assert_no_package_microdata_identities changed
which refusal a tree carrying both problems reports, and round 5 pinned the
sweep's: test_unreadable_history_does_not_blind_the_package_sweep and
test_one_locator_is_read_the_same_way_by_both_readers both failed.

The sweep resolves locators non-raisingly, so what it finds on unreadable
provenance is true either way, and the preflight already prefers the more
serious refusal's message at the hash-only byte boundary. What the sweep
misses is the reason the loop cannot be skipped: an archived element that is
not a mapping is invisible to it, so a table entry carrying a release's
identity only there passes it in silence. The loop now runs immediately
after, still before the release evidence, the lock and publisher I/O, and
the new regression pins that silent case through the fetch path.

peer5 + peer6: 167 passed. peer6 against the pre-fix tree: 36 failed, 55
passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The preflight's only owner check is scoped to the filename being fetched, so
a contradiction two manifests already record about another package-local
file is refused only by _assert_package_file_owner_identities_agree inside
_upsert_manifest -- after the lock and after the publisher read, on a tree
inventory-artifacts already reports invalid. Found while verifying the
locator hoist; pre-existing, and register-artifact runs the same sweep
before taking its own lock.

Both variants are public-vs-public, the only collision family the package
microdata sweep does not already refuse before the read: a declared sha256
disagreement and a disagreement only the content-addressed keys carry, which
is what _effective_recorded_digest resolves.

2 failed, 93 passed; both failures ORDERING VIOLATION.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_assert_shared_owner_identities_agree in the preflight is scoped to the
filename being fetched, so a contradiction two manifests already record
about another package-local file reached _assert_package_file_owner_identities_agree
only inside _upsert_manifest -- after the registration lock and after the
publisher read, on a tree inventory-artifacts already reports invalid, for a
fetch that could never succeed. No fetch repairs such a collision: the
rewrite revises only the owners of its own filename.

The preflight now runs the sweep register-artifact already runs before its
own lock, with check_local_files=False for the same reason registration
passes it: the local bytes are what this fetch is about to replace, and
every owner is compared against the fetched bytes once they are known.
_upsert_manifest keeps the full sweep as the under-lock recheck.

Measured over a 20-tree matrix against the tree with the locator hoist but
without this one, fetch driven with and without --record-revision: exactly 4
cells change, all four the identical refusal moving from LOCKREAD to before
either. Nothing moved from refused to accepted, and nothing moved from
accepted to refused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third measured behaviour change from reading the manifest's own identifiers
rather than the fetch arguments: a microdata_release manifest omitting
source_id or package_id used to be repaired by the fetch that filled the key
in via payload.setdefault, while inventory-artifacts, publish-raw and
register-artifact all refused the same tree on both trees. The preflight now
agrees with them, before the release licence evidence is looked at, which
the paired inventory assertion records. Unreachable on tracked data: of 199
manifests under db/data the two microdata_release ones declare both keys and
the five that omit package_id are kindless.

peer6 at HEAD: 97 passed. Against the pre-fix tree 70e128b: 40 failed, 57
passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
check_local_files=False left the unidentified-entry comparison firing only
from _upsert_manifest, after the lock and the publisher read, on a tree no
fetch could repair -- the same contract, left open on the check the previous
commit added. The two controls pin why the fetched filename must stay exempt:
its bytes are the ones this fetch replaces, and every owner is compared
against them once the publisher has served them.

1 failed, 99 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
check_local_files=False left the owner sweep's unidentified-entry
comparison firing only from _upsert_manifest, after the registration lock
and the publisher read, for a directory no fetch repairs. The preflight now
runs it over every package-local file except the one being fetched, via a
new pending_filename argument: that file's bytes are the ones this fetch
replaces, and they are compared against every owner as soon as the publisher
has served them. Every other caller keeps its current behaviour --
_upsert_manifest still runs the full sweep as the recheck, registration
still passes check_local_files=False under its documented "checks recorded
identities without opening artifacts" contract, and the resource readers
still supply observed digests instead.

The exemption is load-bearing, shown by execution: on a directory whose
shared file is stale and whose sibling records what it must hold, the sweep
without pending_filename refuses and with it passes -- so a plain
check_local_files=True would have refused the fetch that repairs those
bytes.

Measured over a 14-cell matrix against the previous commit: exactly 2 cells
change, both the same refusal moving from LOCKREAD to before either, and
nothing moved between accepted and refused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

#351 landed on main and retired the Supabase mirror. It deleted the mirror loader, query_sources, chronicle/client.py, db/supabase_client.py and the supabase dependency.

This branch carries #226's commits, so merging main conflicts in the same 13 files as #226 plus chronicle/cli.py. The conflicts in chronicle/artifacts.py, chronicle/cli.py, chronicle/source_package.py, chronicle/suite.py, their tests and README's register-artifact hunk predate #351; the others involve #351:

  • db/supabase_client.py (modify/delete): keep the deletion. chronicle_schema(), targets_schema() and the LEDGER_SCHEMA and TARGETS_SCHEMA aliases go with it.
  • chronicle/mirror.py, tests/test_chronicle_mirror.py and tests/test_chronicle_namespace.py: take main's side.
  • chronicle/harness.py: take main's side in the chronicle.mirror import, load_supabase_mirror_files and the load-supabase-mirror subparser. Keep both RawArtifactVerifyReport and SourceArtifactManifestError in the first import hunk.
  • README and docs/agent-source-package-harness.md: keep main's text in each of their Retire the hosted Supabase mirror and query_sources #351 hunks, then add back this PR's bucket-variable sentence in README and its chronicle.db name in the harness doc.
  • docs/storage-architecture.md: keep main's text in the summary, matrix, validation and build-flow hunks, then re-apply this PR's renames and bucket-variable paragraph. Keep the "Environment Variable Rename Window" and "Bucket Cutover" sections without the CHRONICLE_SCHEMA row, the sentence naming POLICYENGINE_SUPABASE_URL, and the two paragraphs on the hosted schema value and the loader's --schema default. The Publisher Revisions hunk predates Retire the hosted Supabase mirror and query_sources #351.

tests/test_chronicle_env.py merges clean, but its four test_supabase_schema_* tests import db.supabase_client, so delete them. Then run ruff check --fix on the touched files: the deletions leave unused imports in chronicle/mirror.py, tests/test_chronicle_mirror.py and tests/test_chronicle_env.py, and CI's lint fails on them.

On main, tests/test_supabase_mirror_retired.py fails if a shipped module imports a Supabase client, if chronicle/mirror.py mentions Supabase, or if live code or docs name load-supabase-mirror, POLICYENGINE_SUPABASE_URL, POLICYENGINE_LEDGER_SCHEMA or five other retired names. In this merge, every one of them sits on this PR's side of a conflict or in db/supabase_client.py.

After the merge, nothing outside tests calls default_chronicle_schema(), so chronicle/env.py can drop it along with DEFAULT_CHRONICLE_SCHEMA and CHRONICLE_SCHEMA_ENV.

@vahid-ahmadi vahid-ahmadi left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things to settle before this merges:

  1. The grandfather freeze conflicts with current main, and with #222's ADR. chronicle/grandfathered_manifests.py freezes the 168 kindless manifests at ba8147a7 (2026-09-04). It says entries are "never added", and tests/test_chronicle_manifest_kind.py::test_the_frozen_list_only_shrinks enforces that. On main 14181d7, roughly 350–380 manifests with file entries (377 by a script count; the exact figure depends on what counts as a file entry) have no kind and are either missing from the list (373) or changed since the freeze (4, including db/data/irs_soi/historic_table_2/manifest.yaml, changed by #291). Most of them (about 300) are read by a source package. After merging main, test_every_kindless_manifest_in_the_tree_is_frozen_unmodified fails, and manifest_kind (chronicle/registration.py:373-388) makes those packages refuse to parse (manifest_kind_missing). CI will flag it, but the #351 merge recipe doesn't cover it, and the fix is a decision rather than a mechanical step. #222 decision 1 says "The manifests that exist at #227's merge commit are grandfathered", which means re-freezing at the merge commit. This PR's docstring and shrink-only test forbid that, leaving several hundred kind: publisher_table edits as the other route. Please pick one and align the ADR and the code.

  2. The FRS registration is still 2023-24. The 2026-09-02 ruling on this PR moved the Chronicle side to FRS 2024-25 (SN 9563, dwp / dwp-frs-2024-25, year 2024) plus WAS/LCFS/ETB. At 4519336, db/data/dwp/frs_2023_24/manifest.yaml still registers dwp-frs-2023-24 (14 tabs pinned from microcosm 2fb2e2f8). None of those 14 SHA-256s appears in microcosm main's uk/source_stages.json today (c4ad232), which cites SN 9563 and 2024_25 throughout. The SPI PUT hash still matches. The pins are accurate for the commit they cite, but as merged, the first registrations would name a release no current consumer uses.

Review drafted with AI assistance and checked against the code at 4519336.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants