Skip to content

Make ledger.db byte-reproducible for identical inputs - #316

Merged
MaxGhenis merged 4 commits into
mainfrom
reproducible-ledger-db
Nov 11, 2026
Merged

MaxGhenis merged 4 commits into
mainfrom
reproducible-ledger-db

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What was wrong

Two builds of the same source package at the same output path wrote different ledger.db bytes. sqlite3 .dump of the two differed in one value only: ledger_builds.created_at, which _insert_build filled from datetime.now(timezone.utc). _write_package_sidecars hashes ledger.db into datapackage.json and ro-crate-metadata.json, so both sidecars changed on every rebuild too. As a result, no rebuilt suite could be checked against a published one by hash.

Reproduced on main at 0f2cd96 by building packages/irs_soi/table_1_1 for 2023 twice at one --out path. ledger.db, datapackage.json and ro-crate-metadata.json all differed, and nothing else did.

Fix

ledger_builds.created_at now follows the SOURCE_DATE_EPOCH convention. The value comes from the first of these that is present:

  1. an explicit source_date_epoch= argument to build_chronicle_db;
  2. the SOURCE_DATE_EPOCH environment variable;
  3. the Unix epoch (1970-01-01T00:00:00+00:00). This fallback is this project's choice; the spec doesn't set one.

The rules for the value:

  • It must be Unix seconds from 0 to 253402300799 (9999-12-31T23:59:59Z), written as date +%s prints them: ASCII digits, no sign, no leading zeros. Anything else raises ValueError, as the spec asks for malformed values.
  • The stamp is computed as the Unix epoch plus a timedelta, which reaches year 9999 on every platform. (datetime.fromtimestamp stops near year 3000 on Windows.)
  • build_chronicle_db resolves it before it touches the filesystem.
  • build-suite and build-bundle also resolve it before preparing their output directory, so a bad value can't wipe a published suite under --replace.

No schema change.

  • The column, its TEXT NOT NULL type and the ledger.relational.v1 schema id are unchanged.
  • The value is still an ISO-8601 UTC string with a +00:00 offset, in the same format as before minus the microseconds.
  • build_id is still a content hash and never sees the stamp.
  • No contract schema, source package or target profile is touched, so none of the approved-agent roles in .github/chronicle-agents.yml apply.

Why not derive it from inputs or drop the column?

  • Dropping the column is a relational schema change.
  • The only input-derived time available is the artifacts' extracted_at. It is a free-form date string (the scaffold template says TODO YYYY-MM-DD), and facts-only builds have no artifacts at all.
  • A fixed default plus SOURCE_DATE_EPOCH is how reproducible build backends usually work. Hatchling, for example, defaults to a fixed 1980 timestamp. A CI release job can still stamp the commit time with SOURCE_DATE_EPOCH=$(git log -1 --format=%ct).

Readers of ledger_builds.created_at (searched on origin/main)

  • chronicle: _insert_build is the only writer. Nothing in chronicle/, db/, policyengine_chronicle/ or scripts/ selects created_at.
    • chronicle/artifacts.py and chronicle/mirror.py read build_id from ledger_builds.
    • export-db-tables copies every column with SELECT * into ledger_builds.jsonl. load-supabase-mirror upserts those rows on build_id, so a reload now writes a stable created_at instead of the latest rebuild's clock.
    • The Supabase DDL is not in this repo; there are no migrations. The value keeps its ISO-8601-with-offset form, so a text or timestamptz column accepts it.
    • db/schema.py's created_at fields belong to the separate SQLModel tables, not ledger_builds.
  • Downstream:
    • git grep of ledger_builds|ledger\.db on microcosm origin/main: no hits.
    • git grep ledger_builds across every local clone under ~/PolicyEngine, ~/TheAxiomFoundation, ~/CosilicoAI and ~: only copies of chronicle.
    • gh search code ledger_builds across PolicyEngine, TheAxiomFoundation, CosilicoAI and MaxGhenis: only chronicle/ files.

Invariants (tested)

Invariant Test
Identical inputs give an identical suite file tree when rebuilt in a new process with a different PYTHONHASHSEED. This includes ledger.db, datapackage.json and ro-crate-metadata.json. test_build_suite_rebuild_in_a_new_process_is_byte_identical
ledger.db bytes are a function of the inputs: identical at different output paths, different exactly when the epoch differs. Hypothesis: test_ledger_db_bytes_are_a_function_of_inputs
build_id does not depend on the build time. same property
created_at parses back to its epoch in UTC, with no fractional seconds. Hypothesis: test_created_at_round_trips_to_its_epoch_in_utc
The environment variable and the explicit argument agree, and the argument wins. Hypothesis: test_environment_and_explicit_epoch_agree (differential)
Stamps sort lexicographically in epoch order. Hypothesis: test_created_at_sorts_in_epoch_order
Exactly the date +%s-form strings 0–253402300799 are accepted; everything else is refused. The oracle is written independently of the implementation's regex. Hypothesis: test_only_date_format_epochs_in_range_are_accepted, plus parametrized cases
The epoch arithmetic agrees with datetime.fromtimestamp across the whole range. Known epochs (0, 2026-01-01, the maximum) format as expected. Hypothesis: test_created_at_matches_the_platform_timestamp_conversion (differential); test_created_at_formats_known_epochs
A malformed value leaves an existing DB, suite or bundle output untouched. test_malformed_source_date_epoch_*

Hypothesis is added as a dev dependency. uv.lock gains only hypothesis 6.168.5 and sortedcontainers.

Limits and follow-ups

  • Byte identity holds on the same Python and SQLite build. The SQLite file header records the version of the library that wrote the file (bytes 96–99: 3053001 for SQLite 3.53.1 here), so a rebuild on a different SQLite differs in the header. The README says so.
  • Rebuilds still need the same --out path, because reports record absolute output paths. This is unchanged and now documented in the README.
  • ledger.db written before this change carries microsecond wall-clock stamps, so it can't be byte-reproduced. Builds from this change on can.
  • ledger.db bytes depend on the order of the input facts and cells, while build_id does not. Today's builders feed a stable order; the IRS SOI 1.1 rebuild matches across hash seeds. This is split out as a follow-up rather than changing insert order here.
  • export-db-tables still stamps its mirror manifest.json with the export time. That is an export event and not part of the suite or its sidecars.
  • Add build-bundle --slim to drop per-suite intermediates #315 freezes chronicle.database.datetime in test_slim_bundle_matches_full_bundle_byte_for_byte. That patch keeps working here: the module still binds datetime, and fromtimestamp is inherited. So tests pass in either merge order. The two PRs do edit neighbouring lines of build_bundle in chronicle/bundle.py: Add build-bundle --slim to drop per-suite intermediates #315 changes its signature and docstring, and this PR adds the build_created_at() preflight right after the docstring. Whichever lands second therefore needs a rebase: keep Add build-bundle --slim to drop per-suite intermediates #315's signature and docstring, put the preflight after it, and drop _FrozenDatetime.

Test plan

  • chronicle build-suite packages/irs_soi/table_1_1 --year 2023 twice at one --out path: every file identical. ledger.db is also identical across two different paths, and sqlite3 .dump is identical.
  • SOURCE_DATE_EPOCH=1767225600 gives created_at 2026-01-01T00:00:00+00:00. SOURCE_DATE_EPOCH=bogus fails with ValueError.
  • ruff check with the locked ruff (0.14.10) on the CI lint surface.
  • Independent Opus 5.5 review (Subfleet, round 1): APPROVE with minor and nit findings. All of them are addressed in 95ab2af.
  • Delta review of 95ab2af
  • Targeted pytest under the heavy-run lock: tests/test_chronicle_reproducible_build.py, database, mirror, suite, bundle, governance and core.
  • CI Chronicle checks.

axiom: n/a: build-infrastructure change, no policy encoded.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 7, 2026 15:58
ledger_builds.created_at was the wall clock, so two builds of the same
package at the same path wrote different ledger.db bytes, and the
datapackage.json and ro-crate-metadata.json sidecars that hash it
differed too. It now follows SOURCE_DATE_EPOCH: an explicit
source_date_epoch argument, then the environment variable, then the
Unix epoch. The column, its type and the relational schema id are
unchanged, and build_id stays a content hash that ignores the stamp.

build-suite and build-bundle check SOURCE_DATE_EPOCH before preparing
their output directory, so a malformed value cannot wipe a published
suite under --replace.

Tests rebuild the IRS SOI Table 1.1 suite at one path and require an
identical file tree, and Hypothesis properties cover the timestamp
round trip, env/argument agreement, ordering, input validation and
ledger.db bytes as a function of inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild test now runs `chronicle build-suite` twice as separate
processes with different PYTHONHASHSEED values, matching the real
check of a rebuild against a published suite. Set-iteration order
cannot hide behind one shared hash seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the reproducible-ledger-db branch from 78d76e2 to abc9284 Compare October 7, 2026 19:59
MaxGhenis and others added 2 commits October 7, 2026 16:15
Following the independent review of #316:

- Accept SOURCE_DATE_EPOCH only in the `date +%s` form (no leading
  zeros), so equal values cannot arrive in two spellings.
- Compute the stamp as Unix epoch + timedelta rather than
  datetime.fromtimestamp, which stops near year 3000 on Windows.
- Say that byte identity holds on the same Python and SQLite build:
  the SQLite header records the library version that wrote the file.
- Tests: check acceptance with an oracle written independently of the
  implementation's regex and with generated digit strings; add known
  epochs and a differential check against fromtimestamp; draw index
  bounds from the data; show build-suite stderr on failure.
- Ignore the .hypothesis/ example database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit e547b3b into main Nov 11, 2026
3 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge audit (PR sweep of 2026-11-11): the sweep merged main into this branch (0d0581c; the conflict resolution adds only .hypothesis/ to .gitignore and one import to bundle.py). An independent Opus 5.5 review approved head 0d0581c with no blocking findings. It searched the merged tree for wall-clock reads, checked insert order and hash-seed independence, ran test_chronicle_reproducible_build.py (30 passed) and the database, mirror and suite tests (100 passed), and rebuilt twice under different PYTHONHASHSEEDs and output paths: ledger.db came out byte-identical. On the same facts, main and this PR differ only in ledger_builds.created_at, and fact values don't change. All three checks passed; the PR was MERGEABLE with no changes requested. This isn't a source-data PR under AGENTS.md (no source package or contract schema changes). Non-blocking: SOURCE_DATE_EPOCH="" is refused rather than treated as unset, and without the variable every build's created_at is 1970-01-01, so a release process that wants a real date should set it.

MaxGhenis added a commit that referenced this pull request Nov 11, 2026
bundle.py keeps this branch's build_bundle docstring and main's
build_created_at() preflight. #316 stamps ledger.db with SOURCE_DATE_EPOCH or
the Unix epoch instead of the clock, so the slim test's frozen-datetime patch
no longer does anything; drop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 11, 2026
Main gained Historic Table 2's state Schedule 1 line 1 and Schedule A lines
7, 8a and 14 (#350), the byte-reproducible ledger.db (#316) and the source
table tax-year test (#333). Only the whole-bundle test's three fact-count
pins conflicted: facts 2,246,063 (2,245,655 here + 408 HT2). Main's
duplicate keys (4,135), irs_soi, tax_unit, tax_year:2022 and California
pins merge as they are. The period, entity and source pins each sum to the
fact count; packages (509), tables (451) and sources (63) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant