Repository navigation
Make ledger.db byte-reproducible for identical inputs - #316
Conversation
ledger_builds.created_at was the wall clock, so two builds of the same package at the same path wrote different ledger.db bytes, and the datapackage.json and ro-crate-metadata.json sidecars that hash it differed too. It now follows SOURCE_DATE_EPOCH: an explicit source_date_epoch argument, then the environment variable, then the Unix epoch. The column, its type and the relational schema id are unchanged, and build_id stays a content hash that ignores the stamp. build-suite and build-bundle check SOURCE_DATE_EPOCH before preparing their output directory, so a malformed value cannot wipe a published suite under --replace. Tests rebuild the IRS SOI Table 1.1 suite at one path and require an identical file tree, and Hypothesis properties cover the timestamp round trip, env/argument agreement, ordering, input validation and ledger.db bytes as a function of inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild test now runs `chronicle build-suite` twice as separate processes with different PYTHONHASHSEED values, matching the real check of a rebuild against a published suite. Set-iteration order cannot hide behind one shared hash seed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
78d76e2 to
abc9284
Compare
Following the independent review of #316: - Accept SOURCE_DATE_EPOCH only in the `date +%s` form (no leading zeros), so equal values cannot arrive in two spellings. - Compute the stamp as Unix epoch + timedelta rather than datetime.fromtimestamp, which stops near year 3000 on Windows. - Say that byte identity holds on the same Python and SQLite build: the SQLite header records the library version that wrote the file. - Tests: check acceptance with an oracle written independently of the implementation's regex and with generated digit strings; add known epochs and a differential check against fromtimestamp; draw index bounds from the data; show build-suite stderr on failure. - Ignore the .hypothesis/ example database. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merge audit (PR sweep of 2026-11-11): the sweep merged main into this branch (0d0581c; the conflict resolution adds only |
bundle.py keeps this branch's build_bundle docstring and main's build_created_at() preflight. #316 stamps ledger.db with SOURCE_DATE_EPOCH or the Unix epoch instead of the clock, so the slim test's frozen-datetime patch no longer does anything; drop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main gained Historic Table 2's state Schedule 1 line 1 and Schedule A lines 7, 8a and 14 (#350), the byte-reproducible ledger.db (#316) and the source table tax-year test (#333). Only the whole-bundle test's three fact-count pins conflicted: facts 2,246,063 (2,245,655 here + 408 HT2). Main's duplicate keys (4,135), irs_soi, tax_unit, tax_year:2022 and California pins merge as they are. The period, entity and source pins each sum to the fact count; packages (509), tables (451) and sources (63) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What was wrong
Two builds of the same source package at the same output path wrote different
ledger.dbbytes.sqlite3 .dumpof the two differed in one value only:ledger_builds.created_at, which_insert_buildfilled fromdatetime.now(timezone.utc)._write_package_sidecarshashesledger.dbintodatapackage.jsonandro-crate-metadata.json, so both sidecars changed on every rebuild too. As a result, no rebuilt suite could be checked against a published one by hash.Reproduced on
mainat 0f2cd96 by buildingpackages/irs_soi/table_1_1for 2023 twice at one--outpath.ledger.db,datapackage.jsonandro-crate-metadata.jsonall differed, and nothing else did.Fix
ledger_builds.created_atnow follows theSOURCE_DATE_EPOCHconvention. The value comes from the first of these that is present:source_date_epoch=argument tobuild_chronicle_db;SOURCE_DATE_EPOCHenvironment variable;1970-01-01T00:00:00+00:00). This fallback is this project's choice; the spec doesn't set one.The rules for the value:
date +%sprints them: ASCII digits, no sign, no leading zeros. Anything else raisesValueError, as the spec asks for malformed values.timedelta, which reaches year 9999 on every platform. (datetime.fromtimestampstops near year 3000 on Windows.)build_chronicle_dbresolves it before it touches the filesystem.build-suiteandbuild-bundlealso resolve it before preparing their output directory, so a bad value can't wipe a published suite under--replace.No schema change.
TEXT NOT NULLtype and theledger.relational.v1schema id are unchanged.+00:00offset, in the same format as before minus the microseconds.build_idis still a content hash and never sees the stamp..github/chronicle-agents.ymlapply.Why not derive it from inputs or drop the column?
extracted_at. It is a free-form date string (the scaffold template saysTODO YYYY-MM-DD), and facts-only builds have no artifacts at all.SOURCE_DATE_EPOCHis how reproducible build backends usually work. Hatchling, for example, defaults to a fixed 1980 timestamp. A CI release job can still stamp the commit time withSOURCE_DATE_EPOCH=$(git log -1 --format=%ct).Readers of
ledger_builds.created_at(searched onorigin/main)_insert_buildis the only writer. Nothing inchronicle/,db/,policyengine_chronicle/orscripts/selectscreated_at.chronicle/artifacts.pyandchronicle/mirror.pyreadbuild_idfromledger_builds.export-db-tablescopies every column withSELECT *intoledger_builds.jsonl.load-supabase-mirrorupserts those rows onbuild_id, so a reload now writes a stablecreated_atinstead of the latest rebuild's clock.textortimestamptzcolumn accepts it.db/schema.py'screated_atfields belong to the separate SQLModel tables, notledger_builds.git grepofledger_builds|ledger\.dbon microcosmorigin/main: no hits.git grep ledger_buildsacross every local clone under~/PolicyEngine,~/TheAxiomFoundation,~/CosilicoAIand~: only copies of chronicle.gh search code ledger_buildsacross PolicyEngine, TheAxiomFoundation, CosilicoAI and MaxGhenis: onlychronicle/files.Invariants (tested)
PYTHONHASHSEED. This includesledger.db,datapackage.jsonandro-crate-metadata.json.test_build_suite_rebuild_in_a_new_process_is_byte_identicalledger.dbbytes are a function of the inputs: identical at different output paths, different exactly when the epoch differs.test_ledger_db_bytes_are_a_function_of_inputsbuild_iddoes not depend on the build time.created_atparses back to its epoch in UTC, with no fractional seconds.test_created_at_round_trips_to_its_epoch_in_utctest_environment_and_explicit_epoch_agree(differential)test_created_at_sorts_in_epoch_orderdate +%s-form strings 0–253402300799 are accepted; everything else is refused. The oracle is written independently of the implementation's regex.test_only_date_format_epochs_in_range_are_accepted, plus parametrized casesdatetime.fromtimestampacross the whole range. Known epochs (0, 2026-01-01, the maximum) format as expected.test_created_at_matches_the_platform_timestamp_conversion(differential);test_created_at_formats_known_epochstest_malformed_source_date_epoch_*Hypothesis is added as a dev dependency.
uv.lockgains onlyhypothesis6.168.5 andsortedcontainers.Limits and follow-ups
--outpath, because reports record absolute output paths. This is unchanged and now documented in the README.ledger.dbwritten before this change carries microsecond wall-clock stamps, so it can't be byte-reproduced. Builds from this change on can.ledger.dbbytes depend on the order of the input facts and cells, whilebuild_iddoes not. Today's builders feed a stable order; the IRS SOI 1.1 rebuild matches across hash seeds. This is split out as a follow-up rather than changing insert order here.export-db-tablesstill stamps its mirrormanifest.jsonwith the export time. That is an export event and not part of the suite or its sidecars.chronicle.database.datetimeintest_slim_bundle_matches_full_bundle_byte_for_byte. That patch keeps working here: the module still bindsdatetime, andfromtimestampis inherited. So tests pass in either merge order. The two PRs do edit neighbouring lines ofbuild_bundleinchronicle/bundle.py: Add build-bundle --slim to drop per-suite intermediates #315 changes its signature and docstring, and this PR adds thebuild_created_at()preflight right after the docstring. Whichever lands second therefore needs a rebase: keep Add build-bundle --slim to drop per-suite intermediates #315's signature and docstring, put the preflight after it, and drop_FrozenDatetime.Test plan
chronicle build-suite packages/irs_soi/table_1_1 --year 2023twice at one--outpath: every file identical.ledger.dbis also identical across two different paths, andsqlite3 .dumpis identical.SOURCE_DATE_EPOCH=1767225600givescreated_at2026-01-01T00:00:00+00:00.SOURCE_DATE_EPOCH=bogusfails withValueError.ruff checkwith the locked ruff (0.14.10) on the CI lint surface.tests/test_chronicle_reproducible_build.py, database, mirror, suite, bundle, governance and core.Chronicle checks.axiom: n/a: build-infrastructure change, no policy encoded.
🤖 Generated with Claude Code