Repository navigation
Let transport CREATE take an optional donor-decoding profile - #1698
Merged
Merged
Conversation
CREATE accepts a `donor_profile` parameter: a canonical JSON document that names a versioned decoding profile for the populace-us donor and states each of its choices. The first profile, populace-us-cps-asec version 1, stages household tenure, legal marital status and full-time study from the donor's raw answers and SPM units, refuses a document written for other donor bytes, and records its counts in CREATE's receipt. No graph selects a profile here. A graph that sets none keeps its CREATE frame and receipt bytes, which frozen fixtures captured at the base commit pin. Seven kernel refs move once: transport.create@1 and @2, currency@1, quantile_map@1 and @2, support_bounds@1 and boundary@1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five no-profile fixtures reproduce byte for byte on main 7ff5a26 with the base commit's transport.py and population_kernels.py in place, so only the recorded base commit changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two guard tests flagged only whole 64-hex digests, two country names and dotted node ids. One helper now reads a shared module's string constants and also flags bare country codes, such as a `country == "nz"` comparison or a country-keyed dict, and digest prefixes of eight or more hex characters. Tests plant each pattern and check ordinary strings pass. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Three reader fixes from the round-1 review of the donor-decoding profile: - Survey codes (A_MARITL, A_ENRLW, A_FTPT, H_TENURE) must have real integer or float storage, plain or pandas nullable. Each value is compared with each code in its own storage, so a complex 1+99j can no longer lose its imaginary part in a float cast and read as code 1. - Each housing cost is checked against its own concept's contract before tenure reconciliation, with the same validator and message as the reader without a profile. Under tenure_wins a negative cost in a contradicting household is refused instead of zeroed; under refuse the bound refuses before the contradiction count. Final validation still runs. - Source-year counts group on the year's text under the reader's lineage contract, so the integer 2022 and the string "2022" are one year and every blank answer is counted once, in the record and in the refusal. Tests go through the authenticated reader with pinned synthetic donors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The term-7 literal guard flags any hex run of 64 or more characters, hex runs of eight or more that hold a letter (all-letter runs included), node ids in any letter case, and "gb" only as a whole string. Its docstring calls it a heuristic and lists what it misses. The planted and ordinary cases cover the round-1 review's examples. - The frozen-baseline provenance and capture module say that toy_v2 holds no codec frame; the capture module's SHA-256 is updated, and the five fixture files are unchanged. - The changelog and ADR say the battery refuses stale support-bounds evidence when it is supplied to it, and that an ordinary graph run re-keys the producer and runs it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard now also flags United Kingdom (joined by spaces, hyphens, underscores or nothing), Britain, British, England, Scotland and Aotearoa, in any letter case. Both production modules still return no literal at this head and at base 7ff5a26. The planted-literal test gains one case for each new name. The module docstring lists exactly what flags; "us" and digit-only runs stay documented exceptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
November 21, 2026 11:57
This was referenced Nov 21, 2026
This was referenced Nov 21, 2026
MaxGhenis
added a commit
that referenced
this pull request
Nov 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Transport CREATE (
transport.create@1and@2) accepts one optional parameter,donor_profile: a closed, canonical JSON document that names a versioned decoding profile for the populace-us donor and states each of its choices. This is the shared change in the Microcosm hub's donor-profile placement ruling (the second of its three pull requests). No graph selects a profile here. New Zealand's selection comes in its own pull request.microcosm.donor-profile/1; profilepopulace-us-cps-asec, version 1; adonor_sha256; and one value for each choice. The parser accepts only canonical text (sorted keys, separators,and:,ensure_ascii=False, no NaN), which is what composition sends (compose.py:4208).parse_donor_profileis atframe/transport.py:889; the value sets live in one place,DONOR_PROFILE_CHOICES(:772).assisted_renter_tenure:rented_public_authorityorrented_private.blank_full_time_study:refuseorread_false.housing_cost_contradicting_tenure:tenure_winsorrefuse.H_TENUREdecides who owns, rents or lives rent free: code 3 (no cash rent) stagesrent_free, and a renter stays a renter whatever its SPM units say. The SPM units' tenure only splits owners into with and without a mortgage; an owner household whose SPM unit says renter is refused, because its mortgage status is unknown. The receipt counts the renter and rent-free households whose SPM unit says owner. Assistance counts if any SPM unit in the household receives it.A_MARITL1-3 read as married, 4 widowed, 5 divorced, 6 separated, 7 never married (2025 ASEC data dictionary, p. 6C-3).enrolled_full_timeisA_ENRLW == 1andA_FTPT == 1. SPM units with no members are counted, not refused. A housing cost that contradicts the staged tenure (rent outside the rented tenures, mortgage interest or principal outsideowned_with_mortgage;_TENURE_COSTS,:990-994) is zeroed undertenure_wins, and the receipt records the counts and amounts. Underrefusethe read raises instead, and the error gives the count of contradicting households for each cost. Before reconciliation reads a cost, the reader checks it against its own concept's contract with the same validator and message as a read without a profile (:1306-1319), so a negative or non-finite cost refuses under either choice and never enters the receipt's counts or totals. The shared decode runs before this check, with or without a profile: it sums each household's person costs, reading a blank raw cost as 0 and a complex one as its real part (concept_mapping.py:2728-2734). This change leaves decode alone. Two households with rent near the largest float (1e308 each) make the profiled read raiseOverflowErrorwhen it totals contradicting amounts (math.fsum,:1337), where a read without a profile succeeds. A rent of 1e300 with a weight of 1e10 recordsinfas the receipt'sweighted_total_amount. No real input comes near either value.donor_sha256names other bytes before it opens the file (frame/transport.py:1514-1521). It reads thespm_unittable only when a profile is set (:1533). Staging (_stage_donor_profile,:1086-1260) refuses missing columns; survey codes (A_MARITL,A_ENRLW,A_FTPT,H_TENURE) that lack real integer or float storage, plain or pandas nullable, or that match no code when compared in their own storage (_profile_codes,:997-1027); non-boolean assistance; blank enrollment answers underrefuse;tenure_typethat disagrees withH_TENURE; and SPM joins that do not hold: duplicate unit ids, persons in no known unit or household, units that span households, households whose units disagree on tenure, and households with no unit. Staging runs before the raw tables are dropped (:1535-1541), and the staged columns attach before concept validation (:1548-1553).build/transport/population_kernels.py:179-183), passes it to the reader (:234), and adds adonor_profilekey to its receipt only when one is set (:329-334): the document's digest, its choices, and the staging counts (tenure by weight, SPM unit counts, blank enrollment answers by source year, marital status, and the rent and mortgage reconciliation). The source-year counts key each year by its text under the reader's lineage contract, so the integer2022and the string"2022"count as one year (_source_years,frame/transport.py:1055-1067)._time_periodseries) and never thespm_unittable. Frozen fixtures captured on commit7ff5a26bc2(main when this branch was cut; both edited modules have the same blobs there and at main2f45706fc2andb9e2972fa1, which the branch has since merged) pin that commit's CREATE frame and receipt bytes for five toy cases, and thepopulace-us-h5-v1codec's frame bytes for four of them (packages/microcosm-build/tests/fixtures/transport_create_baseline/, withprovenance.json);toy_v2runstransport.create@2on a donor written by the same call astoy's, and itscodecfield is null. The capture module,test_support/microcosm_build/transport_create_baseline.py, is test support and ships in no package.What moves (ruling term 10)
The change edits
microcosm/frame/transport.pyandmicrocosm/build/transport/population_kernels.py, so seven kernel refs move once:transport.create@1,transport.create@2,transport.currency@1,transport.quantile_map@1,transport.quantile_map@2,transport.support_bounds@1andtransport.boundary@1.gates.battery@1does not move, and no committed digest moves.Implementation hashes, measured in the repo venv with the locked numpy 2.4.6, pandas 3.0.3 and tables 3.11.1. Base: commit
7ff5a26bc2, and again mainb9e2972fa1, where the two modules keep their base blobse55fa547andf8d24121. Head: commit326215fe3d(eac1d80f,00f82573), and again this head3f30195853, which merges mainb9e2972fa1. Each pair gives the same values. A walk at326215fe3dof the 148 refs that the locally importable registry builders register found exactly these seven moving. Separate probes measured seven registered refs that walk leaves out: the six US post-transfer refs (us_runtime/graph_kernels.py:488) anduk.full.rotated-holdout@1(with the policyengine-uk version emulated). None of those seven moves or reads an edited module. That makes 155 refs measured. The round-1 review's own walk counted 153 measurable refs in a 154-ref roster and found the same seven moving; its count runs one lower than this one, and the ref behind the difference is not identified.transport.create@1,@206c0d7a81ba4b430e8312b8c716b1d75ebc9c0510532086a127c5daffc68c593135f21143c6cc9bc7c355d5bf248a0c2c5731bd969d22bc591f201762bc7c867transport.currency@13ac7e78fdecb85d423e82f932c110e875ed99c14e26d9f62aa9ad6538ba55d681f1d2433c951fbb3acfe90a87e3e45c1891593d9b38630d9511e2aee4868ddc0transport.quantile_map@1d5fd0bc6f366f78c5ff0ffa945ce1495deebc9ac95fa9a6681fabafa3f7fa4130a5542c60fa0dc13d147ad6ace189c59f7e17fcd0cc51bec8c52b7f698b595eetransport.quantile_map@2108b46de228e78596baa57ba8168406f2132983c1b2edf5197813b5ef8ef4fcb251c4ddbfae1dbbc6abb996dbd858087e6df9d92c9cd2c697840a49ca6d936c5transport.support_bounds@1b696801ce697fae16e2cb2942ce4622676d9a855a42984b01efd869106ea42dff6fbc4e63e3ef8681f67f0f8218b3ab39dc3101993dfe3c9dd92c0c1ed92e3cftransport.boundary@1fc9d10670f6a092f4e27450ddad23c2af340b10c9378b6c35c35d4aaec3128a01c63fb85d066fb27adc20543ed21d7e530a69b92cd37cc80dca88638b7617ea6gates.battery@1(NZ toy fixture / toy registry)89924d06…/f2269356…The other 141 walked refs do not move, and no committed file contains any base or head value above, or any round-1 head value (full or 8-hex prefix;
git grep -F -i). The head values for the first five rows differ from round 1's, because round 2 edits the reader's module, which those six refs hash;transport.boundary@1keeps its round-1 value. The merges of main2f45706fc2andb9e2972fa1(#1694 and #1688) bring no Python source underpackages/*/src, and the seven refs and both batteries measure the same at3f30195853as at326215fe3d. No committed file at3f30195853contains any base, head or round-1 head value (8-hex prefix,git grep -F -i). Read with #1683's frame source in place (its head5fba262d3e, measured at round 1),transport.create@1and@2and the battery move again because they hash the frame bundle module; PR B still moves exactly these seven refs there. If #1683 lands first, the create and battery values need measuring again.microcosm-graph/.../keys.py:288-296), so these kernels' nodes and every node downstream of them re-key and run again.build/transport/cli.py:153-168, the hash at:160).build/transport/evidence_contract.py:339), so the battery's header check (checked_header,:695) refuses evidence stored before this change when that evidence is supplied to it (:746-751). An ordinary graph run does not supply it: the producer's node key includes the new hash, so the executor misses its cache and runs the producer again (graph/executor.py:2951-2981).transport.boundary@1hashes CREATE's module andgraph_inputs.pybut not the reader's module (population_kernels.py:355-361). Round 2 of this pull request was such an edit and moved exactly those six. Boundary's nodes still re-key as downstream nodes of CREATE, so all seven refs' nodes re-key either way. The ADR states the six-hash form.Interaction with #1666 and #1683
test_shared_buffer_writes.py, 99 passed) covers both of PR B's modules with no exemption.5fba262d3e) addsframe/shared_columns.py. PR B writes only copies before any Frame exists, so the two do not conflict. Measured at round 1: Frame: share unchanged columns and check them before writing #1683's change touches none of PR B's files; Frame columns are recorded only insidebundle.py; Frame: share unchanged columns and check them before writing #1683's raw-write scanner finds nothing in either of PR B's modules. PR B's reader writes only arrays it allocates or copies (frame/transport.py:1019-1024,:1177-1178,:1213,:1327), plus column assignments in_attach_donor_profile(:1350-1355), all before CREATE builds its Frame (population_kernels.py:282), whose constructor deep-copies.On the real donor
At commit
326215fe3d, this reader (frame/transport.pysha256044d6644…, the same blob at this head) read donor48b9d479…read-only withread_false,tenure_winsandrented_public_authority, with no refusal, in 2.9 s. Every SPM join holds: 59,900 units, none without members, no owner household with SPM renter tenure, and no household whose units disagree. The staged tenure counts 22,938 households owning with a mortgage, 17,649 owning outright, 15,256 renting privately, 830 assisted renters and 567 rent free (525 of them with SPM owner tenure). Rent is zeroed in 10,739 households and mortgage interest in 6,310. Blank enrollment answers are 54,464 for source year 2022, 54,654 for 2023 and none for 2024, the same in all three source-year counters; 4,511 persons are enrolled full time. These match round 1's counts, which Belgium's review measured on round 1's reader.With the same reader,
refusefor blank enrollment answers refuses with the counts by source year above, andrefusefor contradicting costs refuses with 10,739 rent and 6,310 mortgage-interest households, the same messages as round 1's reader. With no profile, the base and head readers return equal person and household tables, weights and lineage arrays on this donor;DonorBankdiffers only in its trailingdonor_profilefield (None).A profiled CREATE has since run through composition and the gate battery on toy data, at a local integration of round 1's head with New Zealand's selection. On the real donor that path is still to run: New Zealand's selection pull request and its Tier 2 preflight do that.
Tests
Run locally, serially, one file at a time, at this head
3f30195853unless marked:test_transport_donor_profile.py: 178 passed (131 at round 1's head, plus 47: 25 reader cases for the three round-1 findings, 3 complex-storage refusals in the id, label and assistance checks, and 19 literal-guard cases).test_transport_create_donor_profile.py: 36 passed. It includes the five frozen no-profile baselines and a profiled and an unprofiled CREATE in both orders. The file is unchanged since round 1.test_transport_population_kernels.py: 38 passed.test_shared_buffer_writes.py(Frame: share unchanged tables under copy-on-write, and refuse raw writes into shared buffers #1666's scanner): 99 passed.test_nz_transport_declarations.py(changed on main by NZ: read owner payments as interest only, add a terminal gate checkpoint, pin the real CREATE inventory #1688): 76 passed, 2 skipped (generator round-trips that need a staged feed).test_nz_spec_package.py(holds the reader's source-text pins): 1208 passed.test_nz_transport_toy_graph.py, at326215fe3d: 45 passed.test_transport_compile_modules.py: 10 passed, 1 failed on a local policyengine-us ABI mismatch (SpecValidationError,engine_abi.py:314), which fails the same way at mainb9e2972fa1; 11 passed on the engine-absent path.6861491a…after a docstring-only fix) rewrites all five baseline fixtures byte for byte with the base commit's two modules in place and with this head's (10 of 10cmpidentical, at326215fe3d; neither module changes after it).provenance.jsonrecords base commit7ff5a26bc2and the capture module's new SHA-256; the five fixture files are unchanged since round 1.test_support/microcosm_frame/shared_source_literals.py). It is a heuristic. It flags, in any letter case,zealand,belgi,united kingdom(spaces, hyphens, underscores or nothing between the words),britain,british,england,scotlandandaotearoa; the codesnz,nzl,bel,ukandgbrstanding alone, andbeorgbas a whole string; country-scoped node ids in any letter case; and hex runs of 64 or more characters of any composition, or of 8 or more with at least one letter. Its docstring lists what it misses: a branch on the donor's own code"us", all-digit runs shorter than 64, prefixes shorter than eight, literals assembled at run time or read from a file, names outside string constants, and country names it does not list (Wales, for one). Planted in a test string,country == "nz", a"be"-keyed dict,"BE.create","GB","deadbeef", 64 digits, an 8-hex digest prefix,"United Kingdom rules","England"and"Aotearoa"each fail the guard.ruff checkandruff format --checkpass on the six Python files.CI needs to show: a plain pass of
test_transport_compile_modules.py(it fails locally only on a policyengine-us ABI mismatch, the same way at the base), the[us]tests, and both Python versions.Review
In-session reviews before this PR opened: ruling compliance, default path, refusals and hash moves, a completeness critic, a fix round, and a verifier that ran mutants (each killed by a named test). After the rebase, a re-measure and an independent check by different methods agreed on every value then reported.
Round 1 of the independent review, at
d456cedd4e, asked for changes: three P1s in the reader and three P2s. All six are taken, in two commits.Commit
dab5d58233(reader fixes and their tests):1+99jread as code 1._profile_codesnow refuses any dtype whose kind is not integer, unsigned or float before it reads a value, then compares each value with each allowed code in its own storage, with no cast (frame/transport.py:997-1027). Tests: a complex code in each of the four columns refuses through the authenticated reader; integer and float storage, plain and nullable, read;2**53 + 1refuses.tenure_winsa negative cost that contradicted tenure was zeroed before concept validation saw it._attach_donor_profilenow validates the ids and every present cost column withvalidate_concept_tablesand raises the plain reader'sInvalid donor concepts: ...message before it reconciles anything (:1306-1319). This check covers only ids and costs, so its message equals the plain reader's when costs are the only violation; with other violations the plain read lists more. The read refuses either way. The final validation still runs (:1553). Tests: a negative rent and a negative mortgage interest refuse under both choices with the plain reader's message.2022and the string"2022"collided and one count was lost._source_yearsnow keys each person's year by its text under_source_id_payloads, the contract the reader already applies tosource_year(:1055-1067); staging computes the keys once (:1132) for the refusal (:1136) and the three counters (:1252-1254). Test: mixed integer and string years give{"2022": 2}in every counter and in the refusal.Commit
326215fe3d(test support and wording):gbas a whole string, and its docstring states what it misses (the Tests section above).provenance.jsonand the capture module's docstring now say thattoy_v2holds no codec frame.The review's "Needs Max" line cites the changelog and ADR text this merge puts on
main. Those are repository docs. The change selects no value for any country and publishes no outward claim or number.After round 1's fixes, a re-measure walked the 148 refs again at
326215fe3dand reproduced the fixer's values. An independent check then ran the reader on 92 survey-code cases, 15 source-year cases and the cost cases, re-read the real donor under every choice and with no profile, and killed 21 of 21 mutants, each by a named test. It raised two points, both taken: decode turns a blank raw cost into 0 before the cost check, so this text no longer says a blank cost refuses, and the guard missed several country names. Commit155ec5de2eadds those names. Commit3f30195853merges mainb9e2972fa1(#1694 and #1688); no file of this pull request changes in the merge.Round 2 of the independent review, at
3f30195853, approved with no finding at P0 or P1. Its reviewer had no shell, so the author reran the seven-ref and NZ toy hash probes at this head: all twelve values and both module digests match the table above, andgit grepfinds none of the base or head values in the tree. This text now carries its P3 notes on the cost pre-check's message and theinfweighted total. Two P3 notes stay open as follow-ups: the literal guard missesgb.node ids and currency codes, and an unreachable error names source years "Source person IDs" (:1586).axiom: n/a: transport infrastructure (donor decoding); no rule, parameter or dataset change
🤖 Generated with Claude Code