Repository navigation
Add IRS Historic Table 2 state facts for refunds, taxes paid, mortgage interest and charitable contributions - #350
Merged
Conversation
MaxGhenis
force-pushed
the
soi-ht2-state-schedule-a-columns
branch
from
October 12, 2026 18:33
9261668 to
d2d27d9
Compare
MaxGhenis
changed the base branch from
fix/irs-cd-ht2-column-mappings
to
main
October 12, 2026 23:16
soi-historic-table-2-state-broad-2022 now reads eight more columns of 22in55cmcsv.csv for every state, each pinned to the TY2022 documentation guide's definition: - N00700/A00700 (Schedule 1 line 1) as taxable_state_local_income_tax_refunds_* - N18300/A18300 (Schedule A line 7) as taxes_paid_deduction_* - N19300/A19300 (Schedule A line 8a) as Table 2.1's mortgage_interest_paid_* - N19700/A19700 (Schedule A line 14) as Table 2.1's charitable_* The congressional-district package's charitable_* move to Table 2.1's concept and label for the same line 14 columns, so state, district and national rows join on concept as well as measure id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 20 IRS SOI packages at bundle year 2023, built against the parent commit, gain 408 TY2022 tax-unit facts, eight per state, and 204 semantic duplicate keys: the district file's state-total rows publish Schedule A lines 8a and 14 under the same concepts as the new state rows. No new geography, period or entity keys and no new warnings. docs/concept-migrations.md records the district charitable_* move to Table 2.1's contributions concept and label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review nits on e3bd56e: the identity test promised measure id, concept and label but checked two; it also rejected a correctly mapped TY2023 twin. The bundle comment now says what the measurement compared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/us-legacy-etl-cleanup.md (chronicle#348) now names the Historic Table 2 state rows for four retired series: - salt_* names taxes_paid_deduction_* (Schedule A line 7) beside the line 5e and 5b measures; the legacy rows named no IRS line; - mortgage interest and charitable rows list Historic Table 2's state rows next to the district file's state totals, separate IRS tabulations of the same lines; - state_local_refunds_* is covered at state grain; HT2's US row and Table 1.4 still go unselected nationally. The QBI row's chronicle#345 note is now past tense. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
force-pushed
the
soi-ht2-state-schedule-a-columns
branch
from
October 13, 2026 01:16
d2d27d9 to
7d096e5
Compare
Confirmation-review nits at 7d096e5: STATE_ONLY_COLUMNS read as "only the state package may read these" while the test accepts matching district readers, so it is now SINGLE_IDENTITY_COLUMNS. The cleanup doc's national refunds gap also names the district file's US row, which publishes N00700/A00700 unselected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 13, 2026 01:57
Brings main 1a3f2c4 (112 commits since #348) into chronicle#350. tests/test_chronicle_bundle.py conflicted in the full-bundle pins. Main moved them for its UK, NZ and Belgian sources; this branch moved them for the 408 Historic Table 2 state facts. Main changed no IRS SOI package, publisher file or guide extract, so the branch's measured deltas apply on top of main's values: fact_count, row_count and coverage fact_count 1,662,948 -> 1,663,356 (+408), and semantic_duplicate_key_count 2,324 -> 2,528 (+204). The irs_soi, tax_unit, CA and tax_year:2022 pins merged cleanly with the same deltas. Both comment blocks are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve tests/test_chronicle_bundle.py: keep main's bundle counts and add this branch's 408 Historic Table 2 facts on top (fact_count 2243075 -> 2243483, semantic duplicate keys 3930 -> 4134, California state rows 270 -> 278, tax_year:2022 period count +408). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve tests/test_chronicle_bundle.py again after chronicle#434 (SSA Supplement Tables 7.B3 and 7.B7): keep main's counts and comments and add this branch's 408 Historic Table 2 facts on top (fact_count 2243268 -> 2243676, semantic duplicate keys 3931 -> 4135, California state rows 276 -> 284). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Nov 11, 2026
#350 raised the bundle test's fact, irs_soi and tax_unit counts by 408 on the same lines this branch raises by 160. Each resolved count is main's value plus 160: fact_count and the consumer-facts row count 2,243,836, irs_soi 44,531, tax_unit 45,836. The semantic duplicate count takes main's 4,135; the eight Table 1.4 concepts appear in no other package on main, so they add none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
added a commit
that referenced
this pull request
Nov 11, 2026
Main gained Historic Table 2's state Schedule 1 line 1 and Schedule A lines 7, 8a and 14 (#350), the byte-reproducible ledger.db (#316) and the source table tax-year test (#333). Only the whole-bundle test's three fact-count pins conflicted: facts 2,246,063 (2,245,655 here + 408 HT2). Main's duplicate keys (4,135), irs_soi, tax_unit, tax_year:2022 and California pins merge as they are. The period, entity and source pins each sum to the fact count; packages (509), tables (451) and sources (63) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Nov 14, 2026
Main fixed the mortgage row and the QBI note (#350), replaced the two SSI gap rows (#434) and dropped the provenance tests (#353) while this branch waited. Conflicts and their resolutions: - docs/us-legacy-etl-cleanup.md: main's SALT, mortgage and charitable rows (#350); this branch's SSI recipients row, which names `ssi_payments` only for the supplement that declares it; main's Table 7.B7 and 7.B3 rows (#434). The merged doc differs from main's in that one SSI row. - pyproject.toml: both dev lists keep main's hypothesis>=6 and gain markdown-it-py>=3.0.0. - tests/test_chronicle_boundaries.py: imports are main's plus functools and MarkdownIt; hashlib and yaml went with the provenance tests. - uv.lock: main's file, then `uv lock`, which adds markdown-it-py to the chronicle package's dev metadata (4 lines). The local uv wrote lock revision 3; line 2 keeps main's revision 5. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Nov 14, 2026
Since #350 the doc's mortgage row writes `N19300`/`A19300` once, after the Historic Table 2 broad package, and names the district package without column codes, so the escaped-pipe mutant now breaks the broad package's claim. The fragment drops the mortgage fix, which #350 made first, and names the SSI row fix and the new dev dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
soi-historic-table-2-state-broad-2022now reads four more column pairs of IRS Historic Table 2 (22in55cmcsv.csv) for each of the 51 states, adding 408 TY2022 facts. I pinned each column to its definition in the TY2022 documentation guide (22incmdocguide.doc), not to the column-name pattern.N00700/A00700(AG/AH)taxable_state_local_income_tax_refunds_returns/_amount(new)irs_soi.returns_with_taxable_state_local_income_tax_refunds/irs_soi.taxable_state_local_income_tax_refunds(new)N18300/A18300(CD/CE)taxes_paid_deduction_returns/_amount(new)irs_soi.returns_with_taxes_paid_deduction/irs_soi.taxes_paid_deduction(new)N19300/A19300(CH/CI)mortgage_interest_paid_returns/_amountN19700/A19700(CP/CQ)charitable_returns/_amountirs_soi.returns_with_contributions_deduction/irs_soi.contributions_deductionBefore this PR, no package selected Schedule 1 line 1 or Schedule A line 7 at state grain, and #348 listed both as gaps in
docs/us-legacy-etl-cleanup.md. For lines 8a and 14, Chronicle read state values only from the district file's state-total rows. The IRS tabulates that file separately, and its state rows differ from Historic Table 2's. For example, the IRS reports $572M for WY line 14 in the district file and $1,204M in Historic Table 2. This PR adds Historic Table 2's own state rows for both lines.I gave the line 8a and line 14 pairs Table 2.1's labels, and all four amount measures carry
value_scale: 1000. I set each measure'ssource_column_idandexpected_column_headerto its IRS variable, and Chronicle stops with an error when a measure's column header in the CSV differs from itsexpected_column_header.District concept move. The district package and Table 2.1's package used different concept ids for line 14.
irs_soi.(returns_with_)charitable_contributions_deduction.irs_soi.(returns_with_)contributions_deduction.The IRS reports Schedule A line 14 in both the district file's
N19700/A19700and Table 2.1's "Contributions deduction: Total" (DG/DH). So this PR moves the districtcharitable_*measures to Table 2.1's concept and label, and consumers can now join state, district and national rows on concept as well asmeasure_id. The district measures keep theirmeasure_id,source_record_id, values and lineage. I recorded the move indocs/concept-migrations.md.New ids. No package on main reads Schedule 1 line 1 or Schedule A line 7 at any grain, so this PR coins ids for both.
taxes_paid_deduction_*after Table 2.1's line-total ids, such asinterest_paid_deduction_*for line 10. The IRS publishes line 7 in Table 2.1 as "Taxes paid deduction: Total" (CA/CB), which Table 2.1's package does not yet select.N00700/A00700"state and local income tax refunds" in its guide, but filers report only the taxable part on Schedule 1 line 1. So I gave the refunds ids thetaxable_prefix, liketaxable_interest_*andtaxable_pension_income_*.Evidence
The evidence folder,
~/PolicyEngine/_reviews/chronicle-ht2-state-schedule-a/on the build Mac, holds the scripts, downloads, bundle measurements, mutation results and the round 1 and 2 reviews.~/PolicyEngine/_reviews/pr-adoption-20261103/holds the round 3 to 5 reviews.Publisher files
22incmdocguide.doc(sha256d99db44b…) and23incmdocguide.doc(94279069…) from irs.gov on 2026-10-12. The bytes match the extracts and hashes intests/data/irs_soi_doc_guides/from Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345.irs.gov/pub/irs-prior.22in55cmcsv.csvmatches its manifest sha256 (f72f8333…).California, TY2022 (amounts in thousands of dollars, as the IRS publishes them):
N00700A00700N18300A18300N19300A19300N19700A19700Ratios to the national tables. I computed each ratio from the repo's publisher files, comparing Historic Table 2 with Table 2.1 and Table 1.4:
Chronicle stores each cell as the IRS published it, and the tests assert neither of these relations:
A18300(line 7) falls short ofA18460(line 5e): DC, ME, MD, MT, OH and OR. MD has the largest dollar gap, $20.8M (0.4% of its line 5e amount). In the TY2023 file, line 7 falls short of line 5e in 8 states.Invariants
csvparse of22in55cmcsv.csv. Each fact equals its cell × scale and carries its IRS variable assource_column_id.N19300,A19300,N19700orA19700declares Table 2.1's measure id, concept and label.N00700,A00700,N18300orA18300declares the same measure id, concept and label for it.test_every_measure_reads_the_column_its_guide_definition_namesandtest_every_definition_names_the_same_variable_in_every_guidecheck this.23in55cmcsv.csvhas the same eight variables at the same column letters, so Add IRS SOI Historic Table 2 TY2023 national, state broad and state EITC facts #295 can copy these eight measures into its TY2023 twin as they stand.Mutation check. I ran eight mutants, and the tests killed all eight. Each made one change:
A18460;value_scalefromcharitable_amount;N19500;taxes_paid_deduction_amountlabel.I recorded all eight in the evidence folder's
mutation-results.txt, andmutate.pythere applies the first seven.Bundle counts
On 2026-10-12 I ran
build-bundle --year 2023over the 20 IRS SOI sources with and without this PR's package changes (basea90ad379, the #345 commit this branch then sat on), and these counts moved:fact_counttax_year:2022tax_unite7e1edf1to 284)semantic_duplicate_key_countThe district package reads lines 8a and 14 from the district file's state-total rows under the same concepts, so 204 of the new facts (51 states × 4 columns) share a semantic key with a district row. Facts from the two files already shared 1,632 such keys at the base, in columns both packages read under one concept.
Nothing else moved: no new geography, period or entity keys, and no new warnings or errors. This PR moves the full-bundle pins in
tests/test_chronicle_bundle.pyfrom main's values ate7e1edf1by the same amounts: fact count 2,243,268 → 2,243,676 and semantic duplicate keys 3,931 → 4,135.Consumer effect
Chronicle stores the 408 new values as raw publisher facts and activates none of them. Microcosm decides what to do with them when it next re-pins Chronicle, and PolicyEngine/microcosm#1245 tracks the re-pin. I read Microcosm at
7253402fbon 2026-10-12 and grep-verified the mappings (analysis in the evidence folder):us/chronicle_feed.jsonthen grows from 40,178 to 40,586 rows if nothing else changes, and its digest moves.charitable_*. Of the eight measures this PR adds, Microcosm maps onlycharitable_*, tocharitable_deductionover itemizers. Microcosm turns them into state targets, and instate_cdit replaces the charitable sibling bridges with these Historic Table 2 state parents.mortgage_interest_paid_*unmapped so that it doesn't double-count the mortgage interest total it already maps. It maps neithertaxes_paid_deduction_*nortaxable_state_local_income_tax_refunds_*.measure_id, and none of its modules selects on either charitable concept id, so Microcosm picks the same targets after the move. One test fixture holds Table 2.1's national concept, which this PR doesn't change.TY2023
On
main, no TY2023 state package carries the broad columns:soi-historic-table-2-state-agi-2023holds only return count and AGI, so this PR adds no TY2023 measures. #295 derives a TY2023 state broad twin from this TY2022 package, and I left a heads-up there. #295's mirror test will expect these eight measures.Dependencies
8826247b). This PR extends Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345's guide-pinning test and reuses its districtmortgage_interest_paid_*ids.130b95bd). In7d096e57I updated five rows of Retire US legacy ETLs whose values don't trace to publisher bytes #348'sdocs/us-legacy-etl-cleanup.md:taxes_paid_deduction_*(line 7) beside the line 5e and 5b measures;state_local_refunds_*row, I named the new state-grain measures and kept the national gap;docs/us-legacy-etl-cleanup.md. Its QBI edit matches this PR's, and its mortgage edit fixes the stale districtinterest_paid_deduction_*id that this PR also replaces. When git merges the two branches, it reports a conflict across the adjacent SALT, mortgage interest and charitable rows. Whichever of the two PRs lands second resolves it by keeping this PR's rows and Check the measure ids the US legacy ETL cleanup doc names #354's SSI row edit. That PR also updates theDOC_MUTATIONS["escaped pipe between column codes"]case in Check the measure ids the US legacy ETL cleanup doc names #354'stests/test_chronicle_boundaries.py. The case edits the mortgage row'sN19300/A19300to nameA19301, and it expects an error that namessoi-congressional-district-2022. In this PR's mortgage row, that pair sits in the state broad package's clause, so the case must expectsoi-historic-table-2-state-broad-2022: no `mortgage_interest_paid_*` measure reads `A19301`instead. The round 3 integration reviewer ran Check the measure ids the US legacy ETL cleanup doc names #354's tests on that combination, and 32 of 33 passed. The escaped-pipe case failed because Check the measure ids the US legacy ETL cleanup doc names #354's measure-id check named the state broad package where the case expected the district package. All 33 passed after the reviewer changed the case's expected message to name the state broad package. The check found no errors in this PR's rows. When Check the measure ids the US legacy ETL cleanup doc names #354 merges main, it also has to keep the two SSI rows below the one it edits, which Add SSA Supplement 2025 Tables 7.B3 and 7.B7 by state #434 filled in.N01700,N06500;N00900/N26270/N25870overlap IRS SOI Table 1.4: add business, rental, royalty, farm-rental, partnership and S-corp net income/loss legs; give HT2/district 'net income (less loss)' columns their own concept ids #346).Merge of main (2026-11-03)
I merged main at
1a3f2c41into this branch asaf3dde13.tests/test_chronicle_bundle.pyconflicted, in the full-bundle pins that both sides moved. I took main's values, added the changes I measured for this PR (+408 facts and +204 semantic duplicate keys) and kept both sides' comments.chronicle/bundle.py,chronicle/source_package.pyandchronicle/sources/. So I rebuilt the 20-source IRS SOI bundle ataf3dde13, and its report matches the branch head's 2026-10-12 report in every count, coverage breakdown, duplicate key and warning (bundle-measure-merge-af3dde13/in the evidence folder).Merge of main (2026-11-10)
I merged main at
0d0b412ainto this branch as33e0dd95. Main had gained 117 commits since1a3f2c41. Among them, #353 retired the legacy SQLitetargetstable and deleteddb/etl_soi.py, and #377 (closing #376) gave each dimension value one Chronicle-declared label.tests/test_chronicle_bundle.pyconflicted, in the full-bundle pins that both sides moved. I took main's values, added the changes I measured for this PR and kept both sides' comments:irs_soisource pin (43,963 → 44,371), thetax_unitentity pin (45,268 → 45,676) and thetax_year:2022+408 line. Theby_sourceandby_entitypins each sum to 2,243,483.docs/us-legacy-etl-cleanup.mdwithout a conflict: main edited other parts of it and added a "Retired: the legacy targets table" section, and this PR's rows came through unchanged.chronicle/bundle.pyandchronicle/source_package.py, but no measure, concept, record set or publisher file this PR reads. Give each dimension value one Chronicle-declared label, keep the package's own wording beside it (consumer_fact.v5) #377's label checks read groupby row labels, filter values anddimension_value_labelsdeclarations, and this PR adds none of those. I didn't rebuild a bundle for this merge. In CI run 42950196112 at33e0dd95, all three checks passed, including theChronicle bundlejob, which ran the full-bundle test (test_build_bundle_writes_merged_consumer_contracttook 6,746 s).Merge of main (2026-11-11)
While CI ran on
33e0dd95, I merged #434 into main ase7e1edf1. #434 added 193 person facts from Tables 7.B3 and 7.B7 of SSA's Annual Statistical Supplement 2025, and it moved full-bundle pins that this PR also moves. So I merged main ate7e1edf1into this branch asd3623f94.tests/test_chronicle_bundle.pyconflicted, in six blocks. I took main's values, added this PR's changes and kept both sides' comments:docs/us-legacy-etl-cleanup.md, this PR's rows sit in the IRS SOI table and Add SSA Supplement 2025 Tables 7.B3 and 7.B7 by state #434's in the other publishers' table.Chronicle Governance
ledger-source-ingestorvalidate-package soi-historic-table-2-state-broad-2022 --year 2023: valid, 3,111 facts, no errors or warnings.build-suitebuilt 3,111 facts, recorded 8,476 source cells and traced every fact to a source cell.test_every_new_fact_is_its_published_cellchecks every new fact.build-suiteconsumer facts and the 20-source IRS SOI bundle both pass validation.build-suiteagent acceptance passes: 0 missing lineage or provenance, 0 row-semantic errors. The round 1 reviewer reran raw-fact validation for both packages this PR changes: 3,111 state facts and 29,760 district facts.e3bd56e9(now39dd12f5). An independent Opus 5.5 agent ran round 2 at7d096e57, after the rebase ontomainand the cleanup-doc commit, and confirmed both verdicts.ledger-source-fidelity: PASS ate3bd56e9(round 1) and7d096e57(round 2). The round 1 reviewer read CA, NY, TX, DC and more states from the CSV and found no invented or repaired values. It also checked the guide hashes and wording, the 2022 form lines, the accounting identity, the Table 2.1 ratios and every bundle delta. It raised three nits and no blockers or should-fix findings. I fixed all three in9261668e(nowe6c39c61): the single-identity test now checks labels and accepts matching later readers such as a TY2023 twin, and I rewrote the bundle comment to say what the measurement compared.csvmodule;N19300/A19300/N19700/A19700;a90ad379→8826247b→130b95bd;a838ee45:SINGLE_IDENTITY_COLUMNSand its test totest_line_7_and_refund_columns_keep_one_identity;ledger-contract: n/a (this PR touches no schema or identity code; the package YAML declares the concept-id move, and I recorded it indocs/concept-migrations.md)ledger-boundary: PASS ate3bd56e9(round 1) and7d096e57(round 2). This PR selects more source columns and aligns one concept; it adds no reconciliation, aging, imputation, activation, solver construction or PolicyEngine-computed facts. Only the tests compare one source with another. Outside the ingestor role's allowed paths in.github/chronicle-agents.yml, this PR edits only documentation:docs/concept-migrations.md, five rows ofdocs/us-legacy-etl-cleanup.mdand achangelog.d/fragment.af3dde13). Two independent Opus 5.5 agents reviewed the first merge of main: one checked the conflict resolution, and the other checked how the branch fits with main and the open PRs. Both approved. In its one nit, the resolution reviewer named stale bundle pins and a stale CI line in this description, and I've since updated the pins and removed the CI line. The integration reviewer raised one nit: once this PR's and Check the measure ids the US legacy ETL cleanup doc names #354's cleanup-doc rows sit together, Check the measure ids the US legacy ETL cleanup doc names #354's escaped-pipe test case must expect the state broad package's message.33e0dd95). An independent Opus 5.5 agent checked the second merge from git objects, without running pytest. It read the hand resolution, main's changes since1a3f2c41(Retire the legacy SQLite targets table and everything that reads or writes it #353's deletions and Give each dimension value one Chronicle-declared label, keep the package's own wording beside it (consumer_fact.v5) #377's label checks among them) and the cleanup doc that git merged on its own, and it approved with nothing blocking.d3623f94). An independent Opus 5.5 agent reviewed the third merge from git objects, without running pytest. It checked the six resolved pins and the static sums (by_source,by_entityand the expected period counts each sum to 2,243,676), and it found all of Add SSA Supplement 2025 Tables 7.B3 and 7.B7 by state #434's test edits intact. It also confirmed that Add SSA Supplement 2025 Tables 7.B3 and 7.B7 by state #434's facts and this PR's can share no semantic key and that every other file matches git's automatic merge, and it approved. Rounds 3 to 5 didn't rerun the ledger judges.Tests
tests/test_chronicle_soi_ht2_state_schedule_a.pycovers:tests/test_chronicle_soi_doc_guide_columns.py:test_schedule_a_lines_shared_across_grains_carry_table_2_1_ids;test_line_7_and_refund_columns_keep_one_identity.tests/test_chronicle_source_package.py: CA values for all eight measures and package counts 2,703 → 3,111.tests/test_chronicle_bundle.py: full-bundle pins +408 facts (total,irs_soi,tax_unitandtax_year:2022), +204 semantic duplicate keys and +8 California facts.test_chronicle_soi_ht2_state_schedule_a.py,test_chronicle_soi_doc_guide_columns.py,test_chronicle_soi_capital_gain_concepts.py), the SOI cases oftests/test_chronicle_source_package.py, and the governance and poverty-coverage tests: 64 passed;tests/test_chronicle_source_package.pyplus the consumer-contract, boundary and catalog tests: 182 passed;ruff check: clean.d61c069a, where the guide-column test compiles selectors from source-record specs, I reran the first seven mutants, and the tests killed all seven again (mutation-results-d61c069a.txt).🤖 Generated with Claude Code