Skip to content

Check the measure ids the US legacy ETL cleanup doc names - #354

Merged
MaxGhenis merged 8 commits into
mainfrom
test/legacy-etl-doc-measure-ids
Nov 15, 2026
Merged

MaxGhenis merged 8 commits into
mainfrom
test/legacy-etl-doc-measure-ids

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On main, test_us_legacy_etl_cleanup_doc_accounts_for_retired_series (added in #348) checks that docs/us-legacy-etl-cleanup.md names each retired module and 15 series it lists, and that the doc's backticked package ids name shipped packages. It skips the measure ids next to them, so a renamed or mistyped measure id passes CI. This PR makes the test check every measure id, row id and IRS column code in the doc's replacement tables against the package the cell names.

On main's doc, the new check reports 2 errors, both in the SSI recipients row. That row lists ssa-ssi-table-7b1-2024, ssa-ssi-monthly-statistics-2024-12 and ssa-annual-statistical-supplement-2025, then ssi_payments, and so claims ssi_payments for all three packages. The supplement declares ssi_payments as a row, with payment_amount as its measure, and the other two packages declare no such id. This PR rewrites the row to name each package's own ids: recipient_count and payment_amount for Table 7.B1, recipient_count for the monthly statistics, and payment_amount with its ssi_payments row for the supplement.

#350 fixed the doc's other drift first. It renamed the mortgage row's district measures to mortgage_interest_paid_*, the ids chronicle#345 gave them (still N19300/A19300; see docs/concept-migrations.md, "Schedule A line 8a relabel"), and it updated the QBI gap note.

Grammar

The test parses every "Chronicle package and measures" cell under the grammar that a comment above the helpers states:

  • A cell splits into clauses at each ; outside parentheses.
  • Within a clause, a backticked id belongs to every package listed since the previous id. So `a`, `b`: `x` claims x for both packages, and `a` `x`, `y` claims both ids for a.
  • The package must declare an exact id as a measure id or a record-set row value id. A prefix_* wildcard must match at least one of its measure ids.
  • An IRS column code (N07225, A06500, ...) must name a source column that a measure matching the id before it reads.
  • A token of none of these kinds fails the test, and so does an id with no package before it.

Table reader

The test reads the doc with markdown-it-py, which implements CommonMark plus GitHub's table extension. A table therefore counts wherever GitHub renders one: indented, without outer pipes, inside a blockquote, list item or <details> block, and under a heading of any form. The test checks every column whose header reads "Chronicle package and measures" after it strips emphasis, code and escape marks. DOC_TABLES pins every table's heading and header, so an added, moved or reheaded table fails until someone lists it. This PR declares markdown-it-py as a dev dependency. The lock already pinned it at 4.0.0 for rich, so the uv.lock diff against main adds 4 lines of metadata to the policyengine-chronicle entry.

The test reads each package's YAML through chronicle.source_package.load_source_package and builds no facts. On this branch's doc it reads 30 replacement cells, checks 72 claims (56 ids, 16 column codes) against 15 packages, and finds no error. It checks packages listed together one by one: all three ACA OEP packages, both Medicaid packages and the three packages that publish net_capital_gains_*.

New tests

  • test_us_legacy_etl_cleanup_doc_check_flags_edited_claims applies 10 edits to the real doc and requires the named errors and no others. One renames a measure, and one mistypes a column code behind an escaped \|. Three of the other eight put a mistyped id in an indented row, a row without outer pipes or a repeated replacement column. The remaining five put it in a second table with a bold header cell, a blockquote, a list item, a <details> block or a new table with an escaped header.
  • test_us_legacy_etl_cleanup_doc_check_reads_any_heading_form checks that indented, closed and setext headings raise no error.
  • test_markdown_tables_read_cells_as_github_renders_them and test_replacement_cells_read_every_replacement_column cover the reader.
  • test_replacement_claim_check_flags_undeclared_ids runs one-row fake tables against fake packages. It covers a nonexistent measure id, a wildcard with no match, a wildcard that would match only a row, a listed package that lacks the id and a wrong column code. It also covers a column code after a row id, an unknown package, an id or column with nothing before it, an id cut off by ;, and an unreadable token.
  • test_replacement_claim_check_accepts_declared_ids covers the forms the doc uses and pins the claims each one produces.

Merge with main

The branch merges main at 4ce631f5, 269 commits past the branch's base (130b95bd, #348). Four files conflicted:

#350 moved N19300/A19300 in the mortgage row to the Historic Table 2 broad package and left the district package without column codes. So the "escaped pipe between column codes" edit now breaks the broad package's claim, and at the merge commit that case alone fails. The next commit points its expected error at soi-historic-table-2-state-broad-2022 and drops the mortgage fix from the changelog fragment.

Review record

The review trail in this PR's comment covers heads c0f750f9 through ebffab44, with an approval at ebffab44. No recorded review covers the merge or the commit after it.

Chronicle governance

This PR leaves the Chronicle source-data boundary alone. It adds no source package, target profile, consumer contract or schema, and it changes no fact. It changes a boundary test, a doc, a changelog fragment and the dev dependency list.

  • Approved Chronicle agent role: n/a (no source package, target profile or contract change)
  • Deterministic checks run: uv run ruff check chronicle policyengine_chronicle db scripts tests; uv run pytest -q tests/test_chronicle_boundaries.py (59 passed)
  • LLM judge verdicts: n/a (no source-data boundary change)

Tests

Run on 2026-11-14 at 1c1edf4a:

  • uv run pytest -q tests/test_chronicle_boundaries.py: 59 passed, including the 33 that -k "cleanup_doc or replacement_ or markdown_tables" selects.
  • With main's SSI recipients row put back, which restores main's doc, the doc test fails with the 2 errors above. With actc_* dropped from the CTC row, it fails because no ctc_* measure reads N11070 or A11070. With the income tax liability row's A06500 changed to A06501, it fails because no income_tax_liability_* measure reads A06501.
  • uv lock --check passes with uv 0.11.7 and 0.12.13.

🤖 Generated with Claude Code

MaxGhenis and others added 6 commits October 12, 2026 22:36
The cleanup doc's test checked only that the backticked package ids in its
"Where the publisher's values live" tables are shipped packages. A renamed
or mistyped measure id next to one passed CI, and two had drifted:

- soi-congressional-district-2022 still named interest_paid_deduction_*,
  which chronicle#345 renamed to mortgage_interest_paid_* (N19300/A19300).
- The SSI row put ssi_payments after three packages; it is a row of
  ssa-annual-statistical-supplement-2025's payment_amount, and the other
  two packages declare neither.

The test now parses each "Chronicle package and measures" cell: an id
belongs to every package listed since the previous id in its clause, must
be a measure id or row value id that package declares (a prefix_* wildcard
must match a measure), and an IRS column code must be a source column a
matching measure reads. Unit tests cover the grammar with fake packages,
including a nonexistent measure id, and a mutation of the real doc. It
reads package YAML only and builds no facts.

The doc fixes those two rows and the stale QBI note that the broad
package's qbi_* read N03270/A03270 "on main" (chronicle#345 repinned them),
worded as chronicle#350 words it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed package

The renamed-measure mutation test reuses the declarations the doc test
loaded, and the cell check looks for a backticked package id rather than
any hyphenated word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #354 found that valid Markdown slipped past the claim check.
An indented row, or a row without outer pipes, ended the table early, so
that row and the ones after it were never read. An escaped pipe split a
cell, which dropped a wrong column code.

A table now runs from its header to the next blank line. Rows may be
indented or drop their outer pipes, and `\|` stays inside its cell. A row
whose cell count differs from the header's fails. The doc test also
requires one table per "Where the publisher's values live" section. The
real-doc mutation test now covers each bypass the review reproduced; the
old reader reported no error for any of them.

The package-path scan is cached, and the grammar comment says that
historical ids go without backticks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review of #354 found two more silent bypasses.
- A repeated header written as **Chronicle package and measures** after a
  blank line started a second table, and the reader skipped its rows.
- A header that named the column twice had only its first copy checked.

Tables are now found as GitHub Markdown finds them: a row followed by a
delimiter row with the same number of cells. Every column whose header,
read without emphasis or code marks, is "Chronicle package and measures"
is checked. Each "Where the publisher's values live" section must also
hold exactly one table with the standard header. Both reproductions are
real-doc mutation cases, and a mutation can now apply several edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The third review of #354 found that a replacement table inside a
blockquote, list item or <details> block was skipped. It also found that
valid heading forms (indented, closing hashes, setext) raised a false
layout error, and that an extra table headed with an escaped asterisk
went unread.

The test now reads the doc with markdown-it-py (CommonMark plus GitHub's
table extension), so a table counts wherever GitHub renders one, under a
heading of any form. Cells are read as GitHub reads them: `\|` is a pipe,
short rows are padded and extra cells are dropped. DOC_TABLES pins every
table's heading and header. A table added, moved or reheaded fails until
it is listed, and the replacement-column check still reads it in the
meantime. markdown-it-py was already locked at 4.0.0 through rich; it is
now a declared dev dependency, and the lock gains metadata only.

Every reproduction from the three reviews is a real-doc mutation case,
and reformatted headings are a passing case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Independent review trail. Each round was GPT-6.1 Sol, adversarial, run through Subfleet.

Round Head Verdict Blocking findings Fixed in
1 c0f750f REQUEST_CHANGES Valid Markdown skipped or split rows: an indented row, a row without outer pipes, an escaped | 44f43f3
2 44f43f3 REQUEST_CHANGES A second table under a bold header was skipped, and a repeated column was half-checked 309dc8a
3 309dc8a REQUEST_CHANGES A table inside a blockquote, list item or <details> block was skipped ebffab4 (markdown-it-py reader, DOC_TABLES pin)
4 ebffab4 APPROVE none —

Every reproduction from rounds 1-3 is now a case in DOC_MUTATIONS.

Round 4 left one nit: a prose line placed directly under a table, with no blank line between, fails the doc test. That is the correct result, not a false failure. GitHub's own GFM spec (cmark-gfm test/spec.txt, the example after "The table is broken at the first empty line") absorbs such a line as an extra table row. GitHub would therefore render it as a row with an empty "Chronicle package and measures" cell, and markdown-it-py reads it the same way. The cmark-gfm extensions.txt fixture cited in that round has a blank line before Hi!, which is why it ends the table there.

Round 4's sandbox could not run the lock check. I ran uv lock --check locally and it passes; the uv.lock diff is metadata only.

Merge note for chronicle#350: whichever PR merges second has one conflicting hunk in docs/us-legacy-etl-cleanup.md, the SALT, mortgage and charitable rows. Take #350's version of those rows. The reader returns the same 30 cells and the same table layout on #350's doc, and #350's new rows pass the check.

MaxGhenis and others added 2 commits November 14, 2026 09:55
Main fixed the mortgage row and the QBI note (#350), replaced the two
SSI gap rows (#434) and dropped the provenance tests (#353) while this
branch waited. Conflicts and their resolutions:

- docs/us-legacy-etl-cleanup.md: main's SALT, mortgage and charitable
  rows (#350); this branch's SSI recipients row, which names
  `ssi_payments` only for the supplement that declares it; main's
  Table 7.B7 and 7.B3 rows (#434). The merged doc differs from main's
  in that one SSI row.
- pyproject.toml: both dev lists keep main's hypothesis>=6 and gain
  markdown-it-py>=3.0.0.
- tests/test_chronicle_boundaries.py: imports are main's plus functools
  and MarkdownIt; hashlib and yaml went with the provenance tests.
- uv.lock: main's file, then `uv lock`, which adds markdown-it-py to
  the chronicle package's dev metadata (4 lines). The local uv wrote
  lock revision 3; line 2 keeps main's revision 5.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Since #350 the doc's mortgage row writes `N19300`/`A19300` once, after
the Historic Table 2 broad package, and names the district package
without column codes, so the escaped-pipe mutant now breaks the broad
package's claim. The fragment drops the mortgage fix, which #350 made
first, and names the SSI row fix and the new dev dependency.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 2f2f672 into main Nov 15, 2026
3 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Landed as 2f2f672, a merge commit whose tree equals git merge-tree of main at 4ce631f and this branch at 1c1edf4.

@MaxGhenis
MaxGhenis deleted the test/legacy-etl-doc-measure-ids branch November 15, 2026 00:41
MaxGhenis added a commit that referenced this pull request Nov 15, 2026
Main is 2f2f672, the landing of chronicle#354, which added markdown-it-py
to both dev lists and four lines to uv.lock. Git merged every file without
a conflict. Both sides' edits to tests/test_chronicle_boundaries.py and
docs/us-legacy-etl-cleanup.md survive.

Git's merged uv.lock named markdown-it-py in the project's entries but held
no package entry for it, because this branch's earlier lock run dropped it
with supabase. So this commit takes main's uv.lock and reruns "uv lock"
(uv 0.11.7, no upgrade). Against main, the lock loses 29 packages:
supabase and the 28 that only it required. It keeps markdown-it-py and
mdurl for #354's dev lists, and adds the five pypdf[crypto] lines.

uv 0.11.7 writes "revision = 3" on line 2. This commit restores main's
"revision = 5" by hand, as eb9df0a did, and "uv lock --check" passes on
the result without rewriting it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant