From c035d0857eae6c378696897f0dded9213a26f740 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 13:50:59 -0700 Subject: [PATCH 01/10] Read a bracketed member settled by its writing as the suffix (#752, phase 2l) A member of the ambiguous credential class alone in brackets or quotes after the name is the suffix where its writing settles it: a period inside it (`John Smith (M.A)`), or capitals in a name of mixed case (`John Smith (MA)`, `Andrew Perkins (JD)`), the case being the name's own words' as rules/S6 asks it. Any other member stays the nickname and reports suffix-or-nickname (`John Smith (Ed)`, `JOHN SMITH (MA)`), and so does one with a name word after it (`Jeffrey (JD) Bricken`). Decided in group, where the recorded case and the following words' tags exist, and read as the suffix directly rather than unwrapped: bare, `M.A` reads as paired initials (#872). The suffix-or-nickname report moves from classify to group with the decision. New rule rules/S13 (a bracketed member settled by its writing), since the position it reads is evidence S1's signal does not; decisions/S13 records the measurements. Closes #810. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/customize.rst | 9 +- docs/design/decisions/README.md | 1 + ...-bracketed-credential-is-not-a-nickname.md | 2 +- ...bracketed-member-settled-by-its-writing.md | 20 +++++ docs/design/decisions/ma-do.md | 4 +- .../decisions/suffix-acronym-collisions.md | 2 +- docs/design/decisions/vocabulary-model.md | 2 + docs/design/rules/README.md | 1 + ...-bracketed-credential-is-not-a-nickname.md | 10 ++- ...bracketed-member-settled-by-its-writing.md | 55 ++++++++++++ docs/design/vocabulary-model-rfc.md | 4 +- docs/release_log.rst | 1 + docs/usage.rst | 10 ++- nameparser/_pipeline/_classify.py | 19 +--- nameparser/_pipeline/_group.py | 88 ++++++++++++++++-- nameparser/_pipeline/_state.py | 4 +- nameparser/_types.py | 3 +- tests/test_suffixes.py | 15 ++-- tests/v2/cases.py | 90 +++++++++++++++++++ tests/v2/pipeline/test_classify.py | 4 +- tests/v2/pipeline/test_state.py | 9 +- tests/v2/rules_budget.json | 3 +- tests/v2/test_cases.py | 7 +- tools/differential/corpus_rules.jsonl | 18 ++++ tools/differential/snapshot.jsonl | 23 ++++- 26 files changed, 352 insertions(+), 54 deletions(-) create mode 100644 docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md create mode 100644 docs/design/rules/S13-bracketed-member-settled-by-its-writing.md diff --git a/AGENTS.md b/AGENTS.md index 73e4a0bae..8f480d093 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -239,7 +239,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `group` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/customize.rst b/docs/customize.rst index fcdbe7a60..851f522e5 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -372,8 +372,13 @@ The full reading of a bare marked acronym, in order: reads as the given name is ``"Smith, Ba"`` (one word, and Title-case has no credential lean) and ``"smith, ba"`` (one word, one case, no lean at all). -- **In brackets**, ``"John Smith (BA)"`` falls through to nickname - parsing. +- **In brackets after the name**, the capitals in a mixed-case name + settle it as the credential too, and so does a period inside the word: + ``"John Smith (BA)"`` and ``"John Smith (B.A)"`` give suffix ``BA`` and + ``B.A``. Any other spelling there, and any spelling inside the name, + falls through to nickname parsing (``"John Smith (Ba)"``, + ``"Jeffrey (JD) Bricken"``), reported as a ``suffix-or-nickname`` + ambiguity. Generation words that are also surnames ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 6bb2d4cfd..04bb680b5 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -58,6 +58,7 @@ Entry conventions: - [S10 — the trailing roman numeral](S10-trailing-roman-numeral.md) - [S12 — a credential split across a space](S12-credential-split-across-a-space.md) - [S11 — a suffix word that is also a name](S11-suffix-word-that-is-also-a-name.md) +- [S13 — a bracketed member settled by its writing](S13-bracketed-member-settled-by-its-writing.md) ## Nicknames & quoted names (N) diff --git a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md index 0b33a3aab..f2f36fde8 100644 --- a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md @@ -4,7 +4,7 @@ - The rule says what the escape does: quotes escape as brackets do, and a clause of two or more words is not suffix-shaped unless an ASCII period ends it, so `(PhD MBA)` keeps its nickname reading with nothing reported, as 1.4.0 read it — 2026-10-21 #807. -Elsewhere: an ambiguous member alone in the brackets staying a nickname, reported, is decided under decisions/suffix-acronym-collisions (its Standing line ending 2026-09-25 #540; 2026-10-21) and decisions/ma-do's 2026-07-17 entry. +Elsewhere: an ambiguous member alone in the brackets staying a nickname, reported, is decided under decisions/suffix-acronym-collisions (its Standing line ending 2026-09-25 #540; 2026-10-21) and decisions/ma-do's 2026-07-17 entry; a member after the name whose writing settles it reading as the suffix is decisions/S13 (2026-10-26 #810). ## Log diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md new file mode 100644 index 000000000..c1459c4fc --- /dev/null +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -0,0 +1,20 @@ +# S13 — a bracketed member settled by its writing + +## Standing + +- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, written with a period inside it or in capitals in a name of mixed case, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname` (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`) — 2026-10-26 #810. +- The case is that of the name's own words, as S6 asks it, never the clause's own, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix — 2026-10-26 #810. +- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix or a maiden clause after it does not count (`John Smith (MA), PhD`, `John Smith (MA) née Jones` → suffix 'MA') — 2026-10-26 #810. +- A settled member is the suffix, not the word read as if written bare: bare, `M.A` reads as paired initials (decisions/S3 (#864)), and the brackets have already set it apart from the name's words — 2026-10-26 #810. + +## Log + +- **2026-10-26 #810 (Derek, 2026-10-23) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME IS THE SUFFIX WHERE ITS WRITING SETTLES IT.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. + - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. + - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. + - **Why the suffix rather than the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly instead, the brackets having already set it apart from the name's words. The decision is made in group, where classify has recorded the name's case and tagged the words after the clause, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. + - **Recorded as a rule of its own**, not in S1: the position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. + - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. + - **Accepted:** a member after a lone name word takes the suffix where its writing settles it, and the word reads as a one-word name reads (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, as bare `Jack MA` reads), where 2.0.0 through 2.6.0 gave family 'Jack' and nickname 'MA'. + - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, 19 of them `Ed`, 3 of those inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. + - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/decisions/ma-do.md b/docs/design/decisions/ma-do.md index 965e9a4e7..f6292776d 100644 --- a/docs/design/decisions/ma-do.md +++ b/docs/design/decisions/ma-do.md @@ -5,7 +5,7 @@ - `ma` and `do` are members of the ambiguous acronym set (`suffix_acronyms_ambiguous`: acronym readings on a word with a `given` or `family` reading, decisions/vocabulary-model 2026-10-25, `uv run python tools/vocab/query.py --word ma`), kept there because both are common surnames; the set has grown past these two since (decisions/suffix-acronym-collisions), and the marking is still what lets `Jack Ma` read family 'Ma' with `suffix-or-name` reported, while `John Smith MA` reads suffix — 2026-07-17. - In a name written in more than one case, a member in capitals leans credential and is taken with nothing to spare (`Jack MA`, `Jack DO` → given 'Jack', suffix 'MA'/'DO'), where `Jack Ma` keeps family 'Ma'; a deliberate break from v1, which read `Jack MA` as family 'MA' — 2026-09-15 #289. - A member written with its periods, one after each letter, counts unambiguously (`Jack M.A.`, `Jack D.O.` → suffix, `given-or-family` reported when only the given name remains), and one trailing period is no such spelling (`Jack Ma.` → family 'Ma.') — 2026-07-17. -- A parenthesized bare `(MA)` or `(DO)` reads as a nickname, no longer escaping to suffix as in 1.x, and reports `suffix-or-nickname` — 2026-07-17. +- A parenthesized bare `(MA)` or `(DO)` reads as a nickname, no longer escaping to suffix as in 1.x, and reports `suffix-or-nickname`, unless it stands after the name in a name of mixed case, where its capitals settle it as the suffix (`John Smith (MA)`; `JOHN SMITH (MA)` and `John Smith (Ma)` keep the nickname), as does a period inside it (`John Smith (M.A)`), decided in decisions/S13's 2026-10-26 #810 entry — 2026-07-17. - After a family comma `do` is spelling-dependent, being the one member of the set that is also particle vocabulary: `Doe, John DO` reads suffix `DO`, while `Doe, John Do`, `Doe, John do` and `DOE, JOHN DO` attach it to the family by P6 (`Doe, John Do` → family 'Do Doe', `particle-or-given` reported); `ma` is no particle, so `Smith, John Ma` keeps middle 'Ma' and `Smith, John MA` reads suffix — 2026-09-18 #531. Comma-less under `FAMILY_FIRST`, only the Title-case spelling attaches (`Doe John Do` → given 'John', family 'Do Doe', with `particle-or-given` besides `suffix-or-name`), while `Doe John do` and `DOE JOHN DO` read suffix, so the comma and the comma-less writing disagree for `do` (rules/P6, decisions/two-input-invariants). Elsewhere: the rest of the case reading — Title-case `Ma` and `Do` leaning surname, an all-lower member or a one-case name taking the positional count (`ANH DO` → family 'DO', `JOHN SMITH MA` → suffix) — is decisions/S6's 2026-09-14 #289 + #516 entry and decisions/S4's 2026-09-15 (spec review) entry, and two shapes stand outside the case reading altogether: a particle run in front takes `do` whatever its case (`Anh van DO` → family 'van DO', reporting nothing; rules/P2, decisions/S5's 2026-09-18 (Derek), #531 — CAPITALS DECIDE FOR `do` entry), and an unambiguous credential in front of a member speaks for it (`John Smith PhD Ma` → suffix 'PhD Ma'; decisions/S7's 2026-09-27 (Derek), #544 — A CREDENTIAL RUN KEEPS ITS AMBIGUOUS MEMBERS entry); the count that goes first at the first slot after a comma, a suffix comma included (`Smith, MA` → suffix, `Smith, Ma` → given, `John Smith, Ma` → suffix), is decisions/S6's 2026-09-14 (Derek) and 2026-10-20 (Derek), #761 entries; the pairing that gives `do` its capitals-only credential reading after a family comma, and its accepted cost, is decisions/S5's 2026-09-18 (Derek), #531 — CAPITALS DECIDE FOR `do` entry; the escape that reads a bracketed member as a nickname is decisions/suffix-acronym-collisions' S1 line; the periods gate is rules/S3's since 2026-10-19; whether a word earns the ambiguous marking is decisions/vocabulary-collisions' C-i and decisions/suffix-acronym-collisions', with `do`'s departure from TITLES recorded in the former and further credentials marked ambiguous in the latter (2026-10-21 #796). @@ -14,7 +14,7 @@ Open: #792 (S1 and S8: whether a bracketed generation ends the credential run, ` ## Log -- 2026-07-17 (M12, Derek-approved) (its `Jack Ma` sentence superseded in part by #289 below, for the all-caps spelling) — ma and do joined the ambiguous acronym set because both are common surnames; the two-word "Jack Ma" is kept intact by S2's words-to-spare guard, while the periods gate governs the dotted spellings ("M.A." counts unambiguously). Documented side effect: parenthesized bare +- 2026-07-17 (M12, Derek-approved) (its `Jack Ma` sentence superseded in part by #289 below, for the all-caps spelling; its parenthesized side effect superseded in part by decisions/S13's 2026-10-26 #810 entry, for a member after the name whose writing settles it) — ma and do joined the ambiguous acronym set because both are common surnames; the two-word "Jack Ma" is kept intact by S2's words-to-spare guard, while the periods gate governs the dotted spellings ("M.A." counts unambiguously). Documented side effect: parenthesized bare "(MA)"/"(DO)" no longer escape to suffix as in 1.x. - 2026-09-15 #289 — AMENDS the sentence above for one spelling and leaves it standing for the other. "The two-word 'Jack Ma' is kept intact by S2's words-to-spare guard" is still true of `Jack Ma`, and false of `Jack MA`: an all-caps member of the ambiguous set inside a mixed-case name is now read as a credential with nothing to spare, so `Jack MA` gives given 'Jack', suffix 'MA' — v1's reading was last 'MA', so this is a deliberate parity break and not a restoration. The periods gate is untouched and still governs the dotted spellings. The two ledger comments that quote this sentence carry the same amendment. See decisions/S6. - 2026-09-18 #531 — AND `do` IS NOW SPELLING-DEPENDENT AFTER A FAMILY COMMA, which neither sentence above anticipated: it is the one member of this set that is also particle vocabulary, so `Doe, John DO` reads suffix `DO` (capitals, mixed-case name) while every other spelling attaches to the family under rules/P6. The pairing and its accepted cost are in decisions/S5's 2026-09-18 `do` bullet. diff --git a/docs/design/decisions/suffix-acronym-collisions.md b/docs/design/decisions/suffix-acronym-collisions.md index 4bd325487..6d5b451e1 100644 --- a/docs/design/decisions/suffix-acronym-collisions.md +++ b/docs/design/decisions/suffix-acronym-collisions.md @@ -10,7 +10,7 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - `ba`, `meng`, `lac`, `si` and `chi` are marked ambiguous rather than removed, so each is read as the ambiguous class is, by the case it is written in, the words to spare and the comma (rules/S6, S4 and C4): `John Smith SI` keeps the credential and reports the fork, while `John Smith Si`, `Anna Ba` and `Wang, Meng` read the surname or the given name, reported — 2026-09-07 #342; 2026-09-25 #540; 2026-10-21 #791. - `meng` and `lac` carry no case mask, since a mask applies in every role and would re-spell the surnames the marking keeps (`wang meng` would render `Wang MEng`) — 2026-09-25 #540. - `ded` is not marked: it is borne as a given name, which is not the position the suffix claim acts on — 2026-09-25 #540. -- A marked member written in brackets or quotes reads as a nickname, S1's escape declining an ambiguous member (`John Smith (MEng)` → nickname 'MEng'), an accepted cost — 2026-09-25 #540; 2026-10-21. +- A marked member written in brackets or quotes reads as a nickname, S1's escape declining an ambiguous member (`John Smith (MEng)` → nickname 'MEng'), an accepted cost, except after the name where its writing settles it, in capitals in a name of mixed case or with a period inside it (`John Smith (MENG)`, `John Smith (M.Eng)` → suffix), decided in decisions/S13's 2026-10-26 #810 entry — 2026-09-25 #540; 2026-10-21. - Under the default order, a marked member that reads as a name is taken into the family by a particle in front of it, with the words between (`John de Smith Chi` → family 'de Smith Chi', where `John de Smith CHI` keeps suffix 'CHI'), an accepted cost — 2026-10-21 #791; 2026-10-21. - 33 more credential readings, listed by `uv run python tools/vocab/query.py --issue 796` and `se` and `om` among them, are marked ambiguous rather than removed: 30 because the surname reading is substantial (US Census 2010 bearers ≥200, or Wikidata family bearers ≥20), a threshold that is a judgement and not a rule the data sets — 2026-10-21. - Where both counts are small, the surname count is weighed against the NPPES count of US healthcare providers writing the word, and the word is marked where the surname reading is the larger (`capa`, `ams`); a provider count is evidence that a credential is live, and a zero says nothing about a credential outside healthcare, so `ei`, whose credential is not a healthcare one, rests on the default under uncertainty instead — 2026-10-21. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index cd848d705..752c9d2d0 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,6 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - A listed credential that is also a name, written in capitals in a name of mixed case, is settled as the credential by its writing: it reports nothing and starts the credential's run where no name word stands behind it with no family comma (2026-10-26, Derek), and keeps its other readings, the clash settled at the sites that choose a reading rather than by retagging the word (decisions/S6) — 2026-10-25 #752, phase 2g, #785. +- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, with a period inside it or in capitals in a name of mixed case, the case being the name's own words' (rules/S13, decisions/S13) — 2026-10-26 #752, phase 2l. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. - `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. @@ -217,3 +218,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. - **2026-10-26 (Derek), #785 — PHASE 2g NARROWED: a capitals member with a name word behind it, and no family comma, does not start the credential's run**, the name word being evidence of initials; `Maria Ana MA Silva`, `Jean Pierre MA DUPONT` and `Mary Jane MA King` keep the readings 1.4.0 through 2.6.0 gave, and `John Smith MA Jones` keeping family 'Jones' is the cost. The record is decisions/S6's entry of this date. +- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 02d9c2088..128341fc3 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -151,6 +151,7 @@ Terms the rules use without defining them each time, each with the rule that def - [S10 — the trailing roman numeral](S10-trailing-roman-numeral.md) - [S12 — a credential split across a space](S12-credential-split-across-a-space.md) - [S11 — a suffix word that is also a name](S11-suffix-word-that-is-also-a-name.md) +- [S13 — a bracketed member settled by its writing](S13-bracketed-member-settled-by-its-writing.md) ## [Nicknames & quoted names (N)](N-nicknames-and-quoted-names.md) diff --git a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md index 33a949c5e..3fc054af3 100644 --- a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md @@ -22,17 +22,19 @@ "John Smith (MBA PhD.)" → suffix="MBA PhD." · boundary ``` -- **A clause that is one member of the ambiguous class (S4) written without periods** is not suffix-shaped, so in a nickname pair it stays a nickname, and the fork is reported, as it is for such a member anywhere in a nickname: the word is also a name, so the brackets may hold the name the person goes by, an accepted cost (decisions/suffix-acronym-collisions). A credential standing in front of the brackets does not speak for it (S7), and a final period makes the clause suffix-shaped, a member's included. +- **A clause that is one member of the ambiguous class (S4)** is not suffix-shaped, so in a nickname pair it stays a nickname, and the fork is reported, as it is for such a member anywhere in a nickname: the word is also a name, so the brackets may hold the name the person goes by, an accepted cost (decisions/suffix-acronym-collisions). A credential standing in front of the brackets does not speak for it (S7), and a final period makes the clause suffix-shaped, a member's included. ``` - "John Smith (MA)" → nickname="MA" - "John Smith (MA)" → ambiguities=("suffix-or-nickname",) - "John Smith PhD (MA)" → nickname="MA" + "John Smith (Ma)" → nickname="Ma" + "John Smith (Ma)" → ambiguities=("suffix-or-nickname",) + "John Smith PhD (Ma)" → nickname="Ma" "John Smith (M.A.)" → suffix="M.A." · boundary "John Smith (MA.)" → suffix="MA." · boundary ``` **Precedence:** +- **A member after the name whose writing settles it** ("John Smith (MA)", "John Smith (M.A)"): S13 reads it as the suffix, and the nickname reading above is left to a member it does not settle. + - **A bracketed clause led by a maiden marker** ("Jane Smith (née Jr.)"): suffix-shaped content is taken here first, and the clause is no maiden name (M3), so M2's delimiter precedence never reaches it. - **A clause M3 reads as a maiden name** ("Jane Smith (née Jones)"): its content is not suffix-shaped, so this rule leaves it to M3; the marker-led clause that is suffix-shaped is the bullet above. diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md new file mode 100644 index 000000000..4be7cdc27 --- /dev/null +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -0,0 +1,55 @@ +# S13 — a bracketed member settled by its writing + +**When:** a clause that is one member of the ambiguous class (S4) stands in brackets or quotes after the name, as in "John Smith (MA)". + +**Then:** where the member's writing settles it, written with a period inside it or in capitals in a name written in more than one case (S6), it is the suffix rather than the nickname, and nothing is reported. + +- **The case is the name's**: it is asked of the name's own words and never of the clause's, so capitals settle nothing in a name written wholly in one case, and Title case leans to the name (S6). +- **After the name** means that no name word follows the clause, though a suffix or a maiden clause may. +- **It is the suffix, not read as if written bare** (S1): the brackets have already set it apart from the name's words, and bare, two letters each followed by a period but the last read as initials (S3). + +**Because:** after the name, brackets hold either a credential or the name the person goes by, and the writing tells the two apart where the count of words cannot: credentials are written in capitals and with their periods, and names are not. Inside the name the same word let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by. So only the clause after the name is settled. + +``` +"John Smith (MA)" → suffix="MA" +"John Smith (MA)" → ambiguities=() +"John Smith (M.A)" → suffix="M.A" +"JOHN SMITH (M.A)" → suffix="M.A" +"Smith, John (MA)" → suffix="MA" +"John Smith PhD (MA)" → suffix="PhD MA" +"John Smith (MA), PhD" → suffix="MA, PhD" +"John Smith (MA) née Jones" → suffix="MA" +"John Smith «MA»" → suffix="MA" +"John Smith 「MA」" → suffix="MA" +"John Smith (Ed)" → nickname="Ed" · boundary +``` + +**Except:** + +- **A member in a name written in one case, or not in capitals**, is settled only by a period inside it: its case says nothing (S6), and it stays the nickname S1 reads. + ``` + "JOHN SMITH (MA)" → nickname="MA" + "JOHN SMITH (MA)" → ambiguities=("suffix-or-nickname",) + "john smith (MA)" → nickname="MA" + "John Smith (Ma)" → nickname="Ma" + ``` +- **A member standing inside the name or before it** stays the nickname, however it is written. + ``` + "Jeffrey (JD) Bricken" → nickname="JD" + "JEFFREY (JD) BRICKEN" → nickname="JD" + "Jean 'JD' Smith" → nickname="JD" + "Jean (M.A) Smith" → nickname="M.A" + "Smith (MA), John" → nickname="MA" + "(MA) John Smith" → nickname="MA" + ``` + +**Accepted:** + +- A member after a lone name word is the suffix where its writing settles it, and the word reads as a one-word name reads, given rather than the family name it was beside a nickname. + ``` + "Jack (MA)" → given="Jack" + ``` + +--- +**Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name, its writing read as S6 reads capitals +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_group.py diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index e734ddf31..86c8b2364 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -128,7 +128,9 @@ Phase 2i carried this out as [rules/H9](rules/H9-title-phrase.md) (the title phr ### Brackets -A bracketed or quoted credential is to escape the nickname reading where its writing settles it (#810): with interior periods, or in capitals in a name of mixed case, the case asked of the whole name. Today [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md) keeps every member of the ambiguous class written without periods a nickname (`John Smith (MA)` → nickname `MA`), and S1 is amended when the escape lands. `(Ed)` stays a nickname, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`). +A bracketed or quoted credential is to escape the nickname reading where its writing settles it (#810): with interior periods, or in capitals in a name of mixed case, the case asked of the whole name. `(Ed)` stays a nickname, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`). + +Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-its-writing.md) (a bracketed member settled by its writing) rather than as an amendment to [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md), since the position it reads is a signal S1 does not: `John Smith (MA)` and `John Smith (M.A)` read suffix, and `JOHN SMITH (MA)` and `Jeffrey (JD) Bricken` keep the nickname. The case is that of the name's own words, as S6 asks it, and a settled member is the suffix rather than the word read bare, since bare, `M.A` reads as initials. ### Readings on the parse diff --git a/docs/release_log.rst b/docs/release_log.rst index f6b6e910d..5d9042c57 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -15,6 +15,7 @@ Release Log - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752` - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``). :issue:`752` + - **Read a bracketed credential that is also a name as a suffix where its writing settles it.** ``John Smith (MA)`` and ``John Smith (M.A)`` give suffix ``MA`` and ``M.A``, as 1.4.0 did, where 2.0.0 through 2.6.0 gave nickname ``MA`` and ``M.A``; ``Andrew Perkins (JD)`` gives suffix ``JD``, where 1.4.0 through 2.6.0 gave nickname ``JD``. Only capitals in a mixed-case name or a period inside the word settle it, and only after the name, so ``John Smith (Ed)``, ``JOHN SMITH (MA)`` and ``Jeffrey (JD) Bricken`` keep the nickname and report a ``suffix-or-nickname`` ambiguity. :issue:`810`, :issue:`752` - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` * 2.6.0 - October 25, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 9f1573da2..8f2594a43 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -843,10 +843,18 @@ parenthesized nicknames that happen to be credentials: The exception to that exception is an *ambiguous* acronym — one that is also a plausible name or set of initials. Standing alone inside -delimiters, it keeps the nickname reading: +delimiters, it keeps the nickname reading unless the way it is written +settles it after the name: in capitals in a name written in mixed case, +or with a period inside it. A name written all in capitals says +nothing about the word, and inside the name the delimiters are read as +a nickname however it is written: .. doctest:: + >>> parse("Andrew Perkins (JD)").suffix + 'JD' + >>> parse("Andrew Perkins (Jd)").nickname + 'Jd' >>> parse("JEFFREY (JD) BRICKEN").nickname 'JD' diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 0b8030042..125e8f617 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -7,7 +7,7 @@ second comma asks for it (#289/#516, #613), so this read is the fallback for every other name. Produces: tokens with vocabulary tags added (text/span/role unchanged), -plus ambiguities (SUFFIX_OR_NICKNAME, CONJUNCTION_OR_INITIAL) and +plus ambiguities (CONJUNCTION_OR_INITIAL) and one_case -- whether the name's own words are written in one case, recorded for the later stages that read it (#289/#516) and left alone where an earlier stage already asked. @@ -72,7 +72,7 @@ PendingAmbiguity, WorkToken, copy_with, ) -from nameparser._types import AmbiguityKind, Role +from nameparser._types import AmbiguityKind from nameparser._pipeline._vocab import ( BUILTIN_SUFFIX_DELIMITERS, SPLIT_CHUNK, SPLIT_PAIR_OPENS, _INITIAL, caps_shape_candidate, composed_len, delimiter_cores, glued_stem, is_cased_paired_initials, @@ -249,7 +249,7 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # role is delimited content, decided by extract's escape and never # at the trailing slot -- 'Bridge (A.B)' is the control that # proves the guard load-bearing (without it the nickname reading - # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report below), and + # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report in group), and # 'Bridge (1.4)' cannot exercise it, a digit chunk never reaching # the shape verdict at all (period_joined_vocab's alphabetic # gate). @@ -515,21 +515,8 @@ def classify(state: ParseState) -> ParseState: for i, tag in split.items(): written[i] = copy_with(written[i], tags=written[i].tags | {tag}) tokens = tuple(written) - # Delimited content whose vocabulary cannot settle it: extract's - # escape sends an UNambiguous suffix straight through ("(MBA)" -> - # suffix) and keeps everything else as a nickname, so an AMBIGUOUS - # acronym in there was a coin the parser had to call. Reported here - # rather than at the escape itself, which runs before tokenize and - # so has no token index to point at. ambiguities = list(state.ambiguities) for i, token in enumerate(tokens): - if (token.role is Role.NICKNAME - and AMBIGUOUS_ACRONYM_TAG in token.tags): - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NICKNAME, - f"delimited {token.text!r} is also a post-nominal; read " - f"as a nickname rather than a suffix", - (i,))) # #383/#479: narrows the fork's own "initial" tag with the same # inputs the fork used, rather than re-deciding from scratch. # Only the casedness test is inherited from the tag -- diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 478b4945b..f2041e596 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1,8 +1,9 @@ """Stage: group. Consumes: tokens (classified), segments, structure, one_case, extracted -(the role + inner span per delimited region, for the #329 pass below -- -the only stage after tokenize that reads it); and, through +(the role + inner span per delimited region, for the #329 pass below and +S13's one-word clause test -- the only stage after tokenize that reads +it), and original (the clause's text); and, through `_comma.decide`, comma_offsets (the own-words span of the name's case contrast) and dropped, which decide extends. Produces: structure -- its head decides a comma form once the words @@ -21,9 +22,14 @@ and tail_reads, rules/S2's trailing run of each main segment read ONCE, after P3's connective joins and before the particle chain and the bound-given join, and split off so neither reaches it (#614), -which assign places and reports rather than peeling again. +which assign places and reports rather than peeling again; and the +SUFFIX role of a one-word nickname that is a member of the ambiguous +class settled by its writing after the name (rules/S13, #810), with the +SUFFIX_OR_NICKNAME report on any other member left a nickname. Reads: token tags (from classify), Lexicon.given_name_titles (the -P5 licence, #369) and Policy.extra_suffix_delimiters, whose +P5 licence, #369), Lexicon.suffix_acronyms_ambiguous (a dotted member in +a clause, which carries no tag there, rules/S13) and +Policy.extra_suffix_delimiters, whose delimiter-core tokens part a tail segment as a comma would and are dropped (v1 suffix_delimiter parity, #549); and, through `_comma.decide`, what that module's own header lists @@ -33,7 +39,7 @@ per-parse state that dissolves with the state (v1 kept per-parse sets for the same reason). -Implements rules P2, P3, P4 and M2, S2's trailing read (#614), C1's +Implements rules P2, P3, P4, M2 and S13, S2's trailing read (#614), C1's decision through _comma, and the group half of M1 (#329: the marker dropped inside EXTRACTED maiden @@ -65,7 +71,7 @@ from enum import IntEnum from typing import assert_never -from nameparser._lexicon import _run_addresses_by_given +from nameparser._lexicon import _normalize, _run_addresses_by_given from nameparser._pipeline._pieces import ( split_credential_report, Peel, TailRead, chain_lead, chain_run_end, is_conj_piece, is_leading_title, is_prefix_piece, @@ -1295,6 +1301,74 @@ def chain(tail: int) -> None: return pieces, ptags, taken +def _alone_in_its_clause(token: WorkToken, state: ParseState) -> bool: + """Whether a delimited token is the whole of its clause.""" + for _, clause in state.extracted: + if clause.start <= token.span.start and token.span.end <= clause.end: + return (state.original[clause.start:clause.end].strip() + == token.text) + return False + + +def _name_word_after(tokens: Sequence[WorkToken], i: int, + dropped: Sequence[int]) -> bool: + """Whether a word of the name proper follows token i: a word with + no role that is not a suffix as written, up to a maiden marker, + the clause from there on being no name of its own (`own_words`).""" + for j in range(i + 1, len(tokens)): + t = tokens[j] + if t.role is not None or j in dropped: + continue + if "vocab:maiden-marker" in t.tags: + return False + if "vocab:suffix" not in t.tags: + return True + return False + + +# rules/S13: "where the member's writing settles it, written with a +# period inside it or in capitals in a name written in more than one +# case (S6), it is the suffix rather than the nickname" -- Then, with +# the Except bullet keeping a member inside the name the nickname; +# extract's S1 escape has already sent every suffix-shaped clause +# through (decisions/S13 (#810)) +def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], + ambiguities: list[PendingAmbiguity]) -> None: + """A one-word nickname that is a member of the ambiguous class + becomes the suffix where its writing settles it and no name word + follows it; any other tagged member stays the nickname and the + fork is reported. Asked here rather than in extract, which runs + before there are words to ask about: the case is the name's + recorded fact (`settled_by_case`, rules/S6), and whether a name + word follows is read off the tags. A dotted member carries no tag + in a clause (classify tags a shape only outside one), so it is + found by its text.""" + lex = state.lexicon + for i, token in enumerate(tokens): + if token.role is not Role.NICKNAME: + continue + tagged = AMBIGUOUS_ACRONYM_TAG in token.tags + # a period inside the word ('M.A'); one ending it is extract's + # escape + dotted = ("." in token.text.rstrip(".") + and _normalize(token.text).replace(".", "") + in lex.suffix_acronyms_ambiguous) + if not (tagged or dotted): + continue + if ((dotted or settled_by_case(token, state.one_case)) + and _alone_in_its_clause(token, state) + and not _name_word_after(tokens, i, state.dropped)): + tokens[i] = copy_with(token, role=Role.SUFFIX) + elif tagged: + # Reported here, where the escape is decided, rather than + # in extract, which has no token index to point at. + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NICKNAME, + f"delimited {token.text!r} is also a post-nominal; read " + f"as a nickname rather than a suffix", + (i,))) + + def group(state: ParseState) -> ParseState: # rules/C1's decision, read once now that classify has tagged # the words and before any join below can reach the part after the @@ -1305,6 +1379,8 @@ def group(state: ParseState) -> ParseState: tokens = list(state.tokens) dropped = list(state.dropped) ambiguities = list(state.ambiguities) + if state.extracted: + _settle_delimited_members(state, tokens, ambiguities) all_pieces: list[tuple[tuple[int, ...], ...]] = [] all_ptags: list[tuple[frozenset[str], ...]] = [] all_reads: list[TailRead | None] = [] diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 2ad1ca947..92d4bc556 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -195,7 +195,9 @@ class ParseState: changes the token COUNT: an unspaced CJK token splits into n+1 pieces, still as sub-slices of the original, and every later index in the segment runs shifts by n); classify -> token tags AND - one_case; group -> structure (its head decides a comma form, #613) + one_case; group -> structure (its head decides a comma form, #613), + the SUFFIX role of a delimited member its writing settles (rules/S13, + #810) AND the SUFFIX/TITLE roles of a postnominal part after the comma, whose segment it empties so no join reaches it, then pieces/piece_tags/dropped AND maiden token roles, plus the diff --git a/nameparser/_types.py b/nameparser/_types.py index 07cd85e74..2f7a36dce 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -444,7 +444,8 @@ class AmbiguityKind(StrEnum): #: Delimited content is an ambiguous suffix acronym, so it reads #: plausibly as either a post-nominal or a nickname -- "JEFFREY #: (JD) BRICKEN" keeps the nickname reading, where the - #: unambiguous "(MBA)" escapes to suffix on vocabulary alone. + #: unambiguous "(MBA)" escapes to suffix on vocabulary alone, and + #: "Andrew Perkins (JD)" on its capitals, which settle it. SUFFIX_OR_NICKNAME = "suffix-or-nickname" #: A trailing word reads plausibly as either a post-nominal or an #: ordinary name part. Covers an ambiguous acronym written without diff --git a/tests/test_suffixes.py b/tests/test_suffixes.py index ea91869ce..044685f64 100644 --- a/tests/test_suffixes.py +++ b/tests/test_suffixes.py @@ -464,26 +464,31 @@ def test_suffix_acronyms_ambiguous_custom_entry_stays_nickname(self) -> None: # A custom suffix_acronyms_ambiguous entry keeps a suffix_acronyms # member classified as a nickname instead of a suffix, confirming # the exception list -- not a hardcoded check -- drives the behavior. + # Title case: capitals in a mixed-case name settle the member as + # the credential (rules/S1, #810). from nameparser.config import Constants C = Constants( suffix_acronyms=['xyz'], suffix_acronyms_ambiguous=['xyz'], ) - hn = HumanName("Andrew Perkins (XYZ)", constants=C) - self.m(hn.nickname, "XYZ", hn) + hn = HumanName("Andrew Perkins (Xyz)", constants=C) + self.m(hn.nickname, "Xyz", hn) self.m(hn.suffix, "", hn) def test_suffix_acronyms_ambiguous_removal_routes_to_suffix(self) -> None: - # Removing 'jd' from a custom suffix_acronyms_ambiguous flips JD + # Removing 'jd' from a custom suffix_acronyms_ambiguous flips Jd # from nickname to suffix. Uses a trailing-position name (unlike the # JEFFREY (JD) BRICKEN regression guard in test_nicknames.py) so # parse_full_name's trailing-run suffix detection actually picks it # up -- see the known mid-name limitation noted on the tests above. + # Title case, which leaves the member a nickname while it is listed + # (rules/S1, #810). from nameparser.config import Constants + assert HumanName("Andrew Perkins (Jd)").nickname == "Jd" C = Constants(suffix_acronyms_ambiguous=[]) - hn = HumanName("Andrew Perkins (JD)", constants=C) + hn = HumanName("Andrew Perkins (Jd)", constants=C) self.m(hn.nickname, "", hn) - self.m(hn.suffix, "JD", hn) + self.m(hn.suffix, "Jd", hn) def test_empty_comma_segment_does_not_drop_following_suffix(self) -> None: # Regression: "Doe, John,, Jr." produced an empty third segment, and diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 9d2009510..d491881c1 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8799,6 +8799,96 @@ def _check_cjk_shape_purity(self) -> None: "2026-07-17)"), Case("paren_period_escapes_nickname", "Andrew Perkins (Ret.)", {"given": "Andrew", "family": "Perkins", "suffix": "Ret."}), + # rules/S13 (#810): a member of the ambiguous class alone in the + # brackets after the name, settled by its writing. Each row is + # paired with a control its writing or its position does not settle. + Case("bracket_member_in_capitals_is_the_suffix", "John Smith (MA)", + {"given": "John", "family": "Smith", "suffix": "MA"}, + notes="capitals in a mixed-case name settle it; 2.0.0 through " + "2.6.0 read nickname 'MA', reported"), + Case("bracket_member_in_title_case_stays_the_nickname", + "John Smith (Ed)", + {"given": "John", "family": "Smith", "nickname": "Ed"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="Title case leans to the name (rules/S6); Ed is " + "Edward's nickname far more often than a credential"), + Case("bracket_member_with_a_period_inside_is_the_suffix", + "John Smith (M.A)", + {"given": "John", "family": "Smith", "suffix": "M.A"}, + notes="the suffix, not the bare reading: bare, M.A reads as " + "paired initials (rules/S3)"), + Case("bracket_member_with_a_period_inside_in_one_case", + "JOHN SMITH (M.A)", + {"given": "JOHN", "family": "SMITH", "suffix": "M.A"}, + notes="the period settles it whatever the name's case"), + Case("bracket_member_in_a_one_case_name_stays_the_nickname", + "JOHN SMITH (MA)", + {"given": "JOHN", "family": "SMITH", "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="capitals settle nothing in a name written in one case; " + "1.4.0 read suffix 'MA'"), + Case("bracket_member_in_a_lower_case_name_stays_the_nickname", + "john smith (MA)", + {"given": "john", "family": "smith", "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="the case asked is the name's own words', never the " + "clause's, so this differs from bare 'john smith MA'; " + "1.4.0 read suffix 'MA'"), + Case("bracket_member_after_a_one_case_name_with_a_nickname", + "JOHN SMITH (Bob) (MA)", + {"given": "JOHN", "family": "SMITH", "nickname": "Bob MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="another clause's case is no name's: 'Bob' does not " + "make the name mixed; 1.4.0 read suffix 'MA'"), + Case("bracket_member_jd_is_the_suffix", "Andrew Perkins (JD)", + {"given": "Andrew", "family": "Perkins", "suffix": "JD"}, + classification="fix(#810)", + notes="1.4.0 through 2.6.0 read nickname 'JD'"), + Case("bracket_member_inside_the_name_stays_the_nickname", + "Jeffrey (JD) Bricken", + {"given": "Jeffrey", "family": "Bricken", "nickname": "JD"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="a name word follows: let out, JD would be a middle name"), + Case("bracket_member_with_a_period_inside_the_name", + "Jean (M.A) Smith", + {"given": "Jean", "family": "Smith", "nickname": "M.A"}, + classification="fix(#810)", + notes="inside the name however it is written; 1.4.0 read " + "middle 'M.A'"), + Case("bracket_member_before_a_family_comma_stays_the_nickname", + "Smith (MA), John", + {"given": "John", "family": "Smith", "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="the given name after the comma is a name word after " + "it; 1.4.0 read suffix 'MA'"), + Case("bracket_member_after_a_family_comma", "Smith, John (MA)", + {"given": "John", "family": "Smith", "suffix": "MA"}), + Case("bracket_member_before_a_suffix_comma", "John Smith (MA), PhD", + {"given": "John", "family": "Smith", "suffix": "MA, PhD"}, + notes="a suffix after the clause is no name word"), + Case("bracket_member_before_a_maiden_clause", + "John Smith (MA) née Jones", + {"given": "John", "family": "Smith", "suffix": "MA", + "maiden": "Jones"}, + classification="fix(#810)", + notes="a maiden clause after it is no name of its own; 1.4.0 " + "read middle 'Smith MA née', last 'Jones'"), + Case("bracket_member_in_quotes", 'John Smith "MA"', + {"given": "John", "family": "Smith", "suffix": "MA"}, + notes="every default nickname pair escapes alike (rules/S1)"), + Case("bracket_member_after_a_lone_name_word", "Jack (MA)", + {"given": "Jack", "suffix": "MA"}, + classification="fix(#810)", + ambiguities=("given-or-family",), + notes="reads as bare 'Jack MA' does (rules/S13's Accepted); " + "1.4.0 read last 'MA', 2.0.0 through 2.6.0 family " + "'Jack', nickname 'MA'"), Case("nickname_quotes", 'John "Jack" Kennedy', {"given": "John", "family": "Kennedy", "nickname": "Jack"}, notes="shape 1's double-quoted Nickname slot, which is the " diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index f1c519e8f..9ffc94756 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -576,8 +576,8 @@ def test_delimited_content_never_joins_the_shape_class() -> None: # control that actually load-bears it: without the guard, the # nickname 'A.B' would gain SHAPE_ACRONYM_TAG and (with the switch # on) `vocab:suffix-ambiguous`, and classify's own nickname check - # would then misreport SUFFIX_OR_NICKNAME on a token the escape - # already decided (measured regression, #516 review round). + # would then misreport SUFFIX_OR_NICKNAME (group's report, since + # #810) on a token the escape already decided (measured regression, #516 review round). for text, word in (("Bridge (1.4)", "1.4"), ("Bridge (A.B)", "A.B")): tags = _tags_by_text(text)[word] assert "shape:acronym" not in tags, text diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index 51b8b9304..696160fbe 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -281,9 +281,8 @@ def test_stage_field_ownership() -> None: # FAMILY_COMMA opt-out, asked again for itself since #613, and # the hook it consults on a vocabulary decline). "script_segment": {"tokens", "segments", "ambiguities"}, - # classify also emits SUFFIX_OR_NICKNAME: the delimiter escape - # that decides it lives in extract_delimited, which has no token - # index to point at, so the report is raised here instead + # classify emits CONJUNCTION_OR_INITIAL; SUFFIX_OR_NICKNAME + # moved to group with the decision it reports (rules/S13, #810) "classify": {"tokens", "ambiguities", "one_case"}, # group also emits PARTICLE_OR_GIVEN: the prefix chain takes # the particle branch of a fork whose given branch _assign @@ -295,7 +294,9 @@ def test_stage_field_ownership() -> None: # and empties the segment of a part it bound, so no join can # reach it (#613's /simplify) # and since #614 the trailing run it reads once at its head, - # handed to assign (`tail_reads`) + # handed to assign (`tail_reads`); and since #810 the SUFFIX + # role of a delimited member its writing settles (rules/S13), + # reporting SUFFIX_OR_NICKNAME on any other "group": {"tokens", "pieces", "piece_tags", "dropped", "ambiguities", "structure", "segments", "tail_reads"}, # assign also records `order`: the effective order it read the diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 1a87ba2a2..578f45a23 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -53,7 +53,7 @@ "R3": {"words": 483, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, "R4": {"words": 991, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, + "S1": {"words": 306, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S2": {"words": 657, "then_words": 119, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, "S3": {"words": 960, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 5, "accepted": 6, "accepted_words": 343}, "S4": {"words": 306, "then_words": 164, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, @@ -65,6 +65,7 @@ "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "S12": {"words": 674, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, + "S13": {"words": 284, "then_words": 132, "because_sentences": 3, "except": 2, "precedence": 0, "accepted": 1, "accepted_words": 35}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 0abbeaacf..d3ad7dc1d 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -271,8 +271,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 1321 + 24 = 1345 on 2026-10-26, #894 (phase 2i) merged over #895 # (phase 2g): 2i's net change from 1568 (1592 - 1568), the two # changes moving disjoint claims. - assert claims == 1345, ( - f"the field sweep checks {claims} claims, recorded as 1345 on " + # 1345 -> 1366 on 2026-10-26, #810 (phase 2l): seven rows whose + # bracketed member stays the nickname, one suffix-or-nickname or + # given-or-family claim each under each order. + assert claims == 1366, ( + f"the field sweep checks {claims} claims, recorded as 1366 on " f"2026-10-26") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 6309a4157..9104f0a48 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -4,6 +4,7 @@ "'Smitty' Jones Jr." "'Smitty' Sir John" "(" +"(MA) John Smith" ", Ana García López" ", Dean" ", Dr. fils" @@ -240,8 +241,11 @@ "J.Smith" "JACK MA" "JANE DOE MA R.T." +"JEFFREY (JD) BRICKEN" "JOHN PROF. MA" "JOHN QUINCY SMITH I" +"JOHN SMITH (M.A)" +"JOHN SMITH (MA)" "JOHN SMITH JR, AD" "JOHN SMITH MA" "JOHN SMITH MA JONES" @@ -258,6 +262,7 @@ "JOSÉ GARCÍA" "JUAN GARCIA Jr." "JUAN GARCIA Y LOPEZ" +"Jack (MA)" "Jack I-V" "Jack I.-V." "Jack MA" @@ -331,11 +336,13 @@ "Jane van der Berg née y Jones" "Jane „JD Smith" "Jean 'JD' Smith" +"Jean (M.A) Smith" "Jean DUPONT" "Jean Marie Fils" "Jean Pierre DUPONT" "Jean Pierre DUPONT-MORETTI" "Jean Pierre MA DUPONT" +"Jeffrey (JD) Bricken" "Jiří Dienstbier ml." "John . Smith" "John Dean Smith" @@ -355,11 +362,16 @@ "John Prof. MA" "John Quincy Smith i" "John Smith" +"John Smith (Ed)" +"John Smith (M.A)" "John Smith (M.A.)" "John Smith (MA PhD)" "John Smith (MA)" +"John Smith (MA) née Jones" +"John Smith (MA), PhD" "John Smith (MA.)" "John Smith (MBA PhD.)" +"John Smith (Ma)" "John Smith (PhD MBA)" "John Smith 1.4" "John Smith DO Jones" @@ -399,6 +411,7 @@ "John Smith Ph. D. Jones" "John Smith PhD" "John Smith PhD (MA)" +"John Smith PhD (Ma)" "John Smith PhD Ed Ma" "John Smith PhD Jones" "John Smith PhD MEng" @@ -434,6 +447,8 @@ "John Smith XYZ" "John Smith Xyz." "John Smith vi" +"John Smith «MA»" +"John Smith 「MA」" "John Smith 博. 士." "John Smith, A-B" "John Smith, A.B." @@ -708,6 +723,7 @@ "Sita Kumari K.C" "Smith" "Smith (Jones)" +"Smith (MA), John" "Smith (née Jones)" "Smith Jones Jr., J.A." "Smith Jr." @@ -738,6 +754,7 @@ "Smith, Jan van Prof." "Smith, Jane Mrs" "Smith, John" +"Smith, John (MA)" "Smith, John MA Jones" "Smith, John MD/PhD" "Smith, John PSM I, Jr." @@ -878,6 +895,7 @@ "john i phd" "john j v" "john j v, PhD" +"john smith (MA)" "john smith EdD" "john smith MEng" "john smith bsc" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index e80dbd87e..961db484b 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -7,6 +7,7 @@ {"name": "'Smitty' Sir John", "title": "Sir", "given": "John", "nickname": "Smitty", "initials": "J."} {"name": "(", "ambiguities": ["UNBALANCED_DELIMITER"]} {"name": "()"} +{"name": "(MA) John Smith", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "김. 민. J. S."} {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} @@ -199,7 +200,7 @@ {"name": "Andrew Perkins \"MBA\"", "given": "Andrew", "family": "Perkins", "suffix": "MBA", "initials": "A. P."} {"name": "Andrew Perkins 'MBA'", "given": "Andrew", "family": "Perkins", "suffix": "MBA", "initials": "A. P."} {"name": "Andrew Perkins (Andy)", "given": "Andrew", "family": "Perkins", "nickname": "Andy", "initials": "A. P."} -{"name": "Andrew Perkins (JD)", "given": "Andrew", "family": "Perkins", "nickname": "JD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "A. P.", "capitalized_force": "Andrew \"Jd\" Perkins"} +{"name": "Andrew Perkins (JD)", "given": "Andrew", "family": "Perkins", "suffix": "JD", "initials": "A. P."} {"name": "Andrew Perkins (M.D)", "given": "Andrew", "family": "Perkins", "suffix": "M.D", "initials": "A. P."} {"name": "Andrew Perkins (MBA)", "given": "Andrew", "family": "Perkins", "suffix": "MBA", "initials": "A. P."} {"name": "Andrew Perkins (Mgr.)", "title": "Mgr.", "given": "Andrew", "family": "Perkins", "initials": "A. P."} @@ -1372,6 +1373,8 @@ {"name": "JOHN PROF. MA", "title": "PROF.", "given": "JOHN", "family": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M.", "capitalized": "Prof. John Ma"} {"name": "JOHN QUINCY SMITH I", "given": "JOHN", "middle": "QUINCY", "family": "SMITH", "suffix": "I", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. Q. S.", "capitalized": "John Quincy Smith I"} {"name": "JOHN SMITH", "given": "JOHN", "family": "SMITH", "initials": "J. S.", "capitalized": "John Smith"} +{"name": "JOHN SMITH (M.A)", "given": "JOHN", "family": "SMITH", "suffix": "M.A", "initials": "J. S.", "capitalized": "John Smith M.A"} +{"name": "JOHN SMITH (MA)", "given": "JOHN", "family": "SMITH", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized": "John \"Ma\" Smith"} {"name": "JOHN SMITH JR, AD", "given": "AD", "family": "JOHN SMITH", "suffix": "JR", "initials": "A. J. S.", "capitalized": "Ad John Smith Jr"} {"name": "JOHN SMITH MA", "given": "JOHN", "family": "SMITH", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized": "John Smith MA"} {"name": "JOHN SMITH MA JONES", "given": "JOHN", "middle": "SMITH MA", "family": "JONES", "initials": "J. S. M. J.", "capitalized": "John Smith Ma Jones"} @@ -1398,6 +1401,7 @@ {"name": "Jacinto López Moreno", "given": "Jacinto", "middle": "López", "family": "Moreno", "initials": "J. L. M."} {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "Jack (MA)", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} {"name": "Jack I.-V.", "given": "Jack", "family": "I.-V.", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} @@ -1529,6 +1533,7 @@ {"name": "Jaume Martí i Cabot", "given": "Jaume", "family": "Martí i Cabot", "initials": "J. M. C."} {"name": "Javier López Díaz", "given": "Javier", "middle": "López", "family": "Díaz", "initials": "J. L. D."} {"name": "Jean 'JD' Smith", "given": "Jean", "family": "Smith", "nickname": "JD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "Jean \"Jd\" Smith"} +{"name": "Jean (M.A) Smith", "given": "Jean", "family": "Smith", "nickname": "M.A", "initials": "J. S."} {"name": "Jean DUPONT", "given": "Jean", "family": "DUPONT", "initials": "J. D.", "capitalized_force": "Jean Dupont"} {"name": "Jean Joseph Antoine Marie de Witte", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Jean Louis Charles d'Orléans-Longueville", "given": "Jean", "middle": "Louis Charles", "family": "d'Orléans-Longueville", "initials": "J. L. C. d.", "capitalized_force": "Jean Louis Charles D'Orléans-Longueville"} @@ -1543,6 +1548,7 @@ {"name": "Jean «Petit» Dupont", "given": "Jean", "family": "Dupont", "nickname": "Petit", "initials": "J. D."} {"name": "Jean-Dominique de Cassini", "given": "Jean-Dominique", "family": "de Cassini", "initials": "J. C."} {"name": "Jean-François Lefebvre de La Barre", "given": "Jean-François", "middle": "Lefebvre", "family": "de La Barre", "initials": "J. L. B.", "capitalized_force": "Jean-François Lefebvre de la Barre"} +{"name": "Jeffrey (JD) Bricken", "given": "Jeffrey", "family": "Bricken", "nickname": "JD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. B.", "capitalized_force": "Jeffrey \"Jd\" Bricken"} {"name": "Jenny \"JJ\" Baker (Johnson)", "given": "Jenny", "family": "Baker", "nickname": "JJ Johnson", "initials": "J. B.", "capitalized_force": "Jenny \"Jj Johnson\" Baker"} {"name": "Jenny (Johnson) Baker", "given": "Jenny", "family": "Baker", "nickname": "Johnson", "initials": "J. B."} {"name": "Jenny Baker", "given": "Jenny", "family": "Baker", "initials": "J. B."} @@ -1649,11 +1655,16 @@ {"name": "John Q. Smith", "given": "John", "middle": "Q.", "family": "Smith", "initials": "J. Q. S."} {"name": "John Quincy Smith i", "given": "John", "middle": "Quincy", "family": "Smith", "suffix": "i", "initials": "J. Q. S.", "capitalized_force": "John Quincy Smith I"} {"name": "John Smith", "given": "John", "family": "Smith", "initials": "J. S."} +{"name": "John Smith (Ed)", "given": "John", "family": "Smith", "nickname": "Ed", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S."} +{"name": "John Smith (M.A)", "given": "John", "family": "Smith", "suffix": "M.A", "initials": "J. S."} {"name": "John Smith (M.A.)", "given": "John", "family": "Smith", "suffix": "M.A.", "initials": "J. S."} {"name": "John Smith (MA PhD)", "given": "John", "family": "Smith", "nickname": "MA PhD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma PhD\" Smith"} -{"name": "John Smith (MA)", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} +{"name": "John Smith (MA)", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} +{"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA", "maiden": "Jones", "initials": "J. S."} +{"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} {"name": "John Smith (MA.)", "given": "John", "family": "Smith", "suffix": "MA.", "initials": "J. S."} {"name": "John Smith (MBA PhD.)", "given": "John", "family": "Smith", "suffix": "MBA PhD.", "initials": "J. S."} +{"name": "John Smith (Ma)", "given": "John", "family": "Smith", "nickname": "Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S."} {"name": "John Smith (PhD MBA)", "given": "John", "family": "Smith", "nickname": "PhD MBA", "initials": "J. S.", "capitalized_force": "John \"PhD Mba\" Smith"} {"name": "John Smith 1.4", "given": "John", "middle": "Smith", "family": "1.4", "initials": "J. S. 1."} {"name": "John Smith B.Tech.", "given": "John", "family": "Smith", "suffix": "B.Tech.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith B.TECH."} @@ -1706,7 +1717,8 @@ {"name": "John Smith Ph. D.", "given": "John", "family": "Smith", "suffix": "Ph. D.", "initials": "J. S."} {"name": "John Smith Ph. D. Jones", "given": "John", "family": "Smith", "suffix": "Ph. D. Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith PhD", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} -{"name": "John Smith PhD (MA)", "given": "John", "family": "Smith", "suffix": "PhD", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith PhD"} +{"name": "John Smith PhD (MA)", "given": "John", "family": "Smith", "suffix": "PhD MA", "initials": "J. S."} +{"name": "John Smith PhD (Ma)", "given": "John", "family": "Smith", "suffix": "PhD", "nickname": "Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S."} {"name": "John Smith PhD Ed Ma", "given": "John", "family": "Smith", "suffix": "PhD Ed Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith PhD ED MA"} {"name": "John Smith PhD Jones", "given": "John", "family": "Smith", "suffix": "PhD Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith PhD MEng", "given": "John", "family": "Smith", "suffix": "PhD MEng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith PhD MENG"} @@ -1746,6 +1758,8 @@ {"name": "John Smith Xyz.", "given": "John", "middle": "Smith", "family": "Xyz.", "initials": "J. S. X."} {"name": "John Smith nee Jones R.A.I.", "given": "John", "family": "Smith", "suffix": "R.A.I.", "maiden": "Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith vi", "given": "John", "family": "Smith", "suffix": "vi", "initials": "J. S.", "capitalized_force": "John Smith VI"} +{"name": "John Smith «MA»", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} +{"name": "John Smith 「MA」", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "John Smith 博. 士.", "given": "John", "middle": "Smith 博.", "family": "士.", "initials": "J. S. 博. 士."} {"name": "John Smith 田.中.", "given": "John", "middle": "Smith", "family": "田.中.", "initials": "J. S. 田."} {"name": "John Smith, 1.4", "given": "1.4", "family": "John Smith", "initials": "1. J. S."} @@ -3100,6 +3114,7 @@ {"name": "Slovák, Ján Matej", "given": "Ján", "middle": "Matej", "family": "Slovák", "initials": "J. M. S."} {"name": "Smith", "given": "Smith", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S."} {"name": "Smith (Jones)", "family": "Smith", "nickname": "Jones", "initials": "S."} +{"name": "Smith (MA), John", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} {"name": "Smith (née Jones)", "family": "Smith", "maiden": "Jones", "initials": "S."} {"name": "Smith , John", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith Cecilia Angela", "order": "FAMILY_FIRST", "given": "Cecilia", "middle": "Angela", "family": "Smith", "initials": "C. A. S."} @@ -3151,6 +3166,7 @@ {"name": "Smith, Jan van Prof.", "title": "Prof.", "given": "Jan", "family": "van Smith", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. S."} {"name": "Smith, Jane Mrs", "title": "Mrs", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Smith, John", "given": "John", "family": "Smith", "initials": "J. S."} +{"name": "Smith, John (MA)", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "Smith, John ,", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, John E, III, Jr", "given": "John", "middle": "E", "family": "Smith", "suffix": "III, Jr", "initials": "J. E. S."} {"name": "Smith, John I", "given": "John", "family": "Smith", "suffix": "I", "initials": "J. S."} @@ -3751,6 +3767,7 @@ {"name": "john j v", "given": "john", "middle": "j", "family": "v", "initials": "j. j. v.", "capitalized": "John J V"} {"name": "john j v, PhD", "given": "john", "family": "j", "suffix": "v, PhD", "initials": "j. j.", "capitalized": "John J V, PhD"} {"name": "john smith", "given": "john", "family": "smith", "initials": "j. s.", "capitalized": "John Smith"} +{"name": "john smith (MA)", "given": "john", "family": "smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "j. s.", "capitalized_force": "John \"Ma\" Smith"} {"name": "john smith EdD", "given": "john", "family": "smith", "suffix": "EdD", "initials": "j. s.", "capitalized": "John Smith EdD", "capitalized_force": "John Smith EDD"} {"name": "john smith MA", "given": "john", "family": "smith", "suffix": "MA", "initials": "j. s.", "capitalized": "John Smith MA"} {"name": "john smith MEng", "given": "john", "middle": "smith", "family": "MEng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s. M.", "capitalized_force": "John Smith Meng"} From afdc69ca3a8bc0f24740e50c51c00d4784b2233d Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 14:06:54 -0700 Subject: [PATCH 02/10] Decide S13 in post_rules, on the roles the parse read (#810 review) The first draft decided in group and called any word after the clause lacking the suffix tag a name word, so a suffix reached by shape (VI, Ph. D., X.Y.Z., a capitals member), fils and a trailing title kept the clause a nickname while their listed twins let it through. Deciding first in post_rules, after the last given, middle or family token, answers for every spelling; one bisect pass maps clauses to tokens, so the pass is linear (test_benchmark's bracket_members row fails on the first draft). Jack (MA) now keeps N3's family name. A dotted member left a nickname inside the name is reported, as S1's widened bullet says. S13's Signal names both its signals and a Precedence bullet against S3; mechanisms.md's CLAUSE-CONTENT-OVERRULES-DELIMITER names the second site; the cross-file Standing clauses become Elsewhere lines; the ctgov count is corrected. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 +- ...bracketed-member-settled-by-its-writing.md | 17 ++-- docs/design/decisions/ma-do.md | 4 +- .../decisions/suffix-acronym-collisions.md | 4 +- docs/design/mechanisms.md | 4 +- ...bracketed-member-settled-by-its-writing.md | 19 ++-- nameparser/_pipeline/_classify.py | 2 +- nameparser/_pipeline/_group.py | 88 ++----------------- nameparser/_pipeline/_post_rules.py | 72 +++++++++++++-- nameparser/_pipeline/_state.py | 10 +-- tests/v2/cases.py | 32 +++++-- tests/v2/pipeline/test_classify.py | 2 +- tests/v2/pipeline/test_state.py | 10 +-- tests/v2/rules_budget.json | 2 +- tests/v2/test_benchmark.py | 1 + tests/v2/test_cases.py | 9 +- tools/differential/corpus_rules.jsonl | 2 + tools/differential/snapshot.jsonl | 6 +- 18 files changed, 155 insertions(+), 133 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8f480d093..2e747ec45 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,7 +186,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams -- and a new chunk reading inherits the old one's exceptions, which #752's first draft did not (`Jack I-V` lost its family to the listed numerals, and the fix's length test counted a chunk's period, so `Jack I.-V.` did too, found by the second review), and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. Capitals between two name words have the same second reading: some writers put their own initials there (`Maria Ana MA Silva`), so a rule that reads a word's capitals as a credential reads the initials too; #785 met it only in ctgov's field moves, no grid having held the shape, and Derek chose the initials (rules/S8: a name word behind capitals keeps them a name word where no family comma stands). A predicate over ADJACENT PAIRS of words meets two shapes no grid of one word holds: an overlapping pair and a joining word beside an initial. #752's split-credential test (phase 2e) read `Dr. Ph.` in `Jane Doe Dr. Ph. D.` as the listed `drph`, leaving `D.` a word of its own (found by the maiden take's candidate grid), and a particle beside a surname written short as the listed `ded` (`DE D JUNIOR`, found only in tse-donors' readings), so put a title, a particle and a connective beside each pair spelling, and a pair overlapping another, in the grid. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams -- and a new chunk reading inherits the old one's exceptions, which #752's first draft did not (`Jack I-V` lost its family to the listed numerals, and the fix's length test counted a chunk's period, so `Jack I.-V.` did too, found by the second review), and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. That holds for a word a rule only asks ABOUT, too: #810's first draft asked whether any word after a bracketed credential lacked the suffix TAG, and kept `John Smith (MA) VI`, `(MA) Ph. D.` and `(MA) Dr.` nicknames while `(MA) III` and `(MA) PhD` escaped; asking the role the parse read for the following word answers for every spelling. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. Capitals between two name words have the same second reading: some writers put their own initials there (`Maria Ana MA Silva`), so a rule that reads a word's capitals as a credential reads the initials too; #785 met it only in ctgov's field moves, no grid having held the shape, and Derek chose the initials (rules/S8: a name word behind capitals keeps them a name word where no family comma stands). A predicate over ADJACENT PAIRS of words meets two shapes no grid of one word holds: an overlapping pair and a joining word beside an initial. #752's split-credential test (phase 2e) read `Dr. Ph.` in `Jane Doe Dr. Ph. D.` as the listed `drph`, leaving `D.` a word of its own (found by the maiden take's candidate grid), and a particle beside a surname written short as the listed `ded` (`DE D JUNIOR`, found only in tse-donors' readings), so put a title, a particle and a connective beside each pair spelling, and a pair overlapping another, in the grid. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. @@ -239,7 +239,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `group` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `post_rules` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index c1459c4fc..d3b2040ed 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -2,19 +2,20 @@ ## Standing -- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, written with a period inside it or in capitals in a name of mixed case, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname` (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`) — 2026-10-26 #810. +- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, written with a period inside it or in capitals in a name of mixed case, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`Jean (M.A) Smith`) (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`) — 2026-10-26 #810. - The case is that of the name's own words, as S6 asks it, never the clause's own, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix — 2026-10-26 #810. -- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix or a maiden clause after it does not count (`John Smith (MA), PhD`, `John Smith (MA) née Jones` → suffix 'MA') — 2026-10-26 #810. +- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`, `John Smith (MA) née Jones` → suffix 'MA'), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. - A settled member is the suffix, not the word read as if written bare: bare, `M.A` reads as paired initials (decisions/S3 (#864)), and the brackets have already set it apart from the name's words — 2026-10-26 #810. ## Log - **2026-10-26 #810 (Derek, 2026-10-23) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME IS THE SUFFIX WHERE ITS WRITING SETTLES IT.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. - - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. + - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4; a dated snapshot, the spike's switch being in no tree) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. - - **Why the suffix rather than the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly instead, the brackets having already set it apart from the name's words. The decision is made in group, where classify has recorded the name's case and tagged the words after the clause, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. - - **Recorded as a rule of its own**, not in S1: the position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - - **Accepted:** a member after a lone name word takes the suffix where its writing settles it, and the word reads as a one-word name reads (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, as bare `Jack MA` reads), where 2.0.0 through 2.6.0 gave family 'Jack' and nickname 'MA'. - - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, 19 of them `Ed`, 3 of those inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. + - **Why the suffix rather than the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly instead, the brackets having already set it apart from the name's words. The decision is made first in post_rules, where every word's role is read, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. + - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× now, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. + - **Recorded as a rule of its own**, not in S1: the position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12). Its Signal line names two signals, the position and the period inside the word, which S3 reads the other way bare (two dotted letters are initials), so S13 is a split candidate; the design-docs review found the first Signal line naming only the position. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. + - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. + - **Accepted:** a member after a lone name word takes the suffix where its writing settles it, and the word stays the family name N3 reads beside a nickname (`Jack (MA)` → family 'Jack', suffix 'MA', nothing reported, where 2.0.0 through 2.6.0 gave nickname 'MA'), while bare `Jack MA` reads given 'Jack', `given-or-family` reported; the decision comes after N3 has read the word. + - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/decisions/ma-do.md b/docs/design/decisions/ma-do.md index f6292776d..464f6ec3e 100644 --- a/docs/design/decisions/ma-do.md +++ b/docs/design/decisions/ma-do.md @@ -5,11 +5,13 @@ - `ma` and `do` are members of the ambiguous acronym set (`suffix_acronyms_ambiguous`: acronym readings on a word with a `given` or `family` reading, decisions/vocabulary-model 2026-10-25, `uv run python tools/vocab/query.py --word ma`), kept there because both are common surnames; the set has grown past these two since (decisions/suffix-acronym-collisions), and the marking is still what lets `Jack Ma` read family 'Ma' with `suffix-or-name` reported, while `John Smith MA` reads suffix — 2026-07-17. - In a name written in more than one case, a member in capitals leans credential and is taken with nothing to spare (`Jack MA`, `Jack DO` → given 'Jack', suffix 'MA'/'DO'), where `Jack Ma` keeps family 'Ma'; a deliberate break from v1, which read `Jack MA` as family 'MA' — 2026-09-15 #289. - A member written with its periods, one after each letter, counts unambiguously (`Jack M.A.`, `Jack D.O.` → suffix, `given-or-family` reported when only the given name remains), and one trailing period is no such spelling (`Jack Ma.` → family 'Ma.') — 2026-07-17. -- A parenthesized bare `(MA)` or `(DO)` reads as a nickname, no longer escaping to suffix as in 1.x, and reports `suffix-or-nickname`, unless it stands after the name in a name of mixed case, where its capitals settle it as the suffix (`John Smith (MA)`; `JOHN SMITH (MA)` and `John Smith (Ma)` keep the nickname), as does a period inside it (`John Smith (M.A)`), decided in decisions/S13's 2026-10-26 #810 entry — 2026-07-17. +- A parenthesized bare `(MA)` or `(DO)` reads as a nickname, no longer escaping to suffix as in 1.x, and reports `suffix-or-nickname` — 2026-07-17. - After a family comma `do` is spelling-dependent, being the one member of the set that is also particle vocabulary: `Doe, John DO` reads suffix `DO`, while `Doe, John Do`, `Doe, John do` and `DOE, JOHN DO` attach it to the family by P6 (`Doe, John Do` → family 'Do Doe', `particle-or-given` reported); `ma` is no particle, so `Smith, John Ma` keeps middle 'Ma' and `Smith, John MA` reads suffix — 2026-09-18 #531. Comma-less under `FAMILY_FIRST`, only the Title-case spelling attaches (`Doe John Do` → given 'John', family 'Do Doe', with `particle-or-given` besides `suffix-or-name`), while `Doe John do` and `DOE JOHN DO` read suffix, so the comma and the comma-less writing disagree for `do` (rules/P6, decisions/two-input-invariants). Elsewhere: the rest of the case reading — Title-case `Ma` and `Do` leaning surname, an all-lower member or a one-case name taking the positional count (`ANH DO` → family 'DO', `JOHN SMITH MA` → suffix) — is decisions/S6's 2026-09-14 #289 + #516 entry and decisions/S4's 2026-09-15 (spec review) entry, and two shapes stand outside the case reading altogether: a particle run in front takes `do` whatever its case (`Anh van DO` → family 'van DO', reporting nothing; rules/P2, decisions/S5's 2026-09-18 (Derek), #531 — CAPITALS DECIDE FOR `do` entry), and an unambiguous credential in front of a member speaks for it (`John Smith PhD Ma` → suffix 'PhD Ma'; decisions/S7's 2026-09-27 (Derek), #544 — A CREDENTIAL RUN KEEPS ITS AMBIGUOUS MEMBERS entry); the count that goes first at the first slot after a comma, a suffix comma included (`Smith, MA` → suffix, `Smith, Ma` → given, `John Smith, Ma` → suffix), is decisions/S6's 2026-09-14 (Derek) and 2026-10-20 (Derek), #761 entries; the pairing that gives `do` its capitals-only credential reading after a family comma, and its accepted cost, is decisions/S5's 2026-09-18 (Derek), #531 — CAPITALS DECIDE FOR `do` entry; the escape that reads a bracketed member as a nickname is decisions/suffix-acronym-collisions' S1 line; the periods gate is rules/S3's since 2026-10-19; whether a word earns the ambiguous marking is decisions/vocabulary-collisions' C-i and decisions/suffix-acronym-collisions', with `do`'s departure from TITLES recorded in the former and further credentials marked ambiguous in the latter (2026-10-21 #796). +Elsewhere: after the name, a member whose writing settles it is the suffix, not the nickname (`John Smith (MA)`, `John Smith (M.A)`; `JOHN SMITH (MA)` and `John Smith (Ma)` keep the nickname) — decisions/S13's 2026-10-26 #810 entry, which supersedes the line above in part. + Open: #792 (S1 and S8: whether a bracketed generation ends the credential run, `W. Russell (Jr.) Withers`), the neighbour of the parenthesized line above, which it does not move: `John Smith (MA) Withers` reads nickname 'MA', family 'Withers'. ## Log diff --git a/docs/design/decisions/suffix-acronym-collisions.md b/docs/design/decisions/suffix-acronym-collisions.md index 6d5b451e1..0e8ecf21a 100644 --- a/docs/design/decisions/suffix-acronym-collisions.md +++ b/docs/design/decisions/suffix-acronym-collisions.md @@ -10,7 +10,7 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - `ba`, `meng`, `lac`, `si` and `chi` are marked ambiguous rather than removed, so each is read as the ambiguous class is, by the case it is written in, the words to spare and the comma (rules/S6, S4 and C4): `John Smith SI` keeps the credential and reports the fork, while `John Smith Si`, `Anna Ba` and `Wang, Meng` read the surname or the given name, reported — 2026-09-07 #342; 2026-09-25 #540; 2026-10-21 #791. - `meng` and `lac` carry no case mask, since a mask applies in every role and would re-spell the surnames the marking keeps (`wang meng` would render `Wang MEng`) — 2026-09-25 #540. - `ded` is not marked: it is borne as a given name, which is not the position the suffix claim acts on — 2026-09-25 #540. -- A marked member written in brackets or quotes reads as a nickname, S1's escape declining an ambiguous member (`John Smith (MEng)` → nickname 'MEng'), an accepted cost, except after the name where its writing settles it, in capitals in a name of mixed case or with a period inside it (`John Smith (MENG)`, `John Smith (M.Eng)` → suffix), decided in decisions/S13's 2026-10-26 #810 entry — 2026-09-25 #540; 2026-10-21. +- A marked member written in brackets or quotes reads as a nickname, S1's escape declining an ambiguous member (`John Smith (MEng)` → nickname 'MEng'), an accepted cost — 2026-09-25 #540; 2026-10-21. - Under the default order, a marked member that reads as a name is taken into the family by a particle in front of it, with the words between (`John de Smith Chi` → family 'de Smith Chi', where `John de Smith CHI` keeps suffix 'CHI'), an accepted cost — 2026-10-21 #791; 2026-10-21. - 33 more credential readings, listed by `uv run python tools/vocab/query.py --issue 796` and `se` and `om` among them, are marked ambiguous rather than removed: 30 because the surname reading is substantial (US Census 2010 bearers ≥200, or Wikidata family bearers ≥20), a threshold that is a judgement and not a rule the data sets — 2026-10-21. - Where both counts are small, the surname count is weighed against the NPPES count of US healthcare providers writing the word, and the word is marked where the surname reading is the larger (`capa`, `ams`); a provider count is evidence that a credential is live, and a zero says nothing about a credential outside healthcare, so `ei`, whose credential is not a healthcare one, rests on the default under uncertainty instead — 2026-10-21. @@ -21,6 +21,8 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - `esq` is a suffix word and not an acronym, a contraction rather than an initialism, so no word is both a suffix acronym and a suffix word — 2026-09-08 #316. - The Wikipedia footnote marks `cva[22]`, `gaee[14]` and `np[18]` are out of the suffix acronyms, and a list imported from a wiki is checked for reference marks before it lands — 2026-10-18. +Elsewhere: after the name, a marked member written in capitals in a name of mixed case or with a period inside it is the suffix, not the nickname (`John Smith (MENG)`, `John Smith (M.Eng)`) — decisions/S13's 2026-10-26 #810 entry. + Elsewhere: the case lean that decides the capitals forms above is decisions/S6's 2026-09-14 #289 and 2026-09-14 (Derek) entries; the chunked dotted spelling passing the period gate (`Wang M.Eng.` → suffix) and the credential run that ends in a member keeping its suffix comma (`John Smith, PhD MEng` → suffix 'PhD MEng'), with the run a member leads keeping the cost (`John Smith MEng PhD` → family 'MEng'), are decisions/S7's 2026-09-27 (Derek), #544 entry headed A CREDENTIAL RUN KEEPS ITS AMBIGUOUS MEMBERS, which this file's 2026-09-27 (#544) entry indexes; the shape-plus-position heuristic this file parked shipped as the two switches of decisions/S9's 2026-09-14 (Derek) entries, the all-caps one reading the comma position by default since its 2026-10-01 (Derek), #564 entry; the roman-numeral accident's narrow retirement is decisions/S3's 2026-09-15 (Derek) entry, which this file's 2026-09-15 twin indexes; and `sa` was a title and suffix dual that position decided by #296's audit (decisions/comma-suffix-arc's audit table, whose `sa` row moving it into the ambiguous set decisions/C8's 2026-08-23 (review of PR #428, four reviewers) entry declined), held out of the marking again by the 2026-10-21 entry, and has been in neither set since decisions/C8's 2026-10-25 #857 entry. Not standing: the #454 bullet's note that the `fix(suffix-routing)` ledger rule stays, the ledgers having been retired for the snapshot (decisions/differential-ledger, #664), and the repair of rules/P6's Accepted clause recorded in the "What #454 got right" bullet, carried out in the same change — 2026-09-07 #342. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index f1a2dd68a..1da3635ab 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -196,9 +196,9 @@ This catalog converts discovery into a one-time cost — it does not remove disc **How it works.** "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. -**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch). +**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), the one site where content overrules the delimiter WITHOUT rejoining the bare path: a member of the ambiguous class settled by its writing after the name (rules/S13) is the suffix directly, because its bare reading would differ (`M.A` reads as initials bare), so it is decided once the parse's roles are read rather than unmasked here (decisions/S13). -**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). +**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Not when the bare reading is the wrong answer for the clause: that is S13's shape. ## CURATED-VOCABULARY-ALTERNATION — the config already splits them diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index 4be7cdc27..ecf839f6b 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -5,7 +5,7 @@ **Then:** where the member's writing settles it, written with a period inside it or in capitals in a name written in more than one case (S6), it is the suffix rather than the nickname, and nothing is reported. - **The case is the name's**: it is asked of the name's own words and never of the clause's, so capitals settle nothing in a name written wholly in one case, and Title case leans to the name (S6). -- **After the name** means that no name word follows the clause, though a suffix or a maiden clause may. +- **After the name** means that no name word follows the clause, though a suffix, a title or a maiden clause may, however each is written. - **It is the suffix, not read as if written bare** (S1): the brackets have already set it apart from the name's words, and bare, two letters each followed by a period but the last read as initials (S3). **Because:** after the name, brackets hold either a credential or the name the person goes by, and the writing tells the two apart where the count of words cannot: credentials are written in capitals and with their periods, and names are not. Inside the name the same word let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by. So only the clause after the name is settled. @@ -19,6 +19,9 @@ "John Smith PhD (MA)" → suffix="PhD MA" "John Smith (MA), PhD" → suffix="MA, PhD" "John Smith (MA) née Jones" → suffix="MA" +"John Smith (MA) Ph. D." → suffix="MA Ph. D." +"John Smith (MA) Dr." → title="Dr." +"John Smith (MA) Dr." → suffix="MA" "John Smith «MA»" → suffix="MA" "John Smith 「MA」" → suffix="MA" "John Smith (Ed)" → nickname="Ed" · boundary @@ -39,17 +42,23 @@ "JEFFREY (JD) BRICKEN" → nickname="JD" "Jean 'JD' Smith" → nickname="JD" "Jean (M.A) Smith" → nickname="M.A" + "Jean (M.A) Smith" → ambiguities=("suffix-or-nickname",) "Smith (MA), John" → nickname="MA" "(MA) John Smith" → nickname="MA" ``` +**Precedence:** + +- **Two letters each followed by a period but the last** ("John Smith (J.D)"): this rule reads the credential where S3 reads the same word bare as paired initials ("John Smith J.D" gives family 'J.D'), the brackets having already set it apart from the name's words. + **Accepted:** -- A member after a lone name word is the suffix where its writing settles it, and the word reads as a one-word name reads, given rather than the family name it was beside a nickname. +- A member after a lone name word is the suffix where its writing settles it, and the word stays the family name it is beside a nickname (N3), where the same member written bare leaves it the given name. ``` - "Jack (MA)" → given="Jack" + "Jack (MA)" → family="Jack" + "Jack (MA)" → suffix="MA" ``` --- -**Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name, its writing read as S6 reads capitals -**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_group.py +**Signal:** position + shape: a member of the ambiguous class alone in brackets or quotes after the name, written with a period inside it; its capitals read as S6 reads them +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_post_rules.py diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 125e8f617..f25197634 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -249,7 +249,7 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # role is delimited content, decided by extract's escape and never # at the trailing slot -- 'Bridge (A.B)' is the control that # proves the guard load-bearing (without it the nickname reading - # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report in group), and + # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report in post_rules), and # 'Bridge (1.4)' cannot exercise it, a digit chunk never reaching # the shape verdict at all (period_joined_vocab's alphabetic # gate). diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index f2041e596..478b4945b 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1,9 +1,8 @@ """Stage: group. Consumes: tokens (classified), segments, structure, one_case, extracted -(the role + inner span per delimited region, for the #329 pass below and -S13's one-word clause test -- the only stage after tokenize that reads -it), and original (the clause's text); and, through +(the role + inner span per delimited region, for the #329 pass below -- +the only stage after tokenize that reads it); and, through `_comma.decide`, comma_offsets (the own-words span of the name's case contrast) and dropped, which decide extends. Produces: structure -- its head decides a comma form once the words @@ -22,14 +21,9 @@ and tail_reads, rules/S2's trailing run of each main segment read ONCE, after P3's connective joins and before the particle chain and the bound-given join, and split off so neither reaches it (#614), -which assign places and reports rather than peeling again; and the -SUFFIX role of a one-word nickname that is a member of the ambiguous -class settled by its writing after the name (rules/S13, #810), with the -SUFFIX_OR_NICKNAME report on any other member left a nickname. +which assign places and reports rather than peeling again. Reads: token tags (from classify), Lexicon.given_name_titles (the -P5 licence, #369), Lexicon.suffix_acronyms_ambiguous (a dotted member in -a clause, which carries no tag there, rules/S13) and -Policy.extra_suffix_delimiters, whose +P5 licence, #369) and Policy.extra_suffix_delimiters, whose delimiter-core tokens part a tail segment as a comma would and are dropped (v1 suffix_delimiter parity, #549); and, through `_comma.decide`, what that module's own header lists @@ -39,7 +33,7 @@ class settled by its writing after the name (rules/S13, #810), with the per-parse state that dissolves with the state (v1 kept per-parse sets for the same reason). -Implements rules P2, P3, P4, M2 and S13, S2's trailing read (#614), C1's +Implements rules P2, P3, P4 and M2, S2's trailing read (#614), C1's decision through _comma, and the group half of M1 (#329: the marker dropped inside EXTRACTED maiden @@ -71,7 +65,7 @@ class settled by its writing after the name (rules/S13, #810), with the from enum import IntEnum from typing import assert_never -from nameparser._lexicon import _normalize, _run_addresses_by_given +from nameparser._lexicon import _run_addresses_by_given from nameparser._pipeline._pieces import ( split_credential_report, Peel, TailRead, chain_lead, chain_run_end, is_conj_piece, is_leading_title, is_prefix_piece, @@ -1301,74 +1295,6 @@ def chain(tail: int) -> None: return pieces, ptags, taken -def _alone_in_its_clause(token: WorkToken, state: ParseState) -> bool: - """Whether a delimited token is the whole of its clause.""" - for _, clause in state.extracted: - if clause.start <= token.span.start and token.span.end <= clause.end: - return (state.original[clause.start:clause.end].strip() - == token.text) - return False - - -def _name_word_after(tokens: Sequence[WorkToken], i: int, - dropped: Sequence[int]) -> bool: - """Whether a word of the name proper follows token i: a word with - no role that is not a suffix as written, up to a maiden marker, - the clause from there on being no name of its own (`own_words`).""" - for j in range(i + 1, len(tokens)): - t = tokens[j] - if t.role is not None or j in dropped: - continue - if "vocab:maiden-marker" in t.tags: - return False - if "vocab:suffix" not in t.tags: - return True - return False - - -# rules/S13: "where the member's writing settles it, written with a -# period inside it or in capitals in a name written in more than one -# case (S6), it is the suffix rather than the nickname" -- Then, with -# the Except bullet keeping a member inside the name the nickname; -# extract's S1 escape has already sent every suffix-shaped clause -# through (decisions/S13 (#810)) -def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], - ambiguities: list[PendingAmbiguity]) -> None: - """A one-word nickname that is a member of the ambiguous class - becomes the suffix where its writing settles it and no name word - follows it; any other tagged member stays the nickname and the - fork is reported. Asked here rather than in extract, which runs - before there are words to ask about: the case is the name's - recorded fact (`settled_by_case`, rules/S6), and whether a name - word follows is read off the tags. A dotted member carries no tag - in a clause (classify tags a shape only outside one), so it is - found by its text.""" - lex = state.lexicon - for i, token in enumerate(tokens): - if token.role is not Role.NICKNAME: - continue - tagged = AMBIGUOUS_ACRONYM_TAG in token.tags - # a period inside the word ('M.A'); one ending it is extract's - # escape - dotted = ("." in token.text.rstrip(".") - and _normalize(token.text).replace(".", "") - in lex.suffix_acronyms_ambiguous) - if not (tagged or dotted): - continue - if ((dotted or settled_by_case(token, state.one_case)) - and _alone_in_its_clause(token, state) - and not _name_word_after(tokens, i, state.dropped)): - tokens[i] = copy_with(token, role=Role.SUFFIX) - elif tagged: - # Reported here, where the escape is decided, rather than - # in extract, which has no token index to point at. - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NICKNAME, - f"delimited {token.text!r} is also a post-nominal; read " - f"as a nickname rather than a suffix", - (i,))) - - def group(state: ParseState) -> ParseState: # rules/C1's decision, read once now that classify has tagged # the words and before any join below can reach the part after the @@ -1379,8 +1305,6 @@ def group(state: ParseState) -> ParseState: tokens = list(state.tokens) dropped = list(state.dropped) ambiguities = list(state.ambiguities) - if state.extracted: - _settle_delimited_members(state, tokens, ambiguities) all_pieces: list[tuple[tuple[int, ...], ...]] = [] all_ptags: list[tuple[frozenset[str], ...]] = [] all_reads: list[TailRead | None] = [] diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 1109c4bde..cc06d32fa 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -5,7 +5,10 @@ under a family comma (#359). structure was always read here, for the rotation gate. Also comma_offsets and dropped, which R1's entry pass below reads to find the separators the writer typed (#436/#437). -Produces: tokens with roles adjusted by the post rules, the stable +Produces: tokens with roles adjusted by the post rules (among them the +SUFFIX role of a delimited member its writing settles after the name, +rules/S13, with the SUFFIX_OR_NICKNAME report on any other member left +a nickname), the stable "joined" tag on a post-nominal continuing the entry before it, and the ambiguity P6's attachment reports at the end of a name read family-first (#467) and after a comma with nothing before it -- after a @@ -15,10 +18,12 @@ Policy.name_order (O6 stands down where the script's order differs from the declared one), Policy.middle_as_family, Policy.extra_suffix_delimiters (R1's entry pass, for the delimiter -cores group drops); Lexicon.given_name_titles. +cores group drops); Lexicon.given_name_titles, and +Lexicon.suffix_acronyms_ambiguous (a dotted member in a clause, which +carries no tag there, S13). Also extracted and one_case, for S13. -Implements rules H1, M4, P1, P6 (its no-comma site, and after a comma -naming no family), O1, O2, O6, O3 and R1 of docs/design/rules/; +Implements rules S13, H1, M4, P1, P6 (its no-comma site, and after a +comma naming no family), O1, O2, O6, O3 and R1 of docs/design/rules/; each is cited at its code below, and H1/P1/O1/O2's history lives in docs/design/decisions/. `suffix_entries` is the R1 entry pass as a state-in/state-out function, for Parser.revise to run over a @@ -26,17 +31,18 @@ """ from __future__ import annotations +import bisect import re -from nameparser._lexicon import _run_addresses_by_given +from nameparser._lexicon import _normalize, _run_addresses_by_given from nameparser._pipeline._assign import ( _attach_particle_tail, _name_positions, ) from nameparser._pipeline._pieces import ( - is_lone_never_given_particle, particle_tail, + is_lone_never_given_particle, particle_tail, settled_by_case, ) from nameparser._pipeline._state import ( - NAME_ROLES, + AMBIGUOUS_ACRONYM_TAG, NAME_ROLES, ParseState, PendingAmbiguity, Structure, WorkToken, _NEVER_FLIPPED, comma_bucket, copy_with, ) @@ -376,9 +382,61 @@ def _family_settled(state: ParseState, tokens: list[WorkToken], _leading_name_piece(state, tokens), tokens) +# rules/S13: "where the member's writing settles it, written with a +# period inside it or in capitals in a name written in more than one +# case (S6), it is the suffix rather than the nickname" -- Then, with +# the Except bullet keeping a member inside the name the nickname; +# extract's S1 escape has already sent every suffix-shaped clause +# through (decisions/S13 (#810)) +def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], + ambiguities: list[PendingAmbiguity]) -> None: + """A nickname that is the whole of its clause and a member of the + ambiguous class becomes the suffix where its writing settles it and + it stands after the last name word; any other tagged member stays + the nickname and the fork is reported, a dotted member left a + nickname inside the name included. Asked here, first in this + stage, because only once roles are read is it known whether a name + word follows: a suffix, a title or a maiden clause after the clause + is no name word, whatever spelling put it there. Reported here, + where the escape is decided, rather than in extract, which has no + token index to point at. A dotted member carries no tag in a + clause (classify tags a shape only outside one), so it is found + by its text.""" + last_name = -1 + alone: set[int] = set() + for i, t in enumerate(tokens): + if t.role in NAME_ROLES: + last_name = i + starts = [t.span.start for t in tokens] + for _, clause in state.extracted: + lo = bisect.bisect_left(starts, clause.start) + if bisect.bisect_left(starts, clause.end) == lo + 1: + alone.add(lo) + lex = state.lexicon + for i, token in enumerate(tokens): + if token.role is not Role.NICKNAME: + continue + # a period inside the word ('M.A'); one ending it is extract's + # escape + dotted = ("." in token.text.rstrip(".") + and _normalize(token.text).replace(".", "") + in lex.suffix_acronyms_ambiguous) + if i > last_name and i in alone and ( + dotted or settled_by_case(token, state.one_case)): + tokens[i] = copy_with(token, role=Role.SUFFIX) + elif dotted or AMBIGUOUS_ACRONYM_TAG in token.tags: + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NICKNAME, + f"delimited {token.text!r} is also a post-nominal; read " + f"as a nickname rather than a suffix", + (i,))) + + def post_rules(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) + if state.extracted: + _settle_delimited_members(state, tokens, ambiguities) titles = _idx(tokens, Role.TITLE) givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 92d4bc556..542b5b837 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -195,9 +195,7 @@ class ParseState: changes the token COUNT: an unspaced CJK token splits into n+1 pieces, still as sub-slices of the original, and every later index in the segment runs shifts by n); classify -> token tags AND - one_case; group -> structure (its head decides a comma form, #613), - the SUFFIX role of a delimited member its writing settles (rules/S13, - #810) + one_case; group -> structure (its head decides a comma form, #613) AND the SUFFIX/TITLE roles of a postnominal part after the comma, whose segment it empties so no join reaches it, then pieces/piece_tags/dropped AND maiden token roles, plus the @@ -206,8 +204,10 @@ class ParseState: assign -> the remaining token roles AND `order`, the effective order it read them under, AND the folded-middle tag on the particles P6's attachment joins after a family comma (#613); - post_rules -> roles again, and the ambiguity P6's no-comma - attachment reports (#467). + post_rules -> roles again (a delimited member S13 settles becomes + the suffix, #810), and the ambiguity P6's no-comma attachment + reports (#467) and SUFFIX_OR_NICKNAME on a member it leaves a + nickname. Ambiguities are recorded by every stage that DECIDES one -- extract (resolved to a token index by tokenize), segment, script_segment, classify, group, assign, and post_rules -- since a diff --git a/tests/v2/cases.py b/tests/v2/cases.py index d491881c1..02ef3429a 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8858,8 +8858,9 @@ def _check_cjk_shape_purity(self) -> None: "Jean (M.A) Smith", {"given": "Jean", "family": "Smith", "nickname": "M.A"}, classification="fix(#810)", - notes="inside the name however it is written; 1.4.0 read " - "middle 'M.A'"), + ambiguities=("suffix-or-nickname",), + notes="inside the name however it is written, and reported as " + "a capitals member is; 1.4.0 read middle 'M.A'"), Case("bracket_member_before_a_family_comma_stays_the_nickname", "Smith (MA), John", {"given": "John", "family": "Smith", "nickname": "MA"}, @@ -8883,12 +8884,29 @@ def _check_cjk_shape_purity(self) -> None: {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), Case("bracket_member_after_a_lone_name_word", "Jack (MA)", - {"given": "Jack", "suffix": "MA"}, + {"family": "Jack", "suffix": "MA"}, classification="fix(#810)", - ambiguities=("given-or-family",), - notes="reads as bare 'Jack MA' does (rules/S13's Accepted); " - "1.4.0 read last 'MA', 2.0.0 through 2.6.0 family " - "'Jack', nickname 'MA'"), + notes="the word stays N3's family name, where bare 'Jack MA' " + "reads given (rules/S13's Accepted); 1.4.0 read last " + "'MA', 2.0.0 through 2.6.0 family 'Jack', nickname 'MA'"), + Case("bracket_member_before_a_shape_read_suffix", + "John Smith (MA) Ph. D.", + {"given": "John", "family": "Smith", "suffix": "MA Ph. D."}, + classification="fix(#810)", + notes="a suffix after the clause is no name word whatever " + "spelling made it one; the first draft asked the " + "suffix tag, which the split Ph. D. does not carry"), + Case("bracket_member_before_a_numeral", "John Smith (MA) VI", + {"given": "John", "family": "Smith", "suffix": "MA VI"}, + classification="fix(#810)", + ambiguities=("suffix-or-name",), + notes="VI is read by shape, not listed (rules/S10)"), + Case("bracket_member_before_a_trailing_title", + "John Smith (MA) Dr.", + {"title": "Dr.", "given": "John", "family": "Smith", + "suffix": "MA"}, + classification="fix(#810)", + notes="a title after the clause is no name word (rules/H5)"), Case("nickname_quotes", 'John "Jack" Kennedy', {"given": "John", "family": "Kennedy", "nickname": "Jack"}, notes="shape 1's double-quoted Nickname slot, which is the " diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index 9ffc94756..fe9af186a 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -576,7 +576,7 @@ def test_delimited_content_never_joins_the_shape_class() -> None: # control that actually load-bears it: without the guard, the # nickname 'A.B' would gain SHAPE_ACRONYM_TAG and (with the switch # on) `vocab:suffix-ambiguous`, and classify's own nickname check - # would then misreport SUFFIX_OR_NICKNAME (group's report, since + # would then misreport SUFFIX_OR_NICKNAME (post_rules' report, since # #810) on a token the escape already decided (measured regression, #516 review round). for text, word in (("Bridge (1.4)", "1.4"), ("Bridge (A.B)", "A.B")): tags = _tags_by_text(text)[word] diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index 696160fbe..d455fe78b 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -282,7 +282,8 @@ def test_stage_field_ownership() -> None: # the hook it consults on a vocabulary decline). "script_segment": {"tokens", "segments", "ambiguities"}, # classify emits CONJUNCTION_OR_INITIAL; SUFFIX_OR_NICKNAME - # moved to group with the decision it reports (rules/S13, #810) + # moved to post_rules with the decision it reports (rules/S13, + # #810) "classify": {"tokens", "ambiguities", "one_case"}, # group also emits PARTICLE_OR_GIVEN: the prefix chain takes # the particle branch of a fork whose given branch _assign @@ -294,9 +295,7 @@ def test_stage_field_ownership() -> None: # and empties the segment of a part it bound, so no join can # reach it (#613's /simplify) # and since #614 the trailing run it reads once at its head, - # handed to assign (`tail_reads`); and since #810 the SUFFIX - # role of a delimited member its writing settles (rules/S13), - # reporting SUFFIX_OR_NICKNAME on any other + # handed to assign (`tail_reads`) "group": {"tokens", "pieces", "piece_tags", "dropped", "ambiguities", "structure", "segments", "tail_reads"}, # assign also records `order`: the effective order it read the @@ -305,7 +304,8 @@ def test_stage_field_ownership() -> None: # post_rules also emits PARTICLE_OR_GIVEN: P6's no-comma # attachment (#467) takes the family branch of a fork whose # other branches assign and group take, so each stage reports - # the side it decides; the comma site is assign's since #613 + # the side it decides; the comma site is assign's since #613; + # and SUFFIX_OR_NICKNAME, with S13's decision (#810) "post_rules": {"tokens", "ambiguities"}, } assert {s.__name__ for s in STAGES} == set(ownership) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 578f45a23..74cbdbcd0 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -65,7 +65,7 @@ "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "S12": {"words": 674, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, - "S13": {"words": 284, "then_words": 132, "because_sentences": 3, "except": 2, "precedence": 0, "accepted": 1, "accepted_words": 35}, + "S13": {"words": 335, "then_words": 138, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 39}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index 3ca873e9b..515e0a2ed 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -294,6 +294,7 @@ def test_a_thousand_names_still_parse_in_reasonable_time( "credential_run": "PhD Ma ", # pieces: S7's anchor pass (#544) "agnomens": "Neto ", # pieces: a gated word scans its front (#890) "carried_agnomens": "MA ma Neto ", # pieces: the scan through carried members + "bracket_members": "(M.A) ", # post_rules: S13's clause pass (#810) } _BASE = 800 diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index d3ad7dc1d..10babff66 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -273,9 +273,12 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # changes moving disjoint claims. # 1345 -> 1366 on 2026-10-26, #810 (phase 2l): seven rows whose # bracketed member stays the nickname, one suffix-or-nickname or - # given-or-family claim each under each order. - assert claims == 1366, ( - f"the field sweep checks {claims} claims, recorded as 1366 on " + # given-or-family claim each under each order; 1366 -> 1369 the same + # day, its review round: 'Jack (MA)' keeps N3's family and loses its + # given-or-family claims (-3), 'John Smith (MA) VI' (+3) and the + # dotted 'Jean (M.A) Smith', reported now (+3). + assert claims == 1369, ( + f"the field sweep checks {claims} claims, recorded as 1369 on " f"2026-10-26") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 9104f0a48..833df6965 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -367,6 +367,8 @@ "John Smith (M.A.)" "John Smith (MA PhD)" "John Smith (MA)" +"John Smith (MA) Dr." +"John Smith (MA) Ph. D." "John Smith (MA) née Jones" "John Smith (MA), PhD" "John Smith (MA.)" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 961db484b..4748a4103 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1401,7 +1401,7 @@ {"name": "Jacinto López Moreno", "given": "Jacinto", "middle": "López", "family": "Moreno", "initials": "J. L. M."} {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} -{"name": "Jack (MA)", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "Jack (MA)", "family": "Jack", "suffix": "MA", "initials": "J."} {"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} {"name": "Jack I.-V.", "given": "Jack", "family": "I.-V.", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} @@ -1533,7 +1533,7 @@ {"name": "Jaume Martí i Cabot", "given": "Jaume", "family": "Martí i Cabot", "initials": "J. M. C."} {"name": "Javier López Díaz", "given": "Javier", "middle": "López", "family": "Díaz", "initials": "J. L. D."} {"name": "Jean 'JD' Smith", "given": "Jean", "family": "Smith", "nickname": "JD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "Jean \"Jd\" Smith"} -{"name": "Jean (M.A) Smith", "given": "Jean", "family": "Smith", "nickname": "M.A", "initials": "J. S."} +{"name": "Jean (M.A) Smith", "given": "Jean", "family": "Smith", "nickname": "M.A", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S."} {"name": "Jean DUPONT", "given": "Jean", "family": "DUPONT", "initials": "J. D.", "capitalized_force": "Jean Dupont"} {"name": "Jean Joseph Antoine Marie de Witte", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Jean Louis Charles d'Orléans-Longueville", "given": "Jean", "middle": "Louis Charles", "family": "d'Orléans-Longueville", "initials": "J. L. C. d.", "capitalized_force": "Jean Louis Charles D'Orléans-Longueville"} @@ -1660,6 +1660,8 @@ {"name": "John Smith (M.A.)", "given": "John", "family": "Smith", "suffix": "M.A.", "initials": "J. S."} {"name": "John Smith (MA PhD)", "given": "John", "family": "Smith", "nickname": "MA PhD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma PhD\" Smith"} {"name": "John Smith (MA)", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} +{"name": "John Smith (MA) Dr.", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} +{"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} {"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA", "maiden": "Jones", "initials": "J. S."} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} {"name": "John Smith (MA.)", "given": "John", "family": "Smith", "suffix": "MA.", "initials": "J. S."} From 909026d8e8b859b94f742775bd33b27d2d4e6f58 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 14:16:13 -0700 Subject: [PATCH 03/10] Refuse a leading period as S13's dotted shape; fix stale stage notes (#810 second review) Co-Authored-By: Claude Opus 5.5 --- .../S13-bracketed-member-settled-by-its-writing.md | 1 + nameparser/_pipeline/_group.py | 4 ++-- nameparser/_pipeline/_post_rules.py | 12 +++++++----- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index d3b2040ed..7a4e671f8 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -16,6 +16,7 @@ - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× now, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. - **Recorded as a rule of its own**, not in S1: the position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12). Its Signal line names two signals, the position and the period inside the word, which S3 reads the other way bare (two dotted letters are initials), so S13 is a split candidate; the design-docs review found the first Signal line naming only the position. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. + - **Left as found by the second review**: one post rule can still put a name word behind a settled clause, P6's attachment after a comma with nothing before it, which runs after this pass (`, Jones (MA) vd` → family 'vd', suffix 'MA', where master read nickname 'MA'); the shape has no name before its comma and nothing in the corpora writes it. A clause holding a character tokenize drops (an emoji, a zero-width space, a nakaguro) beside the member counts as the member alone (`(MA 😀)`), the test counting tokens. - **Accepted:** a member after a lone name word takes the suffix where its writing settles it, and the word stays the family name N3 reads beside a nickname (`Jack (MA)` → family 'Jack', suffix 'MA', nothing reported, where 2.0.0 through 2.6.0 gave nickname 'MA'), while bare `Jack MA` reads given 'Jack', `given-or-family` reported; the decision comes after N3 has read the word. - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 478b4945b..c0479202c 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1,8 +1,8 @@ """Stage: group. Consumes: tokens (classified), segments, structure, one_case, extracted -(the role + inner span per delimited region, for the #329 pass below -- -the only stage after tokenize that reads it); and, through +(the role + inner span per delimited region, for the #329 pass below; +post_rules reads it too, for S13); and, through `_comma.decide`, comma_offsets (the own-words span of the name's case contrast) and dropped, which decide extends. Produces: structure -- its head decides a comma form once the words diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index cc06d32fa..dcc32ba48 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -4,7 +4,8 @@ particle fold reads the opening piece of segment 0, or of segment 1 under a family comma (#359). structure was always read here, for the rotation gate. Also comma_offsets and dropped, which R1's entry pass -below reads to find the separators the writer typed (#436/#437). +below reads to find the separators the writer typed (#436/#437), and +extracted and one_case, which S13's pass reads (#810). Produces: tokens with roles adjusted by the post rules (among them the SUFFIX role of a delimited member its writing settles after the name, rules/S13, with the SUFFIX_OR_NICKNAME report on any other member left @@ -20,7 +21,7 @@ Policy.extra_suffix_delimiters (R1's entry pass, for the delimiter cores group drops); Lexicon.given_name_titles, and Lexicon.suffix_acronyms_ambiguous (a dotted member in a clause, which -carries no tag there, S13). Also extracted and one_case, for S13. +carries no tag there, S13). Implements rules S13, H1, M4, P1, P6 (its no-comma site, and after a comma naming no family), O1, O2, O6, O3 and R1 of docs/design/rules/; @@ -113,8 +114,9 @@ def _mark_suffix_entries(tokens: list[WorkToken], state: ParseState) -> None: # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). decisions/R1 # (#436/#437). # - # After assign, because it keys on Role.SUFFIX; last in post_rules - # by convention only, since no rule here writes SUFFIX or a + # After assign, because it keys on Role.SUFFIX; last in post_rules, + # after S13's pass, the one rule here that writes SUFFIX (#810), and + # by convention otherwise, since no other writes SUFFIX or a # _RENDERS_ELSEWHERE role. Recorded as a tag because a span-less # token (unpickled, or built by replace()) leaves the render # nothing to scan (mechanisms.md#MARK-DONT-STRIP). Every WorkToken @@ -418,7 +420,7 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], continue # a period inside the word ('M.A'); one ending it is extract's # escape - dotted = ("." in token.text.rstrip(".") + dotted = ("." in token.text.strip(".") and _normalize(token.text).replace(".", "") in lex.suffix_acronyms_ambiguous) if i > last_name and i in alone and ( From 07153eb93f1b96b3a6a2f8a4d0371a51c28ad2c4 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 14:47:07 -0700 Subject: [PATCH 04/10] Read a settled bracketed member as its unambiguous twin; S13 reads one signal (#810) Derek's two changes before the PR opens. A member rules/S13 settles now reads as a bracketed unambiguous credential does in the same place, fields and reports alike: post_rules records the settled clauses, and run() folds the stages once more with them escaped in extract and their word tagged as the credential's in classify (`Jack (MA)` reads given 'Jack', suffix 'MA', given-or-family, as `Jack (PhD)` does). A test holds the invariant over comma shapes, heads and the three orders, its negative control recorded (264 of 768 differ with a single fold). S13 now reads only the position, citing S3 for the periods, S6 for the capitals and S1 for the twin reading: its Signal line is a hub of S1, S3 and S6. A maiden clause after a settled member is then taken into the credential's run as after its twin, recorded as Accepted. Co-Authored-By: Claude Opus 5.5 --- ...bracketed-member-settled-by-its-writing.md | 19 +++--- docs/design/decisions/vocabulary-model.md | 4 +- docs/design/mechanisms.md | 4 +- ...bracketed-member-settled-by-its-writing.md | 26 ++++---- docs/design/vocabulary-model-rfc.md | 2 +- nameparser/_pipeline/__init__.py | 18 +++-- nameparser/_pipeline/_classify.py | 12 +++- nameparser/_pipeline/_extract.py | 21 +++++- nameparser/_pipeline/_post_rules.py | 42 +++++++----- nameparser/_pipeline/_state.py | 15 +++-- tests/v2/cases.py | 24 +++++-- tests/v2/pipeline/test_state.py | 5 +- tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 9 ++- tests/v2/test_parser.py | 66 ++++++++++++++++++- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 5 +- 17 files changed, 202 insertions(+), 73 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 7a4e671f8..00b0a6e42 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -2,21 +2,20 @@ ## Standing -- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, written with a period inside it or in capitals in a name of mixed case, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`Jean (M.A) Smith`) (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`) — 2026-10-26 #810. -- The case is that of the name's own words, as S6 asks it, never the clause's own, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix — 2026-10-26 #810. -- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`, `John Smith (MA) née Jones` → suffix 'MA'), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. -- A settled member is the suffix, not the word read as if written bare: bare, `M.A` reads as paired initials (decisions/S3 (#864)), and the brackets have already set it apart from the name's words — 2026-10-26 #810. +- A member of the ambiguous class alone in brackets or quotes after the name, its writing settling it as the credential, reads as its twin, a bracketed unambiguous credential in the same place, does, fields and reports alike (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix; `Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, as `Jack (PhD)`); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`, `Jean (M.A) Smith`) — 2026-10-26 #810. +- The writing that settles it is read by the rules that read writing: a period inside the word, S3's evidence, whatever case the name is in, and capitals in a name of mixed case, S6's, the case being that of the name's own words and never the clause's, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix. S13's own signal is the position alone — 2026-10-26 #810. +- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. ## Log -- **2026-10-26 #810 (Derek, 2026-10-23) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME IS THE SUFFIX WHERE ITS WRITING SETTLES IT.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. +- **2026-10-26 #810 (Derek, 2026-10-23; the twin reading and the single signal Derek, 2026-10-26) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME, SETTLED BY ITS WRITING, READS AS A BRACKETED UNAMBIGUOUS CREDENTIAL DOES.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4; a dated snapshot, the spike's switch being in no tree) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. - - **Why the suffix rather than the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly instead, the brackets having already set it apart from the name's words. The decision is made first in post_rules, where every word's role is read, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. - - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× now, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. - - **Recorded as a rule of its own**, not in S1: the position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12). Its Signal line names two signals, the position and the period inside the word, which S3 reads the other way bare (two dotted letters are initials), so S13 is a split candidate; the design-docs review found the first Signal line naming only the position. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. + - **Why the twin, not the bare reading (Derek, 2026-10-26).** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. It reads instead as a bracketed unambiguous credential reads in the same place, the whole name included, which is what the brackets and the writing together say it is. Whether a member settles is known only once the roles are read, so post_rules records the settled clauses and the pipeline folds once more from the start with them escaped as such a credential is and their word carrying its tag (`vocab:suffix`); the `suffix-or-nickname` report moved to post_rules from classify, so that it is raised where the fork is decided. The second fold costs a whole parse, on a name holding a settled clause only (frames counted with `sys.setprofile`, py3.11, 2026-10-26: `John Smith (MA)` 214 → 416, `Jack (MA)` 174 → 358; `Smith, John` 158 and `John Smith` 149 unchanged; a name holding an unsettled clause pays 3 or 6, `John (Bob) Smith` 204 → 207, `John Smith (Ed)` 214 → 220). The PR's first two drafts made the member the suffix in place instead, leaving the rest of the name read around a nickname: `Jack (MA)` read family 'Jack' where `Jack (PhD)` reads given 'Jack'. tests/v2/test_parser.py's `test_a_settled_bracketed_member_reads_as_its_unambiguous_twin` holds the invariant over a grid of comma shapes, heads and the three orders, its control recorded in its docstring. + - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× after the fix, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. + - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the twin reading S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - - **Left as found by the second review**: one post rule can still put a name word behind a settled clause, P6's attachment after a comma with nothing before it, which runs after this pass (`, Jones (MA) vd` → family 'vd', suffix 'MA', where master read nickname 'MA'); the shape has no name before its comma and nothing in the corpora writes it. A clause holding a character tokenize drops (an emoji, a zero-width space, a nakaguro) beside the member counts as the member alone (`(MA 😀)`), the test counting tokens. - - **Accepted:** a member after a lone name word takes the suffix where its writing settles it, and the word stays the family name N3 reads beside a nickname (`Jack (MA)` → family 'Jack', suffix 'MA', nothing reported, where 2.0.0 through 2.6.0 gave nickname 'MA'), while bare `Jack MA` reads given 'Jack', `given-or-family` reported; the decision comes after N3 has read the word. + - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. A clause holding a character tokenize drops (an emoji, a zero-width space, a nakaguro) beside the member counts as the member alone (`(MA 😀)`), the test counting tokens. + - **Accepted:** a lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was N3's family name (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, where 2.0.0 through 2.6.0 gave family 'Jack', nickname 'MA'); and a maiden clause after a settled member, which lets it settle, is then read as one after its twin is, inside the credential's trailing run (S8: `John Smith (MA) née Jones` → suffix 'MA née Jones', as `John Smith (PhD) née Jones` reads on master). - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 752c9d2d0..3cacf2582 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - A listed credential that is also a name, written in capitals in a name of mixed case, is settled as the credential by its writing: it reports nothing and starts the credential's run where no name word stands behind it with no family comma (2026-10-26, Derek), and keeps its other readings, the clash settled at the sites that choose a reading rather than by retagging the word (decisions/S6) — 2026-10-25 #752, phase 2g, #785. -- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, with a period inside it or in capitals in a name of mixed case, the case being the name's own words' (rules/S13, decisions/S13) — 2026-10-26 #752, phase 2l. +- A member of the ambiguous class alone in brackets or quotes after the name reads as a bracketed unambiguous credential does in the same place where its writing settles it, with a period inside it or in capitals in a name of mixed case, the case being the name's own words' (rules/S13, decisions/S13) — 2026-10-26 #752, phase 2l. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. - `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. @@ -218,4 +218,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. - **2026-10-26 (Derek), #785 — PHASE 2g NARROWED: a capitals member with a name word behind it, and no family comma, does not start the credential's run**, the name word being evidence of initials; `Maria Ana MA Silva`, `Jean Pierre MA DUPONT` and `Mary Jane MA King` keep the readings 1.4.0 through 2.6.0 gave, and `John Smith MA Jones` keeping family 'Jones' is the cost. The record is decisions/S6's entry of this date. -- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. +- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name reads as a bracketed unambiguous credential does in the same place where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix; `Jack (MA)` as `Jack (PhD)`), and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 1da3635ab..81a85372d 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -196,9 +196,9 @@ This catalog converts discovery into a one-time cost — it does not remove disc **How it works.** "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. -**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), the one site where content overrules the delimiter WITHOUT rejoining the bare path: a member of the ambiguous class settled by its writing after the name (rules/S13) is the suffix directly, because its bare reading would differ (`M.A` reads as initials bare), so it is decided once the parse's roles are read rather than unmasked here (decisions/S13). +**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), which decides once the parse's roles are read that a member of the ambiguous class after the name is settled by its writing (rules/S13); `run` then folds the stages again, extract escaping the clause here and classify tagging its word as an unambiguous credential's, so it rejoins the bare path as its twin, a bracketed unambiguous credential, would, not as its own bare word, which reads differently (`M.A` reads as initials bare; decisions/S13). -**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Not when the bare reading is the wrong answer for the clause: that is S13's shape. +**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Where the bare word itself would read wrongly, rejoin it as the twin it is, as S13 does. ## CURATED-VOCABULARY-ALTERNATION — the config already splits them diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index ecf839f6b..b11fb801a 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -1,12 +1,12 @@ # S13 — a bracketed member settled by its writing -**When:** a clause that is one member of the ambiguous class (S4) stands in brackets or quotes after the name, as in "John Smith (MA)". +**When:** a clause that is one member of the ambiguous class (S4) stands in brackets or quotes after the name, with no name word after it, though a suffix, a title or a maiden clause may follow however each is written, as in "John Smith (MA)". -**Then:** where the member's writing settles it, written with a period inside it or in capitals in a name written in more than one case (S6), it is the suffix rather than the nickname, and nothing is reported. +**Then:** where the member's writing settles it as the credential, the way the rules that read the writing say a credential is written, it is no nickname: it reads as a bracketed unambiguous credential does in the same place, and the clash is not reported. -- **The case is the name's**: it is asked of the name's own words and never of the clause's, so capitals settle nothing in a name written wholly in one case, and Title case leans to the name (S6). -- **After the name** means that no name word follows the clause, though a suffix, a title or a maiden clause may, however each is written. -- **It is the suffix, not read as if written bare** (S1): the brackets have already set it apart from the name's words, and bare, two letters each followed by a period but the last read as initials (S3). +- **Written with a period inside it**, whatever case the name is in: the periods are how a credential is written and a name is not (S3). +- **Written in capitals in a name written in more than one case**: the case is asked of the name's own words and never of the clause's, so capitals settle nothing in a name written wholly in one case, and Title case leans to the name (S6). +- **It reads as its twin**, a bracketed unambiguous credential in the same place, fields and reports alike (S1): a lone name word beside it is the given name as beside "(PhD)", and the member is not read as if written bare, where two letters each followed by a period but the last read as initials (S3). **Because:** after the name, brackets hold either a credential or the name the person goes by, and the writing tells the two apart where the count of words cannot: credentials are written in capitals and with their periods, and names are not. Inside the name the same word let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by. So only the clause after the name is settled. @@ -18,11 +18,12 @@ "Smith, John (MA)" → suffix="MA" "John Smith PhD (MA)" → suffix="PhD MA" "John Smith (MA), PhD" → suffix="MA, PhD" -"John Smith (MA) née Jones" → suffix="MA" "John Smith (MA) Ph. D." → suffix="MA Ph. D." "John Smith (MA) Dr." → title="Dr." "John Smith (MA) Dr." → suffix="MA" "John Smith «MA»" → suffix="MA" +"Jack (MA)" → given="Jack" +"Jack (M.A)" → suffix="M.A" "John Smith 「MA」" → suffix="MA" "John Smith (Ed)" → nickname="Ed" · boundary ``` @@ -53,12 +54,15 @@ **Accepted:** -- A member after a lone name word is the suffix where its writing settles it, and the word stays the family name it is beside a nickname (N3), where the same member written bare leaves it the given name. +- A lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was the family name (N3), and the fork is reported as for any lone name word. ``` - "Jack (MA)" → family="Jack" - "Jack (MA)" → suffix="MA" + "Jack (MA)" → ambiguities=("given-or-family",) + ``` +- A maiden clause after a settled member is read as one after its twin is, inside the credential's trailing run (S8), so the clause settles the member and is then taken into the suffix. + ``` + "John Smith (MA) née Jones" → suffix="MA née Jones" ``` --- -**Signal:** position + shape: a member of the ambiguous class alone in brackets or quotes after the name, written with a period inside it; its capitals read as S6 reads them -**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_post_rules.py +**Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name; hub of S1, S3, S6 +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_post_rules.py diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 86c8b2364..9f753e982 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -130,7 +130,7 @@ Phase 2i carried this out as [rules/H9](rules/H9-title-phrase.md) (the title phr A bracketed or quoted credential is to escape the nickname reading where its writing settles it (#810): with interior periods, or in capitals in a name of mixed case, the case asked of the whole name. `(Ed)` stays a nickname, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`). -Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-its-writing.md) (a bracketed member settled by its writing) rather than as an amendment to [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md), since the position it reads is a signal S1 does not: `John Smith (MA)` and `John Smith (M.A)` read suffix, and `JOHN SMITH (MA)` and `Jeffrey (JD) Bricken` keep the nickname. The case is that of the name's own words, as S6 asks it, and a settled member is the suffix rather than the word read bare, since bare, `M.A` reads as initials. +Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-its-writing.md) (a bracketed member settled by its writing) rather than as an amendment to [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md), since the position it reads is a signal S1 does not: `John Smith (MA)` and `John Smith (M.A)` read suffix, and `JOHN SMITH (MA)` and `Jeffrey (JD) Bricken` keep the nickname. The writing is read as the rules that read it say, S3 for the periods and S6 for the capitals in a name of mixed case, and S13's own signal is the position. A settled member reads as its twin, a bracketed unambiguous credential in the same place, does (`Jack (MA)` reads as `Jack (PhD)` does), not as the word read bare, since bare, `M.A` reads as initials. ### Readings on the parse diff --git a/nameparser/_pipeline/__init__.py b/nameparser/_pipeline/__init__.py index 9c91ba088..6a0a8a153 100644 --- a/nameparser/_pipeline/__init__.py +++ b/nameparser/_pipeline/__init__.py @@ -18,7 +18,7 @@ from nameparser._pipeline._post_rules import post_rules from nameparser._pipeline._script_segment import script_segment from nameparser._pipeline._segment import segment -from nameparser._pipeline._state import ParseState +from nameparser._pipeline._state import ParseState, copy_with from nameparser._pipeline._tokenize import tokenize #: The full eight-stage fold. @@ -32,7 +32,17 @@ def run(state: ParseState) -> ParseState: """Fold the stages over the initial state. Pure: each stage returns a new ParseState; parse is total over str input, so no stage may raise on the *content* of a name (malformed policy/config is a - separate, allowed failure mode).""" + separate, allowed failure mode). + + rules/S13's clauses are known only once the roles are read, and a + settled one must read as its twin, a bracketed unambiguous + credential, reads -- the rest of the name included -- so where the + first fold records any, the stages fold once more from the initial + state with them set (#810). The second fold settles nothing new: its + settled clauses are escaped, so post_rules finds no member left.""" + folded = state for stage in STAGES: - state = stage(state) - return state + folded = stage(folded) + if folded.settled_clauses and not state.settled_clauses: + return run(copy_with(state, settled_clauses=folded.settled_clauses)) + return folded diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index f25197634..40d50975f 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -5,7 +5,9 @@ _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the second comma asks for it (#289/#516, #613), so this read is the -fallback for every other name. +fallback for every other name; and settled_clauses on `run`'s second +fold (rules/S13, #810), whose words take an unambiguous credential's +tag. Produces: tokens with vocabulary tags added (text/span/role unchanged), plus ambiguities (CONJUNCTION_OR_INITIAL) and one_case -- whether the name's own words are written in one case, @@ -410,6 +412,14 @@ def classify(state: ParseState) -> ParseState: one_case and i < clause_at and t.role is None))) for i, t in enumerate(state.tokens)) + # rules/S13: "it reads as a bracketed unambiguous credential does in + # the same place" -- Then, on the second fold, the settled word + # taking the tag such a credential's word carries (decisions/S13 + # (#810)) + if state.settled_clauses: + tokens = tuple( + copy_with(t, tags=frozenset({"vocab:suffix"})) + if t.span in state.settled_clauses else t for t in tokens) # A word with a delimiter glued to it joins no neighbour across the # delimiter, so the stem's joining tags stay off it: 'y;' is no # connective to the word after it (rules/C10: "no join, maiden diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 25a931e17..06091716d 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -1,6 +1,8 @@ """Stage: extract_delimited. -Consumes: ParseState.original. +Consumes: ParseState.original, and settled_clauses on `run`'s second +fold (rules/S13, #810), each settled clause escaping as a bracketed +unambiguous credential does. Produces: extracted (role + inner span per delimited region), masked (full regions incl. delimiter chars, skipped by tokenize), UNBALANCED_DELIMITER ambiguities for opens with no close. @@ -99,6 +101,14 @@ def _maiden_marked(content: str, lexicon: Lexicon) -> bool: and not comma_follows_marker(words[run])) +def _settled_within(starts: list[int], inner: Span) -> bool: + """Whether a word rules/S13 settled starts inside the clause: + `starts` is the settled words' start offsets, sorted, so the test + is one bisect and the scan stays linear in the clauses.""" + k = bisect.bisect_left(starts, inner.start) + return k < len(starts) and starts[k] < inner.end + + # rules/N2: "a quote whose open and close are the same character # opens only at a word start and closes only at a word end, so an # apostrophe inside or at the end of a word is literal" @@ -194,6 +204,7 @@ def extract_delimited(state: ParseState) -> ParseState: ) extracted: list[tuple[Role, Span]] = [] masked: list[Span] = [] + settled = sorted(s.start for s in state.settled_clauses) # candidates, not final: each carries the offset of the unmatched # open so ones consumed by a later match can be filtered at the end unbalanced: list[tuple[int, PendingAmbiguity]] = [] @@ -240,8 +251,12 @@ def extract_delimited(state: ParseState) -> ParseState: exhausted.add((role, open_, close)) continue inner = Span(i + len(open_), j) - if inner.start < inner.end and _suffix_shaped( - text[inner.start:inner.end], state.lexicon): + if inner.start < inner.end and (_suffix_shaped( + text[inner.start:inner.end], state.lexicon) + # rules/S13's second fold: a clause the first settled + # escapes as its twin, a bracketed unambiguous + # credential, does (decisions/S13 (#810)) + or (settled and _settled_within(settled, inner))): # v1 parse_nicknames: suffix-shaped delimited content is # left IN PLACE (undelimited) for normal downstream parsing # -- 'Andrew Perkins (MBA)' keeps MBA a suffix, not a diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index dcc32ba48..8c93d3493 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -6,10 +6,10 @@ rotation gate. Also comma_offsets and dropped, which R1's entry pass below reads to find the separators the writer typed (#436/#437), and extracted and one_case, which S13's pass reads (#810). -Produces: tokens with roles adjusted by the post rules (among them the -SUFFIX role of a delimited member its writing settles after the name, -rules/S13, with the SUFFIX_OR_NICKNAME report on any other member left -a nickname), the stable +Produces: tokens with roles adjusted by the post rules; settled_clauses, +the delimited members rules/S13 settles after the name, for the second +fold `run` makes (#810), and the SUFFIX_OR_NICKNAME report on any other +member left a nickname; the stable "joined" tag on a post-nominal continuing the entry before it, and the ambiguity P6's attachment reports at the end of a name read family-first (#467) and after a comma with nothing before it -- after a @@ -51,6 +51,7 @@ from nameparser._policy import PatronymicRule from nameparser._types import ( FOLDED_TAG, UNJOINED_CONJUNCTION_TAG, UNJOINED_TAG, AmbiguityKind, Role, + Span, ) # Ported verbatim from v1 (nameparser/config/regexes.py) -- layering @@ -384,17 +385,20 @@ def _family_settled(state: ParseState, tokens: list[WorkToken], _leading_name_piece(state, tokens), tokens) -# rules/S13: "where the member's writing settles it, written with a -# period inside it or in capitals in a name written in more than one -# case (S6), it is the suffix rather than the nickname" -- Then, with -# the Except bullet keeping a member inside the name the nickname; -# extract's S1 escape has already sent every suffix-shaped clause -# through (decisions/S13 (#810)) +# rules/S13: "where the member's writing settles it as the credential, +# the way the rules that read the writing say a credential is written, +# it is no nickname" -- Then and its first two bullets, with the When's +# "no name word after it"; extract's S1 escape has already sent every +# suffix-shaped clause through, and the twin reading is `run`'s second +# fold (decisions/S13 (#810)) def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], - ambiguities: list[PendingAmbiguity]) -> None: - """A nickname that is the whole of its clause and a member of the - ambiguous class becomes the suffix where its writing settles it and - it stands after the last name word; any other tagged member stays + ambiguities: list[PendingAmbiguity], + ) -> frozenset[Span]: + """The words of the clauses S13 settles: a nickname that is the whole + of its clause and a member of the ambiguous class, where its writing + settles it and it stands after the last name word, `run` then + folding the stages again so it reads as its twin, a bracketed + unambiguous credential, does. Any other tagged member stays the nickname and the fork is reported, a dotted member left a nickname inside the name included. Asked here, first in this stage, because only once roles are read is it known whether a name @@ -415,6 +419,7 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], if bisect.bisect_left(starts, clause.end) == lo + 1: alone.add(lo) lex = state.lexicon + settled: set[Span] = set() for i, token in enumerate(tokens): if token.role is not Role.NICKNAME: continue @@ -425,20 +430,22 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], in lex.suffix_acronyms_ambiguous) if i > last_name and i in alone and ( dotted or settled_by_case(token, state.one_case)): - tokens[i] = copy_with(token, role=Role.SUFFIX) + settled.add(token.span) elif dotted or AMBIGUOUS_ACRONYM_TAG in token.tags: ambiguities.append(PendingAmbiguity( AmbiguityKind.SUFFIX_OR_NICKNAME, f"delimited {token.text!r} is also a post-nominal; read " f"as a nickname rather than a suffix", (i,))) + return frozenset(settled) def post_rules(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) + settled: frozenset[Span] = frozenset() if state.extracted: - _settle_delimited_members(state, tokens, ambiguities) + settled = _settle_delimited_members(state, tokens, ambiguities) titles = _idx(tokens, Role.TITLE) givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) @@ -840,4 +847,5 @@ def post_rules(state: ParseState) -> ParseState: tags=tokens[i].tags | {UNJOINED_CONJUNCTION_TAG}) _mark_suffix_entries(tokens, state) return copy_with(state, tokens=tuple(tokens), - ambiguities=tuple(ambiguities)) + ambiguities=tuple(ambiguities), + settled_clauses=settled or state.settled_clauses) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 542b5b837..707794f94 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -204,10 +204,10 @@ class ParseState: assign -> the remaining token roles AND `order`, the effective order it read them under, AND the folded-middle tag on the particles P6's attachment joins after a family comma (#613); - post_rules -> roles again (a delimited member S13 settles becomes - the suffix, #810), and the ambiguity P6's no-comma attachment - reports (#467) and SUFFIX_OR_NICKNAME on a member it leaves a - nickname. + post_rules -> roles again, settled_clauses (S13's, #810, which + extract and classify read on the second fold `run` makes), and the + ambiguity P6's no-comma attachment reports (#467) and + SUFFIX_OR_NICKNAME on a member S13 leaves a nickname. Ambiguities are recorded by every stage that DECIDES one -- extract (resolved to a token index by tokenize), segment, script_segment, classify, group, assign, and post_rules -- since a @@ -288,6 +288,13 @@ class ParseState: #: did not, without recording the answer (`_one_case`). one_case: bool | None = None ambiguities: tuple[PendingAmbiguity, ...] = () + #: rules/S13's settled clauses, by the span of the one word each + #: holds: post_rules records them on the first fold, and `run` + #: folds the stages once more with them set, extract escaping each + #: as it escapes a bracketed unambiguous credential and classify + #: tagging its word as that credential's twin, so the whole name + #: reads as the twin's does (#810). + settled_clauses: frozenset[Span] = frozenset() def _copy_refusals(cls: type) -> list[str]: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 02ef3429a..ce9a4ded2 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8875,20 +8875,30 @@ def _check_cjk_shape_purity(self) -> None: notes="a suffix after the clause is no name word"), Case("bracket_member_before_a_maiden_clause", "John Smith (MA) née Jones", - {"given": "John", "family": "Smith", "suffix": "MA", - "maiden": "Jones"}, + {"given": "John", "family": "Smith", "suffix": "MA née Jones"}, classification="fix(#810)", - notes="a maiden clause after it is no name of its own; 1.4.0 " - "read middle 'Smith MA née', last 'Jones'"), + ambiguities=("suffix-or-name", "suffix-or-name"), + notes="the clause settles the member and is then read as its " + "twin 'John Smith (PhD) née Jones' reads it, inside the " + "credential's run (rules/S8, S13's Accepted); 1.4.0 read " + "middle 'Smith MA née', last 'Jones'"), Case("bracket_member_in_quotes", 'John Smith "MA"', {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), Case("bracket_member_after_a_lone_name_word", "Jack (MA)", - {"family": "Jack", "suffix": "MA"}, + {"given": "Jack", "suffix": "MA"}, classification="fix(#810)", - notes="the word stays N3's family name, where bare 'Jack MA' " - "reads given (rules/S13's Accepted); 1.4.0 read last " + ambiguities=("given-or-family",), + notes="reads as its twin 'Jack (PhD)' does; 1.4.0 read last " "'MA', 2.0.0 through 2.6.0 family 'Jack', nickname 'MA'"), + Case("bracket_member_with_a_period_after_a_lone_name_word", + "Jack (M.A)", + {"given": "Jack", "suffix": "M.A"}, + classification="fix(#810)", + ambiguities=("given-or-family",), + notes="reads as its twin 'Jack (PhD)' does, where bare " + "'Jack M.A' reads family 'M.A'; 2.0.0 through 2.6.0 " + "read family 'Jack', nickname 'M.A'"), Case("bracket_member_before_a_shape_read_suffix", "John Smith (MA) Ph. D.", {"given": "John", "family": "Smith", "suffix": "MA Ph. D."}, diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index d455fe78b..8f199a5d5 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -305,8 +305,9 @@ def test_stage_field_ownership() -> None: # attachment (#467) takes the family branch of a fork whose # other branches assign and group take, so each stage reports # the side it decides; the comma site is assign's since #613; - # and SUFFIX_OR_NICKNAME, with S13's decision (#810) - "post_rules": {"tokens", "ambiguities"}, + # and SUFFIX_OR_NICKNAME, with S13's decision (#810), whose + # settled clauses it records for `run`'s second fold + "post_rules": {"tokens", "ambiguities", "settled_clauses"}, } assert {s.__name__ for s in STAGES} == set(ownership) # Within the tokens themselves the contract is finer: texts and diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 74cbdbcd0..f0d988f9b 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -65,7 +65,7 @@ "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "S12": {"words": 674, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, - "S13": {"words": 335, "then_words": 138, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 39}, + "S13": {"words": 390, "then_words": 172, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 2, "accepted_words": 71}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 10babff66..34fbbb1b2 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -276,9 +276,12 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # given-or-family claim each under each order; 1366 -> 1369 the same # day, its review round: 'Jack (MA)' keeps N3's family and loses its # given-or-family claims (-3), 'John Smith (MA) VI' (+3) and the - # dotted 'Jean (M.A) Smith', reported now (+3). - assert claims == 1369, ( - f"the field sweep checks {claims} claims, recorded as 1369 on " + # dotted 'Jean (M.A) Smith', reported now (+3); 1369 -> 1381 the same + # day, the twin reading (Derek): 'Jack (MA)' and 'Jack (M.A)' read + # as 'Jack (PhD)' does, a given-or-family claim each (+6), and + # 'John Smith (MA) née Jones' its two suffix-or-name claims (+6). + assert claims == 1381, ( + f"the field sweep checks {claims} claims, recorded as 1381 on " f"2026-10-26") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index c17580fb1..0b8b4e57f 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1,4 +1,5 @@ import dataclasses +import itertools import pickle import re import unicodedata @@ -9,10 +10,11 @@ Lexicon, Locale, Parser, Policy, PolicyPatch, locales, parse, parser_for, ) from nameparser._policy import ( - CapsSuffixes, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, PatronymicRule, + CapsSuffixes, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, + PatronymicRule, ) from nameparser._types import (UNJOINED_CONJUNCTION_TAG, UNJOINED_TAG, - AmbiguityKind, Role, Segmentation) + AmbiguityKind, ParsedName, Role, Segmentation) def test_parser_defaults_and_properties() -> None: @@ -1280,9 +1282,13 @@ def test_revise_reads_a_glued_honorific_on_its_own() -> None: #: a maiden marker the whole name left a word -- in a tail part, or #: taken into #602's credential run behind a credential -- and that #: the value's own sub-parse consumes mid-value, as revise() documents. +#: 'John Smith (MA) née Jones' joined 2026-10-26 (#810): a member +#: rules/S13 settles reads as its twin '(PhD)' does, inside the +#: credential's run, which takes the marker. _MARKER_IN_SUFFIX = frozenset({"Jane Doe PhD nee Smith", "Smith, John, MD - née Jones Smith", - "Smith, John, PhD née Jones"}) + "Smith, John, PhD née Jones", + "John Smith (MA) née Jones"}) def _suffix_bearing_corpus_names() -> list[str]: @@ -2184,3 +2190,57 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( # 長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). assert word not in Parser().lexicon.honorific_tails assert Parser().parse(f"田中{word}").suffix == "" + + +# rules/S13 (#810): a settled bracketed member reads as its twin, a +# bracketed unambiguous credential in the same place, fields and reports +# alike. The twin is MBA, a word no head holds, so the twin's reading +# maps back by replacing its text. 864 texts, about 2s. +_TWIN_HEADS = ( + "John Smith {c}", "Jack {c}", "{c}", "John {c}", "JOHN SMITH {c}", + "Smith, John {c}", "Smith, {c}", ", {c}", "(Bob), {c}", + "John Smith {c}, PhD", "John Smith, PhD {c}", "John Smith PhD {c}", + "Dr. Jack {c}", "Freiherr von Berg {c}", "Juan de la Vega {c}", + "Jane Doe nee Smith {c}", "John Smith {c} née Jones", + "John Smith {c} Dr.", "John Smith {c} VI", "John Smith {c} Ph. D.", + "Smith, Jane {c} Mrs", "Ortega y Gasset, José {c}", + "Smith, Freiherr von Berg {c}", "Doe, John van {c}", +) +_TWIN_MEMBERS = ("MA", "M.A", "JD", "DO", "D.O", "MENG") +_TWIN_PAIRS = (("(", ")"), ('"', '"')) +_TWIN_ORDERS = (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST) + + +def _twin_reading(name: ParsedName, twin: str | None, word: str) -> tuple[ + dict[str, str], list[tuple[str, str]]]: + def back(text: str) -> str: + return text.replace(twin, word) if twin else text + return ({k: back(v) for k, v in name.as_dict().items() if v}, + [(a.kind.value, back(a.detail)) for a in name.ambiguities]) + + +def test_a_settled_bracketed_member_reads_as_its_unambiguous_twin() -> None: + """Negative control, measured 2026-10-26 with `run` patched to a + single fold that makes the member the suffix in place (the first + draft's reading): 264 of 768 settled readings differ from the + twin's, 36 each behind 'Jack', 'John', 'John Smith PhD', + 'John Smith, PhD', before 'née Jones', 'VI' and 'Ph. D.', and 12 + for a clause standing alone ('Jack (MA)' read family 'Jack', where + the twin reads given 'Jack'). With the second fold: 756 settled, 0 + differ.""" + settled = 0 + differ = [] + for (open_, close), word, head, order in itertools.product( + _TWIN_PAIRS, _TWIN_MEMBERS, _TWIN_HEADS, _TWIN_ORDERS): + parser = Parser(policy=Policy(name_order=order)) + name = parser.parse(head.format(c=f"{open_}{word}{close}")) + if (word in name.nickname.split() + or word not in name.suffix.replace(",", " ").split()): + continue + settled += 1 + twin = parser.parse(head.format(c=f"{open_}MBA{close}")) + if _twin_reading(name, None, word) != _twin_reading( + twin, "MBA", word): + differ.append((name.original, order)) + assert settled > 700, settled + assert not differ, differ diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 833df6965..4acf3895e 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -262,6 +262,7 @@ "JOSÉ GARCÍA" "JUAN GARCIA Jr." "JUAN GARCIA Y LOPEZ" +"Jack (M.A)" "Jack (MA)" "Jack I-V" "Jack I.-V." diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 4748a4103..5ab69cff9 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1401,7 +1401,8 @@ {"name": "Jacinto López Moreno", "given": "Jacinto", "middle": "López", "family": "Moreno", "initials": "J. L. M."} {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} -{"name": "Jack (MA)", "family": "Jack", "suffix": "MA", "initials": "J."} +{"name": "Jack (M.A)", "given": "Jack", "suffix": "M.A", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "Jack (MA)", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} {"name": "Jack I.-V.", "given": "Jack", "family": "I.-V.", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} @@ -1662,7 +1663,7 @@ {"name": "John Smith (MA)", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "John Smith (MA) Dr.", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} -{"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA", "maiden": "Jones", "initials": "J. S."} +{"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA née Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA Née Jones"} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} {"name": "John Smith (MA.)", "given": "John", "family": "Smith", "suffix": "MA.", "initials": "J. S."} {"name": "John Smith (MBA PhD.)", "given": "John", "family": "Smith", "suffix": "MBA PhD.", "initials": "J. S."} From 3f0c457e74d67bd4e2c836370354395d82d0c235 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 15:04:27 -0700 Subject: [PATCH 05/10] Fold S13 to a fixed point; settled words in segment's tail test (#810 review) The fix-commit review found three places a settled member read otherwise than its twin. One extra fold left a clause the second fold settled a nickname with its report lost (JOHN SMITH (m.a) (MA)), so run() folds until no new clause settles. Segment's tail test runs before classify tags the settled word, so a dotted member past a second comma drew a comma-structure report (Smith, John, Jr. (M.A)); the test leaves settled words out. "Alone in its clause" compares the clause's text, as extract's escape does for the twin, so a dropped character beside the member no longer lets it settle. Case rows pin each, four failing on the previous commit; S13 cites every module it lives in. Co-Authored-By: Claude Opus 5.5 --- ...bracketed-member-settled-by-its-writing.md | 5 +-- ...bracketed-member-settled-by-its-writing.md | 3 +- nameparser/_pipeline/__init__.py | 30 ++++++++-------- nameparser/_pipeline/_extract.py | 7 ++-- nameparser/_pipeline/_post_rules.py | 7 ++-- nameparser/_pipeline/_segment.py | 7 ++++ tests/v2/cases.py | 34 +++++++++++++++++++ tests/v2/test_cases.py | 8 +++-- tests/v2/test_parser.py | 18 ++++++---- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + 11 files changed, 90 insertions(+), 31 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 00b0a6e42..4291be601 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -11,11 +11,12 @@ - **2026-10-26 #810 (Derek, 2026-10-23; the twin reading and the single signal Derek, 2026-10-26) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME, SETTLED BY ITS WRITING, READS AS A BRACKETED UNAMBIGUOUS CREDENTIAL DOES.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4; a dated snapshot, the spike's switch being in no tree) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. - - **Why the twin, not the bare reading (Derek, 2026-10-26).** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. It reads instead as a bracketed unambiguous credential reads in the same place, the whole name included, which is what the brackets and the writing together say it is. Whether a member settles is known only once the roles are read, so post_rules records the settled clauses and the pipeline folds once more from the start with them escaped as such a credential is and their word carrying its tag (`vocab:suffix`); the `suffix-or-nickname` report moved to post_rules from classify, so that it is raised where the fork is decided. The second fold costs a whole parse, on a name holding a settled clause only (frames counted with `sys.setprofile`, py3.11, 2026-10-26: `John Smith (MA)` 214 → 416, `Jack (MA)` 174 → 358; `Smith, John` 158 and `John Smith` 149 unchanged; a name holding an unsettled clause pays 3 or 6, `John (Bob) Smith` 204 → 207, `John Smith (Ed)` 214 → 220). The PR's first two drafts made the member the suffix in place instead, leaving the rest of the name read around a nickname: `Jack (MA)` read family 'Jack' where `Jack (PhD)` reads given 'Jack'. tests/v2/test_parser.py's `test_a_settled_bracketed_member_reads_as_its_unambiguous_twin` holds the invariant over a grid of comma shapes, heads and the three orders, its control recorded in its docstring. + - **Why the twin, not the bare reading (Derek, 2026-10-26).** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. It reads instead as a bracketed unambiguous credential reads in the same place, the whole name included, which is what the brackets and the writing together say it is. Whether a member settles is known only once the roles are read, so post_rules records the settled clauses and the pipeline folds again from the start with them escaped as such a credential is and their word carrying its tag (`vocab:suffix`), until a fold settles no clause it was not given: a settled word joins the name's own words, which can make a one-case name mixed and settle a second clause (`JOHN SMITH (m.a) (MA)` → suffix 'm.a MA', as `JOHN SMITH (mba) (MA)`), and the set of settled clauses only grows, so the folding ends; the `suffix-or-nickname` report moved to post_rules from classify, so that it is raised where the fork is decided. The second fold costs a whole parse, on a name holding a settled clause only (frames counted with `sys.setprofile`, py3.11, 2026-10-26: `John Smith (MA)` 214 → 416, `Jack (MA)` 174 → 358; `Smith, John` 158 and `John Smith` 149 unchanged; a name holding an unsettled clause pays 3 or 6, `John (Bob) Smith` 204 → 207, `John Smith (Ed)` 214 → 220). The PR's first two drafts made the member the suffix in place instead, leaving the rest of the name read around a nickname: `Jack (MA)` read family 'Jack' where `Jack (PhD)` reads given 'Jack'. tests/v2/test_parser.py's `test_a_settled_bracketed_member_reads_as_its_unambiguous_twin` holds the invariant over a grid of comma shapes, heads and the three orders, its control recorded in its docstring. - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× after the fix, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the twin reading S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. A clause holding a character tokenize drops (an emoji, a zero-width space, a nakaguro) beside the member counts as the member alone (`(MA 😀)`), the test counting tokens. + - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. + - **The third review, of the twin reading.** Three places read the member otherwise than its twin. One more fold was not enough: a clause the second fold settled was left a nickname, its report lost (`JOHN SMITH (m.a) (MA)`, `john smith (M.A) (MA)`), so `run` folds to the fixed point. The part past a second comma is tested for suffix words before classify tags the settled word, so a dotted member there drew a `comma-structure` report its twin does not (`Smith, John, Jr. (M.A)`); segment's test leaves the settled word out. And a clause was called the member alone when it held one token, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) let it settle where its twin, read by extract as the clause's whole text, does not escape (`John Smith (MA 😀)`); the test now compares the clause's text, as extract does. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), though the twin's credential run would take that word (`John Smith (MBA) Ma` → suffix 'MBA Ma'): the position is read on the nickname reading. - **Accepted:** a lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was N3's family name (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, where 2.0.0 through 2.6.0 gave family 'Jack', nickname 'MA'); and a maiden clause after a settled member, which lets it settle, is then read as one after its twin is, inside the credential's trailing run (S8: `John Smith (MA) née Jones` → suffix 'MA née Jones', as `John Smith (PhD) née Jones` reads on master). - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index b11fb801a..8cb1709c9 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -46,6 +46,7 @@ "Jean (M.A) Smith" → ambiguities=("suffix-or-nickname",) "Smith (MA), John" → nickname="MA" "(MA) John Smith" → nickname="MA" + "John Smith (MA) Ma" → nickname="MA" ``` **Precedence:** @@ -65,4 +66,4 @@ --- **Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name; hub of S1, S3, S6 -**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_post_rules.py +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/__init__.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_extract.py, nameparser/_pipeline/_post_rules.py, nameparser/_pipeline/_segment.py diff --git a/nameparser/_pipeline/__init__.py b/nameparser/_pipeline/__init__.py index 6a0a8a153..2d8473968 100644 --- a/nameparser/_pipeline/__init__.py +++ b/nameparser/_pipeline/__init__.py @@ -32,17 +32,19 @@ def run(state: ParseState) -> ParseState: """Fold the stages over the initial state. Pure: each stage returns a new ParseState; parse is total over str input, so no stage may raise on the *content* of a name (malformed policy/config is a - separate, allowed failure mode). - - rules/S13's clauses are known only once the roles are read, and a - settled one must read as its twin, a bracketed unambiguous - credential, reads -- the rest of the name included -- so where the - first fold records any, the stages fold once more from the initial - state with them set (#810). The second fold settles nothing new: its - settled clauses are escaped, so post_rules finds no member left.""" - folded = state - for stage in STAGES: - folded = stage(folded) - if folded.settled_clauses and not state.settled_clauses: - return run(copy_with(state, settled_clauses=folded.settled_clauses)) - return folded + separate, allowed failure mode).""" + # rules/S13: "it reads as a bracketed unambiguous credential does in + # the same place" -- Then; the clauses are known only once the roles + # are read, so the stages fold again from the initial state with + # the settled clauses set, until a fold settles none it was not + # given (decisions/S13 (#810)). A settled word joins the name's own + # words, which can make a one-case name mixed and settle a second + # clause; the set only grows and is bounded by the clauses, so the + # loop ends, and a name with none settled folds once. + while True: + folded = state + for stage in STAGES: + folded = stage(folded) + if folded.settled_clauses <= state.settled_clauses: + return folded + state = copy_with(state, settled_clauses=folded.settled_clauses) diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 06091716d..19c1b52d1 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -253,9 +253,10 @@ def extract_delimited(state: ParseState) -> ParseState: inner = Span(i + len(open_), j) if inner.start < inner.end and (_suffix_shaped( text[inner.start:inner.end], state.lexicon) - # rules/S13's second fold: a clause the first settled - # escapes as its twin, a bracketed unambiguous - # credential, does (decisions/S13 (#810)) + # rules/S13: "it reads as a bracketed unambiguous + # credential does in the same place" -- Then, on a later + # fold: the clause escapes as that credential's does + # (decisions/S13 (#810)) or (settled and _settled_within(settled, inner))): # v1 parse_nicknames: suffix-shaped delimited content is # left IN PLACE (undelimited) for normal downstream parsing diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 8c93d3493..52e412192 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -416,7 +416,10 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], starts = [t.span.start for t in tokens] for _, clause in state.extracted: lo = bisect.bisect_left(starts, clause.start) - if bisect.bisect_left(starts, clause.end) == lo + 1: + # the whole clause, as extract's escape asks it of its twin + if (bisect.bisect_left(starts, clause.end) == lo + 1 + and state.original[clause.start:clause.end].strip() + == tokens[lo].text): alone.add(lo) lex = state.lexicon settled: set[Span] = set() @@ -848,4 +851,4 @@ def post_rules(state: ParseState) -> ParseState: _mark_suffix_entries(tokens, state) return copy_with(state, tokens=tuple(tokens), ambiguities=tuple(ambiguities), - settled_clauses=settled or state.settled_clauses) + settled_clauses=settled | state.settled_clauses) diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index cceecab9e..edcd19f64 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -209,6 +209,13 @@ def titles_and_suffixes(seg: tuple[int, ...]) -> bool: # model of group, the shape mechanisms.md's # READ-WITHOUT-THEN-BIND exists to remove. Last, as the # rarest reading. + # rules/S13: "it reads as a bracketed unambiguous credential does + # in the same place" -- Then, on a later fold: a settled word is + # such a credential, which this test would recognize, though its + # tag comes only in classify (decisions/S13 (#810)) + if state.settled_clauses: + seg = tuple(i for i in seg + if state.tokens[i].span not in state.settled_clauses) if (seg and not suffixy(seg) and not suffixy(seg, case_class()) and not class_run(seg) and not titles_and_suffixes(seg)): texts_joined = " ".join([state.tokens[i].text for i in seg]) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index ce9a4ded2..f0a8018cb 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8882,6 +8882,40 @@ def _check_cjk_shape_purity(self) -> None: "twin 'John Smith (PhD) née Jones' reads it, inside the " "credential's run (rules/S8, S13's Accepted); 1.4.0 read " "middle 'Smith MA née', last 'Jones'"), + Case("bracket_member_settled_by_another_in_a_one_case_name", + "JOHN SMITH (m.a) (MA)", + {"given": "JOHN", "family": "SMITH", "suffix": "m.a MA"}, + classification="fix(#810)", + notes="the settled 'm.a' joins the name's own words, which are " + "then mixed, as its twin 'JOHN SMITH (mba) (MA)' makes " + "them; a fold too few left 'MA' a nickname, unreported"), + Case("bracket_member_settled_by_another_in_a_lower_case_name", + "john smith (M.A) (MA)", + {"given": "john", "family": "smith", "suffix": "M.A MA"}, + classification="fix(#810)", + notes="as its twin 'john smith (MBA) (MA)' reads"), + Case("bracket_member_past_the_second_comma", "Smith, John, Jr. (M.A)", + {"given": "John", "family": "Smith", "suffix": "Jr. M.A"}, + classification="fix(#810)", + notes="no comma-structure report, as its twin '(MBA)' draws " + "none: the part's tail test runs before the settled " + "word is tagged, so it leaves the word out"), + Case("bracket_member_beside_a_dropped_character", + "John Smith (MA \N{GRINNING FACE})", + {"given": "John", "family": "Smith", "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="not the member alone, as its twin, the same clause " + "with PhD, is no suffix-shaped clause"), + Case("bracket_member_before_a_title_case_member", + "John Smith (MA) Ma", + {"given": "John", "middle": "Smith", "family": "Ma", + "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-name", "suffix-or-nickname"), + notes="read on the nickname reading, 'Ma' is a name word after " + "the clause, where the twin's credential run would take " + "it (rules/S13's Except)"), Case("bracket_member_in_quotes", 'John Smith "MA"', {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 34fbbb1b2..934fd90d0 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -279,9 +279,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # dotted 'Jean (M.A) Smith', reported now (+3); 1369 -> 1381 the same # day, the twin reading (Derek): 'Jack (MA)' and 'Jack (M.A)' read # as 'Jack (PhD)' does, a given-or-family claim each (+6), and - # 'John Smith (MA) née Jones' its two suffix-or-name claims (+6). - assert claims == 1381, ( - f"the field sweep checks {claims} claims, recorded as 1381 on " + # 'John Smith (MA) née Jones' its two suffix-or-name claims (+6); + # 1381 -> 1390 the same day, its review round's rows: a member + # beside a dropped character (+3) and before 'Ma' (+6). + assert claims == 1390, ( + f"the field sweep checks {claims} claims, recorded as 1390 on " f"2026-10-26") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 0b8b4e57f..831a59ff0 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2195,7 +2195,7 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( # rules/S13 (#810): a settled bracketed member reads as its twin, a # bracketed unambiguous credential in the same place, fields and reports # alike. The twin is MBA, a word no head holds, so the twin's reading -# maps back by replacing its text. 864 texts, about 2s. +# maps back by replacing its text. 972 texts, about 2s. _TWIN_HEADS = ( "John Smith {c}", "Jack {c}", "{c}", "John {c}", "JOHN SMITH {c}", "Smith, John {c}", "Smith, {c}", ", {c}", "(Bob), {c}", @@ -2205,6 +2205,10 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( "John Smith {c} Dr.", "John Smith {c} VI", "John Smith {c} Ph. D.", "Smith, Jane {c} Mrs", "Ortega y Gasset, José {c}", "Smith, Freiherr von Berg {c}", "Doe, John van {c}", + # a third comma part, and a clause whose settling another makes + # possible, the first settled word making the name mixed case + "Smith, John, Jr. {c}", "JOHN SMITH (m.a) {c}", + "john smith (M.A) {c}", ) _TWIN_MEMBERS = ("MA", "M.A", "JD", "DO", "D.O", "MENG") _TWIN_PAIRS = (("(", ")"), ('"', '"')) @@ -2222,12 +2226,14 @@ def back(text: str) -> str: def test_a_settled_bracketed_member_reads_as_its_unambiguous_twin() -> None: """Negative control, measured 2026-10-26 with `run` patched to a single fold that makes the member the suffix in place (the first - draft's reading): 264 of 768 settled readings differ from the + draft's reading): 300 of 828 settled readings differ from the twin's, 36 each behind 'Jack', 'John', 'John Smith PhD', - 'John Smith, PhD', before 'née Jones', 'VI' and 'Ph. D.', and 12 - for a clause standing alone ('Jack (MA)' read family 'Jack', where - the twin reads given 'Jack'). With the second fold: 756 settled, 0 - differ.""" + 'John Smith, PhD', 'Smith, John, Jr.', before 'née Jones', 'VI' and + 'Ph. D.', and 12 for a clause standing alone ('Jack (MA)' read + family 'Jack', where the twin reads given 'Jack'). Folded to the + fixed point: 864 settled, 0 differ. A member the folds leave a + nickname is skipped here, so the clause a second fold settles is + pinned by case rows instead (bracket_member_settled_by_another_*).""" settled = 0 differ = [] for (open_, close), word, head, order in itertools.product( diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 4acf3895e..95471a693 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -369,6 +369,7 @@ "John Smith (MA PhD)" "John Smith (MA)" "John Smith (MA) Dr." +"John Smith (MA) Ma" "John Smith (MA) Ph. D." "John Smith (MA) née Jones" "John Smith (MA), PhD" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 5ab69cff9..1ebbc673f 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1662,6 +1662,7 @@ {"name": "John Smith (MA PhD)", "given": "John", "family": "Smith", "nickname": "MA PhD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma PhD\" Smith"} {"name": "John Smith (MA)", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "John Smith (MA) Dr.", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} +{"name": "John Smith (MA) Ma", "given": "John", "middle": "Smith", "family": "Ma", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NAME", "SUFFIX_OR_NICKNAME"], "initials": "J. S. M.", "capitalized_force": "John \"Ma\" Smith Ma"} {"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} {"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA née Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA Née Jones"} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} From ebaa7f8f7eda0d29b80d14654119c5f2503f0908 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 15:15:34 -0700 Subject: [PATCH 06/10] Read a settled word in a tail part as its twin's credential (#810 review) The fourth review found the previous commit's segment fix dropping the settled word from the part past a second comma, which lost a report the twin keeps (Smith, John, X.Y.Z. (MA)) and left the word out of the report's text. The word now stays in the part: it passes the suffix test and fails the shape-run test, as the listed credential it reads as does. A case row pins it; stale "second fold" prose now says the later folds. Co-Authored-By: Claude Opus 5.5 --- ...bracketed-member-settled-by-its-writing.md | 2 +- nameparser/_pipeline/_classify.py | 6 ++--- nameparser/_pipeline/_extract.py | 4 ++-- nameparser/_pipeline/_post_rules.py | 4 ++-- nameparser/_pipeline/_segment.py | 22 ++++++++++++------- nameparser/_pipeline/_state.py | 6 ++--- tests/v2/cases.py | 8 +++++++ tests/v2/test_parser.py | 16 ++++++++------ 8 files changed, 42 insertions(+), 26 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 4291be601..b4d386e9f 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -16,7 +16,7 @@ - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the twin reading S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. - - **The third review, of the twin reading.** Three places read the member otherwise than its twin. One more fold was not enough: a clause the second fold settled was left a nickname, its report lost (`JOHN SMITH (m.a) (MA)`, `john smith (M.A) (MA)`), so `run` folds to the fixed point. The part past a second comma is tested for suffix words before classify tags the settled word, so a dotted member there drew a `comma-structure` report its twin does not (`Smith, John, Jr. (M.A)`); segment's test leaves the settled word out. And a clause was called the member alone when it held one token, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) let it settle where its twin, read by extract as the clause's whole text, does not escape (`John Smith (MA 😀)`); the test now compares the clause's text, as extract does. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), though the twin's credential run would take that word (`John Smith (MBA) Ma` → suffix 'MBA Ma'): the position is read on the nickname reading. + - **The third review, of the twin reading.** Three places read the member otherwise than its twin. One more fold was not enough: a clause the second fold settled was left a nickname, its report lost (`JOHN SMITH (m.a) (MA)`, `john smith (M.A) (MA)`), so `run` folds to the fixed point. The part past a second comma is tested for suffix words before classify tags the settled word, so a dotted member there drew a `comma-structure` report its twin does not (`Smith, John, Jr. (M.A)`); segment's tests now read the settled word as that credential: it passes the suffix test, fails the test for a run of shape-admitted words (whose pass had silenced the report on `Smith, John, X.Y.Z. (MA)` where the twin's listed word keeps it), and stays in the report's text (`segment 'Bob MA'`), the fourth review having found a first fix that left it out of the part changing both. And a clause was called the member alone when it held one token, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) let it settle where its twin, read by extract as the clause's whole text, does not escape (`John Smith (MA 😀)`); the test now compares the clause's text, as extract does. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), though the twin's credential run would take that word (`John Smith (MBA) Ma` → suffix 'MBA Ma'): the position is read on the nickname reading. - **Accepted:** a lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was N3's family name (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, where 2.0.0 through 2.6.0 gave family 'Jack', nickname 'MA'); and a maiden clause after a settled member, which lets it settle, is then read as one after its twin is, inside the credential's trailing run (S8: `John Smith (MA) née Jones` → suffix 'MA née Jones', as `John Smith (PhD) née Jones` reads on master). - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 40d50975f..13bdf518c 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -5,8 +5,8 @@ _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the second comma asks for it (#289/#516, #613), so this read is the -fallback for every other name; and settled_clauses on `run`'s second -fold (rules/S13, #810), whose words take an unambiguous credential's +fallback for every other name; and settled_clauses on `run`'s later +folds (rules/S13, #810), whose words take an unambiguous credential's tag. Produces: tokens with vocabulary tags added (text/span/role unchanged), plus ambiguities (CONJUNCTION_OR_INITIAL) and @@ -413,7 +413,7 @@ def classify(state: ParseState) -> ParseState: and t.role is None))) for i, t in enumerate(state.tokens)) # rules/S13: "it reads as a bracketed unambiguous credential does in - # the same place" -- Then, on the second fold, the settled word + # the same place" -- Then, on a later fold, the settled word # taking the tag such a credential's word carries (decisions/S13 # (#810)) if state.settled_clauses: diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 19c1b52d1..7ace440f3 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -1,7 +1,7 @@ """Stage: extract_delimited. -Consumes: ParseState.original, and settled_clauses on `run`'s second -fold (rules/S13, #810), each settled clause escaping as a bracketed +Consumes: ParseState.original, and settled_clauses on `run`'s later +folds (rules/S13, #810), each settled clause escaping as a bracketed unambiguous credential does. Produces: extracted (role + inner span per delimited region), masked (full regions incl. delimiter chars, skipped by tokenize), diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 52e412192..40233ebd9 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -7,8 +7,8 @@ below reads to find the separators the writer typed (#436/#437), and extracted and one_case, which S13's pass reads (#810). Produces: tokens with roles adjusted by the post rules; settled_clauses, -the delimited members rules/S13 settles after the name, for the second -fold `run` makes (#810), and the SUFFIX_OR_NICKNAME report on any other +the delimited members rules/S13 settles after the name, for the later +folds `run` makes (#810), and the SUFFIX_OR_NICKNAME report on any other member left a nickname; the stable "joined" tag on a post-nominal continuing the entry before it, and the ambiguity P6's attachment reports at the end of a name read diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index edcd19f64..9d0c59014 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -1,6 +1,6 @@ """Stage: segment. -Consumes: tokens (role-None main stream), comma_offsets, one_case +Consumes: settled_clauses (rules/S13, on `run`'s later folds), and tokens (role-None main stream), comma_offsets, one_case (where an earlier stage recorded it). Produces: segments (runs of main-token indices; interior segments may be EMPTY -- doubled commas keep their structural position), structure @@ -105,6 +105,16 @@ def case_class() -> bool: # (rules/T4, #727): asked only under that opt-in policy, so the # default pays nothing for it. def suffixy(seg: tuple[int, ...], case: bool | None = None) -> bool: + # rules/S13: "it reads as a bracketed unambiguous credential does + # in the same place" -- Then, on a later fold: a settled word + # passes as such a credential does, its tag coming only in + # classify, and `class_run` refuses it as it refuses that + # credential (decisions/S13 (#810)) + if state.settled_clauses: + seg = tuple([i for i in seg if state.tokens[i].span + not in state.settled_clauses]) + if not seg: + return True return is_wholly_suffix( [state.tokens[i].text for i in seg], state.lexicon, state.policy, one_case=case, @@ -139,10 +149,13 @@ def class_run(seg: tuple[int, ...]) -> bool: # Policy-sensitive by construction: with # `unlisted_dotted_suffixes` off the token is name material, is # no candidate, and the flag stands. + # a settled word is no candidate, as the listed credential it + # reads as is not (rules/S13) return all(ambiguous_class_candidate(state.tokens[i].text, state.lexicon, state.policy) and not ambiguous_class_member(state.tokens[i].text, state.lexicon) + and state.tokens[i].span not in state.settled_clauses for i in seg) def titles_and_suffixes(seg: tuple[int, ...]) -> bool: @@ -209,13 +222,6 @@ def titles_and_suffixes(seg: tuple[int, ...]) -> bool: # model of group, the shape mechanisms.md's # READ-WITHOUT-THEN-BIND exists to remove. Last, as the # rarest reading. - # rules/S13: "it reads as a bracketed unambiguous credential does - # in the same place" -- Then, on a later fold: a settled word is - # such a credential, which this test would recognize, though its - # tag comes only in classify (decisions/S13 (#810)) - if state.settled_clauses: - seg = tuple(i for i in seg - if state.tokens[i].span not in state.settled_clauses) if (seg and not suffixy(seg) and not suffixy(seg, case_class()) and not class_run(seg) and not titles_and_suffixes(seg)): texts_joined = " ".join([state.tokens[i].text for i in seg]) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 707794f94..26ef72ec4 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -205,7 +205,7 @@ class ParseState: order it read them under, AND the folded-middle tag on the particles P6's attachment joins after a family comma (#613); post_rules -> roles again, settled_clauses (S13's, #810, which - extract and classify read on the second fold `run` makes), and the + extract, segment and classify read on `run`'s later folds), and the ambiguity P6's no-comma attachment reports (#467) and SUFFIX_OR_NICKNAME on a member S13 leaves a nickname. Ambiguities are recorded by every stage that DECIDES one -- @@ -289,8 +289,8 @@ class ParseState: one_case: bool | None = None ambiguities: tuple[PendingAmbiguity, ...] = () #: rules/S13's settled clauses, by the span of the one word each - #: holds: post_rules records them on the first fold, and `run` - #: folds the stages once more with them set, extract escaping each + #: holds: post_rules records them, and `run` folds the stages again + #: with them set until no new one settles, extract escaping each #: as it escapes a bracketed unambiguous credential and classify #: tagging its word as that credential's twin, so the whole name #: reads as the twin's does (#810). diff --git a/tests/v2/cases.py b/tests/v2/cases.py index f0a8018cb..eb418f689 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8900,6 +8900,14 @@ def _check_cjk_shape_purity(self) -> None: notes="no comma-structure report, as its twin '(MBA)' draws " "none: the part's tail test runs before the settled " "word is tagged, so it leaves the word out"), + Case("bracket_member_past_the_second_comma_after_a_dotted_shape", + "Smith, John, X.Y.Z. (MA)", + {"given": "John", "family": "Smith", "suffix": "X.Y.Z. MA"}, + classification="fix(#810)", + ambiguities=("comma-structure",), + notes="reported as its twin '(MBA)' is: the settled word is a " + "listed credential, no shape candidate, so the part is no " + "run of shape-admitted words"), Case("bracket_member_beside_a_dropped_character", "John Smith (MA \N{GRINNING FACE})", {"given": "John", "family": "Smith", "nickname": "MA"}, diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 831a59ff0..583f2693d 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2195,7 +2195,7 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( # rules/S13 (#810): a settled bracketed member reads as its twin, a # bracketed unambiguous credential in the same place, fields and reports # alike. The twin is MBA, a word no head holds, so the twin's reading -# maps back by replacing its text. 972 texts, about 2s. +# maps back by replacing its text. 1,044 texts, about 2s. _TWIN_HEADS = ( "John Smith {c}", "Jack {c}", "{c}", "John {c}", "JOHN SMITH {c}", "Smith, John {c}", "Smith, {c}", ", {c}", "(Bob), {c}", @@ -2208,7 +2208,8 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( # a third comma part, and a clause whose settling another makes # possible, the first settled word making the name mixed case "Smith, John, Jr. {c}", "JOHN SMITH (m.a) {c}", - "john smith (M.A) {c}", + "john smith (M.A) {c}", "Smith, John, X.Y.Z. {c}", + "Smith, John, Bob {c}", ) _TWIN_MEMBERS = ("MA", "M.A", "JD", "DO", "D.O", "MENG") _TWIN_PAIRS = (("(", ")"), ('"', '"')) @@ -2226,12 +2227,13 @@ def back(text: str) -> str: def test_a_settled_bracketed_member_reads_as_its_unambiguous_twin() -> None: """Negative control, measured 2026-10-26 with `run` patched to a single fold that makes the member the suffix in place (the first - draft's reading): 300 of 828 settled readings differ from the + draft's reading): 372 of 900 settled readings differ from the twin's, 36 each behind 'Jack', 'John', 'John Smith PhD', - 'John Smith, PhD', 'Smith, John, Jr.', before 'née Jones', 'VI' and - 'Ph. D.', and 12 for a clause standing alone ('Jack (MA)' read - family 'Jack', where the twin reads given 'Jack'). Folded to the - fixed point: 864 settled, 0 differ. A member the folds leave a + 'John Smith, PhD', 'Smith, John, Jr.', 'Smith, John, X.Y.Z.', + 'Smith, John, Bob', before 'née Jones', 'VI' and 'Ph. D.', and 12 + for a clause standing alone ('Jack (MA)' read family 'Jack', where + the twin reads given 'Jack'). Folded to the fixed point: 936 + settled, 0 differ. A member the folds leave a nickname is skipped here, so the clause a second fold settles is pinned by case rows instead (bracket_member_settled_by_another_*).""" settled = 0 From 4d6a316e169a7c54dbabb71c3a5ed87adf644e78 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 15:24:05 -0700 Subject: [PATCH 07/10] Note the re-measure against master 32049ade (#810) Co-Authored-By: Claude Opus 5.5 --- .../decisions/S13-bracketed-member-settled-by-its-writing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index b4d386e9f..3a3a21db9 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -18,5 +18,5 @@ - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. - **The third review, of the twin reading.** Three places read the member otherwise than its twin. One more fold was not enough: a clause the second fold settled was left a nickname, its report lost (`JOHN SMITH (m.a) (MA)`, `john smith (M.A) (MA)`), so `run` folds to the fixed point. The part past a second comma is tested for suffix words before classify tags the settled word, so a dotted member there drew a `comma-structure` report its twin does not (`Smith, John, Jr. (M.A)`); segment's tests now read the settled word as that credential: it passes the suffix test, fails the test for a run of shape-admitted words (whose pass had silenced the report on `Smith, John, X.Y.Z. (MA)` where the twin's listed word keeps it), and stays in the report's text (`segment 'Bob MA'`), the fourth review having found a first fix that left it out of the part changing both. And a clause was called the member alone when it held one token, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) let it settle where its twin, read by extract as the clause's whole text, does not escape (`John Smith (MA 😀)`); the test now compares the clause's text, as extract does. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), though the twin's credential run would take that word (`John Smith (MBA) Ma` → suffix 'MBA Ma'): the position is read on the nickname reading. - **Accepted:** a lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was N3's family name (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, where 2.0.0 through 2.6.0 gave family 'Jack', nickname 'MA'); and a maiden clause after a settled member, which lets it settle, is then read as one after its twin is, inside the credential's trailing run (S8: `John Smith (MA) née Jones` → suffix 'MA née Jones', as `John Smith (PhD) née Jones` reads on master). - - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. + - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the final tree after merging master 32049ade, #896, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. From 523c3696d4693de2b26390edeec23b99592dd4a6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 15:59:35 -0700 Subject: [PATCH 08/10] Read S13 in one pass; record where it differs from the twin (#810) Derek chose the one-pass version: the re-parse loop that read a settled member as its (PhD) twin cost a whole parse and machinery in four stages for a shape that moves 6 real names. A settled clause becomes the suffix in post_rules' single pass again, as 909026d8 had it; the settled_clauses field, run()'s fixed point and the extract, classify and segment hooks are gone, and frames are back near master's (John Smith (MA) 210, master 214). Kept: S13's one signal (position, citing S1, S3 and S6), and from the reviews of the twin draft the text-equality "alone in its clause" test, so a dropped character beside the member leaves it a nickname. Its comma-structure fix is not needed in one pass, the part past a second comma being tested while the clause is still a nickname. Where the rest of the name then reads otherwise than its twin is recorded as S13's Accepted bullets (Jack (MA), a maiden clause after it, a clause only a second pass would settle, a part past a second comma), and the twin test becomes a recorded census of the differences (336 of 1,044 readings, by head). Co-Authored-By: Claude Opus 5.5 --- ...bracketed-member-settled-by-its-writing.md | 16 ++-- docs/design/decisions/vocabulary-model.md | 4 +- docs/design/mechanisms.md | 4 +- ...bracketed-member-settled-by-its-writing.md | 26 +++++-- docs/design/vocabulary-model-rfc.md | 2 +- nameparser/_pipeline/__init__.py | 20 +---- nameparser/_pipeline/_classify.py | 12 +-- nameparser/_pipeline/_extract.py | 22 +----- nameparser/_pipeline/_post_rules.py | 42 +++++------ nameparser/_pipeline/_segment.py | 15 +--- nameparser/_pipeline/_state.py | 15 +--- tests/v2/cases.py | 59 ++++++++------- tests/v2/pipeline/test_state.py | 5 +- tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 11 ++- tests/v2/test_parser.py | 74 +++++++++++-------- tools/differential/corpus_rules.jsonl | 3 + tools/differential/snapshot.jsonl | 9 ++- 18 files changed, 159 insertions(+), 182 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 3a3a21db9..32054418d 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -2,21 +2,23 @@ ## Standing -- A member of the ambiguous class alone in brackets or quotes after the name, its writing settling it as the credential, reads as its twin, a bracketed unambiguous credential in the same place, does, fields and reports alike (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix; `Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, as `Jack (PhD)`); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`, `Jean (M.A) Smith`) — 2026-10-26 #810. +- A member of the ambiguous class alone in brackets or quotes after the name, its writing settling it as the credential, is the suffix, read in one pass, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`, `Jean (M.A) Smith`) — 2026-10-26 #810. - The writing that settles it is read by the rules that read writing: a period inside the word, S3's evidence, whatever case the name is in, and capitals in a name of mixed case, S6's, the case being that of the name's own words and never the clause's, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix. S13's own signal is the position alone — 2026-10-26 #810. - Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. +- Where the rest of the name then reads otherwise than beside a bracketed unambiguous credential in the same place, the member's twin, the difference is an accepted cost, recorded in rules/S13's Accepted bullets and counted by tests/v2/test_parser.py's census (`Jack (MA)` → family 'Jack', where `Jack (PhD)` reads given 'Jack') — 2026-10-26 #810. ## Log -- **2026-10-26 #810 (Derek, 2026-10-23; the twin reading and the single signal Derek, 2026-10-26) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME, SETTLED BY ITS WRITING, READS AS A BRACKETED UNAMBIGUOUS CREDENTIAL DOES.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. +- **2026-10-26 #810 (Derek, 2026-10-23; the single signal and the one pass Derek, 2026-10-26) — A BRACKETED MEMBER OF THE AMBIGUOUS CLASS AFTER THE NAME, SETTLED BY ITS WRITING, IS THE SUFFIX, READ IN ONE PASS.** Carries out item 16 of decisions/vocabulary-model's 2026-10-23 #752 entry, as phase 2l of #752. SUPERSEDES: in decisions/ma-do, the 2026-07-17 entry's side effect, in part (a parenthesized `(MA)` or `(DO)` no longer escaping to suffix), for a member after the name whose writing settles it. #810 asked whether `John Smith (M.A)` should read as `John Smith M.A` does, and found `(M.A)`, `"M.A"` and `(MA)` all reading as a nickname where 1.4.0 read the bracketed ones as a suffix; 2.0.0 through 2.6.0 read each as a nickname (measured 2026-10-26 on the wheels, `John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)`). The escape had refused every member since `ma` joined the class (decisions/ma-do, 2026-07-17), and #796's 33 new members widened its reach. - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4; a dated snapshot, the spike's switch being in no tree) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. - - **Why the twin, not the bare reading (Derek, 2026-10-26).** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. It reads instead as a bracketed unambiguous credential reads in the same place, the whole name included, which is what the brackets and the writing together say it is. Whether a member settles is known only once the roles are read, so post_rules records the settled clauses and the pipeline folds again from the start with them escaped as such a credential is and their word carrying its tag (`vocab:suffix`), until a fold settles no clause it was not given: a settled word joins the name's own words, which can make a one-case name mixed and settle a second clause (`JOHN SMITH (m.a) (MA)` → suffix 'm.a MA', as `JOHN SMITH (mba) (MA)`), and the set of settled clauses only grows, so the folding ends; the `suffix-or-nickname` report moved to post_rules from classify, so that it is raised where the fork is decided. The second fold costs a whole parse, on a name holding a settled clause only (frames counted with `sys.setprofile`, py3.11, 2026-10-26: `John Smith (MA)` 214 → 416, `Jack (MA)` 174 → 358; `Smith, John` 158 and `John Smith` 149 unchanged; a name holding an unsettled clause pays 3 or 6, `John (Bob) Smith` 204 → 207, `John Smith (Ed)` 214 → 220). The PR's first two drafts made the member the suffix in place instead, leaving the rest of the name read around a nickname: `Jack (MA)` read family 'Jack' where `Jack (PhD)` reads given 'Jack'. tests/v2/test_parser.py's `test_a_settled_bracketed_member_reads_as_its_unambiguous_twin` holds the invariant over a grid of comma shapes, heads and the three orders, its control recorded in its docstring. + - **Why the suffix, not the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly, the brackets having already set it apart from the name's words. The decision is made first in post_rules, where every word's role is read, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. Frames counted with `sys.setprofile`, py3.11, 2026-10-26, against master: `Smith, John` 158 and `John Smith` 149 unchanged; `John Smith (MA)` 214 → 210, `Jack (MA)` 174 → 171; a name holding a clause it does not settle pays 2 to 5 (`John (Bob) Smith` 204 → 206, `John Smith (Ed)` 214 → 219). + - **Declined: the twin reading, folding the pipeline again (Derek, 2026-10-26, "the one pass version").** A draft of this PR read a settled member exactly as its twin, a bracketed unambiguous credential in the same place, reads, fields and reports alike: post_rules recorded the settled clauses and the stages folded again from the start, extract escaping each clause and classify giving its word the credential's tag, until no new clause settled (a settled word joining the name's own words can make a one-case name mixed and settle a second clause). Its review rounds found three more places to bring in line with the twin, the part past a second comma among them. It cost a whole parse per settled name (`John Smith (MA)` 214 → 416 frames) and machinery in four stages for a shape that moves 6 real names in the corpora measured here. In one pass the rest of the name is read as beside a nickname, and where that differs from the twin the difference is an accepted cost, recorded in rules/S13's Accepted bullets: a lone name word stays N3's family name (`Jack (MA)`; `Jack (PhD)` reads given 'Jack', reported); a maiden clause after the member stays the maiden name, where the twin's credential run takes it (`John Smith (MA) née Jones`); the case is asked once, so a capitals member settled only once another settled word made the name mixed stays the nickname (`JOHN SMITH (m.a) (MA)`, where the twin `JOHN SMITH (mba) (MA)` reads suffix 'mba MA'); and a part past a second comma is weighed without the member (`Smith, John, X.Y.Z. (MA)` reports nothing where `(MBA)` draws `comma-structure`). tests/v2/test_parser.py's `test_where_a_bracketed_member_reads_otherwise_than_its_twin` records the census of such differences over a grid of comma shapes, heads and the three orders (336 of 1,044 readings on 2026-10-26, by head), so a change that moves the count shows up; the folded draft differed from the twin in none of the settled rows. - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× after the fix, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. - - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the twin reading S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. + - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the escape from the nickname S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - - **Left as found by the second review**: one post rule can still put a name word behind a settled clause on the first fold, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. - - **The third review, of the twin reading.** Three places read the member otherwise than its twin. One more fold was not enough: a clause the second fold settled was left a nickname, its report lost (`JOHN SMITH (m.a) (MA)`, `john smith (M.A) (MA)`), so `run` folds to the fixed point. The part past a second comma is tested for suffix words before classify tags the settled word, so a dotted member there drew a `comma-structure` report its twin does not (`Smith, John, Jr. (M.A)`); segment's tests now read the settled word as that credential: it passes the suffix test, fails the test for a run of shape-admitted words (whose pass had silenced the report on `Smith, John, X.Y.Z. (MA)` where the twin's listed word keeps it), and stays in the report's text (`segment 'Bob MA'`), the fourth review having found a first fix that left it out of the part changing both. And a clause was called the member alone when it held one token, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) let it settle where its twin, read by extract as the clause's whole text, does not escape (`John Smith (MA 😀)`); the test now compares the clause's text, as extract does. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), though the twin's credential run would take that word (`John Smith (MBA) Ma` → suffix 'MBA Ma'): the position is read on the nickname reading. - - **Accepted:** a lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was N3's family name (`Jack (MA)` → given 'Jack', suffix 'MA', `given-or-family` reported, where 2.0.0 through 2.6.0 gave family 'Jack', nickname 'MA'); and a maiden clause after a settled member, which lets it settle, is then read as one after its twin is, inside the credential's trailing run (S8: `John Smith (MA) née Jones` → suffix 'MA née Jones', as `John Smith (PhD) née Jones` reads on master). + - **Left as found by the second review**: one post rule can still put a name word behind a settled clause, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. + - **Kept from the twin draft's reviews.** A clause is the member alone only where its whole text is the member, as extract's escape reads a clause, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) leaves it the nickname, as its twin `(PhD 😀)` is no escape (`John Smith (MA 😀)`); a first draft counted tokens and settled it. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), the position being read on the nickname reading. The draft's comma-structure fix is not needed in one pass: the part past a second comma is tested while the clause is still a nickname, so `Smith, John, Jr. (M.A)` reports nothing, as its twin does. + - **Accepted:** the twin differences the declined bullet above lists, in rules/S13's Accepted bullets. - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the final tree after merging master 32049ade, #896, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0dacaecb6..6ace6c173 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - A listed credential that is also a name, written in capitals in a name of mixed case, is settled as the credential by its writing: it reports nothing and starts the credential's run where no name word stands behind it with no family comma (2026-10-26, Derek), and keeps its other readings, the clash settled at the sites that choose a reading rather than by retagging the word (decisions/S6) — 2026-10-25 #752, phase 2g, #785. -- A member of the ambiguous class alone in brackets or quotes after the name reads as a bracketed unambiguous credential does in the same place where its writing settles it, with a period inside it or in capitals in a name of mixed case, the case being the name's own words' (rules/S13, decisions/S13) — 2026-10-26 #752, phase 2l. +- A member of the ambiguous class alone in brackets or quotes after the name is the suffix where its writing settles it, with a period inside it or in capitals in a name of mixed case, the case being the name's own words' (rules/S13, decisions/S13) — 2026-10-26 #752, phase 2l. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. - `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. @@ -219,4 +219,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `wing` — THE FIRST TITLE BATCH: `wing` HAS A DATED `given` READING.** The order is the 2026-10-24 #752 (Derek) entry's. Decision 10 had already read `wing` as a name at the head of the name, so the reading reaches the parts decision 10 left as listed, after a comma and ending the name: `Lau, Wing Kai` reads given 'Wing', and `John Smith Wing.` family 'Wing.'. The rule and the measurement are decisions/H9's entry of this date. - **2026-10-26 (Derek), #785 — PHASE 2g NARROWED: a capitals member with a name word behind it, and no family comma, does not start the credential's run**, the name word being evidence of initials; `Maria Ana MA Silva`, `Jean Pierre MA DUPONT` and `Mary Jane MA King` keep the readings 1.4.0 through 2.6.0 gave, and `John Smith MA Jones` keeping family 'Jones' is the cost. The record is decisions/S6's entry of this date. -- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name reads as a bracketed unambiguous credential does in the same place where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix; `Jack (MA)` as `Jack (PhD)`), and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. +- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 81a85372d..1da3635ab 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -196,9 +196,9 @@ This catalog converts discovery into a one-time cost — it does not remove disc **How it works.** "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. -**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), which decides once the parse's roles are read that a member of the ambiguous class after the name is settled by its writing (rules/S13); `run` then folds the stages again, extract escaping the clause here and classify tagging its word as an unambiguous credential's, so it rejoins the bare path as its twin, a bracketed unambiguous credential, would, not as its own bare word, which reads differently (`M.A` reads as initials bare; decisions/S13). +**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), the one site where content overrules the delimiter WITHOUT rejoining the bare path: a member of the ambiguous class settled by its writing after the name (rules/S13) is the suffix directly, because its bare reading would differ (`M.A` reads as initials bare), so it is decided once the parse's roles are read rather than unmasked here (decisions/S13). -**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Where the bare word itself would read wrongly, rejoin it as the twin it is, as S13 does. +**Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Not when the bare reading is the wrong answer for the clause: that is S13's shape. ## CURATED-VOCABULARY-ALTERNATION — the config already splits them diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index 8cb1709c9..71a824bb1 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -2,11 +2,11 @@ **When:** a clause that is one member of the ambiguous class (S4) stands in brackets or quotes after the name, with no name word after it, though a suffix, a title or a maiden clause may follow however each is written, as in "John Smith (MA)". -**Then:** where the member's writing settles it as the credential, the way the rules that read the writing say a credential is written, it is no nickname: it reads as a bracketed unambiguous credential does in the same place, and the clash is not reported. +**Then:** where the member's writing settles it as the credential, the way the rules that read the writing say a credential is written, it is no nickname but the suffix, and the clash is not reported. - **Written with a period inside it**, whatever case the name is in: the periods are how a credential is written and a name is not (S3). - **Written in capitals in a name written in more than one case**: the case is asked of the name's own words and never of the clause's, so capitals settle nothing in a name written wholly in one case, and Title case leans to the name (S6). -- **It reads as its twin**, a bracketed unambiguous credential in the same place, fields and reports alike (S1): a lone name word beside it is the given name as beside "(PhD)", and the member is not read as if written bare, where two letters each followed by a period but the last read as initials (S3). +- **It is the suffix, not the word read as if written bare** (S1): the brackets have already set it apart from the name's words, and bare, two letters each followed by a period but the last read as initials (S3). The rest of the name is read as beside a nickname, which is where it can differ from a bracketed unambiguous credential in the same place, its twin, as the Accepted bullets record. **Because:** after the name, brackets hold either a credential or the name the person goes by, and the writing tells the two apart where the count of words cannot: credentials are written in capitals and with their periods, and names are not. Inside the name the same word let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by. So only the clause after the name is settled. @@ -22,7 +22,6 @@ "John Smith (MA) Dr." → title="Dr." "John Smith (MA) Dr." → suffix="MA" "John Smith «MA»" → suffix="MA" -"Jack (MA)" → given="Jack" "Jack (M.A)" → suffix="M.A" "John Smith 「MA」" → suffix="MA" "John Smith (Ed)" → nickname="Ed" · boundary @@ -55,15 +54,26 @@ **Accepted:** -- A lone name word beside a settled member is the given name, as beside its twin, where beside the nickname it was the family name (N3), and the fork is reported as for any lone name word. +- A lone name word beside a settled member stays the family name it is beside a nickname (N3), where beside its twin, "(PhD)", it is the given name, reported. ``` - "Jack (MA)" → ambiguities=("given-or-family",) + "Jack (MA)" → family="Jack" + "Jack (MA)" → ambiguities=() ``` -- A maiden clause after a settled member is read as one after its twin is, inside the credential's trailing run (S8), so the clause settles the member and is then taken into the suffix. +- A maiden clause after a settled member stays the maiden name, where after its twin the credential's trailing run takes it (S8). ``` - "John Smith (MA) née Jones" → suffix="MA née Jones" + "John Smith (MA) née Jones" → maiden="Jones" + ``` +- The case is asked once, of the name's own words without any clause, so a member in capitals beside a settled one in a name otherwise written in one case stays the nickname, where its twin, joining the name's words, makes the name mixed and settles the other too. + ``` + "JOHN SMITH (m.a) (MA)" → nickname="MA" + "JOHN SMITH (m.a) (MA)" → suffix="m.a" + ``` +- A part past the second comma is weighed without the member, so what stands beside it decides whether the part reports, and the report names that alone, where its twin is weighed as a credential of the part. + ``` + "Smith, John, X.Y.Z. (MA)" → ambiguities=() + "Smith, John, Bob (MA)" → ambiguities=("comma-structure",) ``` --- **Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name; hub of S1, S3, S6 -**history:** decisions/S13 · **implemented:** nameparser/_pipeline/__init__.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_extract.py, nameparser/_pipeline/_post_rules.py, nameparser/_pipeline/_segment.py +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_post_rules.py diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 4e5630937..8c5f71dfc 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -130,7 +130,7 @@ Phase 2i carried this out as [rules/H9](rules/H9-title-phrase.md) (the title phr A bracketed or quoted credential is to escape the nickname reading where its writing settles it (#810): with interior periods, or in capitals in a name of mixed case, the case asked of the whole name. `(Ed)` stays a nickname, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`). -Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-its-writing.md) (a bracketed member settled by its writing) rather than as an amendment to [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md), since the position it reads is a signal S1 does not: `John Smith (MA)` and `John Smith (M.A)` read suffix, and `JOHN SMITH (MA)` and `Jeffrey (JD) Bricken` keep the nickname. The writing is read as the rules that read it say, S3 for the periods and S6 for the capitals in a name of mixed case, and S13's own signal is the position. A settled member reads as its twin, a bracketed unambiguous credential in the same place, does (`Jack (MA)` reads as `Jack (PhD)` does), not as the word read bare, since bare, `M.A` reads as initials. +Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-its-writing.md) (a bracketed member settled by its writing) rather than as an amendment to [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md), since the position it reads is a signal S1 does not: `John Smith (MA)` and `John Smith (M.A)` read suffix, and `JOHN SMITH (MA)` and `Jeffrey (JD) Bricken` keep the nickname. The writing is read as the rules that read it say, S3 for the periods and S6 for the capitals in a name of mixed case, and S13's own signal is the position. A settled member is the suffix, decided in one pass, rather than the word read bare, since bare, `M.A` reads as initials; where the rest of the name then reads otherwise than beside a bracketed unambiguous credential (`Jack (MA)` keeps family 'Jack' where `Jack (PhD)` reads given 'Jack'), S13 records the difference as accepted. ### Readings on the parse diff --git a/nameparser/_pipeline/__init__.py b/nameparser/_pipeline/__init__.py index 2d8473968..9c91ba088 100644 --- a/nameparser/_pipeline/__init__.py +++ b/nameparser/_pipeline/__init__.py @@ -18,7 +18,7 @@ from nameparser._pipeline._post_rules import post_rules from nameparser._pipeline._script_segment import script_segment from nameparser._pipeline._segment import segment -from nameparser._pipeline._state import ParseState, copy_with +from nameparser._pipeline._state import ParseState from nameparser._pipeline._tokenize import tokenize #: The full eight-stage fold. @@ -33,18 +33,6 @@ def run(state: ParseState) -> ParseState: a new ParseState; parse is total over str input, so no stage may raise on the *content* of a name (malformed policy/config is a separate, allowed failure mode).""" - # rules/S13: "it reads as a bracketed unambiguous credential does in - # the same place" -- Then; the clauses are known only once the roles - # are read, so the stages fold again from the initial state with - # the settled clauses set, until a fold settles none it was not - # given (decisions/S13 (#810)). A settled word joins the name's own - # words, which can make a one-case name mixed and settle a second - # clause; the set only grows and is bounded by the clauses, so the - # loop ends, and a name with none settled folds once. - while True: - folded = state - for stage in STAGES: - folded = stage(folded) - if folded.settled_clauses <= state.settled_clauses: - return folded - state = copy_with(state, settled_clauses=folded.settled_clauses) + for stage in STAGES: + state = stage(state) + return state diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 13bdf518c..f25197634 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -5,9 +5,7 @@ _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the second comma asks for it (#289/#516, #613), so this read is the -fallback for every other name; and settled_clauses on `run`'s later -folds (rules/S13, #810), whose words take an unambiguous credential's -tag. +fallback for every other name. Produces: tokens with vocabulary tags added (text/span/role unchanged), plus ambiguities (CONJUNCTION_OR_INITIAL) and one_case -- whether the name's own words are written in one case, @@ -412,14 +410,6 @@ def classify(state: ParseState) -> ParseState: one_case and i < clause_at and t.role is None))) for i, t in enumerate(state.tokens)) - # rules/S13: "it reads as a bracketed unambiguous credential does in - # the same place" -- Then, on a later fold, the settled word - # taking the tag such a credential's word carries (decisions/S13 - # (#810)) - if state.settled_clauses: - tokens = tuple( - copy_with(t, tags=frozenset({"vocab:suffix"})) - if t.span in state.settled_clauses else t for t in tokens) # A word with a delimiter glued to it joins no neighbour across the # delimiter, so the stem's joining tags stay off it: 'y;' is no # connective to the word after it (rules/C10: "no join, maiden diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 7ace440f3..25a931e17 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -1,8 +1,6 @@ """Stage: extract_delimited. -Consumes: ParseState.original, and settled_clauses on `run`'s later -folds (rules/S13, #810), each settled clause escaping as a bracketed -unambiguous credential does. +Consumes: ParseState.original. Produces: extracted (role + inner span per delimited region), masked (full regions incl. delimiter chars, skipped by tokenize), UNBALANCED_DELIMITER ambiguities for opens with no close. @@ -101,14 +99,6 @@ def _maiden_marked(content: str, lexicon: Lexicon) -> bool: and not comma_follows_marker(words[run])) -def _settled_within(starts: list[int], inner: Span) -> bool: - """Whether a word rules/S13 settled starts inside the clause: - `starts` is the settled words' start offsets, sorted, so the test - is one bisect and the scan stays linear in the clauses.""" - k = bisect.bisect_left(starts, inner.start) - return k < len(starts) and starts[k] < inner.end - - # rules/N2: "a quote whose open and close are the same character # opens only at a word start and closes only at a word end, so an # apostrophe inside or at the end of a word is literal" @@ -204,7 +194,6 @@ def extract_delimited(state: ParseState) -> ParseState: ) extracted: list[tuple[Role, Span]] = [] masked: list[Span] = [] - settled = sorted(s.start for s in state.settled_clauses) # candidates, not final: each carries the offset of the unmatched # open so ones consumed by a later match can be filtered at the end unbalanced: list[tuple[int, PendingAmbiguity]] = [] @@ -251,13 +240,8 @@ def extract_delimited(state: ParseState) -> ParseState: exhausted.add((role, open_, close)) continue inner = Span(i + len(open_), j) - if inner.start < inner.end and (_suffix_shaped( - text[inner.start:inner.end], state.lexicon) - # rules/S13: "it reads as a bracketed unambiguous - # credential does in the same place" -- Then, on a later - # fold: the clause escapes as that credential's does - # (decisions/S13 (#810)) - or (settled and _settled_within(settled, inner))): + if inner.start < inner.end and _suffix_shaped( + text[inner.start:inner.end], state.lexicon): # v1 parse_nicknames: suffix-shaped delimited content is # left IN PLACE (undelimited) for normal downstream parsing # -- 'Andrew Perkins (MBA)' keeps MBA a suffix, not a diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 40233ebd9..3cd6996de 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -6,10 +6,10 @@ rotation gate. Also comma_offsets and dropped, which R1's entry pass below reads to find the separators the writer typed (#436/#437), and extracted and one_case, which S13's pass reads (#810). -Produces: tokens with roles adjusted by the post rules; settled_clauses, -the delimited members rules/S13 settles after the name, for the later -folds `run` makes (#810), and the SUFFIX_OR_NICKNAME report on any other -member left a nickname; the stable +Produces: tokens with roles adjusted by the post rules (among them the +SUFFIX role of a delimited member its writing settles after the name, +rules/S13, with the SUFFIX_OR_NICKNAME report on any other member left +a nickname), the stable "joined" tag on a post-nominal continuing the entry before it, and the ambiguity P6's attachment reports at the end of a name read family-first (#467) and after a comma with nothing before it -- after a @@ -51,7 +51,6 @@ from nameparser._policy import PatronymicRule from nameparser._types import ( FOLDED_TAG, UNJOINED_CONJUNCTION_TAG, UNJOINED_TAG, AmbiguityKind, Role, - Span, ) # Ported verbatim from v1 (nameparser/config/regexes.py) -- layering @@ -387,18 +386,15 @@ def _family_settled(state: ParseState, tokens: list[WorkToken], # rules/S13: "where the member's writing settles it as the credential, # the way the rules that read the writing say a credential is written, -# it is no nickname" -- Then and its first two bullets, with the When's -# "no name word after it"; extract's S1 escape has already sent every -# suffix-shaped clause through, and the twin reading is `run`'s second -# fold (decisions/S13 (#810)) +# it is no nickname but the suffix" -- Then and its first two bullets, +# with the When's "no name word after it"; +# extract's S1 escape has already sent every suffix-shaped clause +# through (decisions/S13 (#810)) def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], - ambiguities: list[PendingAmbiguity], - ) -> frozenset[Span]: - """The words of the clauses S13 settles: a nickname that is the whole - of its clause and a member of the ambiguous class, where its writing - settles it and it stands after the last name word, `run` then - folding the stages again so it reads as its twin, a bracketed - unambiguous credential, does. Any other tagged member stays + ambiguities: list[PendingAmbiguity]) -> None: + """A nickname that is the whole of its clause and a member of the + ambiguous class becomes the suffix where its writing settles it and + it stands after the last name word; any other tagged member stays the nickname and the fork is reported, a dotted member left a nickname inside the name included. Asked here, first in this stage, because only once roles are read is it known whether a name @@ -416,13 +412,14 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], starts = [t.span.start for t in tokens] for _, clause in state.extracted: lo = bisect.bisect_left(starts, clause.start) - # the whole clause, as extract's escape asks it of its twin + # the whole clause's text, as extract's escape reads a clause: + # a character tokenize drops beside the member (an emoji, a + # zero-width space) leaves it no member alone if (bisect.bisect_left(starts, clause.end) == lo + 1 and state.original[clause.start:clause.end].strip() == tokens[lo].text): alone.add(lo) lex = state.lexicon - settled: set[Span] = set() for i, token in enumerate(tokens): if token.role is not Role.NICKNAME: continue @@ -433,22 +430,20 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], in lex.suffix_acronyms_ambiguous) if i > last_name and i in alone and ( dotted or settled_by_case(token, state.one_case)): - settled.add(token.span) + tokens[i] = copy_with(token, role=Role.SUFFIX) elif dotted or AMBIGUOUS_ACRONYM_TAG in token.tags: ambiguities.append(PendingAmbiguity( AmbiguityKind.SUFFIX_OR_NICKNAME, f"delimited {token.text!r} is also a post-nominal; read " f"as a nickname rather than a suffix", (i,))) - return frozenset(settled) def post_rules(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) - settled: frozenset[Span] = frozenset() if state.extracted: - settled = _settle_delimited_members(state, tokens, ambiguities) + _settle_delimited_members(state, tokens, ambiguities) titles = _idx(tokens, Role.TITLE) givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) @@ -850,5 +845,4 @@ def post_rules(state: ParseState) -> ParseState: tags=tokens[i].tags | {UNJOINED_CONJUNCTION_TAG}) _mark_suffix_entries(tokens, state) return copy_with(state, tokens=tuple(tokens), - ambiguities=tuple(ambiguities), - settled_clauses=settled | state.settled_clauses) + ambiguities=tuple(ambiguities)) diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index 9d0c59014..cceecab9e 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -1,6 +1,6 @@ """Stage: segment. -Consumes: settled_clauses (rules/S13, on `run`'s later folds), and tokens (role-None main stream), comma_offsets, one_case +Consumes: tokens (role-None main stream), comma_offsets, one_case (where an earlier stage recorded it). Produces: segments (runs of main-token indices; interior segments may be EMPTY -- doubled commas keep their structural position), structure @@ -105,16 +105,6 @@ def case_class() -> bool: # (rules/T4, #727): asked only under that opt-in policy, so the # default pays nothing for it. def suffixy(seg: tuple[int, ...], case: bool | None = None) -> bool: - # rules/S13: "it reads as a bracketed unambiguous credential does - # in the same place" -- Then, on a later fold: a settled word - # passes as such a credential does, its tag coming only in - # classify, and `class_run` refuses it as it refuses that - # credential (decisions/S13 (#810)) - if state.settled_clauses: - seg = tuple([i for i in seg if state.tokens[i].span - not in state.settled_clauses]) - if not seg: - return True return is_wholly_suffix( [state.tokens[i].text for i in seg], state.lexicon, state.policy, one_case=case, @@ -149,13 +139,10 @@ def class_run(seg: tuple[int, ...]) -> bool: # Policy-sensitive by construction: with # `unlisted_dotted_suffixes` off the token is name material, is # no candidate, and the flag stands. - # a settled word is no candidate, as the listed credential it - # reads as is not (rules/S13) return all(ambiguous_class_candidate(state.tokens[i].text, state.lexicon, state.policy) and not ambiguous_class_member(state.tokens[i].text, state.lexicon) - and state.tokens[i].span not in state.settled_clauses for i in seg) def titles_and_suffixes(seg: tuple[int, ...]) -> bool: diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 26ef72ec4..542b5b837 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -204,10 +204,10 @@ class ParseState: assign -> the remaining token roles AND `order`, the effective order it read them under, AND the folded-middle tag on the particles P6's attachment joins after a family comma (#613); - post_rules -> roles again, settled_clauses (S13's, #810, which - extract, segment and classify read on `run`'s later folds), and the - ambiguity P6's no-comma attachment reports (#467) and - SUFFIX_OR_NICKNAME on a member S13 leaves a nickname. + post_rules -> roles again (a delimited member S13 settles becomes + the suffix, #810), and the ambiguity P6's no-comma attachment + reports (#467) and SUFFIX_OR_NICKNAME on a member it leaves a + nickname. Ambiguities are recorded by every stage that DECIDES one -- extract (resolved to a token index by tokenize), segment, script_segment, classify, group, assign, and post_rules -- since a @@ -288,13 +288,6 @@ class ParseState: #: did not, without recording the answer (`_one_case`). one_case: bool | None = None ambiguities: tuple[PendingAmbiguity, ...] = () - #: rules/S13's settled clauses, by the span of the one word each - #: holds: post_rules records them, and `run` folds the stages again - #: with them set until no new one settles, extract escaping each - #: as it escapes a bracketed unambiguous credential and classify - #: tagging its word as that credential's twin, so the whole name - #: reads as the twin's does (#810). - settled_clauses: frozenset[Span] = frozenset() def _copy_refusals(cls: type) -> list[str]: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index aff951845..26e5b3e77 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8875,39 +8875,45 @@ def _check_cjk_shape_purity(self) -> None: notes="a suffix after the clause is no name word"), Case("bracket_member_before_a_maiden_clause", "John Smith (MA) née Jones", - {"given": "John", "family": "Smith", "suffix": "MA née Jones"}, + {"given": "John", "family": "Smith", "suffix": "MA", + "maiden": "Jones"}, classification="fix(#810)", - ambiguities=("suffix-or-name", "suffix-or-name"), - notes="the clause settles the member and is then read as its " - "twin 'John Smith (PhD) née Jones' reads it, inside the " - "credential's run (rules/S8, S13's Accepted); 1.4.0 read " - "middle 'Smith MA née', last 'Jones'"), + notes="a maiden clause after it is no name word, and stays the " + "maiden name, where its twin's credential run takes it " + "(rules/S13's Accepted); 1.4.0 read middle 'Smith MA " + "née', last 'Jones'"), Case("bracket_member_settled_by_another_in_a_one_case_name", "JOHN SMITH (m.a) (MA)", - {"given": "JOHN", "family": "SMITH", "suffix": "m.a MA"}, + {"given": "JOHN", "family": "SMITH", "suffix": "m.a", + "nickname": "MA"}, classification="fix(#810)", - notes="the settled 'm.a' joins the name's own words, which are " - "then mixed, as its twin 'JOHN SMITH (mba) (MA)' makes " - "them; a fold too few left 'MA' a nickname, unreported"), + ambiguities=("suffix-or-nickname",), + notes="the case is asked once, of the name's own words without " + "either clause, so 'MA' stays the nickname where its twin " + "'JOHN SMITH (mba) (MA)' makes the name mixed " + "(rules/S13's Accepted)"), Case("bracket_member_settled_by_another_in_a_lower_case_name", "john smith (M.A) (MA)", - {"given": "john", "family": "smith", "suffix": "M.A MA"}, + {"given": "john", "family": "smith", "suffix": "M.A", + "nickname": "MA"}, classification="fix(#810)", - notes="as its twin 'john smith (MBA) (MA)' reads"), + ambiguities=("suffix-or-nickname",), + notes="as the row above; the twin 'john smith (MBA) (MA)' " + "reads suffix 'MBA MA'"), Case("bracket_member_past_the_second_comma", "Smith, John, Jr. (M.A)", {"given": "John", "family": "Smith", "suffix": "Jr. M.A"}, classification="fix(#810)", notes="no comma-structure report, as its twin '(MBA)' draws " - "none: the part's tail test runs before the settled " - "word is tagged, so it leaves the word out"), + "none: the part is tested while the clause is still a " + "nickname, and 'Jr.' alone is a suffix"), Case("bracket_member_past_the_second_comma_after_a_dotted_shape", "Smith, John, X.Y.Z. (MA)", {"given": "John", "family": "Smith", "suffix": "X.Y.Z. MA"}, classification="fix(#810)", - ambiguities=("comma-structure",), - notes="reported as its twin '(MBA)' is: the settled word is a " - "listed credential, no shape candidate, so the part is no " - "run of shape-admitted words"), + notes="no report, where its twin '(MBA)' draws comma-structure: " + "the part is tested while the clause is a nickname, and " + "'X.Y.Z.' alone is a run of shape-admitted words " + "(rules/S13's Accepted)"), Case("bracket_member_beside_a_dropped_character", "John Smith (MA \N{GRINNING FACE})", {"given": "John", "family": "Smith", "nickname": "MA"}, @@ -8928,19 +8934,18 @@ def _check_cjk_shape_purity(self) -> None: {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), Case("bracket_member_after_a_lone_name_word", "Jack (MA)", - {"given": "Jack", "suffix": "MA"}, + {"family": "Jack", "suffix": "MA"}, classification="fix(#810)", - ambiguities=("given-or-family",), - notes="reads as its twin 'Jack (PhD)' does; 1.4.0 read last " - "'MA', 2.0.0 through 2.6.0 family 'Jack', nickname 'MA'"), + notes="the word stays N3's family name, read beside the clause " + "as a nickname, where its twin 'Jack (PhD)' reads given " + "'Jack' (rules/S13's Accepted); 1.4.0 read last 'MA', " + "2.0.0 through 2.6.0 family 'Jack', nickname 'MA'"), Case("bracket_member_with_a_period_after_a_lone_name_word", "Jack (M.A)", - {"given": "Jack", "suffix": "M.A"}, + {"family": "Jack", "suffix": "M.A"}, classification="fix(#810)", - ambiguities=("given-or-family",), - notes="reads as its twin 'Jack (PhD)' does, where bare " - "'Jack M.A' reads family 'M.A'; 2.0.0 through 2.6.0 " - "read family 'Jack', nickname 'M.A'"), + notes="as the row above; bare 'Jack M.A' reads family 'M.A'; " + "2.0.0 through 2.6.0 read family 'Jack', nickname 'M.A'"), Case("bracket_member_before_a_shape_read_suffix", "John Smith (MA) Ph. D.", {"given": "John", "family": "Smith", "suffix": "MA Ph. D."}, diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index 8f199a5d5..d455fe78b 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -305,9 +305,8 @@ def test_stage_field_ownership() -> None: # attachment (#467) takes the family branch of a fork whose # other branches assign and group take, so each stage reports # the side it decides; the comma site is assign's since #613; - # and SUFFIX_OR_NICKNAME, with S13's decision (#810), whose - # settled clauses it records for `run`'s second fold - "post_rules": {"tokens", "ambiguities", "settled_clauses"}, + # and SUFFIX_OR_NICKNAME, with S13's decision (#810) + "post_rules": {"tokens", "ambiguities"}, } assert {s.__name__ for s in STAGES} == set(ownership) # Within the tokens themselves the contract is finer: texts and diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index c14bbeca8..c6a78140f 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -65,7 +65,7 @@ "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "S12": {"words": 674, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, - "S13": {"words": 390, "then_words": 172, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 2, "accepted_words": 71}, + "S13": {"words": 398, "then_words": 180, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 4, "accepted_words": 138}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 934fd90d0..f0c78e3a7 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -281,9 +281,14 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # as 'Jack (PhD)' does, a given-or-family claim each (+6), and # 'John Smith (MA) née Jones' its two suffix-or-name claims (+6); # 1381 -> 1390 the same day, its review round's rows: a member - # beside a dropped character (+3) and before 'Ma' (+6). - assert claims == 1390, ( - f"the field sweep checks {claims} claims, recorded as 1390 on " + # beside a dropped character (+3) and before 'Ma' (+6); 1390 -> 1384 + # the same day, one pass in place of the twin reading (Derek): + # 'Jack (MA)' and 'Jack (M.A)' lose their given-or-family claims and + # 'John Smith (MA) née Jones' its two suffix-or-name claims (-12), + # and the two clauses a second pass would settle keep the nickname, + # reported (+6). + assert claims == 1384, ( + f"the field sweep checks {claims} claims, recorded as 1384 on " f"2026-10-26") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 583f2693d..5904ec404 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1282,13 +1282,9 @@ def test_revise_reads_a_glued_honorific_on_its_own() -> None: #: a maiden marker the whole name left a word -- in a tail part, or #: taken into #602's credential run behind a credential -- and that #: the value's own sub-parse consumes mid-value, as revise() documents. -#: 'John Smith (MA) née Jones' joined 2026-10-26 (#810): a member -#: rules/S13 settles reads as its twin '(PhD)' does, inside the -#: credential's run, which takes the marker. _MARKER_IN_SUFFIX = frozenset({"Jane Doe PhD nee Smith", "Smith, John, MD - née Jones Smith", - "Smith, John, PhD née Jones", - "John Smith (MA) née Jones"}) + "Smith, John, PhD née Jones"}) def _suffix_bearing_corpus_names() -> list[str]: @@ -2192,10 +2188,10 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( assert Parser().parse(f"田中{word}").suffix == "" -# rules/S13 (#810): a settled bracketed member reads as its twin, a -# bracketed unambiguous credential in the same place, fields and reports -# alike. The twin is MBA, a word no head holds, so the twin's reading -# maps back by replacing its text. 1,044 texts, about 2s. +# rules/S13 (#810): where a bracketed member reads differently from its +# twin, a bracketed unambiguous credential in the same place. The twin +# is MBA, a word no head holds, so the twin's reading maps back by +# replacing its text. 1,044 texts, about 2s. _TWIN_HEADS = ( "John Smith {c}", "Jack {c}", "{c}", "John {c}", "JOHN SMITH {c}", "Smith, John {c}", "Smith, {c}", ", {c}", "(Bob), {c}", @@ -2224,31 +2220,49 @@ def back(text: str) -> str: [(a.kind.value, back(a.detail)) for a in name.ambiguities]) -def test_a_settled_bracketed_member_reads_as_its_unambiguous_twin() -> None: - """Negative control, measured 2026-10-26 with `run` patched to a - single fold that makes the member the suffix in place (the first - draft's reading): 372 of 900 settled readings differ from the - twin's, 36 each behind 'Jack', 'John', 'John Smith PhD', - 'John Smith, PhD', 'Smith, John, Jr.', 'Smith, John, X.Y.Z.', - 'Smith, John, Bob', before 'née Jones', 'VI' and 'Ph. D.', and 12 - for a clause standing alone ('Jack (MA)' read family 'Jack', where - the twin reads given 'Jack'). Folded to the fixed point: 936 - settled, 0 differ. A member the folds leave a - nickname is skipped here, so the clause a second fold settles is - pinned by case rows instead (bracket_member_settled_by_another_*).""" - settled = 0 - differ = [] +#: The census, by head and by whether the member was settled or left +#: the nickname, recorded 2026-10-26 (#810). rules/S13's Accepted bullets +#: and its Except bullets name each shape: a lone name word stays N3's +#: family name beside a settled member where its twin's is given; the +#: maiden clause stays the maiden name where the twin's credential run +#: takes it; a part past a second comma reports what is left without +#: the member; a name with no word of its own, or written in one case, +#: settles no capitals; and a clause another settled one would make +#: settle, on a second reading, is left the nickname. +_TWIN_CENSUS = { + ("(Bob), {c}", "nickname"): 24, + (", {c}", "nickname"): 24, + ("JOHN SMITH (m.a) {c}", "nickname"): 24, + ("JOHN SMITH {c}", "nickname"): 24, + ("Jack {c}", "settled"): 36, + ("John Smith {c} née Jones", "settled"): 36, + ("John {c}", "settled"): 36, + ("Smith, John, Bob {c}", "settled"): 36, + ("Smith, John, X.Y.Z. {c}", "settled"): 36, + ("john smith (M.A) {c}", "nickname"): 24, + ("{c}", "nickname"): 24, + ("{c}", "settled"): 12, +} + + +def test_where_a_bracketed_member_reads_otherwise_than_its_twin() -> None: + """A census, not a guarantee: one pass decides S13 once the roles are + read, so the rest of the name is read around the clause as a + nickname, and that differs from the twin's reading in the shapes + _TWIN_CENSUS names. A change that moves a count, or adds a head, is + a change to what rules/S13 accepts: re-record it there and here. + The twin reading folded to a fixed point (the PR's fourth draft, + declined by Derek for one pass) differed in none of the settled + rows.""" + census: dict[tuple[str, str], int] = {} for (open_, close), word, head, order in itertools.product( _TWIN_PAIRS, _TWIN_MEMBERS, _TWIN_HEADS, _TWIN_ORDERS): parser = Parser(policy=Policy(name_order=order)) name = parser.parse(head.format(c=f"{open_}{word}{close}")) - if (word in name.nickname.split() - or word not in name.suffix.replace(",", " ").split()): - continue - settled += 1 twin = parser.parse(head.format(c=f"{open_}MBA{close}")) if _twin_reading(name, None, word) != _twin_reading( twin, "MBA", word): - differ.append((name.original, order)) - assert settled > 700, settled - assert not differ, differ + key = (head, "nickname" if word in name.nickname.split() + else "settled") + census[key] = census.get(key, 0) + 1 + assert census == _TWIN_CENSUS diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b33e35e99..8c200ea90 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -248,6 +248,7 @@ "JOHN QUINCY SMITH I" "JOHN SMITH (M.A)" "JOHN SMITH (MA)" +"JOHN SMITH (m.a) (MA)" "JOHN SMITH JR, AD" "JOHN SMITH MA" "JOHN SMITH MA JONES" @@ -788,12 +789,14 @@ "Smith, John Vi" "Smith, John X" "Smith, John van der Berg Prof." +"Smith, John, Bob (MA)" "Smith, John, MD - née Jones Smith" "Smith, John, PhD - and MD" "Smith, John, PhD - i Soler" "Smith, John, PhD née Jones" "Smith, John, Prof." "Smith, John, Puig - i Soler" +"Smith, John, X.Y.Z. (MA)" "Smith, John, and" "Smith, John; MD" "Smith, Jr." diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 01303bbb2..4c556d005 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1377,6 +1377,7 @@ {"name": "JOHN SMITH", "given": "JOHN", "family": "SMITH", "initials": "J. S.", "capitalized": "John Smith"} {"name": "JOHN SMITH (M.A)", "given": "JOHN", "family": "SMITH", "suffix": "M.A", "initials": "J. S.", "capitalized": "John Smith M.A"} {"name": "JOHN SMITH (MA)", "given": "JOHN", "family": "SMITH", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized": "John \"Ma\" Smith"} +{"name": "JOHN SMITH (m.a) (MA)", "given": "JOHN", "family": "SMITH", "suffix": "m.a", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized": "John \"Ma\" Smith M.A"} {"name": "JOHN SMITH JR, AD", "given": "AD", "family": "JOHN SMITH", "suffix": "JR", "initials": "A. J. S.", "capitalized": "Ad John Smith Jr"} {"name": "JOHN SMITH MA", "given": "JOHN", "family": "SMITH", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized": "John Smith MA"} {"name": "JOHN SMITH MA JONES", "given": "JOHN", "middle": "SMITH MA", "family": "JONES", "initials": "J. S. M. J.", "capitalized": "John Smith Ma Jones"} @@ -1403,8 +1404,8 @@ {"name": "Jacinto López Moreno", "given": "Jacinto", "middle": "López", "family": "Moreno", "initials": "J. L. M."} {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} -{"name": "Jack (M.A)", "given": "Jack", "suffix": "M.A", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} -{"name": "Jack (MA)", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "Jack (M.A)", "family": "Jack", "suffix": "M.A", "initials": "J."} +{"name": "Jack (MA)", "family": "Jack", "suffix": "MA", "initials": "J."} {"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} {"name": "Jack I.-V.", "given": "Jack", "family": "I.-V.", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} @@ -1666,7 +1667,7 @@ {"name": "John Smith (MA) Dr.", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "MA", "initials": "J. S."} {"name": "John Smith (MA) Ma", "given": "John", "middle": "Smith", "family": "Ma", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NAME", "SUFFIX_OR_NICKNAME"], "initials": "J. S. M.", "capitalized_force": "John \"Ma\" Smith Ma"} {"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} -{"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA née Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA Née Jones"} +{"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA", "maiden": "Jones", "initials": "J. S."} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} {"name": "John Smith (MA.)", "given": "John", "family": "Smith", "suffix": "MA.", "initials": "J. S."} {"name": "John Smith (MBA PhD.)", "given": "John", "family": "Smith", "suffix": "MBA PhD.", "initials": "J. S."} @@ -3206,6 +3207,7 @@ {"name": "Smith, John X", "given": "John", "middle": "X", "family": "Smith", "initials": "J. X. S."} {"name": "Smith, John e, III, Jr", "given": "John", "middle": "e", "family": "Smith", "suffix": "III, Jr", "initials": "J. e. S."} {"name": "Smith, John van der Berg Prof.", "title": "Prof.", "given": "John", "middle": "van der Berg", "family": "Smith", "initials": "J. B. S."} +{"name": "Smith, John, Bob (MA)", "given": "John", "family": "Smith", "suffix": "Bob MA", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S."} {"name": "Smith, John, Extra, Jr.", "given": "John", "family": "Smith", "suffix": "Extra, Jr.", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S."} {"name": "Smith, John, MD - née Jones Smith", "given": "John", "family": "Smith", "suffix": "MD - née Jones Smith", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "John Smith MD - Née Jones Smith"} {"name": "Smith, John, PhD - and MD", "given": "John", "family": "Smith", "suffix": "PhD - and MD", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S."} @@ -3213,6 +3215,7 @@ {"name": "Smith, John, PhD née Jones", "given": "John", "family": "Smith", "suffix": "PhD née Jones", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "John Smith PhD Née Jones"} {"name": "Smith, John, Prof.", "title": "Prof.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, John, Puig - i Soler", "given": "John", "family": "Smith", "suffix": "Puig - i Soler", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "John Smith Puig - I Soler"} +{"name": "Smith, John, X.Y.Z. (MA)", "given": "John", "family": "Smith", "suffix": "X.Y.Z. MA", "initials": "J. S."} {"name": "Smith, John, and", "given": "John", "family": "Smith", "suffix": "and", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S."} {"name": "Smith, John; MD", "given": "John;", "family": "Smith", "suffix": "MD", "initials": "J. S."} {"name": "Smith, Jr.", "family": "Smith", "suffix": "Jr.", "initials": "S."} From cd1024cc37d703052da0bfb415ed728228f283c9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 16:12:35 -0700 Subject: [PATCH 09/10] Settle no clause in a name of no word of its own (#810 review) The review of the one-pass commit found every clause in a name with no name word counting as "after the name" (Dr. (MA), (M.A) alone), which S13's When rules out: a clause is settled only behind a name word now. Adds rows for that and for a leading period, a Dr. head to the twin census (re-recorded, 396 of 1,080), an S13 Except example for a stray full stop after the clause, and corrects the "6 real names" count to 3 (ctgov's) and the re-measure's tree. Co-Authored-By: Claude Opus 5.5 --- ...bracketed-member-settled-by-its-writing.md | 6 +++--- docs/design/mechanisms.md | 2 +- ...bracketed-member-settled-by-its-writing.md | 5 ++++- nameparser/_pipeline/_post_rules.py | 5 ++++- tests/v2/cases.py | 11 ++++++++++ tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 7 ++++--- tests/v2/test_parser.py | 20 ++++++++++--------- tools/differential/corpus_rules.jsonl | 3 +++ tools/differential/snapshot.jsonl | 3 +++ 10 files changed, 45 insertions(+), 19 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 32054418d..1acf82208 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -4,7 +4,7 @@ - A member of the ambiguous class alone in brackets or quotes after the name, its writing settling it as the credential, is the suffix, read in one pass, and reports nothing (`John Smith (MA)`, `John Smith (M.A)`, `Smith, John (MA)` → suffix); any other member stays the nickname S1 reads and reports `suffix-or-nickname`, a dotted member inside the name included (`John Smith (Ed)`, `John Smith (Ma)`, `JOHN SMITH (MA)`, `Jean (M.A) Smith`) — 2026-10-26 #810. - The writing that settles it is read by the rules that read writing: a period inside the word, S3's evidence, whatever case the name is in, and capitals in a name of mixed case, S6's, the case being that of the name's own words and never the clause's, so `john smith (MA)` keeps the nickname where bare `john smith MA` reads suffix. S13's own signal is the position alone — 2026-10-26 #810. -- Only a clause after the name is settled: one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. +- Only a clause after the name is settled, behind its last name word: one in a name with no word of its own keeps the nickname (`Dr. (MA)`, `(M.A)`, where the review of the one-pass commit found both settled), and one with a name word after it keeps the nickname however it is written (`Jeffrey (JD) Bricken`, `Jean 'JD' Smith`, `Smith (MA), John`), while a suffix, a title or a maiden clause after it does not count, however it is written (`John Smith (MA), PhD`, `John Smith (MA) Ph. D.`, `John Smith (MA) Dr.`), asked of the roles the parse read rather than of the following words' tags — 2026-10-26 #810. - Where the rest of the name then reads otherwise than beside a bracketed unambiguous credential in the same place, the member's twin, the difference is an accepted cost, recorded in rules/S13's Accepted bullets and counted by tests/v2/test_parser.py's census (`Jack (MA)` → family 'Jack', where `Jack (PhD)` reads given 'Jack') — 2026-10-26 #810. ## Log @@ -13,12 +13,12 @@ - **Why the writing, and why after the name.** After the name, brackets hold either a credential or the name the person goes by, and the count of words cannot tell the two apart there, the name being whole without the clause. The writing can: credentials are written in capitals and with their periods, names are not, which is S6's and S3's ground. Inside the name the same member let out of the brackets would read as a middle name, where the brackets more likely hold the name the person goes by: a spike in the design round (2026-10-23, against 638685b4; a dated snapshot, the spike's switch being in no tree) that escaped every member written in capitals or with an interior period moved `JD` from the nickname to the middle name in two committed rows, `Jean 'JD' Smith` and `JEFFREY (JD) BRICKEN`. So a clause with a name word after it is not settled. - **Why the name's case, not the clause's.** The same spike asked the bracketed word's own case and moved 6 FEC names, every one wrong: the corpus is written wholly in capitals, so capitals there settle nothing, and `(ED)` in such a name is the nickname. Asking the case of the name's own words, the fact S6 records for every other site (`settled_by_case`), moves none of them. The clause's own words are no name of their own, so `john smith (MA)` asks the case of `john smith` and keeps the nickname, where bare `john smith MA` reads suffix; the two differ by construction, the bare word being one of the name's own words. - **Why the suffix, not the bare reading.** S1's escape lets suffix-shaped content read as if written bare. A settled member cannot, because since #872 a word of two dotted single letters in a script with capitals reads bare as paired initials (decisions/S3), so `John Smith (M.A)` let out of its brackets would read family 'M.A'. The clause is read as the suffix directly, the brackets having already set it apart from the name's words. The decision is made first in post_rules, where every word's role is read, and the `suffix-or-nickname` report moved there with it, from classify, so that it is raised where the fork is decided. Frames counted with `sys.setprofile`, py3.11, 2026-10-26, against master: `Smith, John` 158 and `John Smith` 149 unchanged; `John Smith (MA)` 214 → 210, `Jack (MA)` 174 → 171; a name holding a clause it does not settle pays 2 to 5 (`John (Bob) Smith` 204 → 206, `John Smith (Ed)` 214 → 219). - - **Declined: the twin reading, folding the pipeline again (Derek, 2026-10-26, "the one pass version").** A draft of this PR read a settled member exactly as its twin, a bracketed unambiguous credential in the same place, reads, fields and reports alike: post_rules recorded the settled clauses and the stages folded again from the start, extract escaping each clause and classify giving its word the credential's tag, until no new clause settled (a settled word joining the name's own words can make a one-case name mixed and settle a second clause). Its review rounds found three more places to bring in line with the twin, the part past a second comma among them. It cost a whole parse per settled name (`John Smith (MA)` 214 → 416 frames) and machinery in four stages for a shape that moves 6 real names in the corpora measured here. In one pass the rest of the name is read as beside a nickname, and where that differs from the twin the difference is an accepted cost, recorded in rules/S13's Accepted bullets: a lone name word stays N3's family name (`Jack (MA)`; `Jack (PhD)` reads given 'Jack', reported); a maiden clause after the member stays the maiden name, where the twin's credential run takes it (`John Smith (MA) née Jones`); the case is asked once, so a capitals member settled only once another settled word made the name mixed stays the nickname (`JOHN SMITH (m.a) (MA)`, where the twin `JOHN SMITH (mba) (MA)` reads suffix 'mba MA'); and a part past a second comma is weighed without the member (`Smith, John, X.Y.Z. (MA)` reports nothing where `(MBA)` draws `comma-structure`). tests/v2/test_parser.py's `test_where_a_bracketed_member_reads_otherwise_than_its_twin` records the census of such differences over a grid of comma shapes, heads and the three orders (336 of 1,044 readings on 2026-10-26, by head), so a change that moves the count shows up; the folded draft differed from the twin in none of the settled rows. + - **Declined: the twin reading, folding the pipeline again (Derek, 2026-10-26, "the one pass version").** A draft of this PR read a settled member exactly as its twin, a bracketed unambiguous credential in the same place, reads, fields and reports alike: post_rules recorded the settled clauses and the stages folded again from the start, extract escaping each clause and classify giving its word the credential's tag, until no new clause settled (a settled word joining the name's own words can make a one-case name mixed and settle a second clause). Its review rounds found three more places to bring in line with the twin, the part past a second comma among them. It cost a whole parse per settled name (`John Smith (MA)` 214 → 416 frames) and machinery in four stages for a shape that moves 3 real names in the corpora measured here (ctgov-officials'; 35 names hold it, ctgov's 29 and FEC's 6). In one pass the rest of the name is read as beside a nickname, and where that differs from the twin the difference is an accepted cost, recorded in rules/S13's Accepted bullets: a lone name word stays N3's family name (`Jack (MA)`; `Jack (PhD)` reads given 'Jack', reported); a maiden clause after the member stays the maiden name, where the twin's credential run takes it (`John Smith (MA) née Jones`); the case is asked once, so a capitals member settled only once another settled word made the name mixed stays the nickname (`JOHN SMITH (m.a) (MA)`, where the twin `JOHN SMITH (mba) (MA)` reads suffix 'mba MA'); and a part past a second comma is weighed without the member (`Smith, John, X.Y.Z. (MA)` reports nothing where `(MBA)` draws `comma-structure`). tests/v2/test_parser.py's `test_where_a_bracketed_member_reads_otherwise_than_its_twin` records the census of such differences over a grid of comma shapes, heads and the three orders (396 of 1,080 readings on 2026-10-26, by head), so a change that moves the count shows up; the folded draft differed from the twin in none of the settled rows. - **Why post_rules, found by the code review.** The first draft decided in group and called a word after the clause a name word unless it carried the suffix tag. A suffix the tag does not mark then kept the clause a nickname while its listed twin let it through: `John Smith (MA) VI` and `(MA) Ph. D.` against `(MA) III` and `(MA) PhD`, the same for a member read by its capitals (`(MA) DO`), an unlisted dotted credential (`X.Y.Z.`) and `fils`, and a title after the clause (`John Smith (MA) Dr.`, `Doe, Jane (MA) Mrs`). Asking the roles the parse has read answers for every spelling at once. The same draft scanned the whole name once per clause, quadratic in the number of bracketed members (`"(M.A) " * n` 13.7× for 4× the input, against 3.3× after the fix, py3.11, 2026-10-26); the pass now maps clauses to tokens once, and test_benchmark.py's `bracket_members` row guards it, failing on the first draft twice in two runs. - **Recorded as a rule of its own, reading one signal (Derek, 2026-10-26).** The position it reads (after the name) is evidence S1's shape signal does not read (docs/design/AGENTS.md, axis 12), so it is not an S1 clause. What settles the member is the writing, and the writing is other rules' evidence, so S13 cites them rather than reading it itself: the period inside the word is S3's, capitals in a mixed-case name S6's, and the escape from the nickname S1's; its Signal line is the position, a hub of S1, S3 and S6. An earlier draft named the period as a second signal of S13's own, a split candidate at birth. S3 reads two dotted letters bare as initials, and S13's Precedence bullet states that the brackets after the name outrank that reading. The dotted arm has no real-name population in any corpus measured here (none of ctgov-officials' 29 bracketed members, nor FEC's 6, is dotted), so it rests on Derek's decision of 2026-10-23 and on 1.4.0's reading of `John Smith (M.A)` as suffix, not on a measurement. A dotted member left the nickname inside the name is reported, as a tagged one is: the report had keyed on the class tag, which a dotted word in a clause does not carry, while S1's bullet, widened to every member, said it was reported (`Jean (M.A) Smith` reported nothing on master and on the first draft). S1's member bullet keeps the nickname reading for the members this rule does not settle, and its Precedence names this rule. - **Declined: escaping every member.** A spike escaping every bracketed member to its bare reading (2026-10-23, against 638685b4, a dated snapshot like the one above) moved 27 ctgov-officials names and took `Ed` out of the nickname 17 times, `Ed` being the nickname of Edward far more often than a credential written in Title case. - **Left as found by the second review**: one post rule can still put a name word behind a settled clause, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. - **Kept from the twin draft's reviews.** A clause is the member alone only where its whole text is the member, as extract's escape reads a clause, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) leaves it the nickname, as its twin `(PhD 😀)` is no escape (`John Smith (MA 😀)`); a first draft counted tokens and settled it. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), the position being read on the nickname reading. The draft's comma-structure fix is not needed in one pass: the part past a second comma is tested while the clause is still a nickname, so `Smith, John, Jr. (M.A)` reports nothing, as its twin does. - **Accepted:** the twin differences the declined bullet above lists, in rules/S13's Accepted bullets. - - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the final tree after merging master 32049ade, #896, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. + - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the one-pass tree, 523c3696, against master 32049ade, #896, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 1da3635ab..dd2d0e05a 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -196,7 +196,7 @@ This catalog converts discovery into a one-time cost — it does not remove disc **How it works.** "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. -**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), the one site where content overrules the delimiter WITHOUT rejoining the bare path: a member of the ambiguous class settled by its writing after the name (rules/S13) is the suffix directly, because its bare reading would differ (`M.A` reads as initials bare), so it is decided once the parse's roles are read rather than unmasked here (decisions/S13). +**Lives in.** nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch); and nameparser/_pipeline/_post_rules.py (`_settle_delimited_members`), the one site where content overrules the delimiter WITHOUT rejoining the bare path: a member of the ambiguous class settled by its writing after the name (rules/S13) is the suffix directly, because a dotted member's bare reading would differ (`M.A` reads as initials bare), so it is decided once the parse's roles are read rather than unmasked here (decisions/S13). **Reach for it when.** About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). Not when the bare reading is the wrong answer for the clause: that is S13's shape. diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index 71a824bb1..f4455ddfc 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -36,7 +36,7 @@ "john smith (MA)" → nickname="MA" "John Smith (Ma)" → nickname="Ma" ``` -- **A member standing inside the name or before it** stays the nickname, however it is written. +- **A member standing inside the name or before it** stays the nickname, however it is written, and so does one in a name with no name word of its own, or one with a word after it the parse reads as a name word, a stray full stop included. ``` "Jeffrey (JD) Bricken" → nickname="JD" "JEFFREY (JD) BRICKEN" → nickname="JD" @@ -45,6 +45,9 @@ "Jean (M.A) Smith" → ambiguities=("suffix-or-nickname",) "Smith (MA), John" → nickname="MA" "(MA) John Smith" → nickname="MA" + "Dr. (MA)" → nickname="MA" + "(M.A)" → nickname="M.A" + "John Smith (MA)." → nickname="MA" "John Smith (MA) Ma" → nickname="MA" ``` diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 3cd6996de..0f91410d5 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -428,7 +428,10 @@ def _settle_delimited_members(state: ParseState, tokens: list[WorkToken], dotted = ("." in token.text.strip(".") and _normalize(token.text).replace(".", "") in lex.suffix_acronyms_ambiguous) - if i > last_name and i in alone and ( + # after the name: behind its last name word, and a name word + # there is, so a clause in a name of no word of its own + # ('Dr. (MA)') is not settled + if 0 <= last_name < i and i in alone and ( dotted or settled_by_case(token, state.one_case)): tokens[i] = copy_with(token, role=Role.SUFFIX) elif dotted or AMBIGUOUS_ACRONYM_TAG in token.tags: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 26e5b3e77..35e2abccc 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8930,6 +8930,17 @@ def _check_cjk_shape_purity(self) -> None: notes="read on the nickname reading, 'Ma' is a name word after " "the clause, where the twin's credential run would take " "it (rules/S13's Except)"), + Case("bracket_member_with_a_leading_period", "John Smith (.ma)", + {"given": "John", "family": "Smith", "nickname": ".ma"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="a period opening the word is no period inside it"), + Case("bracket_member_in_a_name_of_no_word", "Dr. (MA)", + {"title": "Dr.", "nickname": "MA"}, + classification="fix(#810)", + ambiguities=("suffix-or-nickname",), + notes="no name word, so no clause after the name; the twin " + "'Dr. (MBA)' reads given 'Dr.'"), Case("bracket_member_in_quotes", 'John Smith "MA"', {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index c6a78140f..dd9b246e0 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -65,7 +65,7 @@ "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "S12": {"words": 674, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, - "S13": {"words": 398, "then_words": 180, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 4, "accepted_words": 138}, + "S13": {"words": 431, "then_words": 180, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 4, "accepted_words": 138}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index f0c78e3a7..97a77d08c 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -286,9 +286,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 'Jack (MA)' and 'Jack (M.A)' lose their given-or-family claims and # 'John Smith (MA) née Jones' its two suffix-or-name claims (-12), # and the two clauses a second pass would settle keep the nickname, - # reported (+6). - assert claims == 1384, ( - f"the field sweep checks {claims} claims, recorded as 1384 on " + # reported (+6); 1384 -> 1390 the same day, its review round's rows + # for a leading period and a name of no word (+6). + assert claims == 1390, ( + f"the field sweep checks {claims} claims, recorded as 1390 on " f"2026-10-26") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 5904ec404..d34dc017a 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2191,12 +2191,12 @@ def test_a_declined_cjk_credential_word_does_not_peel_off_a_glued_name( # rules/S13 (#810): where a bracketed member reads differently from its # twin, a bracketed unambiguous credential in the same place. The twin # is MBA, a word no head holds, so the twin's reading maps back by -# replacing its text. 1,044 texts, about 2s. +# replacing its text. 1,080 texts, about 2s. _TWIN_HEADS = ( "John Smith {c}", "Jack {c}", "{c}", "John {c}", "JOHN SMITH {c}", "Smith, John {c}", "Smith, {c}", ", {c}", "(Bob), {c}", "John Smith {c}, PhD", "John Smith, PhD {c}", "John Smith PhD {c}", - "Dr. Jack {c}", "Freiherr von Berg {c}", "Juan de la Vega {c}", + "Dr. Jack {c}", "Dr. {c}", "Freiherr von Berg {c}", "Juan de la Vega {c}", "Jane Doe nee Smith {c}", "John Smith {c} née Jones", "John Smith {c} Dr.", "John Smith {c} VI", "John Smith {c} Ph. D.", "Smith, Jane {c} Mrs", "Ortega y Gasset, José {c}", @@ -2226,12 +2226,15 @@ def back(text: str) -> str: #: family name beside a settled member where its twin's is given; the #: maiden clause stays the maiden name where the twin's credential run #: takes it; a part past a second comma reports what is left without -#: the member; a name with no word of its own, or written in one case, -#: settles no capitals; and a clause another settled one would make -#: settle, on a second reading, is left the nickname. +#: the member; a name written in one case settles no capitals; a name +#: with no word of its own (a title, a nickname, nothing) settles no +#: clause, where the twin is read as the name; and a clause another +#: settled one would make settle, on a second reading, is left the +#: nickname. _TWIN_CENSUS = { - ("(Bob), {c}", "nickname"): 24, - (", {c}", "nickname"): 24, + ("(Bob), {c}", "nickname"): 36, + (", {c}", "nickname"): 36, + ("Dr. {c}", "nickname"): 36, ("JOHN SMITH (m.a) {c}", "nickname"): 24, ("JOHN SMITH {c}", "nickname"): 24, ("Jack {c}", "settled"): 36, @@ -2240,8 +2243,7 @@ def back(text: str) -> str: ("Smith, John, Bob {c}", "settled"): 36, ("Smith, John, X.Y.Z. {c}", "settled"): 36, ("john smith (M.A) {c}", "nickname"): 24, - ("{c}", "nickname"): 24, - ("{c}", "settled"): 12, + ("{c}", "nickname"): 36, } diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 8c200ea90..a3776d878 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -4,6 +4,7 @@ "'Smitty' Jones Jr." "'Smitty' Sir John" "(" +"(M.A)" "(MA) John Smith" ", Ana García López" ", Dean" @@ -146,6 +147,7 @@ "Dr King Jr" "Dr Smith V" "Dr." +"Dr. (MA)" "Dr. Ana García López" "Dr. Freiherr von Berg, Ed" "Dr. John Smith Prof." @@ -376,6 +378,7 @@ "John Smith (MA) Ph. D." "John Smith (MA) née Jones" "John Smith (MA), PhD" +"John Smith (MA)." "John Smith (MA.)" "John Smith (MBA PhD.)" "John Smith (Ma)" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 4c556d005..dd132f3ad 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -7,6 +7,7 @@ {"name": "'Smitty' Sir John", "title": "Sir", "given": "John", "nickname": "Smitty", "initials": "J."} {"name": "(", "ambiguities": ["UNBALANCED_DELIMITER"]} {"name": "()"} +{"name": "(M.A)", "nickname": "M.A", "ambiguities": ["SUFFIX_OR_NICKNAME"]} {"name": "(MA) John Smith", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "김. 민. J. S."} {"name": ","} @@ -759,6 +760,7 @@ {"name": "Dr 김민준씨, Jr.", "title": "Dr", "given": "민준", "family": "김", "suffix": "씨, Jr.", "initials": "민. 김."} {"name": "Dr 김민준씨, V.", "title": "Dr", "given": "민준", "family": "김", "suffix": "씨, V.", "initials": "민. 김."} {"name": "Dr.", "title": "Dr."} +{"name": "Dr. (MA)", "title": "Dr.", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "capitalized_force": "Dr. \"Ma\""} {"name": "Dr. Abdul Salam Hassan, MD", "title": "Dr.", "given": "Abdul Salam", "family": "Hassan", "suffix": "MD", "initials": "A. S. H."} {"name": "Dr. Ana García López", "title": "Dr.", "given": "Ana", "middle": "García", "family": "López", "initials": "A. G. L."} {"name": "Dr. Ardelean Dl Daniel Ionescu Dragoș Ioan Jr.", "order": "FAMILY_FIRST", "title": "Dr.", "given": "Dl", "middle": "Daniel Ionescu Dragoș Ioan", "family": "Ardelean", "suffix": "Jr.", "initials": "D. D. I. D. I. A."} @@ -1669,6 +1671,7 @@ {"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} {"name": "John Smith (MA) née Jones", "given": "John", "family": "Smith", "suffix": "MA", "maiden": "Jones", "initials": "J. S."} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} +{"name": "John Smith (MA).", "given": "John", "middle": "Smith", "family": ".", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S. ..", "capitalized_force": "John \"Ma\" Smith ."} {"name": "John Smith (MA.)", "given": "John", "family": "Smith", "suffix": "MA.", "initials": "J. S."} {"name": "John Smith (MBA PhD.)", "given": "John", "family": "Smith", "suffix": "MBA PhD.", "initials": "J. S."} {"name": "John Smith (Ma)", "given": "John", "family": "Smith", "nickname": "Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S."} From 755133f06cbbef82a4549f3060bc98b9cb6645e7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 16:23:56 -0700 Subject: [PATCH 10/10] Correct the census comment and the re-measure's tree (#810 review) Co-Authored-By: Claude Opus 5.5 --- .../decisions/S13-bracketed-member-settled-by-its-writing.md | 2 +- tests/v2/test_parser.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index 1acf82208..70200e310 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -20,5 +20,5 @@ - **Left as found by the second review**: one post rule can still put a name word behind a settled clause, P6's attachment after a comma with nothing before it, which runs after the pass that settles (`, Jones (MA) vd`); the shape has no name before its comma and nothing in the corpora writes it. - **Kept from the twin draft's reviews.** A clause is the member alone only where its whole text is the member, as extract's escape reads a clause, so a character tokenize drops beside it (an emoji, a zero-width space, a nakaguro) leaves it the nickname, as its twin `(PhD 😀)` is no escape (`John Smith (MA 😀)`); a first draft counted tokens and settled it. A word read as a name after the clause keeps the member a nickname, as it would sit inside the name (`John Smith (MA) Ma`, family 'Ma'), the position being read on the nickname reading. The draft's comma-structure fix is not needed in one pass: the part past a second comma is tested while the clause is still a nickname, so `Smith, John, Jr. (M.A)` reports nothing, as its twin does. - **Accepted:** the twin differences the declined bullet above lists, in rules/S13's Accepted bullets. - - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the one-pass tree, 523c3696, against master 32049ade, #896, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. + - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the one-pass tree with a name word required before the clause, 482691a6, against master 13539131, #897, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index d34dc017a..0a3f4537e 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2228,7 +2228,9 @@ def back(text: str) -> str: #: takes it; a part past a second comma reports what is left without #: the member; a name written in one case settles no capitals; a name #: with no word of its own (a title, a nickname, nothing) settles no -#: clause, where the twin is read as the name; and a clause another +#: clause, where the twin escapes, read as the name ('(MBA)' given +#: 'MBA', 'Dr. (MBA)' given 'Dr.') or as the suffix behind a comma +#: (', (MBA)'); and a clause another #: settled one would make settle, on a second reading, is left the #: nickname. _TWIN_CENSUS = {