From 265634f561d29d9889a92ef81f02fe387cc26d47 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 18:07:55 -0700 Subject: [PATCH 1/7] Read a title that is also a given name as a title only inside a title phrase (#752, phase 2i) rules/H9, the title phrase. A title with a given reading (dean, justice, major) compiles into TITLES_AMBIGUOUS instead of TITLES and is a title only beside a partner: a modifier or style before it, a title that attaches to it, or `of` after it, never a salutation (#853's rule). Outside a phrase it reads as an unlisted word, reported title-or-name where it opens the name. A modifier in the leading run of titles is a title only with a head title after it, `of` after it, or in the object of `of`/`the` there (decision 10, at the head only). `attaches` gains its meaning (TITLE_PHRASES); new Lexicon fields titles_ambiguous, title_modifiers, title_styles, title_phrases. resident/rt/pro, d'affaires and member reclassified; thirteen salutations' given counts move into notes. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 10 +- docs/concepts.rst | 5 +- docs/customize.rst | 41 +- docs/design/decisions/H9-title-phrase.md | 23 + docs/design/decisions/README.md | 1 + docs/design/decisions/salutation-titles.md | 4 +- docs/design/decisions/vocabulary-model.md | 14 +- docs/design/rules/H-titles-and-honorifics.md | 2 +- .../rules/H6-title-before-a-family-comma.md | 5 + docs/design/rules/H9-title-phrase.md | 88 ++++ docs/design/rules/README.md | 1 + docs/design/vocabulary-model-rfc.md | 4 +- docs/migrate.rst | 2 +- docs/release_log.rst | 4 +- nameparser/_config_shim.py | 29 +- nameparser/_lexicon.py | 53 ++- nameparser/_pipeline/_assign.py | 23 +- nameparser/_pipeline/_classify.py | 51 +- nameparser/_pipeline/_state.py | 6 + nameparser/_pipeline/_vocab.py | 151 +++++- nameparser/config/_compiled.py | 446 ++++++++++++++++++ nameparser/config/titles.py | 38 +- nameparser/vocab/README.md | 13 +- nameparser/vocab/bn.toml | 3 +- nameparser/vocab/de.toml | 3 +- nameparser/vocab/en.toml | 72 +-- nameparser/vocab/es.toml | 9 +- nameparser/vocab/fr.toml | 8 +- nameparser/vocab/hi.toml | 3 +- nameparser/vocab/it.toml | 3 +- nameparser/vocab/sr.toml | 3 +- nameparser/vocab/sw.toml | 3 +- tests/v2/cases.py | 110 +++++ tests/v2/pipeline/test_vocab.py | 49 ++ tests/v2/rules_budget.json | 3 +- tests/v2/test_cases.py | 8 +- tests/v2/test_config_shim.py | 14 + tests/v2/test_lexicon.py | 12 +- tests/v2/test_vocab.py | 93 +++- tools/differential/conventions_agreement.json | 11 - tools/differential/corpus_rules.jsonl | 33 ++ tools/differential/snapshot.jsonl | 55 ++- tools/vocab/build.py | 71 ++- 43 files changed, 1424 insertions(+), 156 deletions(-) create mode 100644 docs/design/decisions/H9-title-phrase.md create mode 100644 docs/design/rules/H9-title-phrase.md diff --git a/AGENTS.md b/AGENTS.md index 54cbb98c..0962b16e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,7 +196,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2 and re-measured 2026-10-25 with the key below, the earlier `dphil` example having become a shipped mask in #797: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. -- `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute) +- `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute); `TITLES_AMBIGUOUS` (a title reading beside a `given` one, derived, compiled INSTEAD of into `TITLES`: a title only inside a title phrase, rules/H9, #752 phase 2i), `TITLE_MODIFIERS` and `TITLE_STYLES` (by kind) and `TITLE_PHRASES` (the two-word phrases `attaches` makes), none with a v1 attribute - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a word reading on a word with a name reading, `fils` and the agnomens, gated by the count, rules/S11, since 2026-10-25) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) @@ -244,7 +244,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). -- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), `suffix_words_ambiguous = SUFFIX_WORDS_AMBIGUOUS ∩ suffix_words` with `agnomens` inside it (#752, likewise: deleting the suffix word turns S11's gate off, `test_snapshot_removing_a_suffix_word_turns_its_gate_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. +- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), `suffix_words_ambiguous = SUFFIX_WORDS_AMBIGUOUS ∩ suffix_words` with `agnomens` inside it (#752, likewise: deleting the suffix word turns S11's gate off, `test_snapshot_removing_a_suffix_word_turns_its_gate_off`), `title_modifiers`, `title_styles` and `title_phrases` intersected with the titles (#752 phase 2i, rules/H9, likewise), `titles_ambiguous` passed whole, its words being in no v1 set, except where the caller emptied `titles`, v1's way of turning title detection off, which drops it too (`test_snapshot_cleared_titles_turn_the_title_phrase_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. - **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard. @@ -270,11 +270,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential or particle reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, and a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential. **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. @@ -294,7 +294,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **`HumanName.C` is a property backed by `_C`, but pickles under the public key `'C'`** — `__init__`/direct assignment route through the `C` setter, which calls the shared `_validate_constants` staticmethod (also used by `__init__`) so an invalid value raises `TypeError` immediately instead of surfacing later as an unrelated `AttributeError` deep in parsing (#239). `__getstate__`/`__setstate__` deliberately translate `self._C` ↔ a `'C'` key in the pickled dict (with the usual `CONSTANTS`-singleton-becomes-`None` sentinel) rather than pickling `_C` directly, so the on-disk pickle format hasn't changed across this fix — don't "simplify" that translation away or old pickles/tests that hand-build a state dict with a `'C'` key will break. -**Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is intentionally absent from the default titles (see `docs/customize.rst` — users who need it add it via opt-in `Constants`). Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. +**Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it via opt-in `Constants`. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. **The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") diff --git a/docs/concepts.rst b/docs/concepts.rst index 01ca5629..f31fd576 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -136,8 +136,9 @@ this setting vary with? that varies by **output destination**: the ``spec`` you pass to ``render(spec)``, or a keyword to ``initials()``/``capitalized()``. -"Dean" is a common given name, so it is not in the default titles -vocabulary. But in some data it is more common as a title. The right +"Dean" is a common given name, so the default reads it as a title only +inside a title phrase ("Dean of Students"), and "Dean Martin" keeps +given "Dean". But in some data it is more common as a title. The right reading is a fact about the domain the names come from: that makes it a :class:`~nameparser.Lexicon` entry. A CRM that wraps nicknames in square brackets — ``John [Johnny] Smith`` — instead of quotes is a diff --git a/docs/customize.rst b/docs/customize.rst index ca0c92e9..5747ca56 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -102,8 +102,9 @@ Removing works the same way, and drops the word from recognition: >>> Parser(lexicon=lean).parse("Professor Robert Johns").title '' -A few fields mark a subset of another — ``given_name_titles`` and -``after_given_titles`` over ``titles``, ``particles_ambiguous`` over ``particles``, +A few fields mark a subset of another — ``given_name_titles``, +``after_given_titles``, ``title_modifiers`` and ``title_styles`` over +``titles``, ``particles_ambiguous`` over ``particles``, ``suffix_acronyms_ambiguous`` over ``suffix_acronyms``, ``conjunctions_ambiguous`` over ``conjunctions``, ``honorific_tails`` and ``suffix_words_ambiguous`` over @@ -113,9 +114,10 @@ the marker first. Four of them enforce that — ``particles_ambiguous``, ``suffix_acronyms_ambiguous``, ``honorific_tails`` and ``suffix_words_ambiguous`` raise ``ValueError`` naming the orphans, because an orphan in each of those does real harm rather than nothing. -The other four are deliberately unchecked because an orphan there is +The others are deliberately unchecked because an orphan there is inert, an ``agnomens`` entry being read only for a word -``suffix_words_ambiguous`` marks: +``suffix_words_ambiguous`` marks, and a ``title_modifiers`` or +``title_styles`` entry only for a title: ``given_name_titles`` matches a title run as one space-joined string, or by that run's last word, so a legitimate entry like ``"sir and dame"`` is no single word in ``titles``; and a ``conjunctions_ambiguous`` or @@ -123,6 +125,19 @@ is no single word in ``titles``; and a ``conjunctions_ambiguous`` or conjunction or a title, so ``remove(conjunctions={"e"})`` simply works and the stale marker entry is never consulted. +``titles_ambiguous`` is the one title field that is no subset of +``titles``: a word in it is a title only inside a title phrase, and a +word in both is a title wherever it stands, which is what listing it in +``titles`` asks. ``title_phrases`` holds the two-word phrases a +``titles_ambiguous`` word may stand in (``"chief justice"``, +``"major general"``), each read only where both words are titles: + +.. doctest:: + + >>> chief = Lexicon.default().remove(title_phrases={"chief justice"}) + >>> Parser(lexicon=chief).parse("Chief Justice John Roberts").title + 'Chief' + .. _title-detection-off: Turning title detection off @@ -130,13 +145,17 @@ Turning title detection off The subset rule matters most when clearing a field wholesale. Emptying ``titles`` alone orphans every ``given_name_titles`` entry, so the two -go together: +go together, with ``titles_ambiguous``, whose words are titles inside a +phrase ("Dean of Students") without being in ``titles``: .. doctest:: >>> d = Lexicon.default() >>> lean = d.remove(titles=set(d.titles), - ... given_name_titles=set(d.given_name_titles)) + ... given_name_titles=set(d.given_name_titles), + ... titles_ambiguous=set(d.titles_ambiguous)) + >>> Parser(lexicon=lean).parse("Dean of Students Jane Smith").title + '' >>> Parser(lexicon=lean).parse("Hon Solo").given 'Hon' @@ -405,10 +424,12 @@ free: needs one adds it back with ``Lexicon.default().add(suffix_acronyms={"cha"})``. -The same conservatism is why ``dean`` above isn't in the default -vocabulary in the first place: "Dean" is also a common given name, and -a default that swallowed it as a title would misparse "Dean Martin" for -everyone. +The same conservatism is why ``dean`` above is a title by default only +inside a title phrase ("Dean of Students", "Associate Dean"): "Dean" is +also a common given name, and a default that swallowed it as a title +would misparse "Dean Martin" for everyone. Listing it in ``titles`` +makes it a title wherever it stands. The phrase-only titles are +``Lexicon.titles_ambiguous``. Particles that are also given names ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md new file mode 100644 index 00000000..cb9688e6 --- /dev/null +++ b/docs/design/decisions/H9-title-phrase.md @@ -0,0 +1,23 @@ +# H9 — the title phrase + +## Standing + +- A title that may be a given name is a title only inside a title phrase, and elsewhere reads as a word the vocabulary does not list, reported `title-or-name` where it is the first name word after any titles: `Dean Smith` reads given 'Dean', `Dean of Students Jane Smith` title 'Dean of Students' — 2026-10-25 #752. +- Its partner is a modifier or a style before it, a title before it that attaches to it or a title after it that it attaches to (`attaches`), or `of` after it; a salutation is never one, nor any other title before it (#853's `Mr Dean Smith`), nor a title after it it does not attach to (`SMITH, DEAN MR.`) — 2026-10-25 #752. +- A modifier in the leading run of titles before the first comma is a title only with a head title after it in that run, `of` after it, or in the object of `of` or `the` there; outside that run (ending the name, H5's, or after a comma) it reads as any title does, the given part after a family comma included (`Lau, Wing` → title 'Wing') — 2026-10-25 #752. +- A title that may be a given name which a caller lists in `titles` is a title wherever it stands, which keeps the 1.4 recipe `c.titles.add("dean")`; a v1 caller who empties `titles` turns the phrase reading off with them, while one who replaces the list keeps it (`c.titles.clear(); c.titles.add('dr')` still reads `Dean of Students` as the title), v1 holding no knob for the set — 2026-10-25 #752. + +## Log + +- **2026-10-25 #752, phase 2i — A TITLE THAT MAY BE A GIVEN NAME IS A TITLE ONLY INSIDE A TITLE PHRASE, AND A MODIFIER AT THE HEAD OF THE NAME ONLY WITH A HEAD BESIDE IT.** Carries out items 10, 11 and 12 of decisions/vocabulary-model's 2026-10-23 #752 entry (decision 10 at the head of the name only; the phrase partner; #853's word after a title), measured here on the rule as built rather than on the spike those items were decided on. + THE WORDS IT READS TODAY. `dean`, `justice` and `major` gain a title reading beside a given one (`justice` until now excluded as a given name, #606, its cost `Chief Justice John Roberts` read as given 'Justice'), so the rule's reach is exactly what master read as name words, now reading the title only in a phrase. Declaring given readings on the titles decisions/vocabulary-model's 2026-10-24 entry ordered (`wing` first) is 2j's, one measured PR per batch. The words leave `titles` for `titles_ambiguous` rather than staying in both, and that is a compat decision, not a convenience: `Constants().titles.add("dean")` is the documented 1.4 recipe for a title the default leaves out, and a word in both sets would be read by the phrase test whatever the caller listed, so `Dean Smith` would lose the title the caller asked for. Measured: with dean in both, `test_the_documented_replacements_for_an_in_place_edit_work` and four shim controls failed, `Dean Smith` reading given 'Dean' with dean added to the titles. + THE PARTNER (item 11). A modifier or a style before the word, `of` after it, a title before it that attaches to it, or a title after it that it attaches to: `attaches`, a title property naming title kinds, compiles into two-word phrases (`chief` attaching to office and military words, `major` to military ones; count them with `uv run python -c "from nameparser import Lexicon; print(len(Lexicon.default().title_phrases))"`). A connective other than `of` is no partner: `Dean and Mary Smith` is two given names, as master read it. + OUTSIDE A PHRASE THE WORD IS UNLISTED, EVERYWHERE. Classify tags it no title, and the two vocabulary rosters that ask whether a word is listed at all, the capitals credential's "unlisted" (rules/C6) and the comma contrast's "claimed as non-name text", leave the set out. The first draft left them in, and the grid below found `John Smith, DEAN`, read suffix 'DEAN' by C6 on master, reversed to given 'DEAN', family 'John Smith'. Keeping the word unlisted outside its phrase is also what the design's spike measured for the 2j words, which it took out of `titles` altogether. + THE REPORT. `title-or-name` where the word opens the part carrying the given name, read as a name word, at the leading-title peel; not where a phrase settles it, and not inside the name. It is the rule's own fork: the reader of `Dean Smith` can hesitate. Most of what this PR moves is the report: 1,737 of FEC's 1,790 moved names (a `FAMILY, DEAN` given part), 125 of ctgov's 203. The report goes the direction the RFC's front-of-the-name section states, where the rule reads the word as a name; item 12 of the 2026-10-23 #752 entry worded it "only where the rule decided against a declared name reading", which read literally is the other direction, where the title won. The RFC's direction is the one taken: the fork a reader hesitates over is `Dean Smith`, not `Dean of Students`. + DECISION 10 (item 10). A modifier in the leading run of titles before the first comma is a title with a head title after it in that run (any title that is not a modifier, or H2's shape) or in the object of `of` there (`President of the United States`, `the Queen's Bench`). The spike read only "a head title after it", and on this tree that rule broke `President of the United States`, `Chargé d'Affaires`, `Member of Parliament` and the committed `Queen's Bench` row, which it read given 'Bench' (the cost the 2026-10-23 entry recorded); the object of `of` fixes the first and the last. The first build also carried the object across any connective, so `Mr. and Mrs. Wing Lau` kept title 'Mr. and Mrs. Wing' (the code review); only `of` and `the` carry it now (`the`, to keep `The Grand Lau` a title as master reads it), and it reads given 'Wing'. A third fix of the first round, dropping `of` after a modifier as a partner once `member` was an office, narrowed shapes master read right (`Mother Superior of the Convent Mary Smith`, `Bearer of the Seal John Smith`, found by the second review of the fix commit); it is restored, each with a case row. It also read the whole part before the comma as the head, so a modifier ENDING the name lost H5's trailing title (`Jane Smith Vice.`, title 'Vice.' on master and 2.6.0; the design-docs review); only the leading run is the head now. Reclassified, each a word that heads or completes its phrase rather than qualifying a head after it: `resident` and `rt` to occupation and `pro` to academic (item 10's own, a part after a comma written as the person's role), `d'affaires` to office (it stands behind its head, `chargé`) and `member` to office (the head noun of `Designated Member` and `Member of Parliament`; `Designated Member` was four PSC records the first build broke). After a comma a modifier is read as listed. + #853'S RULE ON ITS OWN (item 12). Measured as the item asks, on the rule itself: a scratch tree of this PR with a `given` reading on the nine words of the item's measurement (`sheikh`, `shaikh`, `wing`, `prince`, `princess`, `sultan`, `sultana`, `mahdi`, `king`), against master `a46b3a6e`, PSC records holding one of them as the forename box's first word or the title box's last (the spike's `psc_pos` count): after another title, 643 fixed and 0 broken (the spike's 644 and 0); alone at the head, 152 fixed; the title box alone, 24 set aside. So the rule lands without declaring any of those words, and its live reach in this PR is `dean`, `justice` and `major`, which master read as name words already. + THE GIVEN READINGS #835 RECORDED ON SALUTATIONS. Thirteen salutations carried a `given` reading as evidence for or against H8's `after_given` (`mr`, `mister`, `herr`, `master`, `señora` and eight more), which under derivation would have made `Mr Smith` read given 'Mr'. Their counts move into the salutation reading's `note`, as `ld`'s did (decisions/vocabulary-model, 2026-10-23 item 2): the README's convention for evidence against marking. No reading moves. A title that is also a particle (`st`, `freiherr`) keeps its given reading, which derives the particle's marker, and is left out of the set: the particle rules read it at the front (rules/P4). Recompute: `uv run python tools/vocab/query.py --word mr`. + THE CORPORA (measured 2026-10-25 against master `a46b3a6e`, py3.11, every local check with `--base`). Committed: 12 entries move (the `Dean …` and `Major …` names gain the report, `Wing Lau` and `Grand Rounds` read the modifier as the given name, the `Dean of Chemistry` rows read the phrase as the title) and 31 names new to the committed corpora join from the rules' examples (`snapshot.py --since a46b3a6e`); CLDR's conventions row `Wing Lau` now agrees on given and title, and no agreement line regresses. PSC (`local_agreement.py psc-individuals`, 1,039,174 records scored, the fetch dated 2026-10-22): 178 fixed, 3 broken, 16 changed, the broken all a role typed in the title box (`British` 2, `Director Uk` 1), the fixed mostly a salutation before `Wing` and `Wing` alone (the first build broke four more, `Designated Member`, which `member`'s reclassification fixed). ctgov-officials 203 moved, 125 of them the report alone; FEC 1,790, 1,737 the report alone (a `FAMILY, DEAN` given part), the other 53 a modifier no longer peeled from the family part by H6 (`FIELD SMITH, JOHN` keeps family 'FIELD SMITH', as 1.4.0 read it) or the FEC placeholder `INFORMATION REQUESTED`; tse-donors 23, 14 the report and 9 a modifier opening a name read as the given name (`ROCK`, `AIR`). Consistency (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE): 35 jrc-names persons and 4 nepal-candidates pairs now alike, none apart, tse unchanged. + THE SWEEPS. A grid of 34,380 parses (twelve fronts, salutations, `Dr.`, a dual second in a title run `Rev. Sr.`, the both-vocabulary heads `Freiherr`, `St` and `St.`, a style, a modifier and `Mr. and Mrs.`, before fifteen heads in their bare, capitals, lower-case and period-marked spellings, eight followers and eleven tails, in the comma shapes with an empty head, a nickname head, a particle surname and a title before the comma, under given-first and family-first orders) moved only the shapes this entry names. Two residuals read and accepted: a phrase extended by a coordinating join behind it (`Chief Dean and Mary Smith` reads title 'Chief Dean and Mary'), and nonsense heads (`Chief Justice General Smith`). + THE REVIEW FIXES (code review and design-docs review of the first build; the corpus figures above were re-taken after both review rounds, on the final tree). A period-joined title in a part holding a contested or modifier word lost its tag (`Lt.Gov. Dean Smith` read given 'Lt.Gov.'): only the two sets' words are read out of the title now. The modifier fixes above. And v1's `c.titles.clear()` left `Dean of Students` a title, its words being in no v1 set: the shim drops `titles_ambiguous` with the titles. + Accepted, in rules/H9: `Mr Justice Smith` (an English High Court judge's style) reads given 'Justice', a salutation being no partner; a modifier whose head is not listed loses the title (`His Grace John Smith` → given 'His', where master read title 'His', given 'Grace', and likewise `Most Worshipful Grand Master` and `Air Cdre`, whose heads are not titles; `Grand Duke of Tuscany`, duke refused by the vocabulary; `Field Marshal Bernard Montgomery` now given 'Field', marshal refused too, where it read title 'Field', given 'Marshal'), and so does one written after its head (`Princess Royal Anne`, `Mother Superior Mary Smith`); the given part after a family comma reads a modifier as listed (`Lau, Wing` → title 'Wing', where `Wing Lau` reads given 'Wing'), the cost decision 10 took by keeping the gate out of comma parts; and a coordinating join behind a phrase extends the title (`Chief Dean and Mary Smith`). diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 917dca85..6bb2d4cf 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -31,6 +31,7 @@ Entry conventions: - [H6 — a title before a family comma](H6-title-before-a-family-comma.md) - [H7 — a joined name unit carrying title vocabulary](H7-joined-name-unit-carrying-title-vocabulary.md) - [H8 — a salutation ending the given part](H8-salutation-ending-the-given-part.md) +- [H9 — the title phrase](H9-title-phrase.md) ## Particles & surname prefixes (P) diff --git a/docs/design/decisions/salutation-titles.md b/docs/design/decisions/salutation-titles.md index 5ce3c47d..5ade3279 100644 --- a/docs/design/decisions/salutation-titles.md +++ b/docs/design/decisions/salutation-titles.md @@ -7,7 +7,7 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr). - A word that addresses or ranks a person and is written in FRONT of the name ships as a title unless it is borne as an ordinary name in the leading position, the position test of vocabulary-collisions' C-i, TITLES having no ambiguous subset to mark a collision in, so the only answers are ship and do not ship — 2026-10-04 #606. - One exception to C-i, for TITLES alone: a word borne as a name only RARELY as the first word of one ships on the precedent of `graf` (`herra`, `batoni`, `hrabia`, `kreivi` and the rest of the graf class, each with its counts on its title reading's `note`), and no word in the particle or suffix sets, and no later title, takes it without its own counts — 2026-10-04 #606. - The cost falls on a name that BEGINS with such a word: `Greve Anna` reads title 'Greve', family 'Anna' and `Herra Wijaya` title 'Herra', family 'Wijaya'; a declared `FAMILY_FIRST` does not help (`Hrabia Anna` reads title 'Hrabia' under it); the family comma keeps the word as the family where no name word stands behind it (`Hrabia, Anna` reads given 'Anna', family 'Hrabia', and so does `Hrabia Jr., Anna`), while a name word behind it makes it the title by H6 (`Hrabia Nowak, Anna` reads title 'Hrabia', family 'Nowak'); a name that ends in the word keeps it (`Gladys Knight`, `Knight, Gladys` read family 'Knight'), unless a period follows it (`Gladys Knight.` reads title 'Knight.', family 'Gladys', by H5, accepted rather than prevented) — 2026-10-04 #606. -- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, `ông` and its pair `bà`, and `marshal` and `justice`, whose accepted costs their entries state: `Field Marshal Bernard Montgomery` reads title 'Field', given 'Marshal'), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. +- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, `ông` and its pair `bà`, and `marshal`, whose accepted cost its entry states: `Field Marshal Bernard Montgomery` reads given 'Field', middle 'Marshal Bernard' since rules/H9 (2026-10-25) read `field`, a modifier, as no title without a head after it, where it read title 'Field', given 'Marshal' before), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. - Accents are not folded, so `paní` and `härra` match only their accented spellings (`Paní Nováková` reads title 'Paní', `Pani Anna` reads given 'Pani'); folding the lexicon would turn both into collisions, and whoever proposes it re-argues the entry — 2026-10-04 #606. - `gravos` was dropped, not excluded, since nothing showed it is the Greek word for Count; `κόμης`, a Greek surname written first in official lists, is neither added nor excluded, and adding it is a decision still to make — 2026-10-04 #606. - `knt` (Knight, beside `kt`) is a suffix acronym with a `Knt` case mask: `Mary Knt` reads suffix 'Knt' — 2026-10-04 #606. @@ -15,6 +15,8 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr). - `owura` (Akan) and `bapak` (Indonesian) are titles, on counts under 100 US Census bearers each and none found on Wikidata, and `malam` (Hausa) and `bay` (Turkish) are excluded, each an `excluded` entry with its counts — 2026-10-17 #662. - The Italian abbreviated salutations and letter-heading honorifics whose letters after the full stop are the word's own ending ship as titles with `cased` masks: `sig.ra`, `sig.na`, `dott.ssa`, `prof.ssa`, `gent.mo`, `gent.ma`, `gent.le`, `ill.mo`, `ill.ma`, `ch.mo`, `ch.ma`, `preg.mo`, `preg.ma` (`Dott.ssa Anna Bianchi` reads title 'Dott.ssa', given 'Anna'; `SIG.RA MARIA ROSSI` repairs to 'Sig.ra Maria Rossi'); the bare `sig` and `ing` are excluded as given names, their period-marked spellings already titles by H2's shape, except `Ing.` after a full name and a comma, which is the ambiguous credential decisions/suffix-acronym-collisions lists (#848) — 2026-10-22 #840. +Elsewhere: `justice`, excluded here as a common leading-position name, is a title only inside a title phrase since 2026-10-25 (`Chief Justice John Roberts` reads title 'Chief Justice', `Justice Smith` given 'Justice') — decisions/H9's 2026-10-25 #752, phase 2i entry. + ## Log - 2026-10-04 #606 (its family-comma remedy superseded in part by #701, decisions/H6) — SALUTATIONS, OFFICES AND RANKS FOR THE LANGUAGES TITLES HAD NONE FOR: the undated bullets below, through the measurement. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 3ae24e43..b539fa6e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -11,14 +11,14 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749. - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. -- Title-phrase modifiers are to count as titles only next to a head title, at the head of the name only and never in a comma part; until phase 2 implements it the `modifier` kind is read as every title is (`Vice John Smith` → title 'Vice', while `John Smith Vice` keeps family 'Vice') — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752. +- Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. - Bearer counts are structured provenance (source and count), never used to compute a reading — 2026-10-15/16. - Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading not taken is to surface as `Ambiguity.alternative` in 2.x and the reading taken as `Token.reading` in 3.0 — 2026-10-23 #752. - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. -- `Dean Smith` reads given `Dean`. The contested-head gate is to cover the words outside `titles` (`dean`, `justice`, `major`), and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752. +- `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. @@ -27,7 +27,8 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. -- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `attaches` is phase 2's and `bare_only`, `leading_only`, `trailing_only` and `separators` phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. +- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `bare_only`, `leading_only`, `trailing_only` and `separators` are phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. `attaches` names the title kinds a title reading stands before in a title phrase, compiled into the phrases it makes — 2026-10-25 #752, phase 2i. +- A title reading beside a `given` reading, on a word with no particle reading, derives `TITLES_AMBIGUOUS` instead of `TITLES`: a title only inside a title phrase (rules/H9); a salutation's given-name counts kept as evidence for H8 sit in its `note`, marking nothing — 2026-10-25 #752, phase 2i. - Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762. - The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749. - The titles take ten kinds beside the generic one, `academic` and `occupation` divided by function rather than subject, and the title keep-outs are exclusions — 2026-10-19 #750, title kinds; 2026-10-19 #750, the rest of the titles. @@ -205,3 +206,10 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - `default = false` is a behavioral property of every category but `name`: the reading is held to the per-entry checks and compiles into no set and no marker, and checks 4, 5, 6 and 9 do not weigh it, so it may stand beside a live reading of the word in another file and beside another language's exclusion of it. `default = true` is refused, being every reading's unwritten value. The first three are the readings with no cue: Spanish `h` (*hijo*) and the German regnal ordinal on `i`, each written with the period that makes a one-letter word an initial (rules/T5), and Czech `st` (*starší*), whose `cs.toml` exclusion it replaces. No reading moves. - The other reserved names keep their reservation and name the phase that needs them: `attaches` phase 2 (its front-of-the-name work), `prefer` and `strict` phase 3, and `bare_only`, `leading_only`, `trailing_only` and `separators` phase 4, the gate's slots carrying position and shape settling a bare letter (rules/P7). Adding the two fields changes `Lexicon`'s pickle layout, as 2.6 did. +- **2026-10-25 #752, phase 2i — A TITLE THAT MAY BE A GIVEN NAME IS DERIVED, AND `attaches` HAS ITS MEANING.** Carries out items 10 to 12 and the `attaches` half of item 15 of the 2026-10-23 #752 entry; the rule and its measurement are decisions/H9's entry of this date (rules/H9). What changes in the vocabulary: + - A title reading beside a `given` reading derives a fourth marker, `TITLES_AMBIGUOUS` (`Lexicon.titles_ambiguous`), and the word compiles into it INSTEAD of `TITLES`, so a caller who lists it as a title has it read as one everywhere, the 1.4 recipe (`c.titles.add("dean")`). A title that is also a particle keeps its given reading for the particle's marker and stays in `TITLES` (`st`, `freiherr`). A `family` reading alone beside a title marks nothing, as before. + - Thirteen salutations carried a `given` reading recorded on 2026-10-24 as H8's evidence (#835), which the derivation would have read as the clash (`Mr Smith` → given 'Mr'): `bwana`, `goodman`, `gospodin`, `herr`, `master`, `mister`, `mr`, `señora`, `señorita`, `signora`, `sra`, `श्री` and `মোঃ`. Their counts move into the salutation reading's `note`, the README's convention for counts kept as evidence against marking (item 2's `ld`). No compiled set moves for them. Recompute: `uv run python tools/vocab/query.py --kind salutation`. + - `attaches`, a behavioral title property, is a non-empty list of title kinds, held to one value per word across files (check 5), and compiles into `TITLE_PHRASES` (`Lexicon.title_phrases`): the two-word phrases the word makes with every title word of those kinds. `chief` attaches to office and military, `major` to military. Two more sets compile from the kinds: `TITLE_MODIFIERS`, the words every title reading of which is a modifier, and `TITLE_STYLES`. + - Reclassified, as item 10 asked and as the build found: `resident` and `rt` to occupation, `pro` to academic, `d'affaires` and `member` to office (decisions/H9). + - `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest. + Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. diff --git a/docs/design/rules/H-titles-and-honorifics.md b/docs/design/rules/H-titles-and-honorifics.md index a214ef9c..45099f4b 100644 --- a/docs/design/rules/H-titles-and-honorifics.md +++ b/docs/design/rules/H-titles-and-honorifics.md @@ -1,5 +1,5 @@ ## Titles & honorifics (H) -Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08); the one exception is a listed salutation ending the given part after a family comma, the comma having marked the surname's place (H8). An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break +Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08); the one exception is a listed salutation ending the given part after a family comma, the comma having marked the surname's place (H8). Some title words need a partner to be titles at all: a title that is also borne as a given name, and a word that only qualifies a head title, are titles only inside a title phrase (H9), so "Dean Smith" and "Wing Lau" read given names where "Dean of Students" and "Wing Commander" read titles. An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break "Rabbi Cohen". diff --git a/docs/design/rules/H6-title-before-a-family-comma.md b/docs/design/rules/H6-title-before-a-family-comma.md index b19a0d96..3d0c4ba2 100644 --- a/docs/design/rules/H6-title-before-a-family-comma.md +++ b/docs/design/rules/H6-title-before-a-family-comma.md @@ -30,6 +30,11 @@ "Freiherr von Berg, Ed" → family="Freiherr von Berg" "Dr. Freiherr von Berg, Ed" → title="Dr." ``` +- **The run stops at a word H9 reads out of the title**: a modifier with no head title behind it, and a title that may be a given name outside a title phrase, open the surname. + ``` + "Wing Smith, John" → family="Wing Smith" · boundary + "Dean Smith, John" → family="Dean Smith" + ``` - **The run stops where nothing but post-nominals stands behind it** (H3). ``` "RA PhD Jr., John" → family="RA" · boundary diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md new file mode 100644 index 00000000..f3eaf8f1 --- /dev/null +++ b/docs/design/rules/H9-title-phrase.md @@ -0,0 +1,88 @@ +# H9 — the title phrase + +**When:** a title word that is also borne as a given name, or a word that only qualifies a head title, stands where a title would be read, as in "Dean Smith" or "Wing Lau". + +**Then:** the word is a title only inside a title phrase, and elsewhere a name word. + +- **A title that may be a given name** has its phrase partner in a modifier or a style before it, in a title before it that attaches to it, in a title after it that it attaches to, or in `of` after it. No other title before it is one, a salutation least of all, and no other title after it: a donor list writes the salutation after the given name. + ``` + "Dean of Students Jane Smith" → title="Dean of Students" + "Associate Dean Smith" → title="Associate Dean" + "Major General John Smith" → title="Major General" + "Chief Justice John Roberts" → title="Chief Justice" + "The Hon. Justice Mary Smith" → title="The Hon. Justice" + "Mr Dean Smith" → given="Dean" + "SMITH, DEAN MR." → given="DEAN" + "General Major John Smith" → given="Major" + ``` +- **A modifier in the leading run of titles before the first comma** has its phrase in a head title after it in that run, in `of` after it, or in the object of `of` or `the` there. A modifier outside that run, ending the name or after a comma, is read as any title is. + ``` + "Wing Commander Lau" → title="Wing Commander" + "Mr Wing Lau" → given="Wing" + "Mr. and Mrs. Wing Lau" → given="Wing" + "President of the United States Barack Obama" → title="President of the United States" + "Mother Superior of the Convent Mary Smith" → title="Mother Superior of the Convent" + "Jane Smith Vice." → title="Vice." + "John Smith, Vice" → title="Vice" + ``` +- **A title that may be a given name, read as a name word where it is the first name word after any titles,** is reported as `title-or-name`; inside a phrase, or later in the name, it is not. + ``` + "Dean Smith" → ambiguities=("title-or-name",) + "Mr Dean Smith" → ambiguities=("title-or-name",) + "Major General John Smith" → ambiguities=() + "John Dean Smith" → ambiguities=() + ``` + +**Because:** a title addresses the name after it, so the front of a name has no count to read, and what tells the person "Dean Smith" from the office in "Dean of Students" is the phrase the word stands in. A word that only qualifies a head title, as wing qualifies commander, says nothing about the person on its own. After a comma the part is read as written there, as decision 10 measured: a lone title word in a part after the name is the person's role, and reading it as a name reversed the name. + +``` +"Dean Smith" → given="Dean" +"Major Smith" → given="Major" +"Wing Lau" → given="Wing" +"Mr Smith" → title="Mr" · boundary +"Wing Commander Lau" → ambiguities=() · boundary +``` + +**Precedence:** + +- **A period-marked word at the front of the name** (H2): the abbreviation shape makes it a title whatever the vocabulary says, so the phrase test never reaches it and nothing is reported. + ``` + "Dean. Smith" → title="Dean." + "Wing. Lau" → title="Wing." + ``` +- **A title run** (H3, H4): a word this rule reads as a name word is no title word to the run, so the run stops before it and does not count it. + ``` + "Mr Dean Smith" → title="Mr" + "Dean" → given="Dean" + ``` +- **A connective after the word** (P3): only `of` makes a phrase of the join, so a word read as a name word before `and` joins the next word as two names join. + ``` + "Dean and Mary Smith" → given="Dean and Mary" + ``` +- **A title that may be a given name which a caller lists in `titles`** is a title wherever it stands, since that is what listing it asks; a modifier is read by this rule however it is listed, and leaving its word out of the modifiers turns the rule off for it. + +**Accepted:** + +- A judge styled with a salutation reads the salutation as the title and the word as the given name, since a salutation is never a partner. + ``` + "Mr Justice Smith" → given="Justice" + ``` +- A modifier whose head is not a listed title loses the title, as with duke, marshal and grace, which the vocabulary does not list as titles, and so does a modifier written after its head with no `of` behind it. + ``` + "His Grace John Smith" → given="His" + "Grand Duke of Tuscany" → given="Grand" + "Field Marshal Bernard Montgomery" → given="Field" + "Princess Royal Anne" → given="Royal" + ``` +- In the given part after a family comma a modifier is read as listed, so the listing form and the written order of one name read it differently, the cost decision 10 took for the parts where a lone title word is a role. + ``` + "Lau, Wing" → title="Wing" + ``` +- Once a partner has made the word a title, a coordinating join behind it takes the next word into the title too. + ``` + "Chief Dean and Mary Smith" → title="Chief Dean and Mary" + ``` + +--- +**Signal:** neighbor: the titles and connectives beside a listed title word in the leading run of titles +**history:** decisions/H9 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_vocab.py, nameparser/_pipeline/_assign.py diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 23f6d368..02d9c208 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -123,6 +123,7 @@ Terms the rules use without defining them each time, each with the rule that def - [H6 — a title before a family comma](H6-title-before-a-family-comma.md) - [H7 — a joined name unit carrying title vocabulary](H7-joined-name-unit-carrying-title-vocabulary.md) - [H8 — a salutation ending the given part](H8-salutation-ending-the-given-part.md) +- [H9 — the title phrase](H9-title-phrase.md) ## [Particles & surname prefixes (P)](P-particles-and-surname-prefixes.md) diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index d3ce8569..694391d1 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -50,7 +50,7 @@ Phase 1 re-encoded the vocabulary and changed no parse. One TOML file per BCP 47 Two choices in phase 1 set up phase 2 and are worth knowing before reading it: - **The marker sets were explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings, recording past curation rather than evidence. Phase 2a (2026-10-25) derived the two that record a clash with a name, `may_be_name` and `may_be_given`, from name readings, each with dated counts or a `note` where no source counts the word (`te`, `tho` and `du`, once thought to have none, have SSA counts), keeping `may_be_initial` as a property (decisions/vocabulary-model). -- **Names a later phase will use are build errors now** (`default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. Phase 2h (2026-10-25) gave `default = false` its meaning, a reading recorded that compiles to nothing, and named the phase each of the others waits for. +- **Names a later phase will use are build errors** (phase 1 reserved `default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. Phase 2h (2026-10-25) gave `default = false` its meaning, a reading recorded that compiles to nothing, and named the phase each of the others waits for; phase 2i gave `attaches` its meaning, the phrase partner below. The reference for the format is nameparser/vocab/README.md; the decisions behind it, with what was declined and measured, are decisions/vocabulary-model's 2026-10-18 entry. @@ -122,7 +122,7 @@ The front of a name has no words-to-spare cue: a title addresses the name after - **Its partner** is a modifier or a style before it (`Her Royal Highness Princess Anne`), `of` after it (`Dean of Students`), or a head title after it that the word `attaches` to (`Major General`). A salutation is never a partner (`SMITH, DEAN MR.` is a given name and a salutation). - **Modifiers** (decision 10, unparked) are titles only beside a head title, at the head of the name only, never in a comma part, where a lone title word is the person's role. -Which title words get a given reading is decided one word at a time (decision 16), in measured batches, and `Dean Smith` keeps reading given `Dean` (decision 13). A title that only ever stands in front stays a title (`lord`, `sir`, `lady`), and a word that opens names in one tradition but is a title class in another (`guru`, with its Devanagari twin) waits for a language pack. Where the rule reads a contested title as a name, the parse reports `title-or-name`. +Phase 2i carried this out as [rules/H9](rules/H9-title-phrase.md) (the title phrase), with `of` after a modifier as a partner too (`Member of Parliament`) and a modifier in a connective's object (`President of the United States`); a title that may be a given name compiles out of `titles`, so one a caller lists there is a title everywhere. Which title words get a given reading is decided one word at a time (decision 16), in measured batches, and `Dean Smith` keeps reading given `Dean` (decision 13). A title that only ever stands in front stays a title (`lord`, `sir`, `lady`), and a word that opens names in one tradition but is a title class in another (`guru`, with its Devanagari twin) waits for a language pack. Where the rule reads a contested title as a name, the parse reports `title-or-name`. **Abbreviated given names** use the same pool from the other side (#789): a `given` reading with `same_as` the full name (`chas` → Charles, Filipino `ma` → María) is to outrank the leading-abbreviation title rule's shape, so that `Ma. Cristina Santos Reyes` reads given `Ma.` where today it reads title `Ma.`. diff --git a/docs/migrate.rst b/docs/migrate.rst index aabce0b5..97a4174f 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -409,7 +409,7 @@ a private ``Constants`` and pass it: >>> from nameparser import HumanName >>> from nameparser.config import Constants - >>> HumanName("Dean Smith").first # not a default title + >>> HumanName("Dean Smith").first # a title only in a phrase 'Dean' >>> dean_constants = Constants() >>> _ = dean_constants.titles.add("dean") # returns the set diff --git a/docs/release_log.rst b/docs/release_log.rst index 458a7410..1bfae0bf 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,7 +12,9 @@ Release Log - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives first ``John``, last ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave first ``Pharm``, middle ``D``, last ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives first ``Pharm``, and one opening with a single letter is still two words (``John Smith, M. D.`` still gives first ``M.``). :issue:`752` - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave last ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives last ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` + - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Mr Dean Smith`` give first ``Dean``, the parse now reporting a ``title-or-name`` ambiguity where the word opens the name. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` + - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``). :issue:`752` + - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` * 2.6.0 - October 25, 2026 diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index 725ed498..51464e09 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1049,7 +1049,9 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: from nameparser.config.suffixes import ( AGNOMENS, GLUED_HONORIFICS, SUFFIX_WORDS_AMBIGUOUS) from nameparser.config.surnames import KOREAN_SURNAMES - from nameparser.config.titles import AFTER_GIVEN_TITLES + from nameparser.config.titles import ( + AFTER_GIVEN_TITLES, TITLE_MODIFIERS, TITLE_PHRASES, TITLE_STYLES, + TITLES_AMBIGUOUS) # Every set field and every capitalization_exceptions key passes # _v1_matchable BEFORE any set algebra below: an entry v1 could # never match, or one Lexicon folds to empty (#582), is dropped @@ -1234,6 +1236,31 @@ def matchable(field: str) -> frozenset[str]: if _normalize(w) not in { _normalize(a) for a in acronyms - ambiguous_acronyms}), agnomens=AGNOMENS & SUFFIX_WORDS_AMBIGUOUS & suffix_words, + # likewise no v1 manager: the title phrase (rules/H9) is + # 2.7 behavior (#752), so its sets ride in the snapshot + # only. A title that may be a given name is not in + # TITLES, so v1's `titles` never held it, and one a v1 + # caller ADDS to `titles` is a title wherever it stands, + # as it was in 1.4 (`c.titles.add("dean")`, the + # documented recipe): the phrase test reads a word of + # titles_ambiguous only where it is not in titles. The + # knob that reaches the other three is deleting a title, + # which turns its modifier, style or phrase reading off + # with it; Lexicon checks none of them, so the + # intersections are kept for legibility, as + # after_given_titles' is. + # ...save where the caller emptied `titles`, v1's way to + # turn title detection off (customize.rst). A caller who + # REPLACES the list keeps the phrase reading: v1 has no + # knob for these words, and what a replaced list means + # for words it never held is not something it says + titles_ambiguous=TITLES_AMBIGUOUS if titles else frozenset(), + title_modifiers=TITLE_MODIFIERS & titles, + title_styles=TITLE_STYLES & titles, + title_phrases=frozenset( + p for p in TITLE_PHRASES + if all(w in titles or w in TITLES_AMBIGUOUS + for w in p.split(" "))), # TupleManager is dict[str, object] (v1 parity: values were # never statically str-typed); every real entry is a str, # same assumption _DelimiterManager's sentinel lookup makes. diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index ee630961..bffc9ae5 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -26,7 +26,8 @@ "suffix_acronyms_ambiguous", "particles", "particles_ambiguous", "conjunctions", "conjunctions_ambiguous", "bound_given_names", "maiden_markers", "surnames", "honorific_tails", "after_given_titles", - "suffix_words_ambiguous", "agnomens", + "suffix_words_ambiguous", "agnomens", "titles_ambiguous", + "title_modifiers", "title_styles", "title_phrases", ) #: (marker, base, why) triples. Each marker QUALIFIES how entries of @@ -71,6 +72,12 @@ #: only where the word is in titles too, so an orphan is never read. #: agnomens -- the same: classify reads it only for a word already in #: suffix_words_ambiguous, so an orphan decides nothing. +#: title_modifiers, title_styles and title_phrases -- the same: the +#: title phrase test (rules/H9) reads them only for words that are +#: titles, so an orphan decides nothing. titles_ambiguous is no subset +#: at all: its words are titles only inside a phrase, so a caller lists +#: one there without listing it in titles, and one in both is a title +#: everywhere. _SUBSET_FIELDS = ( ("particles_ambiguous", "particles", "an orphan emits a spurious particle-or-given ambiguity"), @@ -103,7 +110,7 @@ #: by a genuine multi-token lookahead instead -- #: _pipeline._vocab.maiden_marker_run, longest first -- and only the #: STORAGE rule is shared with titles. -_PHRASE_FIELDS = ("given_name_titles", "maiden_markers") +_PHRASE_FIELDS = ("given_name_titles", "maiden_markers", "title_phrases") #: Every character the parser reads as a full stop at a WORD'S EDGE @@ -720,6 +727,41 @@ class Lexicon: #: entry outside suffix_words_ambiguous is never consulted. Full #: default list: :data:`~nameparser.config.suffixes.AGNOMENS`. agnomens: frozenset[str] = frozenset() + #: Titles that may also be a given name ("dean", "justice", + #: "major"), each read as a title only inside a title phrase: after a + #: modifier, a style, or a title it stands in ``title_phrases`` + #: behind; or before "of" or a title it stands in ``title_phrases`` + #: in front of. Elsewhere it is a name word, and where it opens the + #: name the parse reports ``title-or-name`` (rules/H9): "Dean Smith" + #: and "Mr Major Smith" read given "Dean" and "Major", "Dean of + #: Students Smith" and "Major General Smith" a title. An entry also + #: in ``titles`` is a title wherever it stands, which is what listing + #: it there asks, so the default lists these words here alone. Full + #: default list: + #: :data:`~nameparser.config.titles.TITLES_AMBIGUOUS`. + titles_ambiguous: frozenset[str] = frozenset() + #: Titles that only qualify a head title ("vice", "wing", "grand"). + #: At the head of the name one is a title only with a head title + #: behind it in its run of titles, so "Wing Commander Lau" keeps the + #: title where "Wing Lau" reads "Wing" as the given name; after a + #: comma it is read as any title is (rules/H9). One standing before + #: a word of ``titles_ambiguous`` puts that word in a title phrase. + #: Entries not also in ``titles`` are never consulted. Full default + #: list: :data:`~nameparser.config.titles.TITLE_MODIFIERS`. + title_modifiers: frozenset[str] = frozenset() + #: Honorific styles ("hon", "excellency", "highness"): one standing + #: before a word of ``titles_ambiguous`` puts that word in a title + #: phrase, as in "The Hon. Justice Smith" (rules/H9). Entries not + #: also in ``titles`` are never consulted. Full default list: + #: :data:`~nameparser.config.titles.TITLE_STYLES`. + title_styles: frozenset[str] = frozenset() + #: Two-word title phrases, a title and the title it qualifies + #: ("chief justice", "major general"), stored space-joined: a word + #: of ``titles_ambiguous`` on either side of one is in a title + #: phrase (rules/H9). A phrase whose words are not both in + #: ``titles`` is never consulted. Full default list: + #: :data:`~nameparser.config.titles.TITLE_PHRASES`. + title_phrases: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> #: "PhD"), laid over the word as written, so the one entry @@ -993,7 +1035,8 @@ def _default_lexicon() -> Lexicon: ) from nameparser.config.surnames import KOREAN_SURNAMES from nameparser.config.titles import ( - AFTER_GIVEN_TITLES, GIVEN_NAME_TITLES, TITLES) + AFTER_GIVEN_TITLES, GIVEN_NAME_TITLES, TITLE_MODIFIERS, TITLE_PHRASES, + TITLE_STYLES, TITLES, TITLES_AMBIGUOUS) # every vocabulary constant is a frozenset since #293, so each one # feeds its strictly-typed frozenset[str] field as it stands -- and @@ -1024,6 +1067,10 @@ def _default_lexicon() -> Lexicon: after_given_titles=AFTER_GIVEN_TITLES, suffix_words_ambiguous=SUFFIX_WORDS_AMBIGUOUS, agnomens=AGNOMENS, + titles_ambiguous=TITLES_AMBIGUOUS, + title_modifiers=TITLE_MODIFIERS, + title_styles=TITLE_STYLES, + title_phrases=TITLE_PHRASES, # pass canonical pair-tuples so this strictly-typed call site never # feeds a Mapping to the tuple-annotated field; __post_init__ # still tolerates a Mapping at runtime for interactive use diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 94a09789..e708b0ca 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -105,7 +105,8 @@ tail_reading, trailing_titles, ) from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, ParseState, PendingAmbiguity, Structure, + AMBIGUOUS_ACRONYM_TAG, TITLE_AMBIGUOUS_TAG, ParseState, PendingAmbiguity, + Structure, WorkToken, _AMBIGUOUS_CREDENTIAL_TAGS, _NEVER_FLIPPED, copy_with, ) from nameparser._policy import Policy, Script @@ -148,12 +149,26 @@ def _absorbed(piece: Sequence[int], # here. def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], ptags: tuple[frozenset[str], ...], - tokens: list[WorkToken]) -> int: + tokens: list[WorkToken], + ambiguities: list[PendingAmbiguity]) -> int: """Assign TITLE to the leading title pieces and return the first non-title index.""" n = leading_titles(pieces, ptags, tokens) for k in range(n): _set_roles(tokens, pieces[k], Role.TITLE) + # rules/H9: "read as a name word where it is the first name word + # after any titles" -- the peel stops at it because classify read it + # out of the title, so it is the first piece the peel leaves + # ('Dean Smith', 'Mr Dean Smith') + if (n < len(pieces) and len(pieces[n]) == 1 + and TITLE_AMBIGUOUS_TAG in tokens[pieces[n][0]].tags + and "vocab:title" not in tokens[pieces[n][0]].tags): + i = pieces[n][0] + ambiguities.append(PendingAmbiguity( + AmbiguityKind.TITLE_OR_NAME, + f"{tokens[i].text!r} is a title that may also be a given name, " + f"and stands in no title phrase; read as ", (i,), + field_tail=" rather than a title")) return n @@ -320,7 +335,7 @@ def _assign_main(seg_idx: int, state: ParseState, -- None on every path that returns before resolving one.""" pieces = state.pieces[seg_idx] ptags = state.piece_tags[seg_idx] - n = _peel_leading_titles(pieces, ptags, tokens) + n = _peel_leading_titles(pieces, ptags, tokens, ambiguities) if n == len(pieces): return None # group-flagged suffix pieces (the ph-d merge) are suffixes at ANY @@ -1287,7 +1302,7 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: # before the title reading of the same word (#296, #325). # A name word in the segment makes it v1's walk ('Smith, # John Jr.'). - n = _peel_leading_titles(pieces, ptags, tokens) + n = _peel_leading_titles(pieces, ptags, tokens, ambiguities) # rules/H5: "the title is TRANSPARENT to the suffix # reading: where two or more name words stand, what # stands once the chain is taken reads exactly as it diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 44790893..c2b96792 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -30,7 +30,7 @@ Tags emitted -- stable (API): "particle", "conjunction", "initial"; namespaced (unstable): "vocab:title", "vocab:given-title", -"vocab:after-given-title", +"vocab:after-given-title", "vocab:title-ambiguous", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", "vocab:suffix-word-ambiguous", "vocab:agnomen", "vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", @@ -68,7 +68,8 @@ from nameparser._policy import CapsSuffixes from nameparser._pipeline._state import ( AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, SHAPE_ACRONYM_TAG, - SPLIT_CREDENTIAL_TAG, SPLIT_MEMBER_TAG, ParseState, PendingAmbiguity, + SPLIT_CREDENTIAL_TAG, SPLIT_MEMBER_TAG, TITLE_AMBIGUOUS_TAG, ParseState, + PendingAmbiguity, WorkToken, copy_with, ) from nameparser._types import AmbiguityKind, Role @@ -76,7 +77,7 @@ BUILTIN_SUFFIX_DELIMITERS, SPLIT_CHUNK, SPLIT_PAIR_OPENS, _INITIAL, caps_shape_candidate, composed_len, delimiter_cores, glued_stem, is_cased_paired_initials, is_initial, is_one_case, period_joined_vocab, split_pair, - suffix_as_written, tag_marker_runs, + suffix_as_written, tag_marker_runs, title_readings, ) from nameparser._pipeline._pieces import own_words @@ -334,6 +335,11 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, _JOINING_TAGS = frozenset({"particle", "conjunction", "vocab:particle-ambiguous", "vocab:bound-given"}) +#: What a word read out of the title gives up (rules/H9): the title tag +#: and the two that qualify one, which are read only beside it. +_TITLE_TAGS = frozenset({"vocab:title", "vocab:given-title", + "vocab:after-given-title"}) + def classify(state: ParseState) -> ParseState: # One fold per token, shared by the marker pass and the vocabulary @@ -412,6 +418,45 @@ def classify(state: ParseState) -> ParseState: tokens = tuple( copy_with(t, tags=t.tags - _JOINING_TAGS) if i in stemmed else t for i, t in enumerate(tokens)) + # rules/H9: "the word is a title only inside a title phrase, and + # elsewhere a name word" -- the phrase test (_vocab.title_readings) + # asked per part, of the words carrying no role, and only of a part + # holding one, so that a name with neither pays no frame. A word read out of the title keeps no + # title tag, and every later stage reads it as the name word it is; + # a contested word keeps TITLE_AMBIGUOUS_TAG either way, for the + # report where it opens the name (assign) + lex = state.lexicon + if not (lex.titles_ambiguous.isdisjoint(folded) + and lex.title_modifiers.isdisjoint(folded)): + written = list(tokens) + for at, seg in enumerate(state.segments): + own_part = [i for i in seg if written[i].role is None] + norms = [folded[i] for i in own_part] + if (lex.titles_ambiguous.isdisjoint(norms) + and lex.title_modifiers.isdisjoint(norms)): + continue + readings = title_readings( + [readers[i].text for i in own_part], norms, lex, + head=at == 0, + titled=["vocab:title" in written[i].tags for i in own_part]) + for i, title in zip(own_part, readings): + tags = written[i].tags + n = folded[i] + if n not in lex.titles_ambiguous and n not in lex.title_modifiers: + continue + if n in lex.titles_ambiguous and n not in lex.titles: + tags = tags | {TITLE_AMBIGUOUS_TAG} + if title: + # in a title phrase: a title, and one addressing + # whichever name its vocabulary says + tags = tags | {"vocab:title"} + if n in lex.given_name_titles: + tags = tags | {"vocab:given-title"} + if not title: + tags = tags - _TITLE_TAGS + if tags != written[i].tags: + written[i] = copy_with(written[i], tags=tags) + tokens = tuple(written) # rules/S12: "a credential written split across a space reads as one # word" -- each adjacent pair of a segment asked once, here, and # the answer written on its first word for group's merge and the diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index ac61f05c..2ad1ca94 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -97,6 +97,12 @@ class WorkToken: AMBIGUOUS_WORD_TAG = "vocab:suffix-word-ambiguous" AGNOMEN_TAG = "vocab:agnomen" +#: On a word of `Lexicon.titles_ambiguous` whichever way classify read +#: it (rules/H9): read as a title it carries "vocab:title" too, read +#: out of the title it carries this alone, which is how assign finds +#: the fork to report where the word opens the name. +TITLE_AMBIGUOUS_TAG = "vocab:title-ambiguous" + #: EITHER way a token joins the ambiguous credential class -- the #: vocabulary's claim and the writing's. The two emitters that report #: a fork the peel called and declined ask exactly this: `_group`'s diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 560d38c2..84d05218 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1095,19 +1095,31 @@ def composed_len(text: str) -> int: def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: """Whether the folded word `n` is in ANY of the lexicon's wordlists - (`_lexicon._VOCAB_FIELDS`, the whole roster): the caps shape's - "unlisted" (`caps_shape_candidate`), where a word a caller listed - even as a SURNAME must never become a credential.""" - for field in _VOCAB_FIELDS: + (`_lexicon._VOCAB_FIELDS`, the whole roster but two): the caps + shape's "unlisted" (`caps_shape_candidate`), where a word a caller + listed even as a SURNAME must never become a credential. + + The two left out: `title_phrases`, whose entries are two words and + so never one, and `titles_ambiguous`, a title that may be a given + name, which outside a title phrase is read as the unlisted word it + is there (rules/H9: "and elsewhere a name word") -- a phrase is two + words and a comma part read by this shape is one.""" + for field in _LISTED_FIELDS: if n in getattr(lexicon, field): return True return False +_LISTED_FIELDS = tuple(f for f in _VOCAB_FIELDS + if f not in ("title_phrases", "titles_ambiguous")) + + #: The wordlists that claim a word as something OTHER than name text. #: `surnames` and `bound_given_names` are left out: they say a word IS -#: a name word ('Kim', 'Abdul'), the opposite claim. -_NON_NAME_FIELDS = tuple(f for f in _VOCAB_FIELDS +#: a name word ('Kim', 'Abdul'), the opposite claim; and so is +#: `titles_ambiguous`, a title that may be a given name (rules/H9), with +#: `title_phrases`, whose two-word entries never match one word. +_NON_NAME_FIELDS = tuple(f for f in _LISTED_FIELDS if f not in ("surnames", "bound_given_names")) @@ -1369,6 +1381,117 @@ def surname_unit_tags(text: str, lexicon: Lexicon, return _PARTICLE_ONLY if particle else _NO_TAGS +#: The connective after a title that may be a given name which makes +#: the two a title phrase, 'Dean of Students' (rules/H9). Coordinating +#: connectives are not: 'Dean and Mary Smith' is two people. +_PHRASE_LINKS = frozenset({"of"}) +#: The connectives whose object a modifier after them is in, 'of the +#: United States', 'The Grand Master'; 'and' joins two titles instead. +_OBJECT_LINKS = frozenset({"of", "the"}) + + +# rules/H9: "the word is a title only inside a title phrase, and +# elsewhere a name word" -- both halves, the partners of a title that may +# be a given name and of a modifier at the head of the name, each its +# bullet's (decisions/H9 #752) +def title_readings(texts: Sequence[str], folded: Sequence[str], + lexicon: Lexicon, *, head: bool, + titled: Sequence[bool] | None = None) -> list[bool]: + """Which of these words, one part of the name in order, read as + title words. Where none of them is a title that may be a given name + or a modifier, the listed titles answer alone, which every caller + asks first, in C, so that a name with neither pays no frame. + + A word of `titles_ambiguous` that is not also in `titles` (where a + caller listing it makes it a title everywhere) is a title only + beside a partner: a + modifier, a style, or a title it stands behind in `title_phrases`, + before it; or 'of', or a title it stands in front of in + `title_phrases`, after it. A salutation is never one, nor any other + title before it ('Mr Dean Smith'), nor a title after it that it + does not attach to ('SMITH, DEAN MR.', a given name and the + salutation a donor list writes after it). Its neighbours are asked + as LISTED, so 'Vice Dean' reads 'Vice' as the partner of 'Dean' + whatever the modifier test below makes of 'Vice'. + + Where `head` is true -- the part before the first comma -- a word + of `title_modifiers` is a title only with a head title behind it in + its run of titles: any title that is not a modifier, or a word H2's + shape reads as one, before 'of' ('Mother Superior of the Convent'), + or in the object of 'of' or 'the' there (decision 10 of + decisions/vocabulary-model). Only the leading run is the head: a + modifier outside it is read as listed, one ending the name being + H5's. A + word in both sets takes the modifier test in every part, that run + being its phrase. After a comma a modifier is read as listed: a + part there that is a lone title word is the person's role + ('John Smith, Resident'), and reading it as a name would reverse the + name (decisions/H9). + + `texts` are the words as written, for the shape test; `folded` + their `_normalize`d forms; `titled` which of them a caller already + reads as titles, where that is more than `titles` lists (classify's + period-joined `Lt.Gov.`), else membership in `titles`. Only a word + of the two sets is ever read out of the title.""" + contested = lexicon.titles_ambiguous + modifiers = lexicon.title_modifiers + titles = lexicon.titles + # a word in titles is a title wherever it stands, whatever else it + # is listed as; one of titles_ambiguous alone is listed as a title + # to its neighbours and read by its phrase + reading = [n in titles or n in contested for n in folded] + if titled is not None: + reading = [r or t for r, t in zip(reading, titled)] + listed = list(reading) + phrases = lexicon.title_phrases + last = len(folded) - 1 + for i, n in enumerate(folded): + if n in titles or n not in contested or n in modifiers: + continue + prev = folded[i - 1] if i else None + nxt = folded[i + 1] if i < last else None + reading[i] = bool( + (prev is not None and listed[i - 1] + and (prev in modifiers or prev in lexicon.title_styles + or f"{prev} {n}" in phrases)) + or (nxt is not None + and ((nxt in _PHRASE_LINKS and nxt in lexicon.conjunctions) + or (listed[i + 1] and f"{n} {nxt}" in phrases)))) + # left to right, whether the run so far has passed 'of' or 'the': a + # modifier after it is in the object of a phrase ('President of the + # United States', 'the Queen's Bench'), which no head need follow. A + # coordinating connective opens no object: 'Mr. and Mrs. Wing Lau' + # is two salutations and a given name + linked = [False] * len(folded) + conjunctions = lexicon.conjunctions + for i in range(1, len(folded)): + if reading[i - 1] or folded[i - 1] in conjunctions: + linked[i] = linked[i - 1] or folded[i - 1] in _OBJECT_LINKS + # the head of the name is its LEADING run of titles and connectives: + # a modifier ending the name is H5's trailing run, read as listed + lead = 0 + if head: + while lead < len(folded) and (reading[lead] or folded[lead] in conjunctions + or is_title_shaped(texts[lead])): + lead += 1 + # right to left, so each modifier knows whether a head title stands + # behind it before the run is broken; a connective carries the run + behind = False + for i in range(last, -1, -1): + n = folded[i] + if (n in modifiers and reading[i] + and ((head and i < lead) or (n in contested and n not in titles))): + reading[i] = behind or linked[i] or ( + i < last and folded[i + 1] in _PHRASE_LINKS + and folded[i + 1] in conjunctions) + behind = reading[i] + elif reading[i] or (head and is_title_shaped(texts[i])): + behind = True + elif n not in conjunctions: + behind = False + return reading + + def name_word_count(texts: Sequence[str], lexicon: Lexicon, policy: Policy, letters_one_case: bool | None = None, ) -> int: @@ -1404,14 +1527,22 @@ def name_word_count(texts: Sequence[str], lexicon: Lexicon, # units -- and pay no frame the token loop did not already pay. names: list[bool] = [] has_particle = False - for text in texts: - n = _normalize(text) + # map, not a comprehension, which is a frame of its own on 3.11 + folded = list(map(_normalize, texts)) + # rules/H9: "the word is a title only inside a title phrase" -- the + # part before the comma is the head of the name, read as classify + # reads it + titled = (None if lexicon.titles_ambiguous.isdisjoint(folded) + and lexicon.title_modifiers.isdisjoint(folded) + else title_readings(texts, folded, lexicon, head=True)) + for i, n in enumerate(folded): + text = texts[i] + title = n in lexicon.titles if titled is None else titled[i] if n in lexicon.particles: has_particle = True suffix = (is_suffix_lenient(text, lexicon) if lenient else is_suffix_strict(text, lexicon, letters_one_case)) - names.append(not (suffix or n in lexicon.titles - or is_title_shaped(text))) + names.append(not (suffix or title or is_title_shaped(text))) if not has_particle: return sum(names) count = 0 diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index 13bfd8d6..d5c39175 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -966,6 +966,452 @@ 'শ্রীমতী', }) +TITLES_AMBIGUOUS: frozenset[str] = frozenset({ + 'dean', + 'justice', + 'major', +}) + +TITLE_MODIFIERS: frozenset[str] = frozenset({ + '10th', + '1st', + '2nd', + '3rd', + '4th', + '5th', + '6th', + '7th', + '8th', + '9th', + 'advertising', + 'air', + 'analytics', + 'appellate', + 'army', + 'assistant', + 'assoc', + 'associate', + 'asst', + 'award-winning', + 'ballet', + 'banner', + 'baseball', + 'bearer', + 'behavioral', + 'bench', + 'biblical', + 'blues', + 'british', + 'burlesque', + 'business', + 'civil', + 'classical', + 'computer', + 'corporate', + 'credit', + 'criminal', + 'customs', + 'designated', + 'discovery', + 'district', + 'division', + 'docket', + 'edmi', + 'english', + 'excellent', + 'exec', + 'executive', + 'family', + 'federal', + 'field', + 'film', + 'financial', + 'first', + 'flag', + 'flying', + 'foreign', + 'grand', + 'group', + 'her', + 'hereditary', + 'high', + 'his', + 'information', + 'intelligence', + 'judicial', + "king's", + 'kingdom', + 'knowledge', + 'law', + 'legal', + 'literary', + 'marcher', + 'marketing', + 'mathematics', + 'military', + 'most', + 'municipal', + 'mystery', + 'national', + 'navy', + 'opera', + 'operating', + 'petty', + 'police', + 'political', + 'presiding', + 'prime', + "queen's", + 'radio', + 'rear', + 'revenue', + 'right', + 'risk', + 'rock', + 'royal', + 'security', + 'senior', + 'soccer', + 'social', + 'software', + 'special', + 'staff', + 'state', + 'states', + 'strategy', + 'superior', + 'supreme', + 'tax', + 'technical', + 'television', + 'theater', + 'theatre', + 'travel', + 'uk', + 'united', + 'us', + 'vice', + 'voice', + 'warrant', + 'wing', +}) + +TITLE_STYLES: frozenset[str] = frozenset({ + 'ch.ma', + 'ch.mo', + 'cheikh', + 'doyen', + 'effendi', + 'eminence', + 'excellency', + 'gent.le', + 'gent.ma', + 'gent.mo', + 'hajji', + 'highness', + 'holiness', + 'hon', + 'honorable', + 'honourable', + 'ill.ma', + 'ill.mo', + 'majesty', + 'preg.ma', + 'preg.mo', + 'shaik', + 'shaikh', + 'shayk', + 'shaykh', + 'sheik', + 'sheikh', + 'shekh', + 'الشيخ', + 'الشيخة', + 'महात्मा', + 'सरदार', + 'আলহাজ', + 'আলহাজ্ব', +}) + +TITLE_PHRASES: frozenset[str] = frozenset({ + 'chief 1lt', + 'chief 1sgt', + 'chief 1stlt', + 'chief 1stsgt', + 'chief 2lt', + 'chief 2ndlt', + 'chief a1c', + 'chief ab', + 'chief adjutant', + 'chief adm', + 'chief administrator', + 'chief admiral', + 'chief alderman', + 'chief almoner', + 'chief ambassador', + 'chief amn', + 'chief arbitrator', + 'chief attache', + 'chief attaché', + 'chief bailiff', + 'chief bg', + 'chief bgen', + 'chief brigadier', + 'chief briggen', + 'chief burgess', + 'chief capt', + 'chief captain', + 'chief ccmsgt', + 'chief cdr', + 'chief ceo', + 'chief cfo', + 'chief chair', + 'chief chairs', + 'chief chancellor', + 'chief charge', + 'chief chargé', + 'chief clerk', + 'chief cllr', + 'chief cmsaf', + 'chief cmsgt', + 'chief co-chair', + 'chief co-chairs', + 'chief col', + 'chief colonel', + 'chief commander', + 'chief commander-in-chief', + 'chief commissioner', + 'chief commodore', + 'chief comptroller', + 'chief controller', + 'chief corporal', + 'chief councillor', + 'chief counsel', + 'chief courtier', + 'chief cpl', + 'chief cpo', + 'chief cpt', + 'chief csm', + 'chief cwo-2', + 'chief cwo-3', + 'chief cwo-4', + 'chief cwo-5', + 'chief cwo2', + 'chief cwo3', + 'chief cwo4', + 'chief cwo5', + "chief d'affaires", + 'chief dean', + 'chief delegate', + 'chief deputy', + 'chief detective', + 'chief dir', + 'chief director', + 'chief dpty', + 'chief ens', + 'chief envoy', + 'chief fadm', + 'chief gaf', + 'chief gen', + 'chief general', + 'chief generalissimo', + 'chief gov', + 'chief governor', + 'chief gysgt', + 'chief headman', + 'chief intendant', + 'chief judge', + 'chief justice', + 'chief lcdr', + 'chief lcpl', + 'chief leader', + 'chief lieutenant', + 'chief lt', + 'chief ltc', + 'chief ltcol', + 'chief ltg', + 'chief ltgen', + 'chief ltjg', + 'chief mag', + 'chief mag-judge', + 'chief mag/judge', + 'chief magistrate', + 'chief magistrate-judge', + 'chief maj', + 'chief majgen', + 'chief major', + 'chief mayor', + 'chief mcpo', + 'chief mcpoc', + 'chief mcpon', + 'chief member', + 'chief mg', + 'chief mgysgt', + 'chief minister', + 'chief mpco-cg', + 'chief msg', + 'chief msgt', + 'chief officer', + 'chief pfc', + 'chief po1', + 'chief po2', + 'chief po3', + 'chief prefect', + 'chief premier', + 'chief pres', + 'chief president', + 'chief prin', + 'chief principal', + 'chief private', + 'chief provost', + 'chief pslc', + 'chief pursuivant', + 'chief pv2', + 'chief pvt', + 'chief radm', + 'chief rdml', + 'chief registrar', + 'chief rep', + 'chief representative', + 'chief scpo', + 'chief secretary', + 'chief senator', + 'chief senior-judge', + 'chief sergeant', + 'chief sfc', + 'chief sgm', + 'chief sgt', + 'chief sgtmaj', + 'chief sgtmajmc', + 'chief sheriff', + 'chief sma', + 'chief smsgt', + 'chief sn', + 'chief spc', + 'chief speaker', + 'chief ssg', + 'chief ssgt', + 'chief subaltern', + 'chief subedar', + 'chief swordbearer', + 'chief sysselmann', + 'chief treasurer', + 'chief tsgt', + 'chief vadm', + 'chief vc', + 'chief verderer', + 'chief vizier', + 'chief warden', + 'chief wo-1', + 'chief wo1', + 'chief wo2', + 'chief wo3', + 'chief wo4', + 'chief wo5', + 'chief न्यायमूर्ति', + 'chief বিচারপতি', + 'major 1lt', + 'major 1sgt', + 'major 1stlt', + 'major 1stsgt', + 'major 2lt', + 'major 2ndlt', + 'major a1c', + 'major ab', + 'major adjutant', + 'major adm', + 'major admiral', + 'major amn', + 'major bg', + 'major bgen', + 'major brigadier', + 'major briggen', + 'major capt', + 'major captain', + 'major ccmsgt', + 'major cdr', + 'major cmsaf', + 'major cmsgt', + 'major col', + 'major colonel', + 'major commander', + 'major commander-in-chief', + 'major commodore', + 'major corporal', + 'major cpl', + 'major cpo', + 'major cpt', + 'major csm', + 'major cwo-2', + 'major cwo-3', + 'major cwo-4', + 'major cwo-5', + 'major cwo2', + 'major cwo3', + 'major cwo4', + 'major cwo5', + 'major detective', + 'major ens', + 'major fadm', + 'major gaf', + 'major gen', + 'major general', + 'major generalissimo', + 'major gysgt', + 'major lcdr', + 'major lcpl', + 'major lieutenant', + 'major lt', + 'major ltc', + 'major ltcol', + 'major ltg', + 'major ltgen', + 'major ltjg', + 'major maj', + 'major majgen', + 'major mcpo', + 'major mcpoc', + 'major mcpon', + 'major mg', + 'major mgysgt', + 'major mpco-cg', + 'major msg', + 'major msgt', + 'major officer', + 'major pfc', + 'major po1', + 'major po2', + 'major po3', + 'major private', + 'major pv2', + 'major pvt', + 'major radm', + 'major rdml', + 'major scpo', + 'major sergeant', + 'major sfc', + 'major sgm', + 'major sgt', + 'major sgtmaj', + 'major sgtmajmc', + 'major sma', + 'major smsgt', + 'major sn', + 'major spc', + 'major ssg', + 'major ssgt', + 'major subaltern', + 'major subedar', + 'major tsgt', + 'major vadm', + 'major wo-1', + 'major wo1', + 'major wo2', + 'major wo3', + 'major wo4', + 'major wo5', +}) + NON_GIVEN_NAME_PARTICLES: frozenset[str] = frozenset({ "'t", 'ad-', diff --git a/nameparser/config/titles.py b/nameparser/config/titles.py index c5652db1..9dba3a46 100644 --- a/nameparser/config/titles.py +++ b/nameparser/config/titles.py @@ -13,7 +13,9 @@ "Sir John", "Sister Mary", "Queen Elizabeth". """ -#: **Cannot include things that could also be given names**, e.g. "dean". +#: **Cannot include things that could also be given names**: a title that +#: may be one ("dean") is in TITLES_AMBIGUOUS instead, a title only inside +#: a title phrase, and listing it here makes it a title everywhere. #: Many of these from wikipedia: https://en.wikipedia.org/wiki/Title. #: The parser recognizes chains of these including conjunctions allowing #: recognition titles like "Deputy Secretary of State". @@ -27,6 +29,37 @@ out, so "Doe, John Goodman" keeps middle "Goodman". """ +TITLES_AMBIGUOUS = _compiled.TITLES_AMBIGUOUS +""" +Titles that may be a given name, each a title only inside a title +phrase: "Dean of Students" and "Major General" are titles, where "Dean +Smith" and "Mr Major Smith" read "Dean" and "Major" as the given name. +Kept out of TITLES, where a word is a title wherever it stands, and +derived from a given-name reading on the word. +""" + +TITLE_MODIFIERS = _compiled.TITLE_MODIFIERS +""" +Titles that only qualify a head title ("vice", "wing", "grand"): at the +head of a name each is a title only with a head title behind it in its +run, so "Wing Commander Lau" keeps the title where "Wing Lau" reads +"Wing" as the given name. A subset of TITLES. +""" + +TITLE_STYLES = _compiled.TITLE_STYLES +""" +Honorific styles ("hon", "excellency", "highness"): one standing before +a title that may be a given name makes it part of a title phrase, as in +"The Hon. Justice Smith". A subset of TITLES. +""" + +TITLE_PHRASES = _compiled.TITLE_PHRASES +""" +Two-word title phrases, each a title word and a title it qualifies +("chief justice", "major general"), so a title that may be a given name +reads as a title inside one. +""" + # 1.x name, deprecated in 2.2 and removed in 3.0 (#293). Unlike the # module moves in #293, the constant did not change module, so this @@ -57,4 +90,5 @@ # retired name and drop the live ones instead. # Source order, not alphabetical -- see the note in suffixes.py. __all__ = ["GIVEN_NAME_TITLES", "TITLES", "AFTER_GIVEN_TITLES", - "FIRST_NAME_TITLES"] + "TITLES_AMBIGUOUS", "TITLE_MODIFIERS", "TITLE_STYLES", + "TITLE_PHRASES", "FIRST_NAME_TITLES"] diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 6b12c9a7..4acac3bd 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -100,7 +100,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or |---|---|---| | `title` | `title` | `TITLES` | | `salutation` | `title` | `TITLES` | -| `style` | `title` | `TITLES` | +| `style` | `title` | `TITLES`, `TITLE_STYLES` | | `academic` | `title` | `TITLES` | | `religious` | `title` | `TITLES` | | `nobility` | `title` | `TITLES` | @@ -108,7 +108,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `office` | `title` | `TITLES` | | `occupation` | `title` | `TITLES` | | `kinship` | `title` | `TITLES` | -| `modifier` | `title` | `TITLES` | +| `modifier` | `title` | `TITLES`, `TITLE_MODIFIERS` | | `suffix` | `suffix` | `SUFFIX_WORDS` | | `generation` | `suffix` | `SUFFIX_WORDS` | | `agnomen` | `suffix` | `SUFFIX_WORDS`, `SUFFIX_WORDS_AMBIGUOUS`, `AGNOMENS` | @@ -147,7 +147,7 @@ The title kinds say what the title is: - `office`: a public, judicial, civic, corporate or court office held (`president`, `judge`, `mayor`, `ceo`); - `occupation`: a job DESCRIBED before a name, as bylines and captions write it, household service included (`actor`, `novelist`, `guitarist`, `maid`); - `kinship`: a family word written as a title (`aunt`, `uncle`); -- `modifier`: a word that is a title only inside a phrase with a head title (`vice`, `grand`, `assistant`, `his`, `1st`), the class decision 10 of [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md) parks. Today it is a title wherever it stands, like every other kind. +- `modifier`: a word that is a title only inside a phrase with a head title (`vice`, `grand`, `assistant`, `his`, `1st`), decision 10 of [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md). A word whose every title reading is a modifier compiles into `TITLE_MODIFIERS` too, and in the leading run of titles before the first comma it is a title only with a head title after it in that run, before `of` (`Mother Superior of the Convent`), or inside the object of `of` or `the` there (`President of the United States`); ending the name, or after a comma, it is read as any title is ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase). A word that qualifies the head it stands behind is no modifier in this sense: `d'affaires`, after `chargé`, is an `office`. A word written in a part after a comma as the person's role is not one either: `resident`, `rt` and `pro` were reclassified for that (2026-10-25). A `style` compiles into `TITLE_STYLES`, the one other kind with a set of its own: a style or a modifier before a title that may be a given name puts it in a title phrase (`The Hon. Justice Smith`). `academic` and `occupation` are told apart by what the word does, not its subject: `Dr` addresses the person, `chemist` describes them. A title is written with one kind per file, as a particle is (a convention, not a check), and check 5 holds every title reading of a word to the same `addresses` whatever its kind. `gender` is recorded, as each title is classified, on one marked for one sex (a feminine form, `actress`, `señora`, or one half of a fixed pair, `mr` and `mrs`, `king` and `queen`), and on `mx` as `"n"`; an unmarked noun (`actor`, `doctor`) records none. @@ -175,6 +175,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | Property | Type | Applies to | Unwritten | Effect | |---|---|---|---|---| | `addresses` | behavioral | `title` | `"family"` | `"given"` adds the word to `GIVEN_NAME_TITLES`: `Sir John` reads `John` as the given name ([rules/H1](../../docs/design/rules/H1-title-and-one-name-word.md), a title and one name word) | +| `attaches` | behavioral | `title` | none | `TITLE_PHRASES`: the title kinds this word stands before in a title phrase, a list (`chief = { attaches = ["office", "military"] }`), compiled into the two-word phrases it makes with each title word of those kinds (`chief justice`); a title that may be a given name on either side of one is a title there ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase) | | `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | | `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | @@ -197,12 +198,13 @@ Two of these need more than their row: ### Derived markers -Three sets mark a clash with a name, and all three are derived: a word is in one when it carries the clashing reading and a name reading, in any of its files (check 11, since 2026-10-25, [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md)). Phase 1 wrote two of them as flags, `may_be_name` and `may_be_given = true`, which the build now refuses, naming the reading to write; the third, a suffix word that is also a name, arrived with phase 2h. +Four sets mark a clash with a name, and all four are derived: a word is in one when it carries the clashing reading and a name reading, in any of its files (check 11, since 2026-10-25, [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md)). Phase 1 wrote two of them as flags, `may_be_name` and `may_be_given = true`, which the build now refuses, naming the reading to write; the third, a suffix word that is also a name, arrived with phase 2h, and the fourth, a title that may be a given name, with phase 2i. | Marker | Readings | Compiled into | Effect | |---|---|---|---| | possibly a name, a word | a `generation` or `agnomen` reading and a `given` or `family` reading; a name reading beside any other suffix word kind is refused (check 11), no gate reading that clash | `SUFFIX_WORDS_AMBIGUOUS` | a suffix word that is also a name (`fils`, `neto`) is read as a suffix only where the name has words to spare, as an ambiguous acronym is: `Alexandre Dumas fils` and `Pedro Alves Neto` end in a suffix, `Marie Fils` and `Ana Neto` in a family name, either way reported ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md)). A generation's Title case in a name of mixed case leans it to the name, and its lower case settles it as the suffix wherever the name keeps a family word; an agnomen's writing settles nothing | | possibly a name | an acronym `suffix` reading and a `given` or `family` reading | `SUFFIX_ACRONYMS_AMBIGUOUS` | an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | +| possibly a name, a title | a title reading and a `given` reading, on a word with no particle reading | `TITLES_AMBIGUOUS`, and kept out of `TITLES` | a title that may be a given name (`dean`, `justice`, `major`) is a title only inside a title phrase: after a modifier, a style or a title it stands behind in a phrase `attaches` makes, or before `of` or a title it attaches to. Elsewhere it is a name word, reported where it opens the name: `Dean of Students` and `Major General Smith` read the title, `Dean Smith` and `Mr Major Smith` the given name ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase). A title that is also a particle (`st`, `freiherr`) is left to the particle rules, its `given` reading deriving the particle's marker alone. A salutation's given-name counts kept as evidence for `after_given` sit in the salutation reading's `note` (`mr`, `herr`), so they mark nothing | | possibly given | a `particle` reading and a `given` reading | kept out of `NON_GIVEN_NAME_PARTICLES` | a particle opening the name stays a given name there (`van Gogh`, given `van`) and the fork is reported, rather than starting the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold) | Before phase 2h (2026-10-25) a word reading of the suffix category beside a name reading marked nothing, and no shipped word had one. The marker is the clash, not the counts: a count recorded for a word that should stay unmarked goes in the clashing reading's `note`, as `ld`'s does, since a name reading would mark it. `may_be_initial` is not derived: its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. @@ -257,7 +259,6 @@ These names will mean something in a later phase. Writing one now as a key, in a | Name | Phase | |---|---| -| `attaches` | 2 | | `prefer` | 3 | | `strict` | 3 | | `bare_only` | 4 | @@ -265,7 +266,7 @@ These names will mean something in a later phase. Writing one now as a key, in a | `trailing_only` | 4 | | `separators` | 4 | -Phase 2 gave `default` its meaning (2026-10-25, see [Properties](#properties)) and keeps `attaches` for its front-of-the-name work. The gate's slots carry position and shape already settles a bare letter ([rules/P7](../../docs/design/rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells), so `leading_only`, `trailing_only` and `bare_only` wait for phase 4's shapes, as `separators` waits for its glued words. +Phase 2 gave `default` and `attaches` their meaning (2026-10-25, see [Properties](#properties)). The gate's slots carry position and shape already settles a bare letter ([rules/P7](../../docs/design/rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells), so `leading_only`, `trailing_only` and `bare_only` wait for phase 4's shapes, as `separators` waits for its glued words. Two property values are refused for the same reason, with a message saying a later phase: `addresses = "full"` and `"any"`, and `attached` with a `side` other than `"after"`. diff --git a/nameparser/vocab/bn.toml b/nameparser/vocab/bn.toml index 09fdf5d9..ad639016 100644 --- a/nameparser/vocab/bn.toml +++ b/nameparser/vocab/bn.toml @@ -30,8 +30,7 @@ salutation = { gender = "f", issue = 343, expands = "Mrs., borrowed", after_give [[word]] text = "মোঃ" -salutation = { gender = "m", issue = 343, expands = "Md., Mohammad", note = "a name PREFIX, not a title semantically; TITLES is the functional home because `given` must stay the name the person is addressed by (decisions/indic-honorifics)", excluded = { after_given = "abbreviates Mohammad, a given name, and is read as one where it leads (#853)" } } -given = { wikidata = 37, as_of = 2026-10-24 } +salutation = { gender = "m", issue = 343, expands = "Md., Mohammad", note = "a name PREFIX, not a title semantically; TITLES is the functional home because `given` must stay the name the person is addressed by (decisions/indic-honorifics); borne as a given name: Wikidata 37 people (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)", excluded = { after_given = "abbreviates Mohammad, a given name, and is read as one where it leads (#853)" } } [[word]] text = "মো" diff --git a/nameparser/vocab/de.toml b/nameparser/vocab/de.toml index c0af9f64..0373dbbd 100644 --- a/nameparser/vocab/de.toml +++ b/nameparser/vocab/de.toml @@ -15,8 +15,7 @@ salutation = { gender = "f", after_given = true } [[word]] text = "herr" -salutation = { gender = "m", excluded = { after_given = "a surname (Census 2010 HERR), and middle names are drawn from surnames" } } -given = { wikidata = 3, as_of = 2026-10-24 } +salutation = { gender = "m", excluded = { after_given = "a surname (Census 2010 HERR), and middle names are drawn from surnames" }, note = "borne as a given name: Wikidata 3 people (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { us_census_2010 = 11980, wikidata = 135, as_of = 2026-10-24 } [[word]] diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index c328da1c..6426f89a 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -6,8 +6,7 @@ excluded = { title = { reason = "William, abbreviated, and no title of any kind: [[word]] text = "goodman" -salutation = { gender = "m", expands = "the archaic Mr., 'Goodman Brown'", excluded = { after_given = "a common surname (Benny Goodman), and middle names are drawn from surnames" } } -given = { us_ssa = 22, wikidata = 11, as_of = 2026-10-24 } +salutation = { gender = "m", expands = "the archaic Mr., 'Goodman Brown'", excluded = { after_given = "a common surname (Benny Goodman), and middle names are drawn from surnames" }, note = "borne as a given name: US SSA 22 births, Wikidata 11 people (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { us_census_2010 = 73136, wikidata = 743, as_of = 2026-10-24 } [[word]] @@ -24,8 +23,7 @@ salutation = { gender = "f", after_given = true } [[word]] text = "master" -salutation = { addresses = "given", gender = "m", expands = "the style for a boy, 'Master Harry'", excluded = { after_given = "a surname and a given name, borne as a middle name either way" } } -given = { us_ssa = 476, wikidata = 2, as_of = 2026-10-24 } +salutation = { addresses = "given", gender = "m", expands = "the style for a boy, 'Master Harry'", excluded = { after_given = "a surname and a given name, borne as a middle name either way" }, note = "borne as a given name: US SSA 476 births, Wikidata 2 people (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { us_census_2010 = 2735, wikidata = 61, as_of = 2026-10-24 } [[word]] @@ -39,13 +37,11 @@ salutation = { gender = "f", expands = "the plural of miss, 'the Misses Smith'", [[word]] text = "mr" -salutation = { gender = "m", after_given = true } -given = { us_ssa = 5, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } +salutation = { gender = "m", after_given = true, note = "borne as a given name: US SSA 5 births (2026-10-24); kept for after_given (decisions/H8); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } [[word]] text = "mister" -salutation = { gender = "m", same_as = "mr", excluded = { after_given = "a given name and a surname, borne as a middle name either way" } } -given = { us_ssa = 746, as_of = 2026-10-24 } +salutation = { gender = "m", same_as = "mr", excluded = { after_given = "a given name and a surname, borne as a middle name either way" }, note = "borne as a given name: US SSA 746 births (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { us_census_2010 = 1211, wikidata = 2, as_of = 2026-10-24 } [[word]] @@ -114,6 +110,10 @@ academic = { expands = "the address of an engineer, 'Engr. Cruz' in the Philippi text = "nurse" academic = { expands = "the address of a nurse, 'Nurse Smith'" } +[[word]] +text = "pro" +academic = { issue = 9, expands = "Pro Se Law Clerk", note = "academic rather than modifier since 2026-10-25 (#752): a part after a comma written 'Pro' is the person's role, which decision 10's modifier rule would read out of the title (decisions/vocabulary-model)" } + [[word]] text = "abbess" religious = { gender = "f" } @@ -336,7 +336,7 @@ nobility = { gender = "f" } [[word]] text = "chief" -nobility = { expands = "a chief of a people or clan, 'Chief Obafemi Awolowo'" } +nobility = { expands = "a chief of a people or clan, 'Chief Obafemi Awolowo'", attaches = ["office", "military"], issue = 752, note = "attaches: Chief Justice, Chief Clerk (rules/H9)" } [[word]] text = "chieftain" @@ -677,9 +677,15 @@ military = { expands = "Lieutenant (junior grade)" } text = "maj" military = { expands = "Major" } +[[word]] +text = "major" +military = { issue = 752, decided = 2026-10-25, attaches = ["military"], note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Major General Smith' reads the title, 'Major Smith' and 'Mr Major Smith' the given name; attaches: Major General" } +given = { us_ssa = 21749, wikidata = 65, as_of = 2026-10-25, note = "US SSA: female and male births added" } +family = { us_census_2010 = 20977, wikidata = 400, as_of = 2026-10-25 } + [[word]] text = "marshal" -excluded = { military = { reason = "a given name (Marshal Yanda); the accepted cost is that 'Field Marshal Bernard Montgomery' reads title Field, given Marshal", issue = 606, decided = 2026-10-04 } } +excluded = { military = { reason = "a given name (Marshal Yanda); the accepted cost is that 'Field Marshal Bernard Montgomery' reads given Field, middle Marshal Bernard (title Field, given Marshal until rules/H9, 2026-10-25, read a modifier with no head after it as a name)", issue = 606, decided = 2026-10-04 } } [[word]] text = "mcpo" @@ -956,6 +962,16 @@ office = { issue = 606, note = "borne: ~730 surnames (Australia, the US, England text = "courtier" office = {} +[[word]] +text = "d'affaires" +office = { expands = "the second word of chargé d'affaires", issue = 752, note = "office rather than modifier since 2026-10-25: it stands behind its head, where decision 10's modifier rule asks for a head title after a modifier (rules/H9)" } + +[[word]] +text = "dean" +office = { issue = 752, decided = 2026-10-25, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Dean of Students' and 'Associate Dean Smith' read the title, 'Dean Smith' the given name (decision 13 of decisions/vocabulary-model)" } +given = { us_ssa = 195844, wikidata = 2246, as_of = 2026-10-25, note = "US SSA: female and male births added" } +family = { us_census_2010 = 114030, wikidata = 1128, as_of = 2026-10-25 } + [[word]] text = "delegate" office = {} @@ -1002,7 +1018,9 @@ office = {} [[word]] text = "justice" -excluded = { office = { reason = "a given name (Justice Smith); the accepted cost is that 'Chief Justice John Roberts' reads title Chief, given Justice", issue = 606, decided = 2026-10-04 } } +office = { issue = 752, decided = 2026-10-25, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Chief Justice John Roberts' and 'The Hon. Justice Smith' read the title, 'Justice Smith' the given name; until 2026-10-25 excluded as a given name (#606), the cost being 'Chief Justice' read as title Chief, given Justice" } +given = { us_ssa = 38674, wikidata = 85, as_of = 2026-10-25, note = "US SSA: female and male births added" } +family = { us_census_2010 = 32480, wikidata = 100, as_of = 2026-10-25 } [[word]] text = "leader" @@ -1032,6 +1050,10 @@ office = { same_as = "magistrate-judge" } text = "mayor" office = {} +[[word]] +text = "member" +office = { issue = 752, note = "office rather than modifier since 2026-10-25: the head noun of the phrases it stands in, 'Designated Member' and 'Member of Parliament', which decision 10's modifier rule would read out of the title (rules/H9)" } + [[word]] text = "minister" office = {} @@ -1641,6 +1663,14 @@ occupation = {} text = "researcher" occupation = {} +[[word]] +text = "resident" +occupation = { issue = 752, note = "occupation rather than modifier since 2026-10-25: a part after a comma written 'Resident' is the person's role, which decision 10's modifier rule would read out of the title (decisions/vocabulary-model)" } + +[[word]] +text = "rt" +occupation = { issue = 752, expands = "Right, abbreviated ('Rt Hon'); also a respiratory therapist", note = "occupation rather than modifier since 2026-10-25: a part after a comma written 'RT' is the person's role, which decision 10's modifier rule would read out of the title (decisions/vocabulary-model)" } + [[word]] text = "sailor" occupation = {} @@ -1909,10 +1939,6 @@ modifier = {} text = "customs" modifier = {} -[[word]] -text = "d'affaires" -modifier = { expands = "the second word of chargé d'affaires" } - [[word]] text = "designated" modifier = { issue = 9, expands = "Designated Judge" } @@ -2061,10 +2087,6 @@ modifier = {} text = "mathematics" modifier = {} -[[word]] -text = "member" -modifier = {} - [[word]] text = "military" modifier = {} @@ -2117,10 +2139,6 @@ modifier = {} text = "prime" modifier = {} -[[word]] -text = "pro" -modifier = { issue = 9, expands = "Pro Se Law Clerk" } - [[word]] text = "queen's" modifier = { expands = "Queen's Counsel" } @@ -2133,10 +2151,6 @@ modifier = {} text = "rear" modifier = {} -[[word]] -text = "resident" -modifier = {} - [[word]] text = "revenue" modifier = {} @@ -2145,10 +2159,6 @@ modifier = {} text = "right" modifier = { expands = "'Right Honourable', 'Right Reverend'" } -[[word]] -text = "rt" -modifier = { same_as = "right" } - [[word]] text = "risk" modifier = {} diff --git a/nameparser/vocab/es.toml b/nameparser/vocab/es.toml index 423acc48..35a16eac 100644 --- a/nameparser/vocab/es.toml +++ b/nameparser/vocab/es.toml @@ -11,13 +11,11 @@ salutation = { gender = "m", same_as = "señor" } [[word]] text = "señora" -salutation = { gender = "f", excluded = { after_given = "a given name (US SSA, as Senora), borne as a middle name" } } -given = { us_ssa = 1065, as_of = 2026-10-24, note = "counted as SENORA, the SSA writing no diacritics" } +salutation = { gender = "f", excluded = { after_given = "a given name (US SSA, as Senora), borne as a middle name" }, note = "borne as a given name: US SSA 1,065 births (2026-10-24); counted as SENORA, the SSA writing no diacritics; kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } [[word]] text = "sra" -salutation = { gender = "f", same_as = "señora", after_given = true } -given = { us_ssa = 27, as_of = 2026-10-24 } +salutation = { gender = "f", same_as = "señora", after_given = true, note = "borne as a given name: US SSA 27 births (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { us_census_2010 = 149, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] @@ -30,8 +28,7 @@ salutation = { same_as = "señores", after_given = true } [[word]] text = "señorita" -salutation = { gender = "f", after_given = true } -given = { us_ssa = 6, as_of = 2026-10-24, note = "counted as SENORITA; kept for after_given (decisions/H8)" } +salutation = { gender = "f", after_given = true, note = "borne as a given name: US SSA 6 births (2026-10-24); counted as SENORITA; kept for after_given (decisions/H8); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } [[word]] text = "srta" diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 0852dadd..55ddf79f 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -81,6 +81,10 @@ office = {} text = "chargé" office = { expands = "chargé d'affaires, the head of a mission below an ambassador" } +[[word]] +text = "d'affaires" +office = { expands = "the second word of chargé d'affaires", issue = 752, note = "office rather than modifier since 2026-10-25: it stands behind its head, where decision 10's modifier rule asks for a head title after a modifier (rules/H9)" } + [[word]] text = "compositeur" occupation = {} @@ -89,10 +93,6 @@ occupation = {} text = "historien" occupation = {} -[[word]] -text = "d'affaires" -modifier = { expands = "the second word of chargé d'affaires" } - [[word]] text = "fils" generation = { issue = [679, 752], expands = "son, the Younger: Alexandre Dumas fils", note = "gated by the family reading (rules/S11): Title case leans to the name, so Marie Fils keeps her family name, and lower case is the generation wherever the name keeps a family word" } diff --git a/nameparser/vocab/hi.toml b/nameparser/vocab/hi.toml index 0a2146a8..acd5fa52 100644 --- a/nameparser/vocab/hi.toml +++ b/nameparser/vocab/hi.toml @@ -27,8 +27,7 @@ excluded = { salutation = { reason = "Begum is a trailing name element far more [[word]] text = "श्री" -salutation = { gender = "m", issue = 269, expands = "Shri, Mr.", note = "no Latin twin: Latin sri and shri are given names (their exclusions in hi.toml)", excluded = { after_given = "also an element of given names (Sri, Shri), so where it ends the given part it may be one" } } -given = { wikidata = 1, as_of = 2026-10-24 } +salutation = { gender = "m", issue = 269, expands = "Shri, Mr.", note = "no Latin twin: Latin sri and shri are given names (their exclusions in hi.toml); borne as a given name: Wikidata 1 people (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)", excluded = { after_given = "also an element of given names (Sri, Shri), so where it ends the given part it may be one" } } [[word]] text = "श्रीमती" diff --git a/nameparser/vocab/it.toml b/nameparser/vocab/it.toml index 50ad188f..598cb4c5 100644 --- a/nameparser/vocab/it.toml +++ b/nameparser/vocab/it.toml @@ -6,8 +6,7 @@ excluded = { salutation = { reason = "Sig is borne as a given name (US SSA 50+ m [[word]] text = "signora" -salutation = { gender = "f", after_given = true } -given = { us_ssa = 90, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } +salutation = { gender = "f", after_given = true, note = "borne as a given name: US SSA 90 births (2026-10-24); kept for after_given (decisions/H8); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } [[word]] text = "sig.ra" diff --git a/nameparser/vocab/sr.toml b/nameparser/vocab/sr.toml index 5c478449..106ea6e3 100644 --- a/nameparser/vocab/sr.toml +++ b/nameparser/vocab/sr.toml @@ -2,8 +2,7 @@ schema = 1 [[word]] text = "gospodin" -salutation = { issue = 606, gender = "m", after_given = true } -given = { note = "nam_dict ranks Gospodin a Bulgarian given name, band 5 (2026-10-24); kept for after_given (decisions/H8): a Bulgarian middle name is a patronymic, Gospodinov" } +salutation = { issue = 606, gender = "m", after_given = true, note = "borne as a given name; nam_dict ranks Gospodin a Bulgarian given name, band 5 (2026-10-24); kept for after_given (decisions/H8): a Bulgarian middle name is a patronymic, Gospodinov; kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } [[word]] text = "gospođa" diff --git a/nameparser/vocab/sw.toml b/nameparser/vocab/sw.toml index 4a461517..858d067a 100644 --- a/nameparser/vocab/sw.toml +++ b/nameparser/vocab/sw.toml @@ -2,6 +2,5 @@ schema = 1 [[word]] text = "bwana" -salutation = { gender = "m", expands = "Mr., sir", after_given = true } -given = { us_ssa = 5, as_of = 2026-10-24 } +salutation = { gender = "m", expands = "Mr., sir", after_given = true, note = "borne as a given name: US SSA 5 births (2026-10-24); kept here, not as a `given` reading, which would make the title one only inside a title phrase (rules/H9)" } family = { wikidata = 1, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } diff --git a/tests/v2/cases.py b/tests/v2/cases.py index f7aaa050..6701fa8a 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11860,6 +11860,116 @@ def _check_cjk_shape_purity(self) -> None: notes="the boundary of the row above, kept: #432 reads the " "period as the abbreviation mark it is, so 'V.' is a " "middle initial -- 'Smith, John V.' plus a title"), + Case("contested_title_alone_at_the_head", "Dean Smith", + {"given": "Dean", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: a title that may be a given name stands in no " + "title phrase, so it is the given name (decision 13 of " + "decisions/vocabulary-model), and the fork is reported"), + Case("contested_title_after_a_salutation", "Mr Dean Smith", + {"title": "Mr", "given": "Dean", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#853)", + notes="H9 and #853's rule: a salutation is never a phrase " + "partner, so the word after it is the given name"), + Case("contested_title_before_a_trailing_salutation", + "SMITH, DEAN MR.", + {"title": "MR.", "given": "DEAN", "family": "SMITH"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: no title AFTER the word is a partner unless " + "the word attaches to it -- FEC writes the salutation " + "after the given name, and 'DEAN MR.' read as a phrase " + "moved 99 FEC names in the design's spike"), + Case("contested_title_before_of", "Dean of Students Jane Smith", + {"title": "Dean of Students", "given": "Jane", "family": "Smith"}, + classification="feat(#752)", + notes="H9: 'of' after the word is its partner, the join " + "making the phrase one title"), + Case("contested_title_before_a_coordinating_connective", + "Dean and Mary Smith", + {"given": "Dean and Mary", "family": "Smith"}, + notes="H9: 'and' is no partner, so the word is a name " + "word and the join reads two given names, as before"), + Case("contested_title_after_a_modifier", "Associate Dean Smith", + {"title": "Associate Dean", "family": "Smith"}, + classification="feat(#752)", + notes="H9: a modifier before the word is its partner"), + Case("contested_title_attached_to_after", "Major General John Smith", + {"title": "Major General", "given": "John", "family": "Smith"}, + classification="feat(#752)", + notes="H9: major attaches to a military title after it"), + Case("contested_title_attached_to_before", + "Chief Justice John Roberts", + {"title": "Chief Justice", "given": "John", "family": "Roberts"}, + classification="feat(#752)", + notes="H9: chief attaches to an office, justice's kind; " + "until 2026-10-25 justice was excluded from the titles " + "and this read title 'Chief', given 'Justice'"), + Case("contested_title_after_a_style", "The Hon. Justice Mary Smith", + {"title": "The Hon. Justice", "given": "Mary", "family": "Smith"}, + classification="feat(#752)", + notes="H9: a style before the word is its partner"), + Case("contested_title_after_a_comma_in_capitals", "John Smith, DEAN", + {"given": "John", "family": "Smith", "suffix": "DEAN"}, + ambiguities=("suffix-or-name",), + notes="H9: outside a phrase the word reads as the " + "unlisted word it was, which rules/C6 reads as a " + "capitals credential here"), + Case("lone_modifier_at_the_head", "Wing Lau", + {"given": "Wing", "family": "Lau"}, classification="fix(#752)", + notes="H9, decision 10: a modifier is a title only with a " + "head title after it in its run; read title 'Wing', " + "family 'Lau' before (CLDR's conventions row reads given)"), + Case("modifier_before_its_head", "Wing Commander Lau", + {"title": "Wing Commander", "family": "Lau"}, + notes="H9's boundary: the head title follows"), + Case("modifier_after_a_salutation", "Mr Wing Lau", + {"title": "Mr", "given": "Wing", "family": "Lau"}, + classification="fix(#752)", + notes="H9: a head title before a modifier is not its " + "phrase; read title 'Mr Wing', family 'Lau' before"), + Case("modifier_in_a_connectives_object", + "President of the United States Barack Obama", + {"title": "President of the United States", "given": "Barack", + "family": "Obama"}, + notes="H9: 'united' and 'states' are modifiers with no " + "head after them, standing in the object of 'of the'"), + Case("modifier_before_of", "Mother Superior of the Convent Mary Smith", + {"title": "Mother Superior of the Convent", "given": "Mary", + "family": "Smith"}, + notes="H9: 'of' after a modifier is its phrase, as master read " + "it; dropping that partner in the first review round " + "broke this (the second review)"), + Case("modifier_in_the_object_of_the", "The Grand Lau", + {"title": "The Grand", "family": "Lau"}, + notes="H9: 'the' carries the object as 'of' does, as master " + "read it (the second review of #752 phase 2i)"), + Case("modifier_before_a_family_comma", "Wing Smith, John", + {"given": "John", "family": "Wing Smith"}, + classification="fix(#752)", + notes="H9 at the head of the name before a family comma: " + "H6's peel read title 'Wing', family 'Smith'"), + Case("modifier_in_a_comma_part", "John Smith, Vice", + {"title": "Vice", "given": "John", "family": "Smith"}, + notes="H9: after a comma a modifier reads as any title " + "does, the part being the person's role (decision 10 " + "measured 247 of 367 ctgov comma-part moves reversing the " + "name)"), + Case("contested_title_after_a_period_joined_title", "Lt.Gov. Dean Smith", + {"title": "Lt.Gov.", "given": "Dean", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9 reads only the contested and modifier words out of the " + "title: a period-joined title, which no wordlist holds " + "whole, keeps its tag (the code review of #752 phase 2i)"), + Case("modifier_after_a_salutation_pair", "Mr. and Mrs. Wing Lau", + {"title": "Mr. and Mrs.", "given": "Wing", "family": "Lau"}, + classification="fix(#752)", + notes="H9: only 'of' opens a phrase's object; a coordinating " + "connective between two salutations does not, so the " + "modifier after them has no head (the code review)"), + Case("role_word_in_a_comma_part", "John Smith, Resident", + {"title": "Resident", "given": "John", "family": "Smith"}, + notes="resident is an occupation since 2026-10-25, not a " + "modifier: a part written 'Resident' is the role"), Case("bare_salutation_ending_a_family_comma_given_part", "Doe, Jane Mrs", {"title": "Mrs", "given": "Jane", "family": "Doe"}, diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 45537ca3..ed26ef8c 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -541,7 +541,15 @@ def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: on = Policy(unlisted_caps_suffixes=CapsSuffixes.EVERYWHERE) base = Lexicon.default() assert caps_shape_candidate("ZZQX", base, on, False) + # H9: two fields are left out on purpose -- a title that may + # be a given name reads, outside a phrase, as the unlisted word it + # is there, and a phrase is two words + for field in ("titles_ambiguous", "title_phrases"): + assert caps_shape_candidate( + "ZZQX", base.add(**{field: {"zzqx"}}), on, False), field for field in _VOCAB_FIELDS: + if field in ("titles_ambiguous", "title_phrases"): + continue extra: dict[str, set[str]] = {field: {"zzqx"}} if field in ("honorific_tails", "suffix_words_ambiguous"): # honorific_tails ⊆ suffix_words is a Lexicon invariant, @@ -1108,3 +1116,44 @@ def test_a_comma_part_reads_a_gated_word_by_its_cue() -> None: assert is_wholly_suffix(["fils"], lex, pol, one_case=False) assert not is_wholly_suffix(["Fils"], lex, pol, one_case=False) assert not is_wholly_suffix(["Neto"], lex, pol, one_case=False) + + +# H9: "the word is a title only inside a title phrase" -- each +# partner, and each non-partner, alone +@pytest.mark.parametrize("words, head, expected", [ + (["dean", "smith"], True, [False, False]), + (["mr", "dean", "smith"], True, [True, False, False]), + (["associate", "dean", "smith"], True, [True, True, False]), + (["hon", "justice", "smith"], True, [True, True, False]), + (["chief", "justice", "smith"], True, [True, True, False]), + (["major", "general", "smith"], True, [True, True, False]), + (["dean", "of", "students"], True, [True, False, False]), + (["dean", "and", "mary"], True, [False, False, False]), + # a title after the word it does not attach to is no partner + (["dean", "mr"], False, [False, True]), + (["dean", "general", "smith"], True, [False, True, False]), + # decision 10: a modifier with a head title after it, or in a + # connective's object; at the head only + (["wing", "lau"], True, [False, False]), + (["wing", "lau"], False, [True, False]), + (["wing", "commander", "lau"], True, [True, True, False]), + (["mr", "wing", "lau"], True, [True, False, False]), + (["vice", "dean", "smith"], True, [True, True, False]), + (["president", "of", "the", "united", "states", "barack"], True, + [True, False, False, True, True, False]), + (["wing", "insp.", "lau"], True, [True, False, False]), +]) +def test_title_readings(words: list[str], head: bool, + expected: list[bool]) -> None: + from nameparser._pipeline._vocab import title_readings + folded = [_normalize(w) for w in words] + assert title_readings(words, folded, Lexicon.default(), + head=head) == expected + + +def test_a_word_a_caller_lists_as_a_title_is_one_everywhere() -> None: + # the 1.4-era recipe for a title the default leaves out: listing it + # makes it a title wherever it stands, phrase or none + lex = Lexicon.default().add(titles={"dean"}) + name = Parser(lexicon=lex).parse("Dean Smith") + assert (name.title, name.family, name.ambiguities) == ("Dean", "Smith", ()) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 94d447af..a84e7218 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -22,9 +22,10 @@ "H3": {"words": 142, "then_words": 72, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 113}, "H4": {"words": 390, "then_words": 183, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 1, "accepted_words": 127}, "H5": {"words": 757, "then_words": 289, "because_sentences": 2, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 117}, - "H6": {"words": 280, "then_words": 79, "because_sentences": 3, "except": 3, "precedence": 0, "accepted": 2, "accepted_words": 100}, + "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, + "H9": {"words": 457, "then_words": 179, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 4, "accepted_words": 132}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 217fa2f8..0c7fc308 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -250,9 +250,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 1529 -> 1568 the same day, #891 (phase 2e) merged over it: the # split-credential rows, two run rows and the particle-chain row # (9), the split credential's own report on nine rows (27) and the - # opening-credential row (3), each claim under the three orders. - assert claims == 1568, ( - f"the field sweep checks {claims} claims, recorded as 1568 on " + # opening-credential row (3), each claim under the three orders; + # 1568 -> 1583 the same day: #752's title phrase rows (phase 2i), the + # five that report a title read as a name, under the three orders. + assert claims == 1583, ( + f"the field sweep checks {claims} claims, recorded as 1583 on " f"2026-10-25") diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py index 35ae04ab..09bd9344 100644 --- a/tests/v2/test_config_shim.py +++ b/tests/v2/test_config_shim.py @@ -1311,3 +1311,17 @@ def test_a_first_name_title_lexicon_folds_to_nothing_is_dropped_with_the_warning lexicon, _, _ = c._snapshot() assert "" not in lexicon.given_name_titles assert HumanName("john smith", constants=c).last == "smith" + + +def test_snapshot_cleared_titles_turn_the_title_phrase_off() -> None: + # rules/H9's titles_ambiguous holds words no v1 set holds, so the + # one v1 way to turn title detection off -- emptying `titles` -- has + # to drop them too, or 'Dean of Students' stays a title where 1.4 + # read none (the design-docs review of #752 phase 2i) + c = Constants() + assert HumanName("Dean of Students Jane Smith", constants=c).title == ( + "Dean of Students") # baseline + c.titles.clear() + lexicon, _, _ = c._snapshot() + assert lexicon.titles_ambiguous == frozenset() + assert HumanName("Dean of Students Jane Smith", constants=c).title == "" diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 1b39015c..018100fe 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -805,9 +805,10 @@ def test_every_lexicon_entry_point_rejects_a_buffer_with_a_decode_hint( def test_only_the_phrase_fields_are_exempt_from_the_warning() -> None: # The exemption is exactly the kind of edit that silently widens: # one more field in _PHRASE_FIELDS costs nothing at construction and - # turns a dead entry back into a silent no-op. Two fields, named. - assert _PHRASE_FIELDS == ("given_name_titles", "maiden_markers") - assert len(_PER_WORD_FIELDS) == len(_VOCAB_FIELDS) - 2 + # turns a dead entry back into a silent no-op. Three fields, named. + assert _PHRASE_FIELDS == ("given_name_titles", "maiden_markers", + "title_phrases") + assert len(_PER_WORD_FIELDS) == len(_VOCAB_FIELDS) - 3 @pytest.mark.parametrize("field", _PER_WORD_FIELDS) @@ -1063,9 +1064,10 @@ def test_a_shipped_phrase_entry_is_stored_as_written(field: str) -> None: Over _PHRASE_FIELDS rather than over maiden_markers, so a third phrase field arrives already pinned.""" from nameparser.config.maiden_markers import MAIDEN_MARKERS - from nameparser.config.titles import GIVEN_NAME_TITLES + from nameparser.config.titles import GIVEN_NAME_TITLES, TITLE_PHRASES shipped = {"maiden_markers": MAIDEN_MARKERS, - "given_name_titles": GIVEN_NAME_TITLES}[field] + "given_name_titles": GIVEN_NAME_TITLES, + "title_phrases": TITLE_PHRASES}[field] assert getattr(Lexicon.default(), field) == frozenset(shipped), ( f"{field}: Lexicon rewrote the shipped constant, so the entries " f"it stores are not the ones the data module wrote -- " diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 69113a00..814aea3e 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -90,6 +90,9 @@ def test_the_compiled_sets_are_the_default_lexicon(shipped: dict[str, str]) -> N after_given_titles=c["AFTER_GIVEN_TITLES"], suffix_words_ambiguous=c["SUFFIX_WORDS_AMBIGUOUS"], agnomens=c["AGNOMENS"], + titles_ambiguous=c["TITLES_AMBIGUOUS"], + title_modifiers=c["TITLE_MODIFIERS"], title_styles=c["TITLE_STYLES"], + title_phrases=c["TITLE_PHRASES"], capitalization_exceptions=tuple(c["CAPITALIZATION_EXCEPTIONS"].items()), ) assert rebuilt == lex @@ -277,7 +280,13 @@ def _w(body: str) -> str: # writing; and the four names phase 2 left reserved say which phase ("default-true", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = true }')}, "write default = false"), ("default-not-a-bool", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = "no" }')}, "write default = false"), - ("reserved-attaches", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military"] }')}, "reserved for phase 2"), + ("attaches-not-a-list", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = "military" }')}, "attaches must be a non-empty list of title kinds"), + ("attaches-empty", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = [] }')}, "attaches must be a non-empty list of title kinds"), + ("attaches-twice", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military", "military"] }')}, "attaches must be a non-empty list of title kinds"), + ("attaches-a-suffix-kind", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["credential"] }')}, "attaches names 'credential', which is not a title kind"), + ("attaches-the-generic-kind", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["title"] }')}, "attaches names 'title', which is not a title kind"), + ("attaches-on-a-suffix", {"en.toml": _w('[[word]]\ntext = "phd"\ncredential = { attaches = ["office"] }')}, "unknown key 'attaches'"), + ("attaches-differs-between-files", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military"] }'), "fr.toml": _w('[[word]]\ntext = "chief"\noffice = {}')}, "attaches"), ("reserved-separators", {"th.toml": _w('[[word]]\ntext = "นาย"\nsalutation = { separators = [] }')}, "reserved for phase 4"), # 11, the agnomen's half: an agnomen is gated by its name reading, # so one without would take the family name from two-word names @@ -885,7 +894,8 @@ def _code(cell: str) -> list[str]: return re.findall(r"`([^`]+)`", cell) -def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: +def _sets_fed(kind: str, props: str = "{}", text: str = "zzq", + extra: str = "") -> set[str]: """The compiled constants a one-word file holding a `kind` reading with `props` puts the word into. A particle that does not state may_be_given gets a given reading, as check 11 requires, so the bare @@ -899,11 +909,14 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: entry = 'given = { note = "probe" }\n' + entry if kind == "agnomen": entry = 'family = { note = "probe" }\n' + entry - files = {"en.toml": _w(f'[[word]]\ntext = "{text}"\n{entry}')} + files = {"en.toml": _w(f'[[word]]\ntext = "{text}"\n{entry}{extra}')} with warnings.catch_warnings(): warnings.simplefilter("ignore", build.VocabWarning) # a name reading is inert compiled = build.compile_files(files) - return {name for name, value in compiled.items() if text in value} + # a word is in a set of phrases where it opens one (TITLE_PHRASES) + return {name for name, value in compiled.items() + if text in value or any(isinstance(v, str) and v.startswith(text + " ") + for v in value)} #: For each behavioral property, a reading that sets it away from its @@ -917,16 +930,24 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: "cased": ("title", '{ cased = "ZzQ" }', "zzq"), "after_given": ("salutation", "{ after_given = true }", "zzq"), "default": ("generation", "{ default = false }", "zzq"), + # a phrase needs the head it attaches to: a second title word + "attaches": ("office", '{ attaches = ["occupation"] }', "zzq", + '\n\n[[word]]\ntext = "zzr"\noccupation = {}'), } +#: The title kinds that feed a set of their own besides TITLES (H9). +_TITLE_KIND_SETS = {"modifier": ("TITLE_MODIFIERS",), "style": ("TITLE_STYLES",)} + + @pytest.mark.parametrize("kind", [k for k, c in build.CATEGORY.items() if c == "title"]) def test_every_title_kind_compiles_its_behavioral_properties(kind: str) -> None: # GIVEN_NAME_TITLES is compiled by category, so a classified title # keeps addresses (#750): keyed on the generic kind, a nobility sir # would lose it; and every title kind is allowed `cased`, which the # compiler reads from any reading - assert _sets_fed(kind, '{ addresses = "given" }') == {"TITLES", "GIVEN_NAME_TITLES"} + assert _sets_fed(kind, '{ addresses = "given" }') == { + "TITLES", "GIVEN_NAME_TITLES", *_TITLE_KIND_SETS.get(kind, ())} files = {"en.toml": _w(f'[[word]]\ntext = "zzq"\n{kind} = {{ cased = "ZzQ" }}')} assert build.compile_files(files)["CAPITALIZATION_EXCEPTIONS"] == {"zzq": "ZzQ"} @@ -961,8 +982,9 @@ def test_the_readme_lists_every_property_with_its_type_and_categories() -> None: def test_the_readme_states_each_behavioral_propertys_effect() -> None: rows = {_code(r[0])[0]: r[4] for r in _readme_table("## Properties") if r[1] == "behavioral"} assert set(_PROPERTY_PROBES) == set(rows) - for prop, (kind, props, text) in _PROPERTY_PROBES.items(): - moved = _sets_fed(kind, props, text) ^ _sets_fed(kind, "{}", text) + for prop, (kind, props, text, *extra) in _PROPERTY_PROBES.items(): + moved = (_sets_fed(kind, props, text, *extra) + ^ _sets_fed(kind, "{}", text, *extra)) assert moved, prop # a probe that moves nothing tests nothing assert set(_code(rows[prop])) & {n for n, _ in build.CONSTANTS} == moved, prop @@ -994,6 +1016,7 @@ def test_the_readme_states_each_behavioral_propertys_unwritten_value() -> None: "possibly a name": ("credential", "{}", "{}"), "possibly a name, a word": ("generation", "{}", "{}"), "possibly given": ("preposition", "{ may_be_given = false }", "{}"), + "possibly a name, a title": ("office", "{}", "{}"), } @@ -1026,3 +1049,59 @@ def test_the_readme_lists_the_reserved_names_with_their_phases() -> None: rows = _readme_table("## Names reserved for later phases") assert {_code(r[0])[0]: int(r[1]) for r in rows} == build.RESERVED assert len(rows) == len(build.RESERVED) + + +# H9: "the word is a title only inside a title phrase" -- the +# fourth derived marker and the three sets +# the phrase test reads, compiled from the readings +def test_a_title_with_a_given_reading_compiles_out_of_titles() -> None: + c = _compile({"und.toml": _w( + '[[word]]\ntext = "zzq"\noffice = {}\ngiven = { note = "probe" }')}) + assert "zzq" in c["TITLES_AMBIGUOUS"] + # a word in TITLES is a title wherever it stands, so the marked word + # is not there: a caller listing it there asks for that + assert "zzq" not in c["TITLES"] + + +def test_a_title_particle_with_a_given_reading_stays_a_title() -> None: + # 'st' and 'freiherr': the given reading derives the particle's + # marker, and the particle rules read the word at the front (P4) + c = _compile({"und.toml": _w( + '[[word]]\ntext = "zzq"\ngiven = { note = "probe" }\nreligious = {}\n' + 'saint = {}')}) + assert "zzq" in c["TITLES"] and "zzq" not in c["TITLES_AMBIGUOUS"] + + +def test_a_family_reading_beside_a_title_marks_nothing() -> None: + c = _compile({"und.toml": _w( + '[[word]]\ntext = "zzq"\nsalutation = {}\nfamily = { note = "probe" }')}) + assert "zzq" in c["TITLES"] and "zzq" not in c["TITLES_AMBIGUOUS"] + + +def test_a_modifier_is_one_only_where_every_title_reading_is() -> None: + one = _compile({"en.toml": _w('[[word]]\ntext = "zzq"\nmodifier = {}')}) + both = _compile({ + "en.toml": _w('[[word]]\ntext = "zzq"\nmodifier = {}'), + "fr.toml": _w('[[word]]\ntext = "zzq"\noffice = {}')}) + assert "zzq" in one["TITLE_MODIFIERS"] + # an office in another language heads a phrase of its own + assert "zzq" not in both["TITLE_MODIFIERS"] and "zzq" in both["TITLES"] + + +def test_attaches_compiles_to_a_phrase_with_each_title_of_its_kinds() -> None: + c = _compile({"en.toml": _w( + '[[word]]\ntext = "zzq"\nnobility = { attaches = ["military"] }\n\n' + '[[word]]\ntext = "zzr"\nmilitary = {}\n\n' + '[[word]]\ntext = "zzs"\noffice = {}')}) + assert c["TITLE_PHRASES"] == frozenset({"zzq zzr"}) + + +def test_a_shipped_phrase_names_two_titles() -> None: + # every phrase the build writes is two words, each a title or a + # title that may be a given name, so the phrase test can reach it + lex = Lexicon.default() + for phrase in lex.title_phrases: + words = phrase.split(" ") + assert len(words) == 2, phrase + assert all(w in lex.titles or w in lex.titles_ambiguous + for w in words), phrase diff --git a/tools/differential/conventions_agreement.json b/tools/differential/conventions_agreement.json index 70227219..1cec27cd 100644 --- a/tools/differential/conventions_agreement.json +++ b/tools/differential/conventions_agreement.json @@ -268,16 +268,5 @@ "got": "Van" }, "text": "Van der Walt" - }, - "conventions/Wing Lau/default": { - "given_names": { - "expected": "Wing", - "got": "" - }, - "text": "Wing Lau", - "title": { - "expected": "", - "got": "Wing" - } } } diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index effee77b..d9f43765 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -54,6 +54,7 @@ "Antonio Pavlović mlađi" "Antonio Pavlović mlađi Jones" "Antonio ‌Negri" +"Associate Dean Smith" "Asst. Vice Chancellor John Smith" "Attorney General of Minnesota" "Bahadur, Ram B.K." @@ -74,6 +75,8 @@ "Carod i" "Carod i Rovira, Josep" "Carod y de Rovira i" +"Chief Dean and Mary Smith" +"Chief Justice John Roberts" "DOE, JANE MA VD PHD" "DOE, JANE PHD VD" "DOE, JANE PHD VD MA" @@ -84,6 +87,12 @@ "De La Cruz, Ed" "De La Cruz, M.J. K.L." "De La Cruz, M.J. PhD" +"Dean" +"Dean Smith" +"Dean Smith, John" +"Dean and Mary Smith" +"Dean of Students Jane Smith" +"Dean. Smith" "Del Toro" "Doe Jane Mrs" "Doe nee Smith Jr. Prof., Jane" @@ -164,6 +173,7 @@ "Eric H. Holder, Jr., Secretary of State" "Esq. Smith" "É. y É. Smith" +"Field Marshal Bernard Montgomery" "Freiherr St John Smith" "Freiherr von Berg MA" "Freiherr von Berg, Ed" @@ -186,8 +196,10 @@ "García Márquez, MJ JK" "García Márquez, MJ PhD" "García Márquez, Ms G.J." +"General Major John Smith" "Georg VI." "Graf Lambsdorff, Otto" +"Grand Duke of Tuscany" "Greve Anna" "Greve Smith, Anna" "Greve, Anna" @@ -197,6 +209,7 @@ "Her Majesty Queen Elizabeth" "Her Royal Highness Princess Anne" "His Excellency Lord Duncan" +"His Grace John Smith" "II Van Johnson" "IVOI, PAUL D'" "Ivoi, Paul d'" @@ -279,6 +292,7 @@ "Jane Smith (née PhD)" "Jane Smith (née V)" "Jane Smith (née)" +"Jane Smith Vice." "Jane Smith née Jones" "Jane Smith née Jones PhD" "Jane Smith née PhD" @@ -303,6 +317,7 @@ "Jean Pierre DUPONT-MORETTI" "Jiří Dienstbier ml." "John . Smith" +"John Dean Smith" "John Doctor Smith" "John III" "John III, Dr." @@ -449,6 +464,7 @@ "John Smith, RA Jones" "John Smith, V." "John Smith, V; MD" +"John Smith, Vice" "John Smith, X.Y. P.Q." "John Smith, X.Y.Z." "John Smith, XYZ" @@ -522,12 +538,15 @@ "King Smith, John" "Konstantin VII." "LT.GOV. JOHN DOE" +"Lau, Wing" "Lloret Palol II, JA" "Lloret Palol II, Josep Antoni" "Lord Chancellor" "López, Ana García" "MSc Dr. med. univ." "Mai Le née Nguyen" +"Major General John Smith" +"Major Smith" "Malcolm X" "Mari' Aube'" "Maria Kowalska (z domu Nowak)" @@ -550,9 +569,15 @@ "Mesnil de" "Mohamed Ali Abd Allah" "Morse, Det. Insp. Jane" +"Mother Superior of the Convent Mary Smith" +"Mr Dean Smith" +"Mr Justice Smith" +"Mr Smith" +"Mr Wing Lau" "Mr. Jack Jill" "Mr. Jack and Jill" "Mr. Johnson" +"Mr. and Mrs. Wing Lau" "Mrs. Garcia" "MÜLLER WEIß, HANS" "Ménil Christophe de" @@ -595,8 +620,10 @@ "Ph. D., John" "Pharm D John Smith" "Pieter Bruegel stariji Jones" +"President of the United States Barack Obama" "Prince of Wales" "Prince of Wales Jr" +"Princess Royal Anne" "Prof. Dr. Brühl, Ada" "Prof. Smith, Dr. John" "Prof. Smith, John, PhD" @@ -620,6 +647,7 @@ "SIG.RA MARIA ROSSI" "SMITH VD MA, JOHN" "SMITH VD, JOHN" +"SMITH, DEAN MR." "SMITH, JOHN DO" "SMITH, SAM E." "SMITH, VD DE LA MA" @@ -742,6 +770,7 @@ "Stariji Bruegel" "Steven Hardman, MD, DO, DDS" "Steven Hardman, RN - CRNA" +"The Hon. Justice Mary Smith" "The Right Hon. the President of the Queen's Bench Division" "VI, AD" "Van Buren, Ed" @@ -751,6 +780,10 @@ "Wang M.Eng." "Wang Ma PhD" "Wilhelm I." +"Wing Commander Lau" +"Wing Lau" +"Wing Smith, John" +"Wing. Lau" "Xyz. John Smith" "Xyz. Smith, John" "Xyz. van Johnson" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 4074a4bf..7fabff5a 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -274,8 +274,9 @@ {"name": "Asmar", "given": "Asmar", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "A."} {"name": "Asmar", "order": "FAMILY_FIRST", "family": "Asmar", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "A."} {"name": "Assad, Hafis al-", "given": "Hafis", "family": "al- Assad", "initials": "H. A."} -{"name": "Assoc Dean", "title": "Assoc", "family": "Dean", "initials": "D."} -{"name": "Assoc Dean of Chemistry Robert Johns", "title": "Assoc", "given": "Dean of Chemistry", "middle": "Robert", "family": "Johns", "initials": "D. C. R. J."} +{"name": "Assoc Dean", "title": "Assoc", "family": "Dean", "ambiguities": ["TITLE_OR_NAME"], "initials": "D."} +{"name": "Assoc Dean of Chemistry Robert Johns", "title": "Assoc Dean of Chemistry", "given": "Robert", "family": "Johns", "initials": "R. J."} +{"name": "Associate Dean Smith", "title": "Associate Dean", "family": "Smith", "initials": "S."} {"name": "Asst. Vice Chancellor John Smith", "title": "Asst. Vice Chancellor", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Ataíde, Luís de", "given": "Luís", "family": "de Ataíde", "initials": "L. A."} {"name": "Attorney General of Minnesota", "title": "Attorney", "family": "General of Minnesota", "ambiguities": ["TITLE_OR_NAME"], "initials": "G. M."} @@ -502,8 +503,10 @@ {"name": "Chattel, Frederik Jacobus van Rossum", "given": "Frederik", "middle": "Jacobus van Rossum", "family": "Chattel", "initials": "F. J. R. C."} {"name": "Chemistry Jane Smith", "given": "Chemistry", "middle": "Jane", "family": "Smith", "initials": "C. J. S."} {"name": "Cherice J. (Johnson) Williams", "given": "Cherice", "middle": "J.", "family": "Williams", "nickname": "Johnson", "initials": "C. J. W."} +{"name": "Chief Dean and Mary Smith", "title": "Chief Dean and Mary", "family": "Smith", "initials": "S."} {"name": "Chief Judge J. Leon Holmes", "title": "Chief Judge", "given": "J.", "middle": "Leon", "family": "Holmes", "initials": "J. L. H."} {"name": "Chief Judge Sharon Lovelace Blackburn", "title": "Chief Judge", "given": "Sharon", "middle": "Lovelace", "family": "Blackburn", "initials": "S. L. B."} +{"name": "Chief Justice John Roberts", "title": "Chief Justice", "given": "John", "family": "Roberts", "initials": "J. R."} {"name": "Chief Keef", "title": "Chief", "family": "Keef", "initials": "K."} {"name": "Chikamatsu, Monzaemon", "given": "Monzaemon", "family": "Chikamatsu", "initials": "M. C."} {"name": "Christian Ulrich II.", "given": "Christian", "family": "Ulrich", "suffix": "II.", "initials": "C. U."} @@ -578,14 +581,19 @@ {"name": "De La Cruz, Ed", "given": "Ed", "family": "De La Cruz", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "E. C.", "capitalized_force": "Ed de la Cruz"} {"name": "De La Cruz, M.J. K.L.", "given": "M.J.", "middle": "K.L.", "family": "De La Cruz", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. K. C.", "capitalized_force": "M.J. K.L. de la Cruz"} {"name": "De La Cruz, M.J. PhD", "given": "M.J.", "family": "De La Cruz", "suffix": "PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. C.", "capitalized_force": "M.J. de la Cruz PhD"} -{"name": "Dean Koontz", "given": "Dean", "family": "Koontz", "initials": "D. K."} -{"name": "Dean Martin", "given": "Dean", "family": "Martin", "initials": "D. M."} -{"name": "Dean Ms Hon Solo", "given": "Dean", "middle": "Ms Hon", "family": "Solo", "initials": "D. M. H. S."} -{"name": "Dean Robert Johns", "given": "Dean", "middle": "Robert", "family": "Johns", "initials": "D. R. J."} -{"name": "Dean Smith", "given": "Dean", "family": "Smith", "initials": "D. S."} -{"name": "Dean of Chemistry", "given": "Dean of Chemistry", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "D. C."} -{"name": "Dean of Chemistry Robert Johns", "given": "Dean of Chemistry", "middle": "Robert", "family": "Johns", "initials": "D. C. R. J."} +{"name": "Dean", "given": "Dean", "ambiguities": ["GIVEN_OR_FAMILY", "TITLE_OR_NAME"], "initials": "D."} +{"name": "Dean Koontz", "given": "Dean", "family": "Koontz", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. K."} +{"name": "Dean Martin", "given": "Dean", "family": "Martin", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. M."} +{"name": "Dean Ms Hon Solo", "given": "Dean", "middle": "Ms Hon", "family": "Solo", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. M. H. S."} +{"name": "Dean Robert Johns", "given": "Dean", "middle": "Robert", "family": "Johns", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. R. J."} +{"name": "Dean Smith", "given": "Dean", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S."} +{"name": "Dean Smith, John", "given": "John", "family": "Dean Smith", "initials": "J. D. S."} +{"name": "Dean and Mary Smith", "given": "Dean and Mary", "family": "Smith", "initials": "D. M. S."} +{"name": "Dean of Chemistry", "title": "Dean of Chemistry"} +{"name": "Dean of Chemistry Robert Johns", "title": "Dean of Chemistry", "given": "Robert", "family": "Johns", "initials": "R. J."} +{"name": "Dean of Students Jane Smith", "title": "Dean of Students", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Dean. Robert Johns", "title": "Dean.", "given": "Robert", "family": "Johns", "initials": "R. J."} +{"name": "Dean. Smith", "title": "Dean.", "family": "Smith", "initials": "S."} {"name": "Del Toro", "given": "Del", "family": "Toro", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. T."} {"name": "Del toro", "given": "Del", "family": "toro", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. t.", "capitalized_force": "Del Toro"} {"name": "Della Reese", "given": "Della", "family": "Reese", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. R."} @@ -950,6 +958,7 @@ {"name": "Ferrer Rosa Maria Ferran", "order": "FAMILY_FIRST", "given": "Rosa", "middle": "Maria Ferran", "family": "Ferrer", "initials": "R. M. F. F."} {"name": "Ferrer, Rosa Maria Ferran", "given": "Rosa", "middle": "Maria Ferran", "family": "Ferrer", "initials": "R. M. F. F."} {"name": "Fidus", "given": "Fidus", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "F."} +{"name": "Field Marshal Bernard Montgomery", "given": "Field", "middle": "Marshal Bernard", "family": "Montgomery", "initials": "F. M. B. M."} {"name": "Fien de la Mar", "given": "Fien", "family": "de la Mar", "initials": "F. M."} {"name": "Figueiredo, Jackson de", "given": "Jackson", "family": "de Figueiredo", "initials": "J. F."} {"name": "Finnur Jónsson", "given": "Finnur", "family": "Jónsson", "initials": "F. J."} @@ -1062,6 +1071,7 @@ {"name": "Geir Jón Magnús Brynjar Skeggjason, þingmaður", "given": "þingmaður", "family": "Geir Jón Magnús Brynjar Skeggjason", "initials": "þ. G. J. M. B. S.", "capitalized_force": "Þingmaður Geir Jón Magnús Brynjar Skeggjason"} {"name": "Geleen und Amsterad, Gottfried Huyn von", "given": "Gottfried", "middle": "Huyn", "family": "von Geleen und Amsterad", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "G. H. G. A."} {"name": "Gemmingen, Pleikard Dietrich von", "given": "Pleikard", "middle": "Dietrich", "family": "von Gemmingen", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "P. D. G."} +{"name": "General Major John Smith", "title": "General", "given": "Major", "middle": "John", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "M. J. S."} {"name": "General of Minnesota", "title": "General of Minnesota"} {"name": "Geng, Biao", "given": "Biao", "family": "Geng", "initials": "B. G."} {"name": "Georg VI.", "given": "Georg", "suffix": "VI.", "ambiguities": ["GIVEN_OR_FAMILY", "SUFFIX_OR_NAME"], "initials": "G."} @@ -1099,7 +1109,8 @@ {"name": "Graf Lambsdorff, Otto", "title": "Graf", "given": "Otto", "family": "Lambsdorff", "initials": "O. L."} {"name": "Gragger, Robert", "given": "Robert", "family": "Gragger", "initials": "R. G."} {"name": "Gramont, Antoine de", "given": "Antoine", "family": "de Gramont", "initials": "A. G."} -{"name": "Grand Rounds", "title": "Grand", "family": "Rounds", "initials": "R."} +{"name": "Grand Duke of Tuscany", "given": "Grand", "family": "Duke of Tuscany", "initials": "G. D. T."} +{"name": "Grand Rounds", "given": "Grand", "family": "Rounds", "initials": "G. R."} {"name": "Grandmaison, Louis de", "given": "Louis", "family": "de Grandmaison", "initials": "L. G."} {"name": "Granger Hermione", "order": "FAMILY_FIRST", "given": "Hermione", "family": "Granger", "initials": "H. G."} {"name": "Granger, Hermione", "given": "Hermione", "family": "Granger", "initials": "H. G."} @@ -1217,6 +1228,7 @@ {"name": "Hildebrand, Dietrich von", "given": "Dietrich", "family": "von Hildebrand", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. H."} {"name": "Hildebrand, Rudolf Friedrich Leopold von", "given": "Rudolf", "middle": "Friedrich Leopold", "family": "von Hildebrand", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "R. F. L. H."} {"name": "His Excellency Lord Duncan", "title": "His Excellency Lord", "family": "Duncan", "initials": "D."} +{"name": "His Grace John Smith", "given": "His", "middle": "Grace John", "family": "Smith", "initials": "H. G. J. S."} {"name": "His Holiness", "title": "His", "family": "Holiness", "ambiguities": ["TITLE_OR_NAME"], "initials": "H."} {"name": "His Holiness the Dalai Lama", "title": "His Holiness the Dalai", "family": "Lama", "ambiguities": ["TITLE_OR_NAME"], "initials": "L."} {"name": "Hisham Ibrahim, CPA, CFA", "given": "Hisham", "family": "Ibrahim", "suffix": "CPA, CFA", "initials": "H. I."} @@ -1468,6 +1480,7 @@ {"name": "Jane Smith (née PhD)", "given": "Jane", "family": "Smith", "maiden": "PhD", "initials": "J. S."} {"name": "Jane Smith (née V)", "given": "Jane", "family": "Smith", "maiden": "V", "initials": "J. S."} {"name": "Jane Smith (née)", "given": "Jane", "family": "Smith", "nickname": "née", "initials": "J. S.", "capitalized_force": "Jane \"Née\" Smith"} +{"name": "Jane Smith Vice.", "title": "Vice.", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Jane Smith born Jones", "given": "Jane", "middle": "Smith born", "family": "Jones", "initials": "J. S. b. J.", "capitalized_force": "Jane Smith Born Jones"} {"name": "Jane Smith née Jones", "given": "Jane", "family": "Smith", "maiden": "Jones", "initials": "J. S."} {"name": "Jane Smith née Jones PhD", "given": "Jane", "family": "Smith", "suffix": "PhD", "maiden": "Jones", "initials": "J. S."} @@ -1565,6 +1578,7 @@ {"name": "John A. Kenneth Doe, Jr.", "given": "John", "middle": "A. Kenneth", "family": "Doe", "suffix": "Jr.", "initials": "J. A. K. D."} {"name": "John An Smith", "given": "John", "middle": "An", "family": "Smith", "initials": "J. A. S."} {"name": "John Chinedu Amadi", "given": "John", "middle": "Chinedu", "family": "Amadi", "initials": "J. C. A."} +{"name": "John Dean Smith", "given": "John", "middle": "Dean", "family": "Smith", "initials": "J. D. S."} {"name": "John Doctor Smith", "given": "John", "middle": "Doctor", "family": "Smith", "initials": "J. D. S."} {"name": "John Doe", "given": "John", "family": "Doe", "initials": "J. D."} {"name": "John Doe III", "given": "John", "family": "Doe", "suffix": "III", "initials": "J. D."} @@ -1778,6 +1792,7 @@ {"name": "John Smith, V MD", "given": "John", "family": "Smith", "suffix": "V MD", "initials": "J. S."} {"name": "John Smith, V.", "given": "John", "family": "Smith", "suffix": "V.", "initials": "J. S."} {"name": "John Smith, V; MD", "given": "John", "family": "Smith", "suffix": "V; MD", "initials": "J. S."} +{"name": "John Smith, Vice", "title": "Vice", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith, X.Y. P.Q.", "given": "John", "family": "Smith", "suffix": "X.Y. P.Q.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, X.Y.Z.", "given": "John", "family": "Smith", "suffix": "X.Y.Z.", "initials": "J. S."} {"name": "John Smith, X.Y.Z. MA", "given": "John", "family": "Smith", "suffix": "X.Y.Z. MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -2138,6 +2153,7 @@ {"name": "Larry James Johnson I", "given": "Larry", "middle": "James", "family": "Johnson", "suffix": "I", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "L. J. J."} {"name": "Larry V I", "given": "Larry", "middle": "V", "family": "I", "initials": "L. V. I."} {"name": "Lastname, Firstname", "given": "Firstname", "family": "Lastname", "initials": "F. L."} +{"name": "Lau, Wing", "title": "Wing", "family": "Lau", "initials": "L."} {"name": "Laura Ángeles Lastra Santos", "given": "Laura", "middle": "Ángeles Lastra", "family": "Santos", "initials": "L. Á. L. S."} {"name": "Laureano Veres Acevedo", "given": "Laureano", "middle": "Veres", "family": "Acevedo", "initials": "L. V. A."} {"name": "Lavardin, Jean de Beaumanoir de", "given": "Jean", "middle": "de Beaumanoir", "family": "de Lavardin", "initials": "J. B. L."} @@ -2278,7 +2294,9 @@ {"name": "Maier, Amy I, Jr.", "given": "Amy", "middle": "I", "family": "Maier", "suffix": "Jr.", "initials": "A. I. M."} {"name": "Maier, Amy Lauren I", "given": "Amy", "middle": "Lauren", "family": "Maier", "suffix": "I", "initials": "A. L. M."} {"name": "Majlinda Kraja", "given": "Majlinda", "family": "Kraja", "initials": "M. K."} -{"name": "Major Harris", "given": "Major", "family": "Harris", "initials": "M. H."} +{"name": "Major General John Smith", "title": "Major General", "given": "John", "family": "Smith", "initials": "J. S."} +{"name": "Major Harris", "given": "Major", "family": "Harris", "ambiguities": ["TITLE_OR_NAME"], "initials": "M. H."} +{"name": "Major Smith", "given": "Major", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "M. S."} {"name": "Major. Dona Smith", "title": "Major.", "given": "Dona", "family": "Smith", "initials": "D. S."} {"name": "Major. John Smith, Jr.", "title": "Major.", "given": "John", "family": "Smith", "suffix": "Jr.", "initials": "J. S."} {"name": "Makonnen Haile Selassie", "given": "Makonnen", "middle": "Haile", "family": "Selassie", "initials": "M. H. S."} @@ -2481,10 +2499,15 @@ {"name": "Moreno Casamitjana, Antonio", "given": "Antonio", "family": "Moreno Casamitjana", "initials": "A. M. C."} {"name": "Morris, Robert", "given": "Robert", "family": "Morris", "initials": "R. M."} {"name": "Morse, Det. Insp. Jane", "title": "Det. Insp.", "given": "Jane", "family": "Morse", "initials": "J. M."} +{"name": "Mother Superior of the Convent Mary Smith", "title": "Mother Superior of the Convent", "given": "Mary", "family": "Smith", "initials": "M. S."} {"name": "Mozart Wolfgang Amadeus", "order": "FAMILY_FIRST", "given": "Wolfgang", "middle": "Amadeus", "family": "Mozart", "initials": "W. A. M."} {"name": "Mozart, Wolfgang Amadeus", "given": "Wolfgang", "middle": "Amadeus", "family": "Mozart", "initials": "W. A. M."} +{"name": "Mr Dean Smith", "title": "Mr", "given": "Dean", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S."} +{"name": "Mr Justice Smith", "title": "Mr", "given": "Justice", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "J. S."} {"name": "Mr MA", "title": "Mr", "family": "MA", "initials": "M.", "capitalized_force": "Mr Ma"} {"name": "Mr Md Karim Uddin", "title": "Mr", "given": "Md", "middle": "Karim", "family": "Uddin", "initials": "M. K. U."} +{"name": "Mr Smith", "title": "Mr", "family": "Smith", "initials": "S."} +{"name": "Mr Wing Lau", "title": "Mr", "given": "Wing", "family": "Lau", "initials": "W. L."} {"name": "Mr XXX", "title": "Mr", "family": "XXX", "initials": "X.", "capitalized_force": "Mr Xxx"} {"name": "Mr. & Mrs. John Smith", "title": "Mr. & Mrs.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Mr. Bertram Wilberforce Henry Robert Wooster Jr, MP", "title": "Mr.", "given": "Bertram", "middle": "Wilberforce Henry Robert", "family": "Wooster", "suffix": "Jr, MP", "initials": "B. W. H. R. W."} @@ -2496,6 +2519,7 @@ {"name": "Mr. Van Nguyen", "title": "Mr.", "given": "Van", "family": "Nguyen", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "V. N."} {"name": "Mr. and Mrs. John Smith", "title": "Mr. and Mrs.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Mr. and Mrs. John and Jane Smith", "title": "Mr. and Mrs.", "given": "John and Jane", "family": "Smith", "initials": "J. J. S."} +{"name": "Mr. and Mrs. Wing Lau", "title": "Mr. and Mrs.", "given": "Wing", "family": "Lau", "initials": "W. L."} {"name": "Mrs. Garcia", "title": "Mrs.", "family": "Garcia", "initials": "G."} {"name": "Ms A.J. Smith", "title": "Ms", "given": "A.J.", "family": "Smith", "initials": "A. S."} {"name": "Ms Hon Solo", "title": "Ms Hon", "family": "Solo", "initials": "S."} @@ -2728,10 +2752,12 @@ {"name": "Prats Gonzaléz, Carlos", "given": "Carlos", "family": "Prats Gonzaléz", "initials": "C. P. G."} {"name": "Prema", "given": "Prema", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "P."} {"name": "Prema", "order": "FAMILY_FIRST", "family": "Prema", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "P."} +{"name": "President of the United States Barack Obama", "title": "President of the United States", "given": "Barack", "family": "Obama", "initials": "B. O."} {"name": "Prince Charles", "title": "Prince", "given": "Charles", "initials": "C."} {"name": "Prince Fielder", "title": "Prince", "given": "Fielder", "initials": "F."} {"name": "Prince of Wales", "title": "Prince of Wales"} {"name": "Prince of Wales Jr", "title": "Prince of Wales", "family": "Jr", "initials": "J."} +{"name": "Princess Royal Anne", "title": "Princess", "given": "Royal", "family": "Anne", "initials": "R. A."} {"name": "Prof. Dk. Salima Farida mwana Hamisi Musa M.D. Ph.D.", "title": "Prof. Dk.", "given": "Salima", "middle": "Farida mwana Hamisi", "family": "Musa", "suffix": "M.D. Ph.D.", "initials": "S. F. m. H. M.", "capitalized_force": "Prof. Dk. Salima Farida Mwana Hamisi Musa M.D. Ph.D."} {"name": "Prof. Dr. Ada Cornelia Cesar Martin von Bruhl Jr, MD DDS", "title": "Prof. Dr.", "given": "Ada", "middle": "Cornelia Cesar Martin", "family": "von Bruhl", "suffix": "Jr, MD DDS", "initials": "A. C. C. M. B."} {"name": "Prof. Dr. Ada Cornelia Cesar Martín von Bruhl Jr, M.D., Ph.D.", "title": "Prof. Dr.", "given": "Ada", "middle": "Cornelia Cesar Martín", "family": "von Bruhl", "suffix": "Jr, M.D., Ph.D.", "initials": "A. C. C. M. B."} @@ -2885,6 +2911,7 @@ {"name": "SIG.RA MARIA ROSSI", "title": "SIG.RA", "given": "MARIA", "family": "ROSSI", "initials": "M. R.", "capitalized": "Sig.ra Maria Rossi"} {"name": "SMITH VD MA, JOHN", "given": "JOHN", "family": "SMITH VD MA", "initials": "J. S. M.", "capitalized": "John Smith vd Ma"} {"name": "SMITH VD, JOHN", "given": "JOHN", "family": "SMITH", "suffix": "VD", "initials": "J. S.", "capitalized": "John Smith VD"} +{"name": "SMITH, DEAN MR.", "title": "MR.", "given": "DEAN", "family": "SMITH", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S.", "capitalized": "Mr. Dean Smith"} {"name": "SMITH, JOHN DO", "given": "JOHN", "family": "DO SMITH", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. S.", "capitalized": "John do Smith"} {"name": "SMITH, SAM E.", "given": "SAM", "middle": "E.", "family": "SMITH", "initials": "S. E. S.", "capitalized": "Sam E. Smith"} {"name": "SMITH, VD DE LA MA", "family": "SMITH VD DE LA MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "S. M.", "capitalized": "Smith vd de la Ma"} @@ -3290,6 +3317,7 @@ {"name": "Tham Jun Hoe", "given": "Tham", "middle": "Jun", "family": "Hoe", "initials": "T. J. H."} {"name": "Tham Junhoe", "given": "Tham", "family": "Junhoe", "initials": "T. J."} {"name": "The Hon", "given": "The", "family": "Hon", "initials": "T. H.", "capitalized_force": "the Hon"} +{"name": "The Hon. Justice Mary Smith", "title": "The Hon. Justice", "given": "Mary", "family": "Smith", "initials": "M. S.", "capitalized_force": "the Hon. Justice Mary Smith"} {"name": "The Lord of the Universe", "title": "The Lord of the Universe", "capitalized_force": "the Lord of the Universe"} {"name": "The Right Hon. the President of the Queen's Bench Division", "title": "The Right Hon. the President of the Queen's Bench", "family": "Division", "ambiguities": ["TITLE_OR_NAME"], "initials": "D.", "capitalized_force": "the Right Hon. the President of the Queen'S Bench Division"} {"name": "The Rt Hon", "title": "The Rt", "family": "Hon", "ambiguities": ["TITLE_OR_NAME"], "initials": "H.", "capitalized_force": "the Rt Hon"} @@ -3457,7 +3485,10 @@ {"name": "Willemse, Bertus Wilbers Henri Retief", "given": "Bertus", "middle": "Wilbers Henri Retief", "family": "Willemse", "initials": "B. W. H. R. W."} {"name": "Willemse, Heidi Johannes", "given": "Heidi", "middle": "Johannes", "family": "Willemse", "initials": "H. J. W."} {"name": "Willington Baena Ortiz", "given": "Willington", "middle": "Baena", "family": "Ortiz", "initials": "W. B. O."} -{"name": "Wing Lau", "title": "Wing", "family": "Lau", "initials": "L."} +{"name": "Wing Commander Lau", "title": "Wing Commander", "family": "Lau", "initials": "L."} +{"name": "Wing Lau", "given": "Wing", "family": "Lau", "initials": "W. L."} +{"name": "Wing Smith, John", "given": "John", "family": "Wing Smith", "initials": "J. W. S."} +{"name": "Wing. Lau", "title": "Wing.", "family": "Lau", "initials": "L."} {"name": "Winghe, Jodocus van", "given": "Jodocus", "family": "van Winghe", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. W."} {"name": "Witte, Jean Joseph Antoine Marie de", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Wićazowa, Jěwa-Marja Hańža", "given": "Jěwa-Marja", "middle": "Hańža", "family": "Wićazowa", "initials": "J. H. W."} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 5047e01f..1b24a9ea 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -78,8 +78,8 @@ after_given only on a salutation reading (#835). No post-nominal is both an acronym and a word, by checks 4 and 5; a final check in compile_pool is the backstop. The subset relations -(GIVEN_NAME_TITLES and AFTER_GIVEN_TITLES in TITLES, each marker set in -its base) need no +(GIVEN_NAME_TITLES and AFTER_GIVEN_TITLES in TITLES and TITLES_AMBIGUOUS +together, each marker set in its base) need no check: a flag cannot exist without its reading, and a derived marker is read off the reading it marks. The one soft finding, a name reading on a word with no vocabulary reading and no exclusion, is @@ -224,7 +224,8 @@ def _alnum(text: str) -> str: #: Behavioral properties per category: each compiles into a field, and #: its allowed values are exactly what today's pipeline does. BEHAVIORAL: dict[str, frozenset[str]] = { - "title": frozenset({"addresses", "after_given", "cased", "default"}), + "title": frozenset({"addresses", "after_given", "attaches", "cased", + "default"}), "suffix": frozenset({"acronym", "attached", "cased", "default"}), "particle": frozenset({"cased", "default"}), "conjunction": frozenset({"may_be_initial", "cased", "default"}), @@ -259,12 +260,12 @@ def _alnum(text: str) -> str: ATTACHED_KEYS = frozenset({"side", "issue", "decided", "note"}) #: Names a later phase will give meaning to. Writing one now is an error #: that says so, rather than "unknown", so nothing reads as working. -#: Phase 2 gives `default` its meaning and leaves `attaches` to its -#: front-of-the-name work; the gate's slots carry position, and shape -#: settles a bare letter (rules/P7), so the position and bare-letter -#: names wait with `separators` for phase 4's shapes and glued words. +#: Phase 2 gave `default` and `attaches` their meaning; the gate's slots +#: carry position, and shape settles a bare letter (rules/P7), so the +#: position and bare-letter names wait with `separators` for phase 4's +#: shapes and glued words. RESERVED = { - "attaches": 2, "prefer": 3, "strict": 3, + "prefer": 3, "strict": 3, "bare_only": 4, "leading_only": 4, "trailing_only": 4, "separators": 4, } _TAG = re.compile(r"[a-z]{2,3}(-[A-Z][a-z]{3})?(-([A-Z]{2}|[0-9]{3}))?") @@ -798,6 +799,21 @@ def _check_reading(r: Reading, pool: Pool) -> None: f"for a later phase; phase 1 has 'family' and 'given'") if addresses not in ("family", "given"): raise VocabError(f"{where}: addresses must be 'family' or 'given'") + if "attaches" in p: + # the phrase partner: the title kinds whose words this one + # stands before in a title phrase (rules/H9) + att = p["attaches"] + if (not isinstance(att, list) or not att + or not all(isinstance(k, str) for k in att) + or len(set(att)) != len(att)): + raise VocabError(f"{where}: attaches must be a non-empty " + f"list of title kinds, each once") + for k in att: + if CATEGORY.get(k) != "title" or k == "title": + raise VocabError( + f"{where}: attaches names {k!r}, which is not a " + f"title kind; allowed: " + f"{sorted(t for t, c in CATEGORY.items() if c == 'title' and t != 'title')}") if r.flag("after_given") and r.kind != "salutation": # a salutation is a form of address, where an office or a # rank is borne as a middle name: the reason the given @@ -1092,6 +1108,9 @@ def _behavior(r: Reading, prop: str) -> object: return None if att is None else att.get("side") if prop == "may_be_initial" or prop == "after_given": return r.flag(prop) + if prop == "attaches": + att = r.props.get("attaches") + return None if att is None else frozenset(att) if prop == "default": return r.on_by_default return r.props.get(prop) @@ -1103,6 +1122,10 @@ def _behavior(r: Reading, prop: str) -> object: ("GIVEN_NAME_TITLES", "titles"), ("TITLES", "titles"), ("AFTER_GIVEN_TITLES", "titles"), + ("TITLES_AMBIGUOUS", "titles"), + ("TITLE_MODIFIERS", "titles"), + ("TITLE_STYLES", "titles"), + ("TITLE_PHRASES", "titles"), ("NON_GIVEN_NAME_PARTICLES", "particles"), ("PARTICLES", "particles"), ("CONJUNCTIONS", "conjunctions"), @@ -1125,6 +1148,10 @@ class Compiled(TypedDict): GIVEN_NAME_TITLES: frozenset[str] TITLES: frozenset[str] AFTER_GIVEN_TITLES: frozenset[str] + TITLES_AMBIGUOUS: frozenset[str] + TITLE_MODIFIERS: frozenset[str] + TITLE_STYLES: frozenset[str] + TITLE_PHRASES: frozenset[str] NON_GIVEN_NAME_PARTICLES: frozenset[str] PARTICLES: frozenset[str] CONJUNCTIONS: frozenset[str] @@ -1156,6 +1183,8 @@ def compile_pool(pool: Pool) -> Compiled: titles: set[str] = set() given_titles: set[str] = set() after_given: set[str] = set() + title_kinds: dict[str, set[str]] = {} + attaches: dict[str, set[str]] = {} particles: set[str] = set() never_given: set[str] = set() conjunctions: set[str] = set() @@ -1181,6 +1210,8 @@ def compile_pool(pool: Pool) -> Compiled: given_titles.add(t) if r.flag("after_given"): after_given.add(t) + title_kinds.setdefault(t, set()).add(r.kind) + attaches.setdefault(t, set()).update(p.get("attaches", ())) elif r.category == "suffix": if r.acronym: acronyms.add(t) @@ -1206,10 +1237,32 @@ def compile_pool(pool: Pool) -> Compiled: bound.add(t) elif r.category == "relation": markers.add(t) + contested = titles & given - particles out = Compiled( GIVEN_NAME_TITLES=frozenset(given_titles), - TITLES=frozenset(titles), + # a title that may be a given name compiles into TITLES_AMBIGUOUS + # INSTEAD: a word in TITLES is a title wherever it stands, which + # is what a caller listing one there has always asked for + TITLES=frozenset(titles - contested), AFTER_GIVEN_TITLES=frozenset(after_given), + # the fourth derived marker (check 11): a title that may be a + # given name, a title only inside a title phrase (rules/H9), and + # so not in TITLES above. A + # title that is also a particle ('st', 'freiherr') is left to the + # particle rules, which weigh its given reading at the front + # already (rules/P4) + TITLES_AMBIGUOUS=frozenset(contested), + # a word every title reading of which is a modifier: a title only + # with a head title behind it in its run (decision 10, rules/H9) + TITLE_MODIFIERS=frozenset( + t for t, ks in title_kinds.items() if ks == {"modifier"}), + TITLE_STYLES=frozenset( + t for t, ks in title_kinds.items() if "style" in ks), + # `attaches` resolved to the two-word phrases it licenses, the + # word before and each title word of a kind it names + TITLE_PHRASES=frozenset( + f"{t} {h}" for t, kinds in attaches.items() if kinds + for h, hk in title_kinds.items() if h != t and hk & kinds), NON_GIVEN_NAME_PARTICLES=frozenset(never_given), PARTICLES=frozenset(particles), CONJUNCTIONS=frozenset(conjunctions), From a42fbe4b29e6de482ece2a95d0d49de828e22b47 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 10:25:22 -0700 Subject: [PATCH 2/7] WIP: no report after a family comma; grace, marshal phrase-only; sergeant attaches --- nameparser/_pipeline/_assign.py | 12 +- nameparser/config/_compiled.py | 207 ++++++++++++++++++++++++++++++++ nameparser/vocab/en.toml | 14 ++- 3 files changed, 226 insertions(+), 7 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index e708b0ca..8f25cf17 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -150,9 +150,11 @@ def _absorbed(piece: Sequence[int], def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], ptags: tuple[frozenset[str], ...], tokens: list[WorkToken], - ambiguities: list[PendingAmbiguity]) -> int: + ambiguities: list[PendingAmbiguity] | None) -> int: """Assign TITLE to the leading title pieces and return the first - non-title index.""" + non-title index. `ambiguities` is None where a family comma has + already marked the part as the given name, so no fork is left to + report.""" n = leading_titles(pieces, ptags, tokens) for k in range(n): _set_roles(tokens, pieces[k], Role.TITLE) @@ -160,7 +162,7 @@ def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], # after any titles" -- the peel stops at it because classify read it # out of the title, so it is the first piece the peel leaves # ('Dean Smith', 'Mr Dean Smith') - if (n < len(pieces) and len(pieces[n]) == 1 + if (ambiguities is not None and n < len(pieces) and len(pieces[n]) == 1 and TITLE_AMBIGUOUS_TAG in tokens[pieces[n][0]].tags and "vocab:title" not in tokens[pieces[n][0]].tags): i = pieces[n][0] @@ -1302,7 +1304,9 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: # before the title reading of the same word (#296, #325). # A name word in the segment makes it v1's walk ('Smith, # John Jr.'). - n = _peel_leading_titles(pieces, ptags, tokens, ambiguities) + # rules/H9: "where no comma has marked the given name" -- this + # part is the given name, the family comma said so + n = _peel_leading_titles(pieces, ptags, tokens, None) # rules/H5: "the title is TRANSPARENT to the suffix # reading: where two or more name words stand, what # stands once the chain is taken reads exactly as it diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index d5c39175..96a7b2ac 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -968,8 +968,10 @@ TITLES_AMBIGUOUS: frozenset[str] = frozenset({ 'dean', + 'grace', 'justice', 'major', + 'marshal', }) TITLE_MODIFIERS: frozenset[str] = frozenset({ @@ -1107,6 +1109,7 @@ 'gent.le', 'gent.ma', 'gent.mo', + 'grace', 'hajji', 'highness', 'holiness', @@ -1240,6 +1243,7 @@ 'chief maj', 'chief majgen', 'chief major', + 'chief marshal', 'chief mayor', 'chief mcpo', 'chief mcpoc', @@ -1369,6 +1373,7 @@ 'major ltjg', 'major maj', 'major majgen', + 'major marshal', 'major mcpo', 'major mcpoc', 'major mcpon', @@ -1410,6 +1415,208 @@ 'major wo3', 'major wo4', 'major wo5', + 'sergeant 1lt', + 'sergeant 1sgt', + 'sergeant 1stlt', + 'sergeant 1stsgt', + 'sergeant 2lt', + 'sergeant 2ndlt', + 'sergeant a1c', + 'sergeant ab', + 'sergeant adjutant', + 'sergeant adm', + 'sergeant admiral', + 'sergeant amn', + 'sergeant bg', + 'sergeant bgen', + 'sergeant brigadier', + 'sergeant briggen', + 'sergeant capt', + 'sergeant captain', + 'sergeant ccmsgt', + 'sergeant cdr', + 'sergeant cmsaf', + 'sergeant cmsgt', + 'sergeant col', + 'sergeant colonel', + 'sergeant commander', + 'sergeant commander-in-chief', + 'sergeant commodore', + 'sergeant corporal', + 'sergeant cpl', + 'sergeant cpo', + 'sergeant cpt', + 'sergeant csm', + 'sergeant cwo-2', + 'sergeant cwo-3', + 'sergeant cwo-4', + 'sergeant cwo-5', + 'sergeant cwo2', + 'sergeant cwo3', + 'sergeant cwo4', + 'sergeant cwo5', + 'sergeant detective', + 'sergeant ens', + 'sergeant fadm', + 'sergeant gaf', + 'sergeant gen', + 'sergeant general', + 'sergeant generalissimo', + 'sergeant gysgt', + 'sergeant lcdr', + 'sergeant lcpl', + 'sergeant lieutenant', + 'sergeant lt', + 'sergeant ltc', + 'sergeant ltcol', + 'sergeant ltg', + 'sergeant ltgen', + 'sergeant ltjg', + 'sergeant maj', + 'sergeant majgen', + 'sergeant major', + 'sergeant marshal', + 'sergeant mcpo', + 'sergeant mcpoc', + 'sergeant mcpon', + 'sergeant mg', + 'sergeant mgysgt', + 'sergeant mpco-cg', + 'sergeant msg', + 'sergeant msgt', + 'sergeant officer', + 'sergeant pfc', + 'sergeant po1', + 'sergeant po2', + 'sergeant po3', + 'sergeant private', + 'sergeant pv2', + 'sergeant pvt', + 'sergeant radm', + 'sergeant rdml', + 'sergeant scpo', + 'sergeant sfc', + 'sergeant sgm', + 'sergeant sgt', + 'sergeant sgtmaj', + 'sergeant sgtmajmc', + 'sergeant sma', + 'sergeant smsgt', + 'sergeant sn', + 'sergeant spc', + 'sergeant ssg', + 'sergeant ssgt', + 'sergeant subaltern', + 'sergeant subedar', + 'sergeant tsgt', + 'sergeant vadm', + 'sergeant wo-1', + 'sergeant wo1', + 'sergeant wo2', + 'sergeant wo3', + 'sergeant wo4', + 'sergeant wo5', + 'sgt 1lt', + 'sgt 1sgt', + 'sgt 1stlt', + 'sgt 1stsgt', + 'sgt 2lt', + 'sgt 2ndlt', + 'sgt a1c', + 'sgt ab', + 'sgt adjutant', + 'sgt adm', + 'sgt admiral', + 'sgt amn', + 'sgt bg', + 'sgt bgen', + 'sgt brigadier', + 'sgt briggen', + 'sgt capt', + 'sgt captain', + 'sgt ccmsgt', + 'sgt cdr', + 'sgt cmsaf', + 'sgt cmsgt', + 'sgt col', + 'sgt colonel', + 'sgt commander', + 'sgt commander-in-chief', + 'sgt commodore', + 'sgt corporal', + 'sgt cpl', + 'sgt cpo', + 'sgt cpt', + 'sgt csm', + 'sgt cwo-2', + 'sgt cwo-3', + 'sgt cwo-4', + 'sgt cwo-5', + 'sgt cwo2', + 'sgt cwo3', + 'sgt cwo4', + 'sgt cwo5', + 'sgt detective', + 'sgt ens', + 'sgt fadm', + 'sgt gaf', + 'sgt gen', + 'sgt general', + 'sgt generalissimo', + 'sgt gysgt', + 'sgt lcdr', + 'sgt lcpl', + 'sgt lieutenant', + 'sgt lt', + 'sgt ltc', + 'sgt ltcol', + 'sgt ltg', + 'sgt ltgen', + 'sgt ltjg', + 'sgt maj', + 'sgt majgen', + 'sgt major', + 'sgt marshal', + 'sgt mcpo', + 'sgt mcpoc', + 'sgt mcpon', + 'sgt mg', + 'sgt mgysgt', + 'sgt mpco-cg', + 'sgt msg', + 'sgt msgt', + 'sgt officer', + 'sgt pfc', + 'sgt po1', + 'sgt po2', + 'sgt po3', + 'sgt private', + 'sgt pv2', + 'sgt pvt', + 'sgt radm', + 'sgt rdml', + 'sgt scpo', + 'sgt sergeant', + 'sgt sfc', + 'sgt sgm', + 'sgt sgtmaj', + 'sgt sgtmajmc', + 'sgt sma', + 'sgt smsgt', + 'sgt sn', + 'sgt spc', + 'sgt ssg', + 'sgt ssgt', + 'sgt subaltern', + 'sgt subedar', + 'sgt tsgt', + 'sgt vadm', + 'sgt wo-1', + 'sgt wo1', + 'sgt wo2', + 'sgt wo3', + 'sgt wo4', + 'sgt wo5', }) NON_GIVEN_NAME_PARTICLES: frozenset[str] = frozenset({ diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index 6426f89a..6a3ad1f8 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -70,6 +70,12 @@ style = {} text = "excellency" style = {} +[[word]] +text = "grace" +style = { issue = 752, decided = 2026-10-26, expands = "'His Grace', a duke's or an archbishop's style", note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'His Grace John Smith' reads the title, 'Grace Smith' the given name" } +given = { us_ssa = 536340, wikidata = 3908, as_of = 2026-10-26, note = "US SSA: female and male births added" } +family = { us_census_2010 = 28098, wikidata = 311, as_of = 2026-10-26 } + [[word]] text = "highness" style = {} @@ -685,7 +691,9 @@ family = { us_census_2010 = 20977, wikidata = 400, as_of = 2026-10-25 } [[word]] text = "marshal" -excluded = { military = { reason = "a given name (Marshal Yanda); the accepted cost is that 'Field Marshal Bernard Montgomery' reads given Field, middle Marshal Bernard (title Field, given Marshal until rules/H9, 2026-10-25, read a modifier with no head after it as a name)", issue = 606, decided = 2026-10-04 } } +military = { issue = 752, decided = 2026-10-26, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Field Marshal Bernard Montgomery' reads the title, 'Marshal Yanda' the given name; until 2026-10-26 excluded as a given name (#606)" } +given = { us_ssa = 5559, wikidata = 28, as_of = 2026-10-26, note = "US SSA: female and male births added" } +family = { wikidata = 34, as_of = 2026-10-26 } [[word]] text = "mcpo" @@ -775,11 +783,11 @@ military = { expands = "Senior Chief Petty Officer" } [[word]] text = "sergeant" -military = {} +military = { attaches = ["military"] } [[word]] text = "sgt" -military = { same_as = "sergeant" } +military = { attaches = ["military"], same_as = "sergeant" } [[word]] text = "sfc" From b56729530b4cc3fa81051ee608864dbeabdeff77 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 10:37:41 -0700 Subject: [PATCH 3/7] WIP: Derek's answers -- report only where no comma marked the given name; grace, marshal, sergeant major; coverage --- docs/design/decisions/H9-title-phrase.md | 11 +++++++++-- docs/design/decisions/salutation-titles.md | 4 ++-- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/rules/H9-title-phrase.md | 13 ++++++++----- docs/release_log.rst | 2 +- nameparser/_config_shim.py | 4 ++-- nameparser/_pipeline/_assign.py | 4 ++-- nameparser/_pipeline/_classify.py | 6 ++---- tests/v2/cases.py | 22 +++++++++++++++++++++- tests/v2/pipeline/test_vocab.py | 16 ++++++++++++++++ tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 2 ++ tools/differential/snapshot.jsonl | 8 +++++--- 13 files changed, 72 insertions(+), 24 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index cb9688e6..c69a79f6 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -2,14 +2,16 @@ ## Standing -- A title that may be a given name is a title only inside a title phrase, and elsewhere reads as a word the vocabulary does not list, reported `title-or-name` where it is the first name word after any titles: `Dean Smith` reads given 'Dean', `Dean of Students Jane Smith` title 'Dean of Students' — 2026-10-25 #752. +- A title that may be a given name is a title only inside a title phrase, and elsewhere reads as a word the vocabulary does not list: `Dean Smith` reads given 'Dean', `Dean of Students Jane Smith` title 'Dean of Students' — 2026-10-25 #752. +- The fork is reported `title-or-name` where the word is the first name word after any titles and no comma has marked the given name: `Dean Smith` and `Dean Smith, MD` report, `SMITH, DEAN MR.` does not — 2026-10-26 #752, phase 2i (Derek). +- `grace` (a style) and `marshal` (a rank) are titles that may be given names, `sergeant` attaches to military titles: `His Grace John Smith`, `Field Marshal Bernard Montgomery` and `Sergeant Major John Smith` read the title; `Mr Justice Smith` keeps given 'Justice', no exception behind a salutation — 2026-10-26 #752, phase 2i (Derek). - Its partner is a modifier or a style before it, a title before it that attaches to it or a title after it that it attaches to (`attaches`), or `of` after it; a salutation is never one, nor any other title before it (#853's `Mr Dean Smith`), nor a title after it it does not attach to (`SMITH, DEAN MR.`) — 2026-10-25 #752. - A modifier in the leading run of titles before the first comma is a title only with a head title after it in that run, `of` after it, or in the object of `of` or `the` there; outside that run (ending the name, H5's, or after a comma) it reads as any title does, the given part after a family comma included (`Lau, Wing` → title 'Wing') — 2026-10-25 #752. - A title that may be a given name which a caller lists in `titles` is a title wherever it stands, which keeps the 1.4 recipe `c.titles.add("dean")`; a v1 caller who empties `titles` turns the phrase reading off with them, while one who replaces the list keeps it (`c.titles.clear(); c.titles.add('dr')` still reads `Dean of Students` as the title), v1 holding no knob for the set — 2026-10-25 #752. ## Log -- **2026-10-25 #752, phase 2i — A TITLE THAT MAY BE A GIVEN NAME IS A TITLE ONLY INSIDE A TITLE PHRASE, AND A MODIFIER AT THE HEAD OF THE NAME ONLY WITH A HEAD BESIDE IT.** Carries out items 10, 11 and 12 of decisions/vocabulary-model's 2026-10-23 #752 entry (decision 10 at the head of the name only; the phrase partner; #853's word after a title), measured here on the rule as built rather than on the spike those items were decided on. +- **2026-10-25 #752, phase 2i (its report in a family comma's given part, and its costs `His Grace John Smith` and `Field Marshal Bernard Montgomery`, superseded by 2026-10-26 #752, phase 2i (Derek)) — A TITLE THAT MAY BE A GIVEN NAME IS A TITLE ONLY INSIDE A TITLE PHRASE, AND A MODIFIER AT THE HEAD OF THE NAME ONLY WITH A HEAD BESIDE IT.** Carries out items 10, 11 and 12 of decisions/vocabulary-model's 2026-10-23 #752 entry (decision 10 at the head of the name only; the phrase partner; #853's word after a title), measured here on the rule as built rather than on the spike those items were decided on. THE WORDS IT READS TODAY. `dean`, `justice` and `major` gain a title reading beside a given one (`justice` until now excluded as a given name, #606, its cost `Chief Justice John Roberts` read as given 'Justice'), so the rule's reach is exactly what master read as name words, now reading the title only in a phrase. Declaring given readings on the titles decisions/vocabulary-model's 2026-10-24 entry ordered (`wing` first) is 2j's, one measured PR per batch. The words leave `titles` for `titles_ambiguous` rather than staying in both, and that is a compat decision, not a convenience: `Constants().titles.add("dean")` is the documented 1.4 recipe for a title the default leaves out, and a word in both sets would be read by the phrase test whatever the caller listed, so `Dean Smith` would lose the title the caller asked for. Measured: with dean in both, `test_the_documented_replacements_for_an_in_place_edit_work` and four shim controls failed, `Dean Smith` reading given 'Dean' with dean added to the titles. THE PARTNER (item 11). A modifier or a style before the word, `of` after it, a title before it that attaches to it, or a title after it that it attaches to: `attaches`, a title property naming title kinds, compiles into two-word phrases (`chief` attaching to office and military words, `major` to military ones; count them with `uv run python -c "from nameparser import Lexicon; print(len(Lexicon.default().title_phrases))"`). A connective other than `of` is no partner: `Dean and Mary Smith` is two given names, as master read it. OUTSIDE A PHRASE THE WORD IS UNLISTED, EVERYWHERE. Classify tags it no title, and the two vocabulary rosters that ask whether a word is listed at all, the capitals credential's "unlisted" (rules/C6) and the comma contrast's "claimed as non-name text", leave the set out. The first draft left them in, and the grid below found `John Smith, DEAN`, read suffix 'DEAN' by C6 on master, reversed to given 'DEAN', family 'John Smith'. Keeping the word unlisted outside its phrase is also what the design's spike measured for the 2j words, which it took out of `titles` altogether. @@ -21,3 +23,8 @@ THE SWEEPS. A grid of 34,380 parses (twelve fronts, salutations, `Dr.`, a dual second in a title run `Rev. Sr.`, the both-vocabulary heads `Freiherr`, `St` and `St.`, a style, a modifier and `Mr. and Mrs.`, before fifteen heads in their bare, capitals, lower-case and period-marked spellings, eight followers and eleven tails, in the comma shapes with an empty head, a nickname head, a particle surname and a title before the comma, under given-first and family-first orders) moved only the shapes this entry names. Two residuals read and accepted: a phrase extended by a coordinating join behind it (`Chief Dean and Mary Smith` reads title 'Chief Dean and Mary'), and nonsense heads (`Chief Justice General Smith`). THE REVIEW FIXES (code review and design-docs review of the first build; the corpus figures above were re-taken after both review rounds, on the final tree). A period-joined title in a part holding a contested or modifier word lost its tag (`Lt.Gov. Dean Smith` read given 'Lt.Gov.'): only the two sets' words are read out of the title now. The modifier fixes above. And v1's `c.titles.clear()` left `Dean of Students` a title, its words being in no v1 set: the shim drops `titles_ambiguous` with the titles. Accepted, in rules/H9: `Mr Justice Smith` (an English High Court judge's style) reads given 'Justice', a salutation being no partner; a modifier whose head is not listed loses the title (`His Grace John Smith` → given 'His', where master read title 'His', given 'Grace', and likewise `Most Worshipful Grand Master` and `Air Cdre`, whose heads are not titles; `Grand Duke of Tuscany`, duke refused by the vocabulary; `Field Marshal Bernard Montgomery` now given 'Field', marshal refused too, where it read title 'Field', given 'Marshal'), and so does one written after its head (`Princess Royal Anne`, `Mother Superior Mary Smith`); the given part after a family comma reads a modifier as listed (`Lau, Wing` → title 'Wing', where `Wing Lau` reads given 'Wing'), the cost decision 10 took by keeping the gate out of comma parts; and a coordinating join behind a phrase extends the title (`Chief Dean and Mary Smith`). +- **2026-10-26 #752, phase 2i (Derek) — NO REPORT WHERE A COMMA HAS MARKED THE GIVEN NAME; `grace` AND `marshal` ARE PHRASE-ONLY TITLES; `Mr Justice` STAYS A GIVEN NAME.** SUPERSEDES: in the 2026-10-25 #752, phase 2i entry, the report in a family comma's given part, and its accepted costs for `His Grace John Smith` and `Field Marshal Bernard Montgomery`. Derek's answers to the questions the PR raised, relayed 2026-10-26: + - THE REPORT. In `SMITH, DEAN` the family comma has already marked `Dean` as the given name, so no reader hesitates, the test AGENTS.md sets for a reportable fork (its example is `Smith, John V`). The leading-title peel of a family comma's given part reports nothing now; the comma-less name and the part before a suffix comma still report (`Dean Smith, MD`). FEC's 1,737 report-only moves, every one a `FAMILY, DEAN` given part, are gone. + - `grace` AND `marshal`. Each gains a title reading beside a dated `given` one, as `justice` did: `grace` a style ('His Grace'), `marshal` a military rank, `marshal`'s exclusion (#606) replaced. A modifier before each is its partner (`His`, `Field`), so `His Grace John Smith` reads title 'His Grace' (master read title 'His', given 'Grace') and `Field Marshal Bernard Montgomery` title 'Field Marshal' (1.4.0 through master read title 'Field', given 'Marshal'). `sergeant` and `sgt` attach to military titles, so `Sergeant Major John Smith` reads title 'Sergeant Major'. Bearer counts are on the words (`uv run python tools/vocab/query.py --word grace`). + - `Mr Justice`. An English High Court judge is styled `Mr Justice Smith`, so Derek asked for counts before deciding an exception. Measured 2026-10-26: the PSC register (fetch dated 2026-10-22) holds 17 records whose display name has a salutation and then `Justice`, and all 17 file `Justice` in the forename box; no record files it in the title box. ctgov-officials and fec-individuals hold none, and the committed corpora none outside rules/H9's own example. Every filer who writes it reads it as the given name, so no exception: rules/H9's Accepted clause stands. + - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). PSC 178 fixed, 3 broken, 16 changed, as before. FEC 53 moved, none for the report alone. ctgov-officials 318 moved, 240 for the report alone (`dean` 115, `grace` 123), and tse-donors 94, 85 for the report alone (`grace` 69, `dean` 14, `marshal` 2): `Grace` opening a name now reports, `grace` being a common given name. The committed corpora: `His Grace John Smith` and `Field Marshal Bernard Montgomery` read the title, `SMITH, DEAN MR.` drops its report. Consistency unchanged (jrc 35 and nepal 4 now alike, none apart). diff --git a/docs/design/decisions/salutation-titles.md b/docs/design/decisions/salutation-titles.md index 5ade3279..68c30933 100644 --- a/docs/design/decisions/salutation-titles.md +++ b/docs/design/decisions/salutation-titles.md @@ -7,7 +7,7 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr). - A word that addresses or ranks a person and is written in FRONT of the name ships as a title unless it is borne as an ordinary name in the leading position, the position test of vocabulary-collisions' C-i, TITLES having no ambiguous subset to mark a collision in, so the only answers are ship and do not ship — 2026-10-04 #606. - One exception to C-i, for TITLES alone: a word borne as a name only RARELY as the first word of one ships on the precedent of `graf` (`herra`, `batoni`, `hrabia`, `kreivi` and the rest of the graf class, each with its counts on its title reading's `note`), and no word in the particle or suffix sets, and no later title, takes it without its own counts — 2026-10-04 #606. - The cost falls on a name that BEGINS with such a word: `Greve Anna` reads title 'Greve', family 'Anna' and `Herra Wijaya` title 'Herra', family 'Wijaya'; a declared `FAMILY_FIRST` does not help (`Hrabia Anna` reads title 'Hrabia' under it); the family comma keeps the word as the family where no name word stands behind it (`Hrabia, Anna` reads given 'Anna', family 'Hrabia', and so does `Hrabia Jr., Anna`), while a name word behind it makes it the title by H6 (`Hrabia Nowak, Anna` reads title 'Hrabia', family 'Nowak'); a name that ends in the word keeps it (`Gladys Knight`, `Knight, Gladys` read family 'Knight'), unless a period follows it (`Gladys Knight.` reads title 'Knight.', family 'Gladys', by H5, accepted rather than prevented) — 2026-10-04 #606. -- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, `ông` and its pair `bà`, and `marshal`, whose accepted cost its entry states: `Field Marshal Bernard Montgomery` reads given 'Field', middle 'Marshal Bernard' since rules/H9 (2026-10-25) read `field`, a modifier, as no title without a head after it, where it read title 'Field', given 'Marshal' before), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. +- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, and `ông` and its pair `bà`), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. - Accents are not folded, so `paní` and `härra` match only their accented spellings (`Paní Nováková` reads title 'Paní', `Pani Anna` reads given 'Pani'); folding the lexicon would turn both into collisions, and whoever proposes it re-argues the entry — 2026-10-04 #606. - `gravos` was dropped, not excluded, since nothing showed it is the Greek word for Count; `κόμης`, a Greek surname written first in official lists, is neither added nor excluded, and adding it is a decision still to make — 2026-10-04 #606. - `knt` (Knight, beside `kt`) is a suffix acronym with a `Knt` case mask: `Mary Knt` reads suffix 'Knt' — 2026-10-04 #606. @@ -15,7 +15,7 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr). - `owura` (Akan) and `bapak` (Indonesian) are titles, on counts under 100 US Census bearers each and none found on Wikidata, and `malam` (Hausa) and `bay` (Turkish) are excluded, each an `excluded` entry with its counts — 2026-10-17 #662. - The Italian abbreviated salutations and letter-heading honorifics whose letters after the full stop are the word's own ending ship as titles with `cased` masks: `sig.ra`, `sig.na`, `dott.ssa`, `prof.ssa`, `gent.mo`, `gent.ma`, `gent.le`, `ill.mo`, `ill.ma`, `ch.mo`, `ch.ma`, `preg.mo`, `preg.ma` (`Dott.ssa Anna Bianchi` reads title 'Dott.ssa', given 'Anna'; `SIG.RA MARIA ROSSI` repairs to 'Sig.ra Maria Rossi'); the bare `sig` and `ing` are excluded as given names, their period-marked spellings already titles by H2's shape, except `Ing.` after a full name and a comma, which is the ambiguous credential decisions/suffix-acronym-collisions lists (#848) — 2026-10-22 #840. -Elsewhere: `justice`, excluded here as a common leading-position name, is a title only inside a title phrase since 2026-10-25 (`Chief Justice John Roberts` reads title 'Chief Justice', `Justice Smith` given 'Justice') — decisions/H9's 2026-10-25 #752, phase 2i entry. +Elsewhere: `justice` and `marshal`, excluded here as common leading-position names, are titles only inside a title phrase since 2026-10-25 and 2026-10-26 (`Chief Justice John Roberts` reads title 'Chief Justice' and `Field Marshal Bernard Montgomery` title 'Field Marshal', `Justice Smith` and `Marshal Yanda` the given name) — decisions/H9's 2026-10-25 #752, phase 2i entry; decisions/H9's 2026-10-26 #752, phase 2i (Derek) entry. ## Log diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index b539fa6e..d27ab81b 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. -- `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. +- `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice`, `major`, `grace` and `marshal`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index f3eaf8f1..37df0b8c 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -11,6 +11,9 @@ "Major General John Smith" → title="Major General" "Chief Justice John Roberts" → title="Chief Justice" "The Hon. Justice Mary Smith" → title="The Hon. Justice" + "His Grace John Smith" → title="His Grace" + "Field Marshal Bernard Montgomery" → title="Field Marshal" + "Sergeant Major John Smith" → title="Sergeant Major" "Mr Dean Smith" → given="Dean" "SMITH, DEAN MR." → given="DEAN" "General Major John Smith" → given="Major" @@ -25,10 +28,12 @@ "Jane Smith Vice." → title="Vice." "John Smith, Vice" → title="Vice" ``` -- **A title that may be a given name, read as a name word where it is the first name word after any titles,** is reported as `title-or-name`; inside a phrase, or later in the name, it is not. +- **A title that may be a given name, read as a name word where it is the first name word after any titles and no comma has marked the given name,** is reported as `title-or-name`; inside a phrase, later in the name, or in the given part after a family comma, it is not. ``` "Dean Smith" → ambiguities=("title-or-name",) "Mr Dean Smith" → ambiguities=("title-or-name",) + "Dean Smith, MD" → ambiguities=("title-or-name",) + "SMITH, DEAN MR." → ambiguities=() "Major General John Smith" → ambiguities=() "John Dean Smith" → ambiguities=() ``` @@ -63,15 +68,13 @@ **Accepted:** -- A judge styled with a salutation reads the salutation as the title and the word as the given name, since a salutation is never a partner. +- A judge styled with a salutation reads the salutation as the title and the word as the given name, since a salutation is never a partner; the people who write a salutation before the word file it as their forename. ``` "Mr Justice Smith" → given="Justice" ``` -- A modifier whose head is not a listed title loses the title, as with duke, marshal and grace, which the vocabulary does not list as titles, and so does a modifier written after its head with no `of` behind it. +- A modifier whose head is not a listed title loses the title, as with duke, which the vocabulary refuses, and so does a modifier written after its head with no `of` behind it. ``` - "His Grace John Smith" → given="His" "Grand Duke of Tuscany" → given="Grand" - "Field Marshal Bernard Montgomery" → given="Field" "Princess Royal Anne" → given="Royal" ``` - In the given part after a family comma a modifier is read as listed, so the listing form and the written order of one name read it differently, the cost decision 10 took for the parts where a lone title word is a role. diff --git a/docs/release_log.rst b/docs/release_log.rst index 1bfae0bf..e23df453 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,7 +12,7 @@ Release Log - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives first ``John``, last ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave first ``Pharm``, middle ``D``, last ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives first ``Pharm``, and one opening with a single letter is still two words (``John Smith, M. D.`` still gives first ``M.``). :issue:`752` - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave last ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives last ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Mr Dean Smith`` give first ``Dean``, the parse now reporting a ``title-or-name`` ambiguity where the word opens the name. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` + - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``, the parse now reporting a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``). Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``). :issue:`752` - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index 51464e09..bf722fab 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1255,8 +1255,8 @@ def matchable(field: str) -> frozenset[str]: # knob for these words, and what a replaced list means # for words it never held is not something it says titles_ambiguous=TITLES_AMBIGUOUS if titles else frozenset(), - title_modifiers=TITLE_MODIFIERS & titles, - title_styles=TITLE_STYLES & titles, + title_modifiers=TITLE_MODIFIERS & (titles | TITLES_AMBIGUOUS), + title_styles=TITLE_STYLES & (titles | TITLES_AMBIGUOUS), title_phrases=frozenset( p for p in TITLE_PHRASES if all(w in titles or w in TITLES_AMBIGUOUS diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 8f25cf17..e160ebf9 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -1304,8 +1304,8 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: # before the title reading of the same word (#296, #325). # A name word in the segment makes it v1's walk ('Smith, # John Jr.'). - # rules/H9: "where no comma has marked the given name" -- this - # part is the given name, the family comma said so + # rules/H9: "no comma has marked the given name" -- here the + # family comma has, so the peel reports nothing n = _peel_leading_titles(pieces, ptags, tokens, None) # rules/H5: "the title is TRANSPARENT to the suffix # reading: where two or more name words stand, what diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index c2b96792..0b803004 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -447,11 +447,9 @@ def classify(state: ParseState) -> ParseState: if n in lex.titles_ambiguous and n not in lex.titles: tags = tags | {TITLE_AMBIGUOUS_TAG} if title: - # in a title phrase: a title, and one addressing - # whichever name its vocabulary says + # in a title phrase: a title (its given-name title + # tag, if any, `_tags_for` wrote already) tags = tags | {"vocab:title"} - if n in lex.given_name_titles: - tags = tags | {"vocab:given-title"} if not title: tags = tags - _TITLE_TAGS if tags != written[i].tags: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 6701fa8a..85fa585c 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11874,7 +11874,7 @@ def _check_cjk_shape_purity(self) -> None: Case("contested_title_before_a_trailing_salutation", "SMITH, DEAN MR.", {"title": "MR.", "given": "DEAN", "family": "SMITH"}, - ambiguities=("title-or-name",), classification="feat(#752)", + classification="feat(#752)", notes="H9: no title AFTER the word is a partner unless " "the word attaches to it -- FEC writes the salutation " "after the given name, and 'DEAN MR.' read as a phrase " @@ -11914,6 +11914,26 @@ def _check_cjk_shape_purity(self) -> None: notes="H9: outside a phrase the word reads as the " "unlisted word it was, which rules/C6 reads as a " "capitals credential here"), + Case("contested_style_after_a_modifier", "His Grace John Smith", + {"title": "His Grace", "given": "John", "family": "Smith"}, + classification="fix(#752)", + notes="H9: grace is a style that may be a given name, and 'his' " + "is its partner; master read title 'His', given 'Grace' " + "(Derek, 2026-10-26)"), + Case("contested_style_alone", "Grace Smith", + {"given": "Grace", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: no partner, so the given name, reported"), + Case("contested_rank_after_a_modifier", "Field Marshal Bernard Montgomery", + {"title": "Field Marshal", "given": "Bernard", + "family": "Montgomery"}, classification="fix(#752)", + notes="H9: marshal is a rank that may be a given name, 'field' " + "its partner; 1.4.0 through 2.6.0 read title 'Field', " + "given 'Marshal' (#606's accepted cost)"), + Case("contested_rank_attached_to", "Sergeant Major John Smith", + {"title": "Sergeant Major", "given": "John", "family": "Smith"}, + classification="feat(#752)", + notes="H9: sergeant attaches to a military title after it"), Case("lone_modifier_at_the_head", "Wing Lau", {"given": "Wing", "family": "Lau"}, classification="fix(#752)", notes="H9, decision 10: a modifier is a title only with a " diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index ed26ef8c..1746734e 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -1157,3 +1157,19 @@ def test_a_word_a_caller_lists_as_a_title_is_one_everywhere() -> None: lex = Lexicon.default().add(titles={"dean"}) name = Parser(lexicon=lex).parse("Dean Smith") assert (name.title, name.family, name.ambiguities) == ("Dean", "Smith", ()) + + +def test_a_contested_title_in_a_phrase_addresses_as_its_vocabulary_says() -> None: + # a title that may be a given name and addresses by the given name + # (no shipped word is both yet) takes the given-name title tag once a + # partner makes it a title, so H1 reads the one name word after it as + # the given name, as 'Sir John' reads + lex = Lexicon.default().add(titles_ambiguous={"zed"}, + given_name_titles={"zed"}) + phrase = Parser(lexicon=lex).parse("Hon Zed John") + assert (phrase.title, phrase.given, phrase.family) == ("Hon Zed", "John", "") + # the tag classify documents for a given-name title, which a word + # read into the title by its phrase carries as one listed there does + assert "vocab:given-title" in phrase.tokens[1].tags + alone = Parser(lexicon=lex).parse("Zed John") + assert (alone.title, alone.given, alone.family) == ("", "Zed", "John") diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index a84e7218..9b57eb97 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -25,7 +25,7 @@ "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 457, "then_words": 179, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 4, "accepted_words": 132}, + "H9": {"words": 473, "then_words": 195, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 4, "accepted_words": 139}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index d9f43765..b89acc08 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -90,6 +90,7 @@ "Dean" "Dean Smith" "Dean Smith, John" +"Dean Smith, MD" "Dean and Mary Smith" "Dean of Students Jane Smith" "Dean. Smith" @@ -655,6 +656,7 @@ "STEVEN HARDMAN, MD, DO, DDS" "Salam, abd Allah" "Sean O'Connor" +"Sergeant Major John Smith" "Sheik Abu Bakar" "Sheik abdul Jr Smith" "Sheik abdul salam" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 7fabff5a..3c782527 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -588,6 +588,7 @@ {"name": "Dean Robert Johns", "given": "Dean", "middle": "Robert", "family": "Johns", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. R. J."} {"name": "Dean Smith", "given": "Dean", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S."} {"name": "Dean Smith, John", "given": "John", "family": "Dean Smith", "initials": "J. D. S."} +{"name": "Dean Smith, MD", "given": "Dean", "family": "Smith", "suffix": "MD", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S."} {"name": "Dean and Mary Smith", "given": "Dean and Mary", "family": "Smith", "initials": "D. M. S."} {"name": "Dean of Chemistry", "title": "Dean of Chemistry"} {"name": "Dean of Chemistry Robert Johns", "title": "Dean of Chemistry", "given": "Robert", "family": "Johns", "initials": "R. J."} @@ -958,7 +959,7 @@ {"name": "Ferrer Rosa Maria Ferran", "order": "FAMILY_FIRST", "given": "Rosa", "middle": "Maria Ferran", "family": "Ferrer", "initials": "R. M. F. F."} {"name": "Ferrer, Rosa Maria Ferran", "given": "Rosa", "middle": "Maria Ferran", "family": "Ferrer", "initials": "R. M. F. F."} {"name": "Fidus", "given": "Fidus", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "F."} -{"name": "Field Marshal Bernard Montgomery", "given": "Field", "middle": "Marshal Bernard", "family": "Montgomery", "initials": "F. M. B. M."} +{"name": "Field Marshal Bernard Montgomery", "title": "Field Marshal", "given": "Bernard", "family": "Montgomery", "initials": "B. M."} {"name": "Fien de la Mar", "given": "Fien", "family": "de la Mar", "initials": "F. M."} {"name": "Figueiredo, Jackson de", "given": "Jackson", "family": "de Figueiredo", "initials": "J. F."} {"name": "Finnur Jónsson", "given": "Finnur", "family": "Jónsson", "initials": "F. J."} @@ -1228,7 +1229,7 @@ {"name": "Hildebrand, Dietrich von", "given": "Dietrich", "family": "von Hildebrand", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. H."} {"name": "Hildebrand, Rudolf Friedrich Leopold von", "given": "Rudolf", "middle": "Friedrich Leopold", "family": "von Hildebrand", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "R. F. L. H."} {"name": "His Excellency Lord Duncan", "title": "His Excellency Lord", "family": "Duncan", "initials": "D."} -{"name": "His Grace John Smith", "given": "His", "middle": "Grace John", "family": "Smith", "initials": "H. G. J. S."} +{"name": "His Grace John Smith", "title": "His Grace", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "His Holiness", "title": "His", "family": "Holiness", "ambiguities": ["TITLE_OR_NAME"], "initials": "H."} {"name": "His Holiness the Dalai Lama", "title": "His Holiness the Dalai", "family": "Lama", "ambiguities": ["TITLE_OR_NAME"], "initials": "L."} {"name": "Hisham Ibrahim, CPA, CFA", "given": "Hisham", "family": "Ibrahim", "suffix": "CPA, CFA", "initials": "H. I."} @@ -2911,7 +2912,7 @@ {"name": "SIG.RA MARIA ROSSI", "title": "SIG.RA", "given": "MARIA", "family": "ROSSI", "initials": "M. R.", "capitalized": "Sig.ra Maria Rossi"} {"name": "SMITH VD MA, JOHN", "given": "JOHN", "family": "SMITH VD MA", "initials": "J. S. M.", "capitalized": "John Smith vd Ma"} {"name": "SMITH VD, JOHN", "given": "JOHN", "family": "SMITH", "suffix": "VD", "initials": "J. S.", "capitalized": "John Smith VD"} -{"name": "SMITH, DEAN MR.", "title": "MR.", "given": "DEAN", "family": "SMITH", "ambiguities": ["TITLE_OR_NAME"], "initials": "D. S.", "capitalized": "Mr. Dean Smith"} +{"name": "SMITH, DEAN MR.", "title": "MR.", "given": "DEAN", "family": "SMITH", "initials": "D. S.", "capitalized": "Mr. Dean Smith"} {"name": "SMITH, JOHN DO", "given": "JOHN", "family": "DO SMITH", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. S.", "capitalized": "John do Smith"} {"name": "SMITH, SAM E.", "given": "SAM", "middle": "E.", "family": "SMITH", "initials": "S. E. S.", "capitalized": "Sam E. Smith"} {"name": "SMITH, VD DE LA MA", "family": "SMITH VD DE LA MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "S. M.", "capitalized": "Smith vd de la Ma"} @@ -2983,6 +2984,7 @@ {"name": "Sepp Anne", "order": "FAMILY_FIRST", "given": "Anne", "family": "Sepp", "initials": "A. S."} {"name": "Sepp, Anne", "given": "Anne", "family": "Sepp", "initials": "A. S."} {"name": "Seppo Ilmarinen", "given": "Seppo", "family": "Ilmarinen", "initials": "S. I."} +{"name": "Sergeant Major John Smith", "title": "Sergeant Major", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Sergio Marcos González", "given": "Sergio", "middle": "Marcos", "family": "González", "initials": "S. M. G."} {"name": "Serrano Poncela, Segundo", "given": "Segundo", "family": "Serrano Poncela", "initials": "S. S. P."} {"name": "Sert y Badia, José Maria", "given": "José", "middle": "Maria", "family": "Sert y Badia", "initials": "J. M. S. B."} From 637a0eca5f6faa7ef3e207f74baaa4bd86e15eb8 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 10:48:30 -0700 Subject: [PATCH 4/7] Act on Derek's answers: no report after a family comma; grace and marshal phrase-only; attaches may name a word - the title-or-name report is silent where a family comma has marked the given name (SMITH, DEAN) - grace (style) and marshal (military) gain given readings: His Grace, Field Marshal read as titles; marshal's #606 exclusion replaced - attaches may name a title word: sergeant/sgt attach to major, major to general, so Sgt Marshal Smith keeps given Marshal - a style or modifier before a contested word is a partner only where it is read as a title (Grace Dean Smith) - drop a dead given-title tag line in classify; shim mirrors titles_ambiguous - Mr Justice: PSC's 17 salutation + Justice records all file Justice as a forename; no exception Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 8 +- docs/design/decisions/vocabulary-model.md | 6 +- docs/design/rules/H9-title-phrase.md | 9 +- nameparser/_config_shim.py | 6 +- nameparser/_pipeline/_vocab.py | 10 +- nameparser/config/_compiled.py | 301 +--------------------- nameparser/vocab/README.md | 2 +- nameparser/vocab/en.toml | 6 +- tests/v2/cases.py | 30 ++- tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 11 +- tests/v2/test_vocab.py | 11 + tools/differential/corpus_rules.jsonl | 4 + tools/differential/snapshot.jsonl | 4 + tools/vocab/build.py | 21 +- 15 files changed, 108 insertions(+), 323 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index c69a79f6..3e949d0d 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -4,7 +4,7 @@ - A title that may be a given name is a title only inside a title phrase, and elsewhere reads as a word the vocabulary does not list: `Dean Smith` reads given 'Dean', `Dean of Students Jane Smith` title 'Dean of Students' — 2026-10-25 #752. - The fork is reported `title-or-name` where the word is the first name word after any titles and no comma has marked the given name: `Dean Smith` and `Dean Smith, MD` report, `SMITH, DEAN MR.` does not — 2026-10-26 #752, phase 2i (Derek). -- `grace` (a style) and `marshal` (a rank) are titles that may be given names, `sergeant` attaches to military titles: `His Grace John Smith`, `Field Marshal Bernard Montgomery` and `Sergeant Major John Smith` read the title; `Mr Justice Smith` keeps given 'Justice', no exception behind a salutation — 2026-10-26 #752, phase 2i (Derek). +- `grace` (a style) and `marshal` (a rank) are titles that may be given names, `sergeant` attaches to the word `major` and `major` to `general` (`attaches` naming a word where a kind would reach too far): `His Grace John Smith`, `Field Marshal Bernard Montgomery` and `Sergeant Major John Smith` read the title; `Mr Justice Smith` keeps given 'Justice', no exception behind a salutation — 2026-10-26 #752, phase 2i (Derek). - Its partner is a modifier or a style before it, a title before it that attaches to it or a title after it that it attaches to (`attaches`), or `of` after it; a salutation is never one, nor any other title before it (#853's `Mr Dean Smith`), nor a title after it it does not attach to (`SMITH, DEAN MR.`) — 2026-10-25 #752. - A modifier in the leading run of titles before the first comma is a title only with a head title after it in that run, `of` after it, or in the object of `of` or `the` there; outside that run (ending the name, H5's, or after a comma) it reads as any title does, the given part after a family comma included (`Lau, Wing` → title 'Wing') — 2026-10-25 #752. - A title that may be a given name which a caller lists in `titles` is a title wherever it stands, which keeps the 1.4 recipe `c.titles.add("dean")`; a v1 caller who empties `titles` turns the phrase reading off with them, while one who replaces the list keeps it (`c.titles.clear(); c.titles.add('dr')` still reads `Dean of Students` as the title), v1 holding no knob for the set — 2026-10-25 #752. @@ -23,8 +23,10 @@ THE SWEEPS. A grid of 34,380 parses (twelve fronts, salutations, `Dr.`, a dual second in a title run `Rev. Sr.`, the both-vocabulary heads `Freiherr`, `St` and `St.`, a style, a modifier and `Mr. and Mrs.`, before fifteen heads in their bare, capitals, lower-case and period-marked spellings, eight followers and eleven tails, in the comma shapes with an empty head, a nickname head, a particle surname and a title before the comma, under given-first and family-first orders) moved only the shapes this entry names. Two residuals read and accepted: a phrase extended by a coordinating join behind it (`Chief Dean and Mary Smith` reads title 'Chief Dean and Mary'), and nonsense heads (`Chief Justice General Smith`). THE REVIEW FIXES (code review and design-docs review of the first build; the corpus figures above were re-taken after both review rounds, on the final tree). A period-joined title in a part holding a contested or modifier word lost its tag (`Lt.Gov. Dean Smith` read given 'Lt.Gov.'): only the two sets' words are read out of the title now. The modifier fixes above. And v1's `c.titles.clear()` left `Dean of Students` a title, its words being in no v1 set: the shim drops `titles_ambiguous` with the titles. Accepted, in rules/H9: `Mr Justice Smith` (an English High Court judge's style) reads given 'Justice', a salutation being no partner; a modifier whose head is not listed loses the title (`His Grace John Smith` → given 'His', where master read title 'His', given 'Grace', and likewise `Most Worshipful Grand Master` and `Air Cdre`, whose heads are not titles; `Grand Duke of Tuscany`, duke refused by the vocabulary; `Field Marshal Bernard Montgomery` now given 'Field', marshal refused too, where it read title 'Field', given 'Marshal'), and so does one written after its head (`Princess Royal Anne`, `Mother Superior Mary Smith`); the given part after a family comma reads a modifier as listed (`Lau, Wing` → title 'Wing', where `Wing Lau` reads given 'Wing'), the cost decision 10 took by keeping the gate out of comma parts; and a coordinating join behind a phrase extends the title (`Chief Dean and Mary Smith`). -- **2026-10-26 #752, phase 2i (Derek) — NO REPORT WHERE A COMMA HAS MARKED THE GIVEN NAME; `grace` AND `marshal` ARE PHRASE-ONLY TITLES; `Mr Justice` STAYS A GIVEN NAME.** SUPERSEDES: in the 2026-10-25 #752, phase 2i entry, the report in a family comma's given part, and its accepted costs for `His Grace John Smith` and `Field Marshal Bernard Montgomery`. Derek's answers to the questions the PR raised, relayed 2026-10-26: +- **2026-10-26 #752, phase 2i (Derek) — NO REPORT WHERE A COMMA HAS MARKED THE GIVEN NAME; `grace` AND `marshal` ARE PHRASE-ONLY TITLES; `Mr Justice` STAYS A GIVEN NAME.** SUPERSEDES: in the 2026-10-25 #752, phase 2i entry, the report in a family comma's given part, its accepted costs for `His Grace John Smith` and `Field Marshal Bernard Montgomery`, and `major` attaching to the military kind. Derek's answers to the questions the PR raised, relayed 2026-10-26: - THE REPORT. In `SMITH, DEAN` the family comma has already marked `Dean` as the given name, so no reader hesitates, the test AGENTS.md sets for a reportable fork (its example is `Smith, John V`). The leading-title peel of a family comma's given part reports nothing now; the comma-less name and the part before a suffix comma still report (`Dean Smith, MD`). FEC's 1,737 report-only moves, every one a `FAMILY, DEAN` given part, are gone. - - `grace` AND `marshal`. Each gains a title reading beside a dated `given` one, as `justice` did: `grace` a style ('His Grace'), `marshal` a military rank, `marshal`'s exclusion (#606) replaced. A modifier before each is its partner (`His`, `Field`), so `His Grace John Smith` reads title 'His Grace' (master read title 'His', given 'Grace') and `Field Marshal Bernard Montgomery` title 'Field Marshal' (1.4.0 through master read title 'Field', given 'Marshal'). `sergeant` and `sgt` attach to military titles, so `Sergeant Major John Smith` reads title 'Sergeant Major'. Bearer counts are on the words (`uv run python tools/vocab/query.py --word grace`). + - `grace` AND `marshal`. Each gains a title reading beside a dated `given` one, as `justice` did: `grace` a style ('His Grace'), `marshal` a military rank, `marshal`'s exclusion (#606) replaced. A modifier before each is its partner (`His`, `Field`), so `His Grace John Smith` reads title 'His Grace' (master read title 'His', given 'Grace') and `Field Marshal Bernard Montgomery` title 'Field Marshal' (1.4.0 through master read title 'Field', given 'Marshal'). `sergeant` and `sgt` attach to the word `major`, so `Sergeant Major John Smith` reads title 'Sergeant Major' (1.4.0 through master read title 'Sergeant', given 'Major'). + - `attaches` MAY NAME A TITLE WORD, and `major` and `sergeant` name one. The review of these changes found that making `marshal` a contested military word widened every attachment to the military kind: the build made `sergeant marshal`, `sgt marshal` and `major marshal` phrases, so `Sgt Marshal Smith` lost the given name master and 1.4.0 read ('Marshal'), and `Major Marshal Smith` read title 'Major Marshal'. A kind reaches every word that later joins it, so a contested word joining the kind becomes a partner nobody chose; `sergeant` and `sgt` now attach to `major` and `major` to `general`, the one phrase each was for. `chief` keeps its kinds (office and military), its phrases with a contested word being real ranks and offices (`Chief Justice`, `Chief Marshal`). Each with a case row. + - A STYLE OR MODIFIER IS A PARTNER ONLY WHERE IT IS A TITLE THERE. `grace` is now both a style and a title that may be a given name, so in `Grace Dean Smith` the rule as worded made `Grace` the partner of `Dean`; the tree read given 'Grace' anyway (the run stops at a name word), but the two readings disagreed. The partner before is now asked as read, left to right, and rules/H9 says so. Bearer counts are on the words (`uv run python tools/vocab/query.py --word grace`). - `Mr Justice`. An English High Court judge is styled `Mr Justice Smith`, so Derek asked for counts before deciding an exception. Measured 2026-10-26: the PSC register (fetch dated 2026-10-22) holds 17 records whose display name has a salutation and then `Justice`, and all 17 file `Justice` in the forename box; no record files it in the title box. ctgov-officials and fec-individuals hold none, and the committed corpora none outside rules/H9's own example. Every filer who writes it reads it as the given name, so no exception: rules/H9's Accepted clause stands. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). PSC 178 fixed, 3 broken, 16 changed, as before. FEC 53 moved, none for the report alone. ctgov-officials 318 moved, 240 for the report alone (`dean` 115, `grace` 123), and tse-donors 94, 85 for the report alone (`grace` 69, `dean` 14, `marshal` 2): `Grace` opening a name now reports, `grace` being a common given name. The committed corpora: `His Grace John Smith` and `Field Marshal Bernard Montgomery` read the title, `SMITH, DEAN MR.` drops its report. Consistency unchanged (jrc 35 and nepal 4 now alike, none apart). diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index d27ab81b..8248d31e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. - Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. -- `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice`, `major`, `grace` and `marshal`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. +- `Dean Smith` reads given `Dean`. The contested-head gate covers `dean`, `justice` and `major`, and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752; 2026-10-25 #752, phase 2i. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. @@ -27,7 +27,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. -- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `bare_only`, `leading_only`, `trailing_only` and `separators` are phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. `attaches` names the title kinds a title reading stands before in a title phrase, compiled into the phrases it makes — 2026-10-25 #752, phase 2i. +- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `bare_only`, `leading_only`, `trailing_only` and `separators` are phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. `attaches` names the title kinds, or the title words, a title reading stands before in a title phrase, compiled into the phrases it makes — 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2i (Derek). - A title reading beside a `given` reading, on a word with no particle reading, derives `TITLES_AMBIGUOUS` instead of `TITLES`: a title only inside a title phrase (rules/H9); a salutation's given-name counts kept as evidence for H8 sit in its `note`, marking nothing — 2026-10-25 #752, phase 2i. - Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762. - The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749. @@ -213,3 +213,5 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - Reclassified, as item 10 asked and as the build found: `resident` and `rt` to occupation, `pro` to academic, `d'affaires` and `member` to office (decisions/H9). - `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest. Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. +- **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. + diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 37df0b8c..6c66f49e 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -4,7 +4,7 @@ **Then:** the word is a title only inside a title phrase, and elsewhere a name word. -- **A title that may be a given name** has its phrase partner in a modifier or a style before it, in a title before it that attaches to it, in a title after it that it attaches to, or in `of` after it. No other title before it is one, a salutation least of all, and no other title after it: a donor list writes the salutation after the given name. +- **A title that may be a given name** has its phrase partner in a modifier or a style before it that is a title there, in a title before it that attaches to it, in a title after it that it attaches to, or in `of` after it. No other title before it is one, a salutation least of all, and no other title after it: a donor list writes the salutation after the given name. ``` "Dean of Students Jane Smith" → title="Dean of Students" "Associate Dean Smith" → title="Associate Dean" @@ -17,6 +17,8 @@ "Mr Dean Smith" → given="Dean" "SMITH, DEAN MR." → given="DEAN" "General Major John Smith" → given="Major" + "Grace Dean Smith" → given="Grace" + "Sgt Marshal Smith" → given="Marshal" ``` - **A modifier in the leading run of titles before the first comma** has its phrase in a head title after it in that run, in `of` after it, or in the object of `of` or `the` there. A modifier outside that run, ending the name or after a comma, is read as any title is. ``` @@ -34,6 +36,7 @@ "Mr Dean Smith" → ambiguities=("title-or-name",) "Dean Smith, MD" → ambiguities=("title-or-name",) "SMITH, DEAN MR." → ambiguities=() + ", Dean" → ambiguities=() "Major General John Smith" → ambiguities=() "John Dean Smith" → ambiguities=() ``` @@ -81,6 +84,10 @@ ``` "Lau, Wing" → title="Wing" ``` +- A phrase that is the whole given part after a family comma leaves the given name empty, as a lone title there does. + ``` + "Smith, Sgt Major" → title="Sgt Major" + ``` - Once a partner has made the word a title, a coordinating join behind it takes the next word into the title too. ``` "Chief Dean and Mary Smith" → title="Chief Dean and Mary" diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index bf722fab..44a63808 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1255,8 +1255,10 @@ def matchable(field: str) -> frozenset[str]: # knob for these words, and what a replaced list means # for words it never held is not something it says titles_ambiguous=TITLES_AMBIGUOUS if titles else frozenset(), - title_modifiers=TITLE_MODIFIERS & (titles | TITLES_AMBIGUOUS), - title_styles=TITLE_STYLES & (titles | TITLES_AMBIGUOUS), + title_modifiers=TITLE_MODIFIERS & ( + titles | (TITLES_AMBIGUOUS if titles else frozenset())), + title_styles=TITLE_STYLES & ( + titles | (TITLES_AMBIGUOUS if titles else frozenset())), title_phrases=frozenset( p for p in TITLE_PHRASES if all(w in titles or w in TITLES_AMBIGUOUS diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 84d05218..1f978907 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1412,7 +1412,10 @@ def title_readings(texts: Sequence[str], folded: Sequence[str], does not attach to ('SMITH, DEAN MR.', a given name and the salutation a donor list writes after it). Its neighbours are asked as LISTED, so 'Vice Dean' reads 'Vice' as the partner of 'Dean' - whatever the modifier test below makes of 'Vice'. + whatever the modifier test below makes of 'Vice', save a word before + it that is itself a title that may be a given name, which is a + partner only where its own phrase made it a title ('Grace Dean + Smith' reads 'Grace' as the given name, and 'Dean' with it). Where `head` is true -- the part before the first comma -- a word of `title_modifiers` is a title only with a head title behind it in @@ -1451,7 +1454,10 @@ def title_readings(texts: Sequence[str], folded: Sequence[str], prev = folded[i - 1] if i else None nxt = folded[i + 1] if i < last else None reading[i] = bool( - (prev is not None and listed[i - 1] + # the word before as it was READ, left to right: a title that + # may be a given name and stands in no phrase is no partner + # ('Grace Dean Smith') + (prev is not None and reading[i - 1] and (prev in modifiers or prev in lexicon.title_styles or f"{prev} {n}" in phrases)) or (nxt is not None diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index 96a7b2ac..9583a39e 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -1251,6 +1251,7 @@ 'chief member', 'chief mg', 'chief mgysgt', + 'chief military', 'chief minister', 'chief mpco-cg', 'chief msg', @@ -1314,309 +1315,9 @@ 'chief wo5', 'chief न्यायमूर्ति', 'chief বিচারপতি', - 'major 1lt', - 'major 1sgt', - 'major 1stlt', - 'major 1stsgt', - 'major 2lt', - 'major 2ndlt', - 'major a1c', - 'major ab', - 'major adjutant', - 'major adm', - 'major admiral', - 'major amn', - 'major bg', - 'major bgen', - 'major brigadier', - 'major briggen', - 'major capt', - 'major captain', - 'major ccmsgt', - 'major cdr', - 'major cmsaf', - 'major cmsgt', - 'major col', - 'major colonel', - 'major commander', - 'major commander-in-chief', - 'major commodore', - 'major corporal', - 'major cpl', - 'major cpo', - 'major cpt', - 'major csm', - 'major cwo-2', - 'major cwo-3', - 'major cwo-4', - 'major cwo-5', - 'major cwo2', - 'major cwo3', - 'major cwo4', - 'major cwo5', - 'major detective', - 'major ens', - 'major fadm', - 'major gaf', - 'major gen', 'major general', - 'major generalissimo', - 'major gysgt', - 'major lcdr', - 'major lcpl', - 'major lieutenant', - 'major lt', - 'major ltc', - 'major ltcol', - 'major ltg', - 'major ltgen', - 'major ltjg', - 'major maj', - 'major majgen', - 'major marshal', - 'major mcpo', - 'major mcpoc', - 'major mcpon', - 'major mg', - 'major mgysgt', - 'major mpco-cg', - 'major msg', - 'major msgt', - 'major officer', - 'major pfc', - 'major po1', - 'major po2', - 'major po3', - 'major private', - 'major pv2', - 'major pvt', - 'major radm', - 'major rdml', - 'major scpo', - 'major sergeant', - 'major sfc', - 'major sgm', - 'major sgt', - 'major sgtmaj', - 'major sgtmajmc', - 'major sma', - 'major smsgt', - 'major sn', - 'major spc', - 'major ssg', - 'major ssgt', - 'major subaltern', - 'major subedar', - 'major tsgt', - 'major vadm', - 'major wo-1', - 'major wo1', - 'major wo2', - 'major wo3', - 'major wo4', - 'major wo5', - 'sergeant 1lt', - 'sergeant 1sgt', - 'sergeant 1stlt', - 'sergeant 1stsgt', - 'sergeant 2lt', - 'sergeant 2ndlt', - 'sergeant a1c', - 'sergeant ab', - 'sergeant adjutant', - 'sergeant adm', - 'sergeant admiral', - 'sergeant amn', - 'sergeant bg', - 'sergeant bgen', - 'sergeant brigadier', - 'sergeant briggen', - 'sergeant capt', - 'sergeant captain', - 'sergeant ccmsgt', - 'sergeant cdr', - 'sergeant cmsaf', - 'sergeant cmsgt', - 'sergeant col', - 'sergeant colonel', - 'sergeant commander', - 'sergeant commander-in-chief', - 'sergeant commodore', - 'sergeant corporal', - 'sergeant cpl', - 'sergeant cpo', - 'sergeant cpt', - 'sergeant csm', - 'sergeant cwo-2', - 'sergeant cwo-3', - 'sergeant cwo-4', - 'sergeant cwo-5', - 'sergeant cwo2', - 'sergeant cwo3', - 'sergeant cwo4', - 'sergeant cwo5', - 'sergeant detective', - 'sergeant ens', - 'sergeant fadm', - 'sergeant gaf', - 'sergeant gen', - 'sergeant general', - 'sergeant generalissimo', - 'sergeant gysgt', - 'sergeant lcdr', - 'sergeant lcpl', - 'sergeant lieutenant', - 'sergeant lt', - 'sergeant ltc', - 'sergeant ltcol', - 'sergeant ltg', - 'sergeant ltgen', - 'sergeant ltjg', - 'sergeant maj', - 'sergeant majgen', 'sergeant major', - 'sergeant marshal', - 'sergeant mcpo', - 'sergeant mcpoc', - 'sergeant mcpon', - 'sergeant mg', - 'sergeant mgysgt', - 'sergeant mpco-cg', - 'sergeant msg', - 'sergeant msgt', - 'sergeant officer', - 'sergeant pfc', - 'sergeant po1', - 'sergeant po2', - 'sergeant po3', - 'sergeant private', - 'sergeant pv2', - 'sergeant pvt', - 'sergeant radm', - 'sergeant rdml', - 'sergeant scpo', - 'sergeant sfc', - 'sergeant sgm', - 'sergeant sgt', - 'sergeant sgtmaj', - 'sergeant sgtmajmc', - 'sergeant sma', - 'sergeant smsgt', - 'sergeant sn', - 'sergeant spc', - 'sergeant ssg', - 'sergeant ssgt', - 'sergeant subaltern', - 'sergeant subedar', - 'sergeant tsgt', - 'sergeant vadm', - 'sergeant wo-1', - 'sergeant wo1', - 'sergeant wo2', - 'sergeant wo3', - 'sergeant wo4', - 'sergeant wo5', - 'sgt 1lt', - 'sgt 1sgt', - 'sgt 1stlt', - 'sgt 1stsgt', - 'sgt 2lt', - 'sgt 2ndlt', - 'sgt a1c', - 'sgt ab', - 'sgt adjutant', - 'sgt adm', - 'sgt admiral', - 'sgt amn', - 'sgt bg', - 'sgt bgen', - 'sgt brigadier', - 'sgt briggen', - 'sgt capt', - 'sgt captain', - 'sgt ccmsgt', - 'sgt cdr', - 'sgt cmsaf', - 'sgt cmsgt', - 'sgt col', - 'sgt colonel', - 'sgt commander', - 'sgt commander-in-chief', - 'sgt commodore', - 'sgt corporal', - 'sgt cpl', - 'sgt cpo', - 'sgt cpt', - 'sgt csm', - 'sgt cwo-2', - 'sgt cwo-3', - 'sgt cwo-4', - 'sgt cwo-5', - 'sgt cwo2', - 'sgt cwo3', - 'sgt cwo4', - 'sgt cwo5', - 'sgt detective', - 'sgt ens', - 'sgt fadm', - 'sgt gaf', - 'sgt gen', - 'sgt general', - 'sgt generalissimo', - 'sgt gysgt', - 'sgt lcdr', - 'sgt lcpl', - 'sgt lieutenant', - 'sgt lt', - 'sgt ltc', - 'sgt ltcol', - 'sgt ltg', - 'sgt ltgen', - 'sgt ltjg', - 'sgt maj', - 'sgt majgen', 'sgt major', - 'sgt marshal', - 'sgt mcpo', - 'sgt mcpoc', - 'sgt mcpon', - 'sgt mg', - 'sgt mgysgt', - 'sgt mpco-cg', - 'sgt msg', - 'sgt msgt', - 'sgt officer', - 'sgt pfc', - 'sgt po1', - 'sgt po2', - 'sgt po3', - 'sgt private', - 'sgt pv2', - 'sgt pvt', - 'sgt radm', - 'sgt rdml', - 'sgt scpo', - 'sgt sergeant', - 'sgt sfc', - 'sgt sgm', - 'sgt sgtmaj', - 'sgt sgtmajmc', - 'sgt sma', - 'sgt smsgt', - 'sgt sn', - 'sgt spc', - 'sgt ssg', - 'sgt ssgt', - 'sgt subaltern', - 'sgt subedar', - 'sgt tsgt', - 'sgt vadm', - 'sgt wo-1', - 'sgt wo1', - 'sgt wo2', - 'sgt wo3', - 'sgt wo4', - 'sgt wo5', }) NON_GIVEN_NAME_PARTICLES: frozenset[str] = frozenset({ diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 4acac3bd..4a529da1 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -175,7 +175,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | Property | Type | Applies to | Unwritten | Effect | |---|---|---|---|---| | `addresses` | behavioral | `title` | `"family"` | `"given"` adds the word to `GIVEN_NAME_TITLES`: `Sir John` reads `John` as the given name ([rules/H1](../../docs/design/rules/H1-title-and-one-name-word.md), a title and one name word) | -| `attaches` | behavioral | `title` | none | `TITLE_PHRASES`: the title kinds this word stands before in a title phrase, a list (`chief = { attaches = ["office", "military"] }`), compiled into the two-word phrases it makes with each title word of those kinds (`chief justice`); a title that may be a given name on either side of one is a title there ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase) | +| `attaches` | behavioral | `title` | none | `TITLE_PHRASES`: the title kinds, or the title words, this word stands before in a title phrase, a list (`chief = { attaches = ["office", "military"] }`, `sergeant = { attaches = ["major"] }`, a word where a kind would reach too far), compiled into the two-word phrases it makes with each title word of those kinds and each word named (`chief justice`, `sergeant major`); a title that may be a given name on either side of one is a title there ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase) | | `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | | `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index 6a3ad1f8..968ef329 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -685,7 +685,7 @@ military = { expands = "Major" } [[word]] text = "major" -military = { issue = 752, decided = 2026-10-25, attaches = ["military"], note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Major General Smith' reads the title, 'Major Smith' and 'Mr Major Smith' the given name; attaches: Major General" } +military = { issue = 752, decided = 2026-10-25, attaches = ["general"], note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Major General Smith' reads the title, 'Major Smith' and 'Mr Major Smith' the given name; attaches: Major General" } given = { us_ssa = 21749, wikidata = 65, as_of = 2026-10-25, note = "US SSA: female and male births added" } family = { us_census_2010 = 20977, wikidata = 400, as_of = 2026-10-25 } @@ -783,11 +783,11 @@ military = { expands = "Senior Chief Petty Officer" } [[word]] text = "sergeant" -military = { attaches = ["military"] } +military = { attaches = ["major"], issue = 752, decided = 2026-10-26, note = "attaches: Sergeant Major (rules/H9); a word, not the military kind, which made Sgt Marshal Smith a title" } [[word]] text = "sgt" -military = { attaches = ["military"], same_as = "sergeant" } +military = { attaches = ["major"], issue = 752, decided = 2026-10-26, note = "attaches: Sergeant Major (rules/H9); a word, not the military kind, which made Sgt Marshal Smith a title", same_as = "sergeant" } [[word]] text = "sfc" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 85fa585c..cf97eee0 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11932,8 +11932,36 @@ def _check_cjk_shape_purity(self) -> None: "given 'Marshal' (#606's accepted cost)"), Case("contested_rank_attached_to", "Sergeant Major John Smith", {"title": "Sergeant Major", "given": "John", "family": "Smith"}, - classification="feat(#752)", + classification="fix(#752)", notes="H9: sergeant attaches to a military title after it"), + Case("contested_rank_after_a_rank_that_attaches_elsewhere", + "Sgt Marshal Smith", + {"title": "Sgt", "given": "Marshal", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: sergeant attaches to the word 'major', not to every " + "military title; attaching it to the kind made 'Sgt " + "Marshal' a phrase and lost the given name (the review of " + "the fix commits)"), + Case("contested_rank_alone", "Marshal Yanda", + {"given": "Marshal", "family": "Yanda"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: no partner, so the given name, reported"), + Case("contested_rank_after_a_modifier_air", "Air Marshal John Smith", + {"title": "Air Marshal", "given": "John", "family": "Smith"}, + classification="fix(#752)", + notes="H9: 'air' is a modifier and marshal's partner"), + Case("contested_style_with_no_partner_before_a_contested_word", + "Grace Dean Smith", + {"given": "Grace", "middle": "Dean", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: a style is a partner only where it is a title there; " + "'Grace' stands in no phrase, so neither does 'Dean'"), + Case("contested_title_after_an_empty_head", ", Dean", + {"given": "Dean"}, + notes="H9: the comma has marked the given name, so no report"), + Case("contested_title_after_a_nickname_head", "(Bob), Dean", + {"given": "Dean", "nickname": "Bob"}, + notes="H9: the comma has marked the given name, so no report"), Case("lone_modifier_at_the_head", "Wing Lau", {"given": "Wing", "family": "Lau"}, classification="fix(#752)", notes="H9, decision 10: a modifier is a title only with a " diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 9b57eb97..74309042 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -25,7 +25,7 @@ "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 473, "then_words": 195, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 4, "accepted_words": 139}, + "H9": {"words": 478, "then_words": 200, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 5, "accepted_words": 162}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 0c7fc308..9c7f34b2 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -252,10 +252,13 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # (9), the split credential's own report on nine rows (27) and the # opening-credential row (3), each claim under the three orders; # 1568 -> 1583 the same day: #752's title phrase rows (phase 2i), the - # five that report a title read as a name, under the three orders. - assert claims == 1583, ( - f"the field sweep checks {claims} claims, recorded as 1583 on " - f"2026-10-25") + # five that report a title read as a name, under the three orders; + # 1583 -> 1592 on 2026-10-26: Derek's answers on #894, three rows + # that report ('Sgt Marshal', 'Marshal Yanda', 'Grace Dean'), the + # one that stopped ('SMITH, DEAN MR.') offset by 'Grace Smith'. + assert claims == 1592, ( + f"the field sweep checks {claims} claims, recorded as 1592 on " + f"2026-10-26") def test_every_case_name_is_unique() -> None: diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 814aea3e..93ecf780 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -285,6 +285,7 @@ def _w(body: str) -> str: ("attaches-twice", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military", "military"] }')}, "attaches must be a non-empty list of title kinds"), ("attaches-a-suffix-kind", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["credential"] }')}, "attaches names 'credential', which is not a title kind"), ("attaches-the-generic-kind", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["title"] }')}, "attaches names 'title', which is not a title kind"), + ("attaches-a-word-with-no-title-reading", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["zzq"] }')}, "attaches names 'zzq', which is no title kind and no title word"), ("attaches-on-a-suffix", {"en.toml": _w('[[word]]\ntext = "phd"\ncredential = { attaches = ["office"] }')}, "unknown key 'attaches'"), ("attaches-differs-between-files", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military"] }'), "fr.toml": _w('[[word]]\ntext = "chief"\noffice = {}')}, "attaches"), ("reserved-separators", {"th.toml": _w('[[word]]\ntext = "นาย"\nsalutation = { separators = [] }')}, "reserved for phase 4"), @@ -1105,3 +1106,13 @@ def test_a_shipped_phrase_names_two_titles() -> None: assert len(words) == 2, phrase assert all(w in lex.titles or w in lex.titles_ambiguous for w in words), phrase + + +def test_attaches_may_name_one_title_word() -> None: + # a word where a kind would reach too far: 'sergeant' before 'major' + # only, not before every military title (rules/H9) + c = _compile({"en.toml": _w( + '[[word]]\ntext = "zzq"\nmilitary = { attaches = ["zzr"] }\n\n' + '[[word]]\ntext = "zzr"\nmilitary = {}\n\n' + '[[word]]\ntext = "zzs"\nmilitary = {}')}) + assert c["TITLE_PHRASES"] == frozenset({"zzq zzr"}) diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b89acc08..90b22686 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -5,6 +5,7 @@ "'Smitty' Sir John" "(" ", Ana García López" +", Dean" ", Dr. fils" ", Jane" ", Jane Mrs" @@ -199,6 +200,7 @@ "García Márquez, Ms G.J." "General Major John Smith" "Georg VI." +"Grace Dean Smith" "Graf Lambsdorff, Otto" "Grand Duke of Tuscany" "Greve Anna" @@ -657,6 +659,7 @@ "Salam, abd Allah" "Sean O'Connor" "Sergeant Major John Smith" +"Sgt Marshal Smith" "Sheik Abu Bakar" "Sheik abdul Jr Smith" "Sheik abdul salam" @@ -759,6 +762,7 @@ "Smith, Pharm D" "Smith, RA PhD Ma" "Smith, RN - CRNA" +"Smith, Sgt Major" "Smith, Sir" "Smith, Sr." "Smith, V; MD" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 3c782527..6095b66b 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -10,6 +10,7 @@ {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "김. 민. J. S."} {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} +{"name": ", Dean", "given": "Dean", "initials": "D."} {"name": ", Dr. fils", "title": "Dr.", "family": "fils", "initials": "f.", "capitalized_force": "Dr. Fils"} {"name": ", Jane", "given": "Jane", "initials": "J."} {"name": ", Jane Mrs", "title": "Mrs", "family": "Jane", "initials": "J."} @@ -1107,6 +1108,7 @@ {"name": "Gottfried Huyn von Geleen und Amsterad", "given": "Gottfried", "middle": "Huyn", "family": "von Geleen und Amsterad", "initials": "G. H. G. A."} {"name": "Gottfried IV.", "given": "Gottfried", "suffix": "IV.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "G."} {"name": "Goëß, Johann von", "given": "Johann", "family": "von Goëß", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. G."} +{"name": "Grace Dean Smith", "given": "Grace", "middle": "Dean", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "G. D. S."} {"name": "Graf Lambsdorff, Otto", "title": "Graf", "given": "Otto", "family": "Lambsdorff", "initials": "O. L."} {"name": "Gragger, Robert", "given": "Robert", "family": "Gragger", "initials": "R. G."} {"name": "Gramont, Antoine de", "given": "Antoine", "family": "de Gramont", "initials": "A. G."} @@ -2996,6 +2998,7 @@ {"name": "Señ Baba Wagué Amdy Raas Sañsé MP", "given": "Señ", "middle": "Baba Wagué Amdy Raas", "family": "Sañsé", "suffix": "MP", "initials": "S. B. W. A. R. S."} {"name": "Señor Carlos García", "title": "Señor", "given": "Carlos", "family": "García", "initials": "C. G."} {"name": "Señora María García", "title": "Señora", "given": "María", "family": "García", "initials": "M. G."} +{"name": "Sgt Marshal Smith", "title": "Sgt", "given": "Marshal", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "M. S."} {"name": "Shahnon Ahmad", "given": "Shahnon", "family": "Ahmad", "initials": "S. A."} {"name": "Shanti Gaonkar", "given": "Shanti", "family": "Gaonkar", "initials": "S. G."} {"name": "Shehu Muhammed", "given": "Shehu", "family": "Muhammed", "initials": "S. M."} @@ -3191,6 +3194,7 @@ {"name": "Smith, Prof.", "title": "Prof.", "family": "Smith", "initials": "S."} {"name": "Smith, RA PhD Ma", "title": "RA", "given": "PhD", "middle": "Ma", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. M. S.", "capitalized_force": "Ra PhD Ma Smith"} {"name": "Smith, RN - CRNA", "family": "Smith", "suffix": "RN - CRNA", "initials": "S."} +{"name": "Smith, Sgt Major", "title": "Sgt Major", "family": "Smith", "initials": "S."} {"name": "Smith, Sir", "title": "Sir", "given": "Smith", "initials": "S."} {"name": "Smith, Sr.", "family": "Smith", "suffix": "Sr.", "initials": "S."} {"name": "Smith, V; MD", "given": "V;", "family": "Smith", "suffix": "MD", "initials": "V. S."} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 1b24a9ea..10504b5f 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -808,11 +808,15 @@ def _check_reading(r: Reading, pool: Pool) -> None: or len(set(att)) != len(att)): raise VocabError(f"{where}: attaches must be a non-empty " f"list of title kinds, each once") + # a title kind, or one title word where a kind would reach + # too far ('sergeant' attaching to 'major', not to every + # military title); a word is checked against the pool in + # check_pool for k in att: - if CATEGORY.get(k) != "title" or k == "title": + if k in CATEGORY and (CATEGORY[k] != "title" or k == "title"): raise VocabError( f"{where}: attaches names {k!r}, which is not a " - f"title kind; allowed: " + f"title kind or a title word; the kinds: " f"{sorted(t for t, c in CATEGORY.items() if c == 'title' and t != 'title')}") if r.flag("after_given") and r.kind != "salutation": # a salutation is a form of address, where an office or a @@ -1064,6 +1068,16 @@ def check_pool(pool: Pool) -> None: f"(rules/S11); classify the word as one, or " f"keep the counts in this reading's `note`") + # the words `attaches` names: each a word with a live title reading, + # else the phrase it asks for can never be made + title_words = {t for t, rs in live.items() + if any(r.category == "title" for r in rs)} + for r in pool.readings: + for k in r.props.get("attaches", ()) if r.category == "title" else (): + if k not in CATEGORY and k not in title_words: + raise VocabError(f"{r.where()}: attaches names {k!r}, which " + f"is no title kind and no title word") + # 9: a bound given name that is also a particle must be one that may # be given -- the bound join reads it as a given name. Lexicon raises # on the same contradiction; this says which file and word @@ -1262,7 +1276,8 @@ def compile_pool(pool: Pool) -> Compiled: # word before and each title word of a kind it names TITLE_PHRASES=frozenset( f"{t} {h}" for t, kinds in attaches.items() if kinds - for h, hk in title_kinds.items() if h != t and hk & kinds), + for h, hk in title_kinds.items() + if h != t and (hk & kinds or h in kinds)), NON_GIVEN_NAME_PARTICLES=frozenset(never_given), PARTICLES=frozenset(particles), CONJUNCTIONS=frozenset(conjunctions), From 2aebd821859141fc32c0a8f8e493af1f9db11474 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 12:58:53 -0700 Subject: [PATCH 5/7] Report a contested title only where a listed phrase holds it (#752, phase 2i) Derek's answer to the `Grace` report volume: a title that may be a given name and that no phrase in `title_phrases` holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no fork a reader hesitates over. `Grace Smith` and `GRACE SMITH` report nothing; `Dean Smith`, `Marshal Yanda`, `Justice Smith` and `Major Smith` still report, each held by a phrase (`chief` attaches to office and military; `major` attaches to `general`). Derived from the phrases `attaches` compiles, no new property. Measured against master a46b3a6e with --base: ctgov 195 moved, 117 report-only (was 318/240); tse 25, 16 report-only (was 94/85); FEC 53, unchanged. rules/H9 and decisions/H9 amended. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 8 ++++++-- docs/design/rules/H9-title-phrase.md | 5 ++++- docs/release_log.rst | 2 +- nameparser/_pipeline/_assign.py | 19 +++++++++++++------ nameparser/_pipeline/_vocab.py | 9 +++++++++ nameparser/vocab/README.md | 2 +- nameparser/vocab/en.toml | 2 +- tests/v2/cases.py | 20 +++++++++++++++----- tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 8 +++++--- tools/differential/corpus_rules.jsonl | 3 +++ tools/differential/snapshot.jsonl | 5 ++++- 12 files changed, 63 insertions(+), 22 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 3e949d0d..697ac620 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -3,7 +3,7 @@ ## Standing - A title that may be a given name is a title only inside a title phrase, and elsewhere reads as a word the vocabulary does not list: `Dean Smith` reads given 'Dean', `Dean of Students Jane Smith` title 'Dean of Students' — 2026-10-25 #752. -- The fork is reported `title-or-name` where the word is the first name word after any titles and no comma has marked the given name: `Dean Smith` and `Dean Smith, MD` report, `SMITH, DEAN MR.` does not — 2026-10-26 #752, phase 2i (Derek). +- The fork is reported `title-or-name` where the word is the first name word after any titles and no comma has marked the given name: `Dean Smith` and `Dean Smith, MD` report, `SMITH, DEAN MR.` does not — 2026-10-26 #752, phase 2i (Derek). Only where a phrase the vocabulary lists holds the word: `Grace Smith` reports nothing, `Marshal Yanda` does — 2026-10-26 #752, phase 2i, the report's reach (Derek). - `grace` (a style) and `marshal` (a rank) are titles that may be given names, `sergeant` attaches to the word `major` and `major` to `general` (`attaches` naming a word where a kind would reach too far): `His Grace John Smith`, `Field Marshal Bernard Montgomery` and `Sergeant Major John Smith` read the title; `Mr Justice Smith` keeps given 'Justice', no exception behind a salutation — 2026-10-26 #752, phase 2i (Derek). - Its partner is a modifier or a style before it, a title before it that attaches to it or a title after it that it attaches to (`attaches`), or `of` after it; a salutation is never one, nor any other title before it (#853's `Mr Dean Smith`), nor a title after it it does not attach to (`SMITH, DEAN MR.`) — 2026-10-25 #752. - A modifier in the leading run of titles before the first comma is a title only with a head title after it in that run, `of` after it, or in the object of `of` or `the` there; outside that run (ending the name, H5's, or after a comma) it reads as any title does, the given part after a family comma included (`Lau, Wing` → title 'Wing') — 2026-10-25 #752. @@ -23,10 +23,14 @@ THE SWEEPS. A grid of 34,380 parses (twelve fronts, salutations, `Dr.`, a dual second in a title run `Rev. Sr.`, the both-vocabulary heads `Freiherr`, `St` and `St.`, a style, a modifier and `Mr. and Mrs.`, before fifteen heads in their bare, capitals, lower-case and period-marked spellings, eight followers and eleven tails, in the comma shapes with an empty head, a nickname head, a particle surname and a title before the comma, under given-first and family-first orders) moved only the shapes this entry names. Two residuals read and accepted: a phrase extended by a coordinating join behind it (`Chief Dean and Mary Smith` reads title 'Chief Dean and Mary'), and nonsense heads (`Chief Justice General Smith`). THE REVIEW FIXES (code review and design-docs review of the first build; the corpus figures above were re-taken after both review rounds, on the final tree). A period-joined title in a part holding a contested or modifier word lost its tag (`Lt.Gov. Dean Smith` read given 'Lt.Gov.'): only the two sets' words are read out of the title now. The modifier fixes above. And v1's `c.titles.clear()` left `Dean of Students` a title, its words being in no v1 set: the shim drops `titles_ambiguous` with the titles. Accepted, in rules/H9: `Mr Justice Smith` (an English High Court judge's style) reads given 'Justice', a salutation being no partner; a modifier whose head is not listed loses the title (`His Grace John Smith` → given 'His', where master read title 'His', given 'Grace', and likewise `Most Worshipful Grand Master` and `Air Cdre`, whose heads are not titles; `Grand Duke of Tuscany`, duke refused by the vocabulary; `Field Marshal Bernard Montgomery` now given 'Field', marshal refused too, where it read title 'Field', given 'Marshal'), and so does one written after its head (`Princess Royal Anne`, `Mother Superior Mary Smith`); the given part after a family comma reads a modifier as listed (`Lau, Wing` → title 'Wing', where `Wing Lau` reads given 'Wing'), the cost decision 10 took by keeping the gate out of comma parts; and a coordinating join behind a phrase extends the title (`Chief Dean and Mary Smith`). -- **2026-10-26 #752, phase 2i (Derek) — NO REPORT WHERE A COMMA HAS MARKED THE GIVEN NAME; `grace` AND `marshal` ARE PHRASE-ONLY TITLES; `Mr Justice` STAYS A GIVEN NAME.** SUPERSEDES: in the 2026-10-25 #752, phase 2i entry, the report in a family comma's given part, its accepted costs for `His Grace John Smith` and `Field Marshal Bernard Montgomery`, and `major` attaching to the military kind. Derek's answers to the questions the PR raised, relayed 2026-10-26: +- **2026-10-26 #752, phase 2i (Derek) (`Grace` opening a name reporting, superseded by 2026-10-26 #752, phase 2i, the report's reach (Derek)) — NO REPORT WHERE A COMMA HAS MARKED THE GIVEN NAME; `grace` AND `marshal` ARE PHRASE-ONLY TITLES; `Mr Justice` STAYS A GIVEN NAME.** SUPERSEDES: in the 2026-10-25 #752, phase 2i entry, the report in a family comma's given part, its accepted costs for `His Grace John Smith` and `Field Marshal Bernard Montgomery`, and `major` attaching to the military kind. Derek's answers to the questions the PR raised, relayed 2026-10-26: - THE REPORT. In `SMITH, DEAN` the family comma has already marked `Dean` as the given name, so no reader hesitates, the test AGENTS.md sets for a reportable fork (its example is `Smith, John V`). The leading-title peel of a family comma's given part reports nothing now; the comma-less name and the part before a suffix comma still report (`Dean Smith, MD`). FEC's 1,737 report-only moves, every one a `FAMILY, DEAN` given part, are gone. - `grace` AND `marshal`. Each gains a title reading beside a dated `given` one, as `justice` did: `grace` a style ('His Grace'), `marshal` a military rank, `marshal`'s exclusion (#606) replaced. A modifier before each is its partner (`His`, `Field`), so `His Grace John Smith` reads title 'His Grace' (master read title 'His', given 'Grace') and `Field Marshal Bernard Montgomery` title 'Field Marshal' (1.4.0 through master read title 'Field', given 'Marshal'). `sergeant` and `sgt` attach to the word `major`, so `Sergeant Major John Smith` reads title 'Sergeant Major' (1.4.0 through master read title 'Sergeant', given 'Major'). - `attaches` MAY NAME A TITLE WORD, and `major` and `sergeant` name one. The review of these changes found that making `marshal` a contested military word widened every attachment to the military kind: the build made `sergeant marshal`, `sgt marshal` and `major marshal` phrases, so `Sgt Marshal Smith` lost the given name master and 1.4.0 read ('Marshal'), and `Major Marshal Smith` read title 'Major Marshal'. A kind reaches every word that later joins it, so a contested word joining the kind becomes a partner nobody chose; `sergeant` and `sgt` now attach to `major` and `major` to `general`, the one phrase each was for. `chief` keeps its kinds (office and military), its phrases with a contested word being real ranks and offices (`Chief Justice`, `Chief Marshal`). Each with a case row. - A STYLE OR MODIFIER IS A PARTNER ONLY WHERE IT IS A TITLE THERE. `grace` is now both a style and a title that may be a given name, so in `Grace Dean Smith` the rule as worded made `Grace` the partner of `Dean`; the tree read given 'Grace' anyway (the run stops at a name word), but the two readings disagreed. The partner before is now asked as read, left to right, and rules/H9 says so. Bearer counts are on the words (`uv run python tools/vocab/query.py --word grace`). - `Mr Justice`. An English High Court judge is styled `Mr Justice Smith`, so Derek asked for counts before deciding an exception. Measured 2026-10-26: the PSC register (fetch dated 2026-10-22) holds 17 records whose display name has a salutation and then `Justice`, and all 17 file `Justice` in the forename box; no record files it in the title box. ctgov-officials and fec-individuals hold none, and the committed corpora none outside rules/H9's own example. Every filer who writes it reads it as the given name, so no exception: rules/H9's Accepted clause stands. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). PSC 178 fixed, 3 broken, 16 changed, as before. FEC 53 moved, none for the report alone. ctgov-officials 318 moved, 240 for the report alone (`dean` 115, `grace` 123), and tse-donors 94, 85 for the report alone (`grace` 69, `dean` 14, `marshal` 2): `Grace` opening a name now reports, `grace` being a common given name. The committed corpora: `His Grace John Smith` and `Field Marshal Bernard Montgomery` read the title, `SMITH, DEAN MR.` drops its report. Consistency unchanged (jrc 35 and nepal 4 now alike, none apart). +- **2026-10-26 #752, phase 2i, the report's reach (Derek) — A TITLE NO LISTED PHRASE HOLDS REPORTS NOTHING STANDING ALONE.** SUPERSEDES: in the 2026-10-26 #752, phase 2i (Derek) entry, `Grace` opening a name reporting. Derek's answer to the volume that entry measured (`grace` 123 of ctgov's 240 report-only moves, 69 of tse's 85), relayed 2026-10-26: keep `Grace Smith` from reporting, by a rule derived from each word's phrases rather than a flag on the word. + - THE RULE. A contested word some phrase in `title_phrases` holds can head a title of itself, so a reader of `Dean Smith` can hesitate; a word no listed phrase holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no title anyone hesitates over, the test AGENTS.md sets for a reportable fork. The question is asked of the phrases `attaches` compiles (`_vocab.in_a_title_phrase`), so no property was added. + - WHICH WORDS FALL WHERE (the default vocabulary, 2026-10-26). Report: `dean`, `justice` and `marshal`, each held by a phrase only because `chief` attaches to their kinds (office, military: `chief dean`, `chief justice`, `chief marshal`), and `major`, which attaches to `general` and which `sergeant`, `sgt` and `chief` attach to. Silent: `grace`, a style, held by none, since no title attaches to the style kind. The partners the rule reads but `title_phrases` does not hold — a modifier or style before the word, `of` after it — are the same for every contested word, so they cannot tell one word from another, and the derivation does not read them. Its consequence: a word reports only while a phrase holds it, so narrowing `chief` to a list of words, or a caller's `Lexicon` with no phrases, silences the report for the words it drops. + - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). ctgov-officials 195 moved, 117 for the report alone (`dean` 115); tse-donors 25, 16 for the report alone (`dean` 14, `marshal` 2); fec-individuals 53, as before. The difference from the entry above is exactly the `grace` names (123 and 69). The committed corpora: `Grace Dean Smith` drops its report, the one move. diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 6c66f49e..6353ce6e 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -30,11 +30,14 @@ "Jane Smith Vice." → title="Vice." "John Smith, Vice" → title="Vice" ``` -- **A title that may be a given name, read as a name word where it is the first name word after any titles and no comma has marked the given name,** is reported as `title-or-name`; inside a phrase, later in the name, or in the given part after a family comma, it is not. +- **A title that may be a given name, read as a name word where it is the first name word after any titles and no comma has marked the given name,** is reported as `title-or-name` where a phrase the vocabulary lists holds it; inside a phrase, later in the name, or in the given part after a family comma, it is not. A word no listed phrase holds is a title only behind a style or a modifier, so standing alone it is no title a reader would hesitate over. ``` "Dean Smith" → ambiguities=("title-or-name",) + "Marshal Yanda" → ambiguities=("title-or-name",) "Mr Dean Smith" → ambiguities=("title-or-name",) "Dean Smith, MD" → ambiguities=("title-or-name",) + "Grace Smith" → ambiguities=() + "GRACE SMITH" → ambiguities=() "SMITH, DEAN MR." → ambiguities=() ", Dean" → ambiguities=() "Major General John Smith" → ambiguities=() diff --git a/docs/release_log.rst b/docs/release_log.rst index e23df453..fe14fe0c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,7 +12,7 @@ Release Log - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives first ``John``, last ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave first ``Pharm``, middle ``D``, last ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives first ``Pharm``, and one opening with a single letter is still two words (``John Smith, M. D.`` still gives first ``M.``). :issue:`752` - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave last ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives last ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``, the parse now reporting a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``). Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` + - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only behind a style such as ``His``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``). :issue:`752` - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index e160ebf9..b7d7541f 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -91,7 +91,8 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline._vocab import ( - _ROMAN, effective_script, is_initial_shaped, is_listed_numeral, + _ROMAN, effective_script, in_a_title_phrase, is_initial_shaped, + is_listed_numeral, is_suffix_lenient, numeral_leans_to_the_name, numeral_settled_by_case, reads_as_a_reported_numeral, reads_as_generation, resolve_script_set, @@ -150,7 +151,8 @@ def _absorbed(piece: Sequence[int], def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], ptags: tuple[frozenset[str], ...], tokens: list[WorkToken], - ambiguities: list[PendingAmbiguity] | None) -> int: + ambiguities: list[PendingAmbiguity] | None, + lexicon: Lexicon) -> int: """Assign TITLE to the leading title pieces and return the first non-title index. `ambiguities` is None where a family comma has already marked the part as the given name, so no fork is left to @@ -161,10 +163,13 @@ def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], # rules/H9: "read as a name word where it is the first name word # after any titles" -- the peel stops at it because classify read it # out of the title, so it is the first piece the peel leaves - # ('Dean Smith', 'Mr Dean Smith') + # ('Dean Smith', 'Mr Dean Smith'); and rules/H9: "a word no listed + # phrase holds is a title only behind a style or a modifier", so + # standing alone it reports nothing ('Grace Smith') if (ambiguities is not None and n < len(pieces) and len(pieces[n]) == 1 and TITLE_AMBIGUOUS_TAG in tokens[pieces[n][0]].tags - and "vocab:title" not in tokens[pieces[n][0]].tags): + and "vocab:title" not in tokens[pieces[n][0]].tags + and in_a_title_phrase(tokens[pieces[n][0]].text, lexicon)): i = pieces[n][0] ambiguities.append(PendingAmbiguity( AmbiguityKind.TITLE_OR_NAME, @@ -337,7 +342,8 @@ def _assign_main(seg_idx: int, state: ParseState, -- None on every path that returns before resolving one.""" pieces = state.pieces[seg_idx] ptags = state.piece_tags[seg_idx] - n = _peel_leading_titles(pieces, ptags, tokens, ambiguities) + n = _peel_leading_titles(pieces, ptags, tokens, ambiguities, + state.lexicon) if n == len(pieces): return None # group-flagged suffix pieces (the ph-d merge) are suffixes at ANY @@ -1306,7 +1312,8 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: # John Jr.'). # rules/H9: "no comma has marked the given name" -- here the # family comma has, so the peel reports nothing - n = _peel_leading_titles(pieces, ptags, tokens, None) + n = _peel_leading_titles(pieces, ptags, tokens, None, + state.lexicon) # rules/H5: "the title is TRANSPARENT to the suffix # reading: where two or more name words stand, what # stands once the chain is taken reads exactly as it diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 1f978907..cf42b20f 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1390,6 +1390,15 @@ def surname_unit_tags(text: str, lexicon: Lexicon, _OBJECT_LINKS = frozenset({"of", "the"}) +# rules/H9: "a word no listed phrase holds is a title only behind a +# style or a modifier" -- whether the word heads a title of itself, asked +# of the phrases `attaches` compiled rather than of a property of its own +def in_a_title_phrase(text: str, lexicon: Lexicon) -> bool: + """Whether some phrase in `lexicon.title_phrases` holds `text`.""" + n = _normalize(text) + return any(n in phrase.split(" ") for phrase in lexicon.title_phrases) + + # rules/H9: "the word is a title only inside a title phrase, and # elsewhere a name word" -- both halves, the partners of a title that may # be a given name and of a modifier at the head of the name, each its diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 4a529da1..f2496805 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -204,7 +204,7 @@ Four sets mark a clash with a name, and all four are derived: a word is in one w |---|---|---|---| | possibly a name, a word | a `generation` or `agnomen` reading and a `given` or `family` reading; a name reading beside any other suffix word kind is refused (check 11), no gate reading that clash | `SUFFIX_WORDS_AMBIGUOUS` | a suffix word that is also a name (`fils`, `neto`) is read as a suffix only where the name has words to spare, as an ambiguous acronym is: `Alexandre Dumas fils` and `Pedro Alves Neto` end in a suffix, `Marie Fils` and `Ana Neto` in a family name, either way reported ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md)). A generation's Title case in a name of mixed case leans it to the name, and its lower case settles it as the suffix wherever the name keeps a family word; an agnomen's writing settles nothing | | possibly a name | an acronym `suffix` reading and a `given` or `family` reading | `SUFFIX_ACRONYMS_AMBIGUOUS` | an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | -| possibly a name, a title | a title reading and a `given` reading, on a word with no particle reading | `TITLES_AMBIGUOUS`, and kept out of `TITLES` | a title that may be a given name (`dean`, `justice`, `major`) is a title only inside a title phrase: after a modifier, a style or a title it stands behind in a phrase `attaches` makes, or before `of` or a title it attaches to. Elsewhere it is a name word, reported where it opens the name: `Dean of Students` and `Major General Smith` read the title, `Dean Smith` and `Mr Major Smith` the given name ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase). A title that is also a particle (`st`, `freiherr`) is left to the particle rules, its `given` reading deriving the particle's marker alone. A salutation's given-name counts kept as evidence for `after_given` sit in the salutation reading's `note` (`mr`, `herr`), so they mark nothing | +| possibly a name, a title | a title reading and a `given` reading, on a word with no particle reading | `TITLES_AMBIGUOUS`, and kept out of `TITLES` | a title that may be a given name (`dean`, `justice`, `major`) is a title only inside a title phrase: after a modifier, a style or a title it stands behind in a phrase `attaches` makes, or before `of` or a title it attaches to. Elsewhere it is a name word, reported where it opens the name if a phrase `attaches` makes holds it (`grace`, held by none, is a title only behind a style or a modifier and reports nothing): `Dean of Students` and `Major General Smith` read the title, `Dean Smith` and `Mr Major Smith` the given name ([rules/H9](../../docs/design/rules/H9-title-phrase.md), the title phrase). A title that is also a particle (`st`, `freiherr`) is left to the particle rules, its `given` reading deriving the particle's marker alone. A salutation's given-name counts kept as evidence for `after_given` sit in the salutation reading's `note` (`mr`, `herr`), so they mark nothing | | possibly given | a `particle` reading and a `given` reading | kept out of `NON_GIVEN_NAME_PARTICLES` | a particle opening the name stays a given name there (`van Gogh`, given `van`) and the fork is reported, rather than starting the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold) | Before phase 2h (2026-10-25) a word reading of the suffix category beside a name reading marked nothing, and no shipped word had one. The marker is the clash, not the counts: a count recorded for a word that should stay unmarked goes in the clashing reading's `note`, as `ld`'s does, since a name reading would mark it. `may_be_initial` is not derived: its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index 968ef329..563b1a6c 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -72,7 +72,7 @@ style = {} [[word]] text = "grace" -style = { issue = 752, decided = 2026-10-26, expands = "'His Grace', a duke's or an archbishop's style", note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'His Grace John Smith' reads the title, 'Grace Smith' the given name" } +style = { issue = 752, decided = 2026-10-26, expands = "'His Grace', a duke's or an archbishop's style", note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'His Grace John Smith' reads the title, 'Grace Smith' the given name, unreported, since no listed phrase holds it" } given = { us_ssa = 536340, wikidata = 3908, as_of = 2026-10-26, note = "US SSA: female and male births added" } family = { us_census_2010 = 28098, wikidata = 311, as_of = 2026-10-26 } diff --git a/tests/v2/cases.py b/tests/v2/cases.py index cf97eee0..ed8b34e4 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11922,8 +11922,14 @@ def _check_cjk_shape_purity(self) -> None: "(Derek, 2026-10-26)"), Case("contested_style_alone", "Grace Smith", {"given": "Grace", "family": "Smith"}, - ambiguities=("title-or-name",), classification="feat(#752)", - notes="H9: no partner, so the given name, reported"), + classification="feat(#752)", + notes="H9: no partner, so the given name; and no listed phrase " + "holds 'grace', a title only behind a style or a " + "modifier, so nothing is reported (Derek, 2026-10-26)"), + Case("contested_style_alone_in_capitals", "GRACE SMITH", + {"given": "GRACE", "family": "SMITH"}, + classification="feat(#752)", + notes="H9: the row above in capitals, as TSE writes it"), Case("contested_rank_after_a_modifier", "Field Marshal Bernard Montgomery", {"title": "Field Marshal", "given": "Bernard", "family": "Montgomery"}, classification="fix(#752)", @@ -11945,7 +11951,9 @@ def _check_cjk_shape_purity(self) -> None: Case("contested_rank_alone", "Marshal Yanda", {"given": "Marshal", "family": "Yanda"}, ambiguities=("title-or-name",), classification="feat(#752)", - notes="H9: no partner, so the given name, reported"), + notes="H9: no partner, so the given name, reported: a listed " + "phrase holds 'marshal' ('chief marshal'), so it may head " + "a title"), Case("contested_rank_after_a_modifier_air", "Air Marshal John Smith", {"title": "Air Marshal", "given": "John", "family": "Smith"}, classification="fix(#752)", @@ -11953,9 +11961,11 @@ def _check_cjk_shape_purity(self) -> None: Case("contested_style_with_no_partner_before_a_contested_word", "Grace Dean Smith", {"given": "Grace", "middle": "Dean", "family": "Smith"}, - ambiguities=("title-or-name",), classification="feat(#752)", + classification="feat(#752)", notes="H9: a style is a partner only where it is a title there; " - "'Grace' stands in no phrase, so neither does 'Dean'"), + "'Grace' stands in no phrase, so neither does 'Dean', and " + "neither reports: 'Grace' is in no listed phrase and " + "'Dean' is not the first name word"), Case("contested_title_after_an_empty_head", ", Dean", {"given": "Dean"}, notes="H9: the comma has marked the given name, so no report"), diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 74309042..f30118f3 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -25,7 +25,7 @@ "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 478, "then_words": 200, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 5, "accepted_words": 162}, + "H9": {"words": 514, "then_words": 236, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 5, "accepted_words": 162}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 9c7f34b2..c2b98a1c 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -255,9 +255,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # five that report a title read as a name, under the three orders; # 1583 -> 1592 on 2026-10-26: Derek's answers on #894, three rows # that report ('Sgt Marshal', 'Marshal Yanda', 'Grace Dean'), the - # one that stopped ('SMITH, DEAN MR.') offset by 'Grace Smith'. - assert claims == 1592, ( - f"the field sweep checks {claims} claims, recorded as 1592 on " + # one that stopped ('SMITH, DEAN MR.') offset by 'Grace Smith'; + # 1592 -> 1586 the same day: 'Grace Smith' and 'Grace Dean Smith' + # stopped reporting, no listed phrase holding 'grace'. + assert claims == 1586, ( + f"the field sweep checks {claims} claims, recorded as 1586 on " f"2026-10-26") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 90b22686..95ad048d 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -180,6 +180,7 @@ "Freiherr von Berg MA" "Freiherr von Berg, Ed" "Freiherr von Richthofen V" +"GRACE SMITH" "Gabriel García Márquez" "Gabriel García Márquez Jr." "Gabriel García Márquez, PhD" @@ -201,6 +202,7 @@ "General Major John Smith" "Georg VI." "Grace Dean Smith" +"Grace Smith" "Graf Lambsdorff, Otto" "Grand Duke of Tuscany" "Greve Anna" @@ -561,6 +563,7 @@ "Marie FILS" "Marie Fils" "Marquess of Bath" +"Marshal Yanda" "Mary Beth Smith" "Mary Jane King" "Mary Jane King Smith" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 6095b66b..4c6d73c3 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1032,6 +1032,7 @@ {"name": "G. V. M.", "given": "G.", "middle": "V.", "family": "M.", "initials": "G. V. M."} {"name": "GEORGE SHANAHAN IV", "given": "GEORGE", "family": "SHANAHAN", "suffix": "IV", "initials": "G. S.", "capitalized": "George Shanahan IV"} {"name": "GOMINA MUSA Abɖu-Tɔyib", "order": "FAMILY_FIRST", "given": "MUSA", "middle": "Abɖu-Tɔyib", "family": "GOMINA", "initials": "M. A. G.", "capitalized_force": "Musa Abɖu-Tɔyib Gomina"} +{"name": "GRACE SMITH", "given": "GRACE", "family": "SMITH", "initials": "G. S.", "capitalized": "Grace Smith"} {"name": "GREGORY HOUSE M.D.", "given": "GREGORY", "family": "HOUSE", "suffix": "M.D.", "initials": "G. H.", "capitalized": "Gregory House M.D."} {"name": "Gabriel García Márquez", "given": "Gabriel", "middle": "García", "family": "Márquez", "initials": "G. G. M."} {"name": "Gabriel García Márquez Jr.", "given": "Gabriel", "middle": "García", "family": "Márquez", "suffix": "Jr.", "initials": "G. G. M."} @@ -1108,7 +1109,8 @@ {"name": "Gottfried Huyn von Geleen und Amsterad", "given": "Gottfried", "middle": "Huyn", "family": "von Geleen und Amsterad", "initials": "G. H. G. A."} {"name": "Gottfried IV.", "given": "Gottfried", "suffix": "IV.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "G."} {"name": "Goëß, Johann von", "given": "Johann", "family": "von Goëß", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. G."} -{"name": "Grace Dean Smith", "given": "Grace", "middle": "Dean", "family": "Smith", "ambiguities": ["TITLE_OR_NAME"], "initials": "G. D. S."} +{"name": "Grace Dean Smith", "given": "Grace", "middle": "Dean", "family": "Smith", "initials": "G. D. S."} +{"name": "Grace Smith", "given": "Grace", "family": "Smith", "initials": "G. S."} {"name": "Graf Lambsdorff, Otto", "title": "Graf", "given": "Otto", "family": "Lambsdorff", "initials": "O. L."} {"name": "Gragger, Robert", "given": "Robert", "family": "Gragger", "initials": "R. G."} {"name": "Gramont, Antoine de", "given": "Antoine", "family": "de Gramont", "initials": "A. G."} @@ -2375,6 +2377,7 @@ {"name": "Marković, Irena", "given": "Irena", "family": "Marković", "initials": "I. M."} {"name": "Markus von Wickenburg", "given": "Markus", "family": "von Wickenburg", "initials": "M. W."} {"name": "Marquess of Bath", "title": "Marquess of Bath"} +{"name": "Marshal Yanda", "given": "Marshal", "family": "Yanda", "ambiguities": ["TITLE_OR_NAME"], "initials": "M. Y."} {"name": "Marsick, Martin Pierre", "given": "Martin", "middle": "Pierre", "family": "Marsick", "initials": "M. P. M."} {"name": "Martha Nava García", "given": "Martha", "middle": "Nava", "family": "García", "initials": "M. N. G."} {"name": "Martin Pierre Marsick", "given": "Martin", "middle": "Pierre", "family": "Marsick", "initials": "M. P. M."} From d730db9a9ed6bd76a0f924a3e0e079cbfd5fe488 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 13:07:54 -0700 Subject: [PATCH 6/7] Ask the phrase test of readable phrases only, and cache it (#752, phase 2i) The review of 2aebd821: - in_a_title_phrase split every phrase per call, 175 frames on each name opening with Grace; the phrase words are now cached per vocabulary, as _marker_heads is (163 frames, below the parent's 169). - it counted phrases classify never reads (a caller's 'your grace', a three-word phrase), so one made Grace report; only two-word phrases of titles count now, as title_phrases' docstring says. - rules/H9's reason omitted 'of' after the word, which every contested word takes as a partner ('Grace of Monaco'); the rule, the code quotes and the release log say so now. - the titles_ambiguous and TITLE_OR_NAME docstrings name the condition. - 'Justice Smith' and 'Major Smith' rows pin the report surviving. No reading of the default vocabulary moves: its phrase words are the same. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 2 +- docs/design/rules/H9-title-phrase.md | 2 +- docs/release_log.rst | 2 +- nameparser/_lexicon.py | 5 ++++- nameparser/_pipeline/_assign.py | 4 ++-- nameparser/_pipeline/_vocab.py | 25 +++++++++++++++++++----- nameparser/_types.py | 5 +++++ tests/v2/cases.py | 12 ++++++++++++ tests/v2/pipeline/test_vocab.py | 20 +++++++++++++++++++ tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 7 ++++--- 11 files changed, 71 insertions(+), 15 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 697ac620..43749650 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -31,6 +31,6 @@ - `Mr Justice`. An English High Court judge is styled `Mr Justice Smith`, so Derek asked for counts before deciding an exception. Measured 2026-10-26: the PSC register (fetch dated 2026-10-22) holds 17 records whose display name has a salutation and then `Justice`, and all 17 file `Justice` in the forename box; no record files it in the title box. ctgov-officials and fec-individuals hold none, and the committed corpora none outside rules/H9's own example. Every filer who writes it reads it as the given name, so no exception: rules/H9's Accepted clause stands. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). PSC 178 fixed, 3 broken, 16 changed, as before. FEC 53 moved, none for the report alone. ctgov-officials 318 moved, 240 for the report alone (`dean` 115, `grace` 123), and tse-donors 94, 85 for the report alone (`grace` 69, `dean` 14, `marshal` 2): `Grace` opening a name now reports, `grace` being a common given name. The committed corpora: `His Grace John Smith` and `Field Marshal Bernard Montgomery` read the title, `SMITH, DEAN MR.` drops its report. Consistency unchanged (jrc 35 and nepal 4 now alike, none apart). - **2026-10-26 #752, phase 2i, the report's reach (Derek) — A TITLE NO LISTED PHRASE HOLDS REPORTS NOTHING STANDING ALONE.** SUPERSEDES: in the 2026-10-26 #752, phase 2i (Derek) entry, `Grace` opening a name reporting. Derek's answer to the volume that entry measured (`grace` 123 of ctgov's 240 report-only moves, 69 of tse's 85), relayed 2026-10-26: keep `Grace Smith` from reporting, by a rule derived from each word's phrases rather than a flag on the word. - - THE RULE. A contested word some phrase in `title_phrases` holds can head a title of itself, so a reader of `Dean Smith` can hesitate; a word no listed phrase holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no title anyone hesitates over, the test AGENTS.md sets for a reportable fork. The question is asked of the phrases `attaches` compiles (`_vocab.in_a_title_phrase`), so no property was added. + - THE RULE. A contested word some phrase in `title_phrases` holds can head a title of itself, so a reader of `Dean Smith` can hesitate; a word no listed phrase holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no title anyone hesitates over, the test AGENTS.md sets for a reportable fork. The question is asked of the phrases `attaches` compiles (`_vocab.in_a_title_phrase`), so no property was added, and only of the phrases classify can read, two words each a title or a title that may be a given name: a caller's `your grace`, whose `your` is no title, makes `Grace Smith` report nothing (the review of this change). The review also measured the first version splitting every phrase per call, 175 frames on each name opening with `Grace`; the words are now cached per vocabulary. - WHICH WORDS FALL WHERE (the default vocabulary, 2026-10-26). Report: `dean`, `justice` and `marshal`, each held by a phrase only because `chief` attaches to their kinds (office, military: `chief dean`, `chief justice`, `chief marshal`), and `major`, which attaches to `general` and which `sergeant`, `sgt` and `chief` attach to. Silent: `grace`, a style, held by none, since no title attaches to the style kind. The partners the rule reads but `title_phrases` does not hold — a modifier or style before the word, `of` after it — are the same for every contested word, so they cannot tell one word from another, and the derivation does not read them. Its consequence: a word reports only while a phrase holds it, so narrowing `chief` to a list of words, or a caller's `Lexicon` with no phrases, silences the report for the words it drops. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). ctgov-officials 195 moved, 117 for the report alone (`dean` 115); tse-donors 25, 16 for the report alone (`dean` 14, `marshal` 2); fec-individuals 53, as before. The difference from the entry above is exactly the `grace` names (123 and 69). The committed corpora: `Grace Dean Smith` drops its report, the one move. diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 6353ce6e..d756c6e4 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -30,7 +30,7 @@ "Jane Smith Vice." → title="Vice." "John Smith, Vice" → title="Vice" ``` -- **A title that may be a given name, read as a name word where it is the first name word after any titles and no comma has marked the given name,** is reported as `title-or-name` where a phrase the vocabulary lists holds it; inside a phrase, later in the name, or in the given part after a family comma, it is not. A word no listed phrase holds is a title only behind a style or a modifier, so standing alone it is no title a reader would hesitate over. +- **A title that may be a given name, read as a name word where it is the first name word after any titles and no comma has marked the given name,** is reported as `title-or-name` where a phrase the vocabulary lists holds it; inside a phrase, later in the name, or in the given part after a family comma, it is not. A word no listed phrase holds is a title only behind a style or a modifier, or before `of`, the partners every such word shares, so standing alone it is no title a reader would hesitate over. ``` "Dean Smith" → ambiguities=("title-or-name",) "Marshal Yanda" → ambiguities=("title-or-name",) diff --git a/docs/release_log.rst b/docs/release_log.rst index fe14fe0c..6542cb02 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,7 +12,7 @@ Release Log - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives first ``John``, last ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave first ``Pharm``, middle ``D``, last ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives first ``Pharm``, and one opening with a single letter is still two words (``John Smith, M. D.`` still gives first ``M.``). :issue:`752` - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave last ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives last ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only behind a style such as ``His``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` + - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``). :issue:`752` - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index bffc9ae5..dc45a396 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -732,7 +732,10 @@ class Lexicon: #: modifier, a style, or a title it stands in ``title_phrases`` #: behind; or before "of" or a title it stands in ``title_phrases`` #: in front of. Elsewhere it is a name word, and where it opens the - #: name the parse reports ``title-or-name`` (rules/H9): "Dean Smith" + #: name the parse reports ``title-or-name`` if a phrase of + #: ``title_phrases`` holds it (rules/H9), so "Grace Smith", which no + #: phrase holds, reports nothing, and emptying ``title_phrases`` + #: silences the report: "Dean Smith" #: and "Mr Major Smith" read given "Dean" and "Major", "Dean of #: Students Smith" and "Major General Smith" a title. An entry also #: in ``titles`` is a title wherever it stands, which is what listing diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index b7d7541f..d54b853e 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -164,8 +164,8 @@ def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], # after any titles" -- the peel stops at it because classify read it # out of the title, so it is the first piece the peel leaves # ('Dean Smith', 'Mr Dean Smith'); and rules/H9: "a word no listed - # phrase holds is a title only behind a style or a modifier", so - # standing alone it reports nothing ('Grace Smith') + # phrase holds is a title only behind a style or a modifier, or + # before `of`", so standing alone it reports nothing ('Grace Smith') if (ambiguities is not None and n < len(pieces) and len(pieces[n]) == 1 and TITLE_AMBIGUOUS_TAG in tokens[pieces[n][0]].tags and "vocab:title" not in tokens[pieces[n][0]].tags diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index cf42b20f..a73d29d8 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1391,12 +1391,27 @@ def surname_unit_tags(text: str, lexicon: Lexicon, # rules/H9: "a word no listed phrase holds is a title only behind a -# style or a modifier" -- whether the word heads a title of itself, asked -# of the phrases `attaches` compiled rather than of a property of its own +# style or a modifier, or before `of`" -- whether the word heads a title +# of itself, asked of the phrases `attaches` compiled rather than of a +# property of its own def in_a_title_phrase(text: str, lexicon: Lexicon) -> bool: - """Whether some phrase in `lexicon.title_phrases` holds `text`.""" - n = _normalize(text) - return any(n in phrase.split(" ") for phrase in lexicon.title_phrases) + """Whether a phrase of `lexicon.title_phrases` that classify can + read holds `text`.""" + return _normalize(text) in _phrase_words( + lexicon.title_phrases, lexicon.titles, lexicon.titles_ambiguous) + + +@functools.lru_cache(maxsize=16) +def _phrase_words(phrases: frozenset[str], titles: frozenset[str], + contested: frozenset[str]) -> frozenset[str]: + """The words of the phrases classify can read: two words, each a + title or a title that may be a given name, as its pair test asks. + A phrase it never consults makes no word report.""" + listed = titles | contested + pairs = (p.split(" ") for p in phrases) + return frozenset(w for ws in pairs + if len(ws) == 2 and all(w in listed for w in ws) + for w in ws) # rules/H9: "the word is a title only inside a title phrase, and diff --git a/nameparser/_types.py b/nameparser/_types.py index efe70fdf..ed19c7ea 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -590,6 +590,11 @@ class AmbiguityKind(StrEnum): #: and whether the title word inside it is a title is the fork. #: Not the title-vs-given-name collision on a word like "Baron", #: which is a question about the VOCABULARY and is not this kind. + #: A title that may also be a given name ("Dean", rules/H9) reports + #: it too, where it opens the name outside a title phrase and no + #: comma has marked the given name: "Dean Smith" gives given "Dean". + #: Only a word some phrase of ``Lexicon.title_phrases`` holds + #: reports, so "Grace Smith" does not. TITLE_OR_NAME = "title-or-name" #: An ambiguous particle is either a particle or a name in its own #: right -- "Van Johnson" is the actor's given name, a bare diff --git a/tests/v2/cases.py b/tests/v2/cases.py index ed8b34e4..52d1c7e5 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11940,6 +11940,18 @@ def _check_cjk_shape_purity(self) -> None: {"title": "Sergeant Major", "given": "John", "family": "Smith"}, classification="fix(#752)", notes="H9: sergeant attaches to a military title after it"), + Case("contested_office_alone_held_by_chief", "Justice Smith", + {"given": "Justice", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: no partner, so the given name, reported: 'chief " + "justice' holds the word, through chief's attaching to " + "the office kind"), + Case("contested_rank_alone_held_by_its_own_phrase", "Major Smith", + {"given": "Major", "family": "Smith"}, + ambiguities=("title-or-name",), classification="feat(#752)", + notes="H9: no partner, so the given name, reported: 'major " + "general' and 'sergeant major' hold the word, phrases " + "naming it rather than its kind"), Case("contested_rank_after_a_rank_that_attaches_elsewhere", "Sgt Marshal Smith", {"title": "Sgt", "given": "Marshal", "family": "Smith"}, diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 1746734e..15b3e622 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -1,3 +1,4 @@ +import dataclasses import unicodedata import pytest @@ -1173,3 +1174,22 @@ def test_a_contested_title_in_a_phrase_addresses_as_its_vocabulary_says() -> Non assert "vocab:given-title" in phrase.tokens[1].tags alone = Parser(lexicon=lex).parse("Zed John") assert (alone.title, alone.given, alone.family) == ("", "Zed", "John") + + +@pytest.mark.parametrize("phrases, reports", [ + (frozenset(), False), + # a title before the word: classify reads the pair, so the word may + # head a title of itself and opening a name it reports + (frozenset({"chief grace"}), True), + # 'your' is no title, so classify never reads the pair, and a phrase + # it never reads makes no word report + (frozenset({"your grace"}), False), + # classify reads two-word phrases only + (frozenset({"chief of grace"}), False), +]) +def test_only_a_phrase_classify_can_read_makes_a_contested_word_report( + phrases: frozenset[str], reports: bool) -> None: + lex = dataclasses.replace(Lexicon.default(), title_phrases=phrases) + kinds = [a.kind.value for a in Parser(lexicon=lex).parse( + "Grace Smith").ambiguities] + assert kinds == (["title-or-name"] if reports else []) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index f30118f3..73f843b2 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -25,7 +25,7 @@ "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 514, "then_words": 236, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 5, "accepted_words": 162}, + "H9": {"words": 523, "then_words": 245, "because_sentences": 3, "except": 0, "precedence": 4, "accepted": 5, "accepted_words": 162}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index c2b98a1c..23a01e56 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -257,9 +257,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # that report ('Sgt Marshal', 'Marshal Yanda', 'Grace Dean'), the # one that stopped ('SMITH, DEAN MR.') offset by 'Grace Smith'; # 1592 -> 1586 the same day: 'Grace Smith' and 'Grace Dean Smith' - # stopped reporting, no listed phrase holding 'grace'. - assert claims == 1586, ( - f"the field sweep checks {claims} claims, recorded as 1586 on " + # stopped reporting, no listed phrase holding 'grace'; 1586 -> 1592 + # the same day: 'Justice Smith' and 'Major Smith', which report. + assert claims == 1592, ( + f"the field sweep checks {claims} claims, recorded as 1592 on " f"2026-10-26") From 21b34ea7a457520fb4aca67f97b408e8b7fe4b57 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 13:12:14 -0700 Subject: [PATCH 7/7] Say which title phrases the parse reads (#752, phase 2i) The second review of d730db9a: title_phrases' docstring said a phrase needs both words in titles, where classify and the report read titles or titles_ambiguous; and titles_ambiguous' said any phrase holding the word makes it report, where only a readable one does. decisions/H9's frame figure now says 175 more (344 against 169). Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 2 +- nameparser/_lexicon.py | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 43749650..ecfed54c 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -31,6 +31,6 @@ - `Mr Justice`. An English High Court judge is styled `Mr Justice Smith`, so Derek asked for counts before deciding an exception. Measured 2026-10-26: the PSC register (fetch dated 2026-10-22) holds 17 records whose display name has a salutation and then `Justice`, and all 17 file `Justice` in the forename box; no record files it in the title box. ctgov-officials and fec-individuals hold none, and the committed corpora none outside rules/H9's own example. Every filer who writes it reads it as the given name, so no exception: rules/H9's Accepted clause stands. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). PSC 178 fixed, 3 broken, 16 changed, as before. FEC 53 moved, none for the report alone. ctgov-officials 318 moved, 240 for the report alone (`dean` 115, `grace` 123), and tse-donors 94, 85 for the report alone (`grace` 69, `dean` 14, `marshal` 2): `Grace` opening a name now reports, `grace` being a common given name. The committed corpora: `His Grace John Smith` and `Field Marshal Bernard Montgomery` read the title, `SMITH, DEAN MR.` drops its report. Consistency unchanged (jrc 35 and nepal 4 now alike, none apart). - **2026-10-26 #752, phase 2i, the report's reach (Derek) — A TITLE NO LISTED PHRASE HOLDS REPORTS NOTHING STANDING ALONE.** SUPERSEDES: in the 2026-10-26 #752, phase 2i (Derek) entry, `Grace` opening a name reporting. Derek's answer to the volume that entry measured (`grace` 123 of ctgov's 240 report-only moves, 69 of tse's 85), relayed 2026-10-26: keep `Grace Smith` from reporting, by a rule derived from each word's phrases rather than a flag on the word. - - THE RULE. A contested word some phrase in `title_phrases` holds can head a title of itself, so a reader of `Dean Smith` can hesitate; a word no listed phrase holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no title anyone hesitates over, the test AGENTS.md sets for a reportable fork. The question is asked of the phrases `attaches` compiles (`_vocab.in_a_title_phrase`), so no property was added, and only of the phrases classify can read, two words each a title or a title that may be a given name: a caller's `your grace`, whose `your` is no title, makes `Grace Smith` report nothing (the review of this change). The review also measured the first version splitting every phrase per call, 175 frames on each name opening with `Grace`; the words are now cached per vocabulary. + - THE RULE. A contested word some phrase in `title_phrases` holds can head a title of itself, so a reader of `Dean Smith` can hesitate; a word no listed phrase holds is a title only behind a style or a modifier (`His Grace`), so standing alone it is no title anyone hesitates over, the test AGENTS.md sets for a reportable fork. The question is asked of the phrases `attaches` compiles (`_vocab.in_a_title_phrase`), so no property was added, and only of the phrases classify can read, two words each a title or a title that may be a given name: a caller's `your grace`, whose `your` is no title, makes `Grace Smith` report nothing (the review of this change). The review also measured the first version splitting every phrase per call, 175 frames more on each name opening with `Grace` (344 against 169); the words are now cached per vocabulary. - WHICH WORDS FALL WHERE (the default vocabulary, 2026-10-26). Report: `dean`, `justice` and `marshal`, each held by a phrase only because `chief` attaches to their kinds (office, military: `chief dean`, `chief justice`, `chief marshal`), and `major`, which attaches to `general` and which `sergeant`, `sgt` and `chief` attach to. Silent: `grace`, a style, held by none, since no title attaches to the style kind. The partners the rule reads but `title_phrases` does not hold — a modifier or style before the word, `of` after it — are the same for every contested word, so they cannot tell one word from another, and the derivation does not read them. Its consequence: a word reports only while a phrase holds it, so narrowing `chief` to a list of words, or a caller's `Lexicon` with no phrases, silences the report for the words it drops. - MEASURED (2026-10-26, against master `a46b3a6e`, every local check with `--base`). ctgov-officials 195 moved, 117 for the report alone (`dean` 115); tse-donors 25, 16 for the report alone (`dean` 14, `marshal` 2); fec-individuals 53, as before. The difference from the entry above is exactly the `grace` names (123 and 69). The committed corpora: `Grace Dean Smith` drops its report, the one move. diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index dc45a396..c64ab7cd 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -733,7 +733,7 @@ class Lexicon: #: behind; or before "of" or a title it stands in ``title_phrases`` #: in front of. Elsewhere it is a name word, and where it opens the #: name the parse reports ``title-or-name`` if a phrase of - #: ``title_phrases`` holds it (rules/H9), so "Grace Smith", which no + #: ``title_phrases`` the parse can read holds it (rules/H9), so "Grace Smith", which no #: phrase holds, reports nothing, and emptying ``title_phrases`` #: silences the report: "Dean Smith" #: and "Mr Major Smith" read given "Dean" and "Major", "Dean of @@ -761,8 +761,9 @@ class Lexicon: #: Two-word title phrases, a title and the title it qualifies #: ("chief justice", "major general"), stored space-joined: a word #: of ``titles_ambiguous`` on either side of one is in a title - #: phrase (rules/H9). A phrase whose words are not both in - #: ``titles`` is never consulted. Full default list: + #: phrase (rules/H9). A phrase is consulted only if it is two + #: words, each in ``titles`` or ``titles_ambiguous``. Full default + #: list: #: :data:`~nameparser.config.titles.TITLE_PHRASES`. title_phrases: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's