From 1d37da752ef7da6d353353d1fa9c0f413470d688 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Mon, 26 Oct 2026 19:38:32 -0700 Subject: [PATCH] Read the given names Sultana and Sultan as names outside a title phrase (#752 phase 2j, #853) The fifth and sixth title batches of phase 2j, combined by Derek. `sultana` and `sultan` gain dated given readings, so each is a title only inside a title phrase, unreported. `Sultana Begum`, `Sultan Khan`, `Mr Sultan Khan` and `Khan, Sultan` read the given name, where 1.4.0 through 2.6.0 read a title; `His Majesty Sultan Hassanal Bolkiah` and `Sultan of Brunei` keep it. The cost, the class accepted for `king`: a sultan written with the bare title reads it as the given name. Measured against master 6ba78d0d: PSC 111 fixed, 1 broken (a title-box Sultan); ctgov 41, FEC 21, tse 1; jrc 9 and nepal 6 now alike, 0 apart. `sultana` alone: PSC 28 fixed, 0 broken, every move a fix. AGENTS.md records #901's lesson: a word whose reading moves may be its class's worked example. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions/H9-title-phrase.md | 7 +++++ docs/design/decisions/vocabulary-model.md | 1 + docs/design/rules/H9-title-phrase.md | 4 ++- docs/design/vocabulary-model-rfc.md | 2 +- docs/release_log.rst | 1 + nameparser/config/_compiled.py | 4 +-- nameparser/vocab/ar.toml | 4 ++- nameparser/vocab/tr.toml | 4 ++- tests/v2/cases.py | 32 +++++++++++++++++++++++ tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 8 +++--- tools/differential/corpus_rules.jsonl | 2 ++ tools/differential/snapshot.jsonl | 2 ++ 14 files changed, 64 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0c6232f4..eb62a4b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -271,7 +271,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. A word whose reading moves may be the worked example of the class it is leaving: `king` was the rules' and tests' title-that-is-also-a-surname across 35 files when #901 gave it a given reading, and five tests went vacuous before the review found them. Grep rules, decisions Standing lines, the RFCs, AGENTS.md, docs/*.rst, code comments and tests for the word before measuring, and replace it with a stand-in checked to read exactly as the word read on master in every example (`pope` there). A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index f7a0b4cc..6eddcdb7 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -56,3 +56,10 @@ - `king` WAS THE WORKED EXAMPLE OF A TITLE THAT IS ALSO A SURNAME, in rules H1, H3 to H8, M2, S2 and S8, their decisions Standing lines, AGENTS.md, docs/usage.rst, two code comments and eleven case rows and tests. Each now uses `pope`, chosen because it reads as `king` read on master in every one of those examples (19 inputs checked against `queen`, `bishop`, `pope`, `baron`, `judge`, `sheriff`, `mayor` and `emperor`; only `pope` and `queen` matched all, `pope` being the commoner surname). decisions/v1-xfail-triage's "`king` stays in TITLES" is superseded there. - MEASURED (2026-10-26, against master `957509d1`, py3.11, every local check with `--base`). PSC (1,039,174 scored): 58 records fixed, none broken, 5 changed. fec-individuals 73 moved, all fixes: `AA, KING …` given names (51) and `KING AA, AA` family parts that H6 had peeled (22). ctgov-officials 12, all fixes: officials whose given name is King (`King A. Aa`), and one whose credential master read as the given name (`Aa King, AA Aa` → given 'Aa', family 'King', the credentials suffixes). tse-donors, ukec-individuals, jrc-names and nepal-candidates pairs (0 now alike, 0 apart: both spellings of a person move together), CLDR and GND agreement do not move. The committed corpora move 11 names before the rules examples were rewritten: the three royal rows, `King Curtis` (#348's example, fixed), `King Smith, John` and `Mary Jane King.` (1.4.0's readings), a maiden clause, and four that only lose H4's `title-or-name`, the word no longer being title vocabulary (`Dr. King`, `Dr King Jr`, `Dr. King MD`; `Smith and King` reports `given-or-family` instead). Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local fec-individuals --base ` (and `ctgov-officials`). +- **2026-10-26 #752, phase 2j, `sultana` and `sultan` (Derek) — THE FIFTH AND SIXTH TITLE BATCHES, COMBINED.** The batches decisions/vocabulary-model's 2026-10-24 #752 (Derek) entry put fifth and sixth; measured apart and together, `sultana` costing nothing and `sultan` only the royal class accepted for `king` (the entry above), Derek combined them on 2026-10-26. Each gains a dated `given` reading (`sultan`'s on its tr.toml entry, the other two files' readings deriving the same marker) and is a title only inside a title phrase; no listed phrase holds either, so nothing is reported. `Sultana Begum`, `Mrs Sultana Begum`, `Sultan Khan`, `Mr Sultan Khan`, `Khan, Sultan` and `Sultan Bin Zayed` read the given name, where 1.4.0 through 2.6.0 read the title (and `Mrs Sultana`, `Mr Sultan`). A style keeps the title (`His Majesty Sultan Hassanal Bolkiah`), as does `of` (`Sultan of Brunei`), and the Ottoman writing after the name was already a name word (`Hurrem Sultan`). The other sites read as for `king`. + - THE COUNTS, recorded on the words (`uv run python tools/name_usage.py Sultana Sultan`, 2026-10-26): `sultana` given US SSA 275 births, Wikidata 19 (Bangladesh 5), nam_dict band 3 in Romania and Arabia/Persia, family Census 3,471, Wikidata 34; `sultan` given US SSA 2,396, Wikidata 151 (Turkey 28, Saudi Arabia 13), nam_dict band 7 in Turkey, family Census 3,363, Wikidata 106 (Ottoman Empire 26, the title written after the name). + - THE COST, `sultan`'s only: a sultan written with the bare title in front reads it as the given name (`Sultan Hassanal Bolkiah` → given 'Sultan', where 1.4.0 through 2.6.0 read title 'Sultan', given 'Hassanal'; so `Sultan Qaboos bin Said`), the class accepted for `king`; in PSC one filer typed `Sultan` in the title box before a name, the one broken record. Sized afterwards, at the review's prompting, in the news text of jrc-names: 135 persons (266 spellings) open with `Sultan` or `Sultana`, by a reading of each about 40 rulers (mostly Ottoman sultans and today's Gulf and Malay rulers), about 80 people whose given name is Sultan, and 13 mosques and other places, a cost about four times `king`'s 10 there. + - FOUND BY THE REVIEW, NOT FIXED HERE: A v1 `remove` OF A WORD 2.6.0 LISTED NOW RAISES. A title that gains a given reading compiles out of `TITLES` (the 2026-10-25 #752, phase 2i entry of decisions/vocabulary-model), so `Constants().titles` no longer holds it and v1's strict `remove` raises `KeyError`: `c.titles.remove('sultan')`, which 2.6.0 accepted (and which then read `Sultan Khan` with first 'Sultan'), raises on this tree, and so does it for every word phase 2j moved out of the titles 2.6.0 listed (`wing`, `mahdi`, `bard`, `buddha`, `siddha`, `gyani`, `king`, `sultana`); phase 2i's own words were never in 2.6.0's titles, so removing them raised before too. The shim is not to raise on a config v1 accepted (AGENTS.md), so this is a defect of the batches, left for its own change before 2.7.0 ships. + - NOT REACHED, for `sheikh`'s batch. `sheikh` is a style, so under rules/H9 it partners a contested word behind it: `Sheikh Sultan Khan` keeps title 'Sheikh Sultan', as master reads it and as `Sheikh Dean Smith` reads, where in Gulf names the word is the person's given name. + - MEASURED (2026-10-26, against master `6ba78d0d`, py3.11, every local check with `--base`). Both words: PSC (1,039,174 scored) 111 records fixed, 1 broken (the title box above), 6 changed, each still wrong for another reason, mostly a two-word surname box; ctgov-officials 41 moved; fec-individuals 21; tse-donors 1; spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE) jrc-names 9 persons and nepal-candidates 6 pairs now alike, none apart. `sultana` alone: PSC 28 fixed, none broken, 1 changed (a two-word surname box); ctgov 7, every one a fix, 6 of them `Aa Sultana, CRED` names whose credential master read as the given name (`Nasreen Sultana, MBBS` → given 'Nasreen', family 'Sultana', suffix 'MBBS', reported `suffix-or-name`); FEC 5 (`AA, SULTANA`); tse 1; nepal 1. The committed corpora and CLDR and GND agreement do not move. + Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`). diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index b53601e8..32f64e88 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -218,6 +218,7 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest. Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. +- **2026-10-26 #752, phase 2j, `sultana` and `sultan` (Derek) — THE FIFTH AND SIXTH TITLE BATCHES, IN ONE PR: both have dated `given` readings**, `sultan`'s on its tr.toml entry, so each is a title only inside a title phrase. `Sultana Begum` and `Sultan Khan` read the given name; a sultan written with the bare title does too (`Sultan Hassanal Bolkiah`), the cost accepted for `king`. Derek combined the two batches on 2026-10-26. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `king` (Derek) — THE FOURTH TITLE BATCH: `king` has a dated `given` reading**, US SSA 37,890 births and Census rank 34 as a surname, so it is a title only inside a title phrase. `King Curtis` and `Mr King Smith` read given 'King'; a monarch written with the bare title does too (`King Henry`), the cost Derek accepted on 2026-10-26. `pope` replaces `king` as the worked example of a title that is also a surname wherever the rules and tests used it. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `buddha`, `siddha` and `gyani` (#869) — THE THIRD TITLE BATCH: each has a declared `given` reading**, `siddha`'s and `gyani`'s without a count, which their notes explain: the first halves of Nepali compound given names, which no source outside nepal-candidates counts. `Buddha Bahadur Tamang` reads given 'Buddha', middle 'Bahadur'; `guru` stays a title for a `ne` pack (the 2026-10-24 #752 (Derek) entry). The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `mahdi` and `bard` (#867) — THE SECOND TITLE BATCH: both words have a dated `given` reading.** `Mahdi Karimi`, `Mr Mahdi Karimi`, `Karimi, Mahdi` and `Bard Larsen` read the given name, where 1.4.0 through 2.6.0 read a title; no listed phrase holds either word, so nothing is reported. The measurement is decisions/H9's entry of this date. diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 1f5f19c4..ea17f1e9 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -22,6 +22,7 @@ "Mr Mahdi Karimi" → given="Mahdi" "Buddha Bahadur Tamang" → given="Buddha" "King Curtis" → given="King" + "Sultana Begum" → given="Sultana" ``` - **A modifier in the leading run of titles before the first comma** has its phrase in a head title after it in that run, in `of` after it, or in the object of `of` or `the` there. A modifier outside that run, ending the name or after a comma, is read as any title is, save one that may be a given name, which is read everywhere as it is in that run. ``` @@ -119,9 +120,10 @@ "Gyani Zail Singh" → given="Gyani" "Lord Buddha Tamang" → given="Buddha" ``` -- A monarch written with the bare title in front reads it as the given name, a style before it or `of` after it keeping the title; the record data at hand writes the word there as a given name, and only news text writes monarchs that way. +- A monarch written with the bare title in front reads it as the given name, a style before it or `of` after it keeping the title; the record data at hand writes the word there almost only as a given name, and monarchs that way mostly in news text. ``` "King Henry" → given="King" + "Sultan Hassanal Bolkiah" → given="Sultan" "His Majesty King Charles" → title="His Majesty King" ``` - Once a partner has made the word a title, a coordinating join behind it takes the next word into the title too. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index f9f03e51..998bc1b0 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -118,7 +118,7 @@ A reading whose writing offers no cue is one the gate could only guess, so it is The front of a name has no words-to-spare cue: a title addresses the name after it. What it has is the contested-head rule. **A title word that also has a name reading is a title only inside a title phrase:** -- **The title before it is not a partner.** A contested word after another title is the given name (`Mr Sultan Khan`, `Ms Princess Ada Obi`, #853), unless that title is its partner; today `Mr Sultan Khan` reads title `Mr Sultan`. +- **The title before it is not a partner.** A contested word after another title is the given name (`Mr Sultan Khan`, `Ms Princess Ada Obi`, #853), unless that title is its partner; `Mr Sultan Khan` read title `Mr Sultan` until phase 2j gave `sultan` its given reading. - **Its partner** is a modifier or a style before it (`Her Royal Highness Princess Anne`), `of` after it (`Dean of Students`), or a head title after it that the word `attaches` to (`Major General`). A salutation is never a partner (`SMITH, DEAN MR.` is a given name and a salutation). - **Modifiers** (decision 10, unparked) are titles only beside a head title, at the head of the name only, never in a comma part, where a lone title word is the person's role; a modifier that is also a given name (`wing`, phase 2j) is read by that test in every part. diff --git a/docs/release_log.rst b/docs/release_log.rst index 1a1ae388..51a8bb5d 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -14,6 +14,7 @@ Release Log - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752` - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` + - **Fix the given names Sultan and Sultana reading as titles.** ``Sultan Khan`` and ``Sultana Begum`` give first ``Sultan`` and ``Sultana``, where 1.4.0 through 2.6.0 gave title ``Sultan`` and ``Sultana`` with no first name; ``Mr Sultan Khan`` and ``Khan, Sultan`` read the same. A sultan written with the bare title reads it as a first name too (``Sultan Hassanal Bolkiah`` gives first ``Sultan``), where a style keeps it (``His Majesty Sultan Hassanal Bolkiah``). :issue:`752`, :issue:`853` - **Fix the given name King reading as a title.** ``King Curtis`` and ``Mr King Smith`` give first ``King``, where 1.4.0 through 2.6.0 gave title ``King`` and ``Mr King``; ``SMITH, KING`` reads the same. A monarch written with the bare title reads it as a first name too (``King Henry`` gives first ``King``, last ``Henry``), where a style keeps it (``His Majesty King Henry``). :issue:`752`, :issue:`348` - **Fix the Nepali given names Buddha, Siddha and Gyani reading as titles.** ``Buddha Bahadur Tamang`` and ``Gyani Maya Gurung`` give first ``Buddha`` and ``Gyani``, middle ``Bahadur`` and ``Maya``, where 1.4.0 through 2.6.0 gave title ``Buddha`` and ``Gyani``, first ``Bahadur`` and ``Maya``; ``Siddha Raj Joshi`` and ``Tamang, Buddha Bahadur`` read the same. ``Guru`` is still a title. The Sikh title written ``Gyani`` reads as a given name too (``Gyani Zail Singh``); its usual spelling ``Giani`` is still a title. :issue:`869`, :issue:`752` - **Fix the given names Mahdi and Bard reading as titles.** ``Mahdi Karimi`` and ``Bard Larsen`` give first ``Mahdi`` and ``Bard``, where 1.4.0 through 2.6.0 gave title ``Mahdi`` and ``Bard`` with no first name; ``Mr Mahdi Karimi`` and ``Karimi, Mahdi`` read the same. Inside a title phrase either is still a title (``Mahdi of Sudan``), but the epithet with its article is not: ``Muhammad Ahmad, the Mahdi`` gives first ``the Mahdi``, last ``Muhammad Ahmad``, where 1.4.0 through 2.6.0 gave title ``the Mahdi``. :issue:`867`, :issue:`752` diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index b3fc0c04..393d3289 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -719,8 +719,6 @@ 'subaltern', 'subedar', 'suffragist', - 'sultan', - 'sultana', 'superior', 'supreme', 'surgeon', @@ -971,6 +969,8 @@ 'major', 'marshal', 'siddha', + 'sultan', + 'sultana', 'wing', }) diff --git a/nameparser/vocab/ar.toml b/nameparser/vocab/ar.toml index d509bee1..0b979b7a 100644 --- a/nameparser/vocab/ar.toml +++ b/nameparser/vocab/ar.toml @@ -124,7 +124,9 @@ nobility = {} [[word]] text = "sultana" -nobility = { gender = "f" } +nobility = { gender = "f", issue = 752, decided = 2026-10-26, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Sultana Begum' reads the given name, unreported, since no listed phrase holds it" } +given = { us_ssa = 275, wikidata = 19, as_of = 2026-10-26, note = "US SSA: female births; Wikidata is a lower bound (Bangladesh 5); nam_dict band 3 in Romania and Arabia/Persia" } +family = { us_census_2010 = 3471, wikidata = 34, as_of = 2026-10-26 } [[word]] text = "أمير" diff --git a/nameparser/vocab/tr.toml b/nameparser/vocab/tr.toml index c41cb853..c311137d 100644 --- a/nameparser/vocab/tr.toml +++ b/nameparser/vocab/tr.toml @@ -11,7 +11,9 @@ style = { expands = "the Ottoman efendi, in its conventional Latin spelling" } [[word]] text = "sultan" -nobility = { note = "no gender: Ottoman usage gave it to imperial women too, written after the name (Hürrem Sultan)" } +nobility = { note = "no gender: Ottoman usage gave it to imperial women too, written after the name (Hürrem Sultan)", issue = 752, decided = 2026-10-26 } +given = { us_ssa = 2396, wikidata = 151, as_of = 2026-10-26, note = "US SSA: male births; Wikidata is a lower bound (Turkey 28, Saudi Arabia 13); nam_dict band 7 in Turkey, 5 in Central Asia; a title only inside a title phrase (rules/H9): 'Sultan Khan' and 'Mr Sultan Khan' read the given name, unreported" } +family = { us_census_2010 = 3363, wikidata = 106, as_of = 2026-10-26, note = "Wikidata: Ottoman Empire 26, the title written after the name, which reads as a name word" } [[word]] text = "baba" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index c7443fc8..89314289 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -12579,6 +12579,38 @@ def _check_cjk_shape_purity(self) -> None: {"title": "Pope", "given": "Francis"}, notes="'pope' has no given reading: still the title that " "addresses by the given name, H1"), + Case("given_name_title_sultana_at_the_head", "Sultana Begum", + {"given": "Sultana", "family": "Begum"}, classification="fix(#752)", + notes="H9; read title 'Sultana', no given name, 1.4.0 through " + "2.6.0 (#752 phase 2j)"), + Case("given_name_title_sultana_before_a_credential_comma", + "Nasreen Sultana, MBBS", + {"given": "Nasreen", "family": "Sultana", "suffix": "MBBS"}, + ambiguities=("suffix-or-name",), classification="fix(#752)", + notes="two name words now stand before the comma, so the " + "credential is the suffix; read given 'MBBS', family " + "'Nasreen Sultana' before"), + Case("given_name_title_sultan_after_a_salutation", "Mr Sultan Khan", + {"title": "Mr", "given": "Sultan", "family": "Khan"}, + classification="fix(#853)", + notes="#853's own example; read title 'Mr Sultan' before"), + Case("given_name_title_sultan_after_a_family_comma", "Khan, Sultan", + {"given": "Sultan", "family": "Khan"}, classification="fix(#752)", + notes="H9; read title 'Sultan' before"), + Case("sultan_behind_a_style_keeps_the_title", + "His Majesty Sultan Hassanal Bolkiah", + {"title": "His Majesty Sultan", "given": "Hassanal", + "family": "Bolkiah"}, + notes="H9: a style is a partner, as master and 1.4.0 read it"), + Case("ruler_with_a_bare_leading_sultan", "Sultan Hassanal Bolkiah", + {"given": "Sultan", "middle": "Hassanal", "family": "Bolkiah"}, + classification="fix(#752)", + notes="H9's Accepted clause, the class accepted for 'king' " + "(Derek, 2026-10-26); read title 'Sultan' before"), + Case("sultan_behind_sheikh_keeps_the_title", "Sheikh Sultan Khan", + {"title": "Sheikh Sultan", "family": "Khan"}, + notes="'sheikh' is a style, so it partners the word: master's " + "reading, left for sheikh's own batch"), Case("given_name_modifier_after_a_style", "Hon. Wing Lau", {"title": "Hon.", "given": "Wing", "family": "Lau"}, notes="H9: the modifier's test, not a contested word's style " diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 76e8ec8c..a5f50f16 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -25,7 +25,7 @@ "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 660, "then_words": 263, "because_sentences": 3, "except": 0, "precedence": 5, "accepted": 9, "accepted_words": 340}, + "H9": {"words": 660, "then_words": 263, "because_sentences": 3, "except": 0, "precedence": 5, "accepted": 9, "accepted_words": 342}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 714, "then_words": 435, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 97a77d08..99d6ae1d 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -287,9 +287,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 'John Smith (MA) née Jones' its two suffix-or-name claims (-12), # and the two clauses a second pass would settle keep the nickname, # reported (+6); 1384 -> 1390 the same day, its review round's rows - # for a leading period and a name of no word (+6). - assert claims == 1390, ( - f"the field sweep checks {claims} claims, recorded as 1390 on " + # for a leading period and a name of no word (+6); 1390 -> 1393 the + # same day, #752 phase 2j's 'Nasreen Sultana, MBBS', the credential + # behind two name words reported suffix-or-name under three orders. + assert claims == 1393, ( + f"the field sweep checks {claims} claims, recorded as 1393 on " f"2026-10-26") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index d0f6acd9..ce71c6b6 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -857,6 +857,8 @@ "Stariji Bruegel" "Steven Hardman, MD, DO, DDS" "Steven Hardman, RN - CRNA" +"Sultan Hassanal Bolkiah" +"Sultana Begum" "The Hon. Justice Mary Smith" "The Hon. Wing Lau" "The Mahdi Muhammad Ahmad" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 6137c6f0..68c314c6 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -3355,6 +3355,8 @@ {"name": "Sulaiman, Salimah", "given": "Salimah", "family": "Sulaiman", "initials": "S. S."} {"name": "Sule", "given": "Sule", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S."} {"name": "Sule", "order": "FAMILY_FIRST", "family": "Sule", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S."} +{"name": "Sultan Hassanal Bolkiah", "given": "Sultan", "middle": "Hassanal", "family": "Bolkiah", "initials": "S. H. B."} +{"name": "Sultana Begum", "given": "Sultana", "family": "Begum", "initials": "S. B."} {"name": "Suresh a/l Muthu", "given": "Suresh", "middle": "a/l", "family": "Muthu", "initials": "S. a. M.", "capitalized_force": "Suresh A/L Muthu"} {"name": "Surāqa Ibn-Mirdās al-Aṣġar", "given": "Surāqa", "middle": "Ibn-Mirdās", "family": "al-Aṣġar", "initials": "S. I. a.", "capitalized_force": "Surāqa ibn-Mirdās al-Aṣġar"} {"name": "Suzuki san", "order": "FAMILY_FIRST", "given": "san", "family": "Suzuki", "initials": "s. S.", "capitalized_force": "San Suzuki"}