From 2c100ffcf35ca129f62cd4c724ea45b2d4051649 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 02:17:12 -0700 Subject: [PATCH 1/4] Read the name Sheikh and its spellings as names outside a title phrase (#752 phase 2j, #853) The ninth and last title batch of phase 2j. `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` gain dated given readings, vetted by where UK PSC's filers put each, so each is a title only inside a title phrase, unreported: `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed` and `Cheikh Anta Diop` read the given name, while `His Highness Sheikh Ahmed Khan` and `Sheikh of Kuwait` keep the title. `shaykh` stays a title. A ruler written with the bare title reads it as the given name, the cost recorded for Derek to decide. The batch rejoins all 12 jrc-names persons the shaikh batch split. `Sheik` was rules/P5's and H1's given-name title; `Sir` replaces it, reading as `Sheik` did on master in every example and order. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 7 ++ docs/design/decisions/P5-bound-given-names.md | 4 +- docs/design/decisions/vocabulary-model.md | 1 + .../rules/H1-title-and-one-name-word.md | 2 +- docs/design/rules/H9-title-phrase.md | 2 + docs/design/rules/P5-bound-given-names.md | 16 ++--- docs/release_log.rst | 1 + nameparser/_pipeline/_post_rules.py | 2 +- nameparser/config/_compiled.py | 10 +-- nameparser/vocab/ar.toml | 15 +++-- nameparser/vocab/fr.toml | 4 +- tests/v2/cases.py | 66 +++++++++++++++++-- tests/v2/pipeline/test_group.py | 6 +- tests/v2/test_parser.py | 28 ++++---- tools/differential/corpus_rules.jsonl | 10 +-- tools/differential/local_corpora.py | 4 +- tools/differential/local_nepal.py | 3 +- tools/differential/snapshot.jsonl | 12 ++-- 18 files changed, 137 insertions(+), 56 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 6dba575f..431c5499 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -76,3 +76,10 @@ - THE COST, accepted by Derek on 2026-10-27, shown the share of royals below beside `king`'s, `sultan`'s and `shaikh`'s and that a style or `of` keeps the title: a royal written with the bare title in front reads it as the given name (`Prince Harry` → given 'Prince', family 'Harry', where 2.3.0 through 2.6.0 read title 'Prince', given 'Harry'), the class accepted for `king`, `sultan` and `shaikh`. In jrc-names 43 persons have a spelling opening with a bare `Prince` or `Princess` (one not followed by `of`): by a reading of each, about 33 are royals, 8 people whose given name is Prince, and 2 no person at all (a fictional character, a place). In PSC 12 of the 233 moved records file the word in the title box (10 broken, 2 changed). - MEASURED (2026-10-27, against master `ac81de9c`, py3.11, every local check with `--base`). PSC 205 records fixed, 10 broken, every one `Prince` (6) or `Princess` (4) filed in the title box, which the scorer now sets aside, 18 changed, most now reading the given name right and splitting a two-word surname box, 2 of them title-box filings; ctgov-officials 6 moved, all fixes: 4 open with the word, 3 of them before a credential comma, 1 glues an initial to it (`J.Prince Aa, CRED`), and 1 is `Aa Prince, CRED`, whose credential master read as the given name; fec-individuals 43, all fixes, every one `AA, PRINCE` or `AA, PRINCESS` in the listing form, 28 of which had read no family name; nepal-candidates 3, fixes; jrc-names 101 spellings moved, the royals above and `Archduchess of X Princess Y`, a title before the word that is no partner of it; tse-donors and ukec-individuals none; spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE) jrc-names 4 persons now alike and 0 apart, nepal-candidates 3 and 0. The committed corpora move by `Prince Charles` and `Prince Fielder` (corpus_issues, #348's examples) beside the batch's own rule examples; CLDR and GND agreement do not move. A v1 `Constants().titles` keeps both words with no edit (the v1 title list entry above). Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). +- **2026-10-27 #752, phase 2j, `sheikh` and its spellings — THE NINTH AND LAST TITLE BATCH.** `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` gain dated `given` readings and are titles only inside a title phrase, as `shaikh` became in the seventh batch; no listed phrase holds any of them, so nothing is reported. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed`, `Khan, Sheikh`, `Cheikh Anta Diop` and `Shekh Abdul Rahim` read the word as the given name, where 1.4.0 through 2.6.0 read the title. A style keeps it (`His Highness Sheikh Ahmed Khan`, as 1.4.0 read it), as does a modifier (`Grand Sheikh Ahmed`) or `of` (`Sheikh of Kuwait`), and the period-marked spelling at the front is a title by H2's shape (`Sheikh. Ahmed Khan`). The other sites read as for `shaikh`: before a family comma (`Sheikh Smith, Ahmed` → family 'Sheikh Smith', 1.4.0's reading), ending the name, alone after a full name and a comma, and past a second comma. A style read as a name word partners nothing (`Sheikh Sultan Khan` → given 'Sheikh', middle 'Sultan', where master read title 'Sheikh Sultan'), and behind a style it partners the contested word as before (`His Highness Sheikh Sultan Khan`), closing the NOT REACHED bullet of the 2026-10-26 `sultana` and `sultan` entry above. + - WHICH SPELLINGS. Vetted by where UK PSC's filers put each (the first word of the forename box against any word of the title box, 2026-10-27): `sheikh` 174 against 19, `sheik` 23 against 1, `shekh` 6 against 1, `shaik` 14 against 0, `cheikh` 14 against 0. `shaykh`, the scholarly transliteration, stays a title everywhere: no filer writes it in front and no source counts it as a given name; the one jrc-names person it now holds apart from its other spellings (below) is the cost. + - THE COUNTS, recorded on the words (`uv run python tools/name_usage.py Sheikh Shekh Sheik Shaykh Shaik Cheikh`, 2026-10-27): `sheikh` given US SSA 436, Wikidata 21, family Census 6,462 (rank 5,384), Wikidata 60; `sheik` given SSA 39, family Census 401, Wikidata 4; `shaik` given SSA 10, family Census 1,053 (93.73% asian/PI), Wikidata 14; `cheikh` given SSA 512, Wikidata 51 (Senegal 34), family Census 112, Wikidata 14; `shekh` no US or Wikidata count, its evidence the filers above and 378 of Nepal's local candidates. + - THE COST, for Derek to decide: a ruler, cleric or leader written with the bare title in front reads it as the given name (`Sheikh Mohammed bin Rashid Al Maktoum` → given 'Sheikh', middle 'Mohammed bin Rashid', where 1.4.0 through 2.6.0 read title 'Sheikh', given 'Mohammed'), the class accepted for `king`, `sultan`, `shaikh` and `prince`. In jrc-names 174 persons have a spelling opening with a bare form of the word: by a reading of each, about 40 Gulf rulers and members of their houses, about 55 clerics and militia, tribal or political leaders, about 60 people whose own name opens with it (most of them Senegalese and Mauritanian `Cheikh`s, beside Bangladeshi and Malaysian bearers), and the rest unclear. In PSC 16 of the 244 moved records file the word in the title box (13 broken, 3 changed), while Qatari and Emirati ruling-house members there file it in the forename box. + - MEASURED (2026-10-27, against master `79a4111f`, py3.11, every local check with `--base`). PSC (1,039,144 scored) 209 records fixed, 13 broken, every one the word filed in the title box (12 `Sheikh`, 1 `Sheik`), 22 changed, most now reading the given name and splitting the filer's middle box differently; ctgov-officials 10 moved, 8 fixes (4 `Aa Sheikh, CRED` whose credential master read as the given name, 4 opening with `Sheikh` or `Cheikh`) and 2 lower-case or credential-run names still wrong; fec-individuals 26, all fixes, every one `AA, SPELLING…`, 18 of which had read no family name; nepal-candidates 267, every one opening with `Shekh` (265) or `Sheikh` (2); jrc-names 571 spellings; tse-donors and ukec-individuals none. Spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE): jrc-names 37 persons now alike and 1 apart (spelled `Shaykh` beside `Sheikh` and `Cheikh`), and all 12 persons the `shaikh` entry above left apart are alike again; nepal-candidates 280 pairs now alike and 0 apart, the romanized `Shekh` now reading as its Devanagari does. The committed corpora move by two names in corpus_issues, #367's `Sheik Abu Bakar` (given 'Sheik', family 'Abu Bakar') and `Sheik abdul salam` (given 'Sheik', middle 'abdul', family 'salam'), beside the batch's own rule examples, where `Sir` now stands for `Sheik` (below); CLDR and GND agreement do not move. A v1 `Constants().titles` keeps all five with no edit (the v1 title list entry above). + - WORKED EXAMPLES MOVED. `Sheik` was rules/P5's given-name title licensing the bound join (`Sheik abdul salam` and its family-first and suffix rows) and rules/H1's `Sheik Abu Bakar`; `Sir` replaces it, reading as `Sheik` read on master in every one of them under all three orders. The title-run row `Sir Sheikh abdul rahman` is `Dr. Sir abdul rahman`. + Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). diff --git a/docs/design/decisions/P5-bound-given-names.md b/docs/design/decisions/P5-bound-given-names.md index 512e0ce8..413caf24 100644 --- a/docs/design/decisions/P5-bound-given-names.md +++ b/docs/design/decisions/P5-bound-given-names.md @@ -5,9 +5,9 @@ - The join is vocabulary-driven: a bound given-name word joins the name word after it, and needs a name word to spare — 2026-06-30. - `abd` is a bound given-name word and the credential ABD, position deciding at the two ends (`Jane Smith ABD` → suffix, `Smith, Abd` → suffix 'Abd'); the three-token `abd al rahman` does not join, `al` chaining forward as a particle — 2026-08-18. - The join never absorbs a lone maiden marker, and the reserve counts the name the marker pass leaves (`abdul Berg née Jones` → family 'Berg') — 2026-08-21 #411; 2026-08-21 #418. -- A given-name title licenses the join with one word to spare, read through the predicate H1 reads a title run with: `Sheik abdul salam` → given 'abdul salam', `Dr. abdul salam` → family 'salam'. The licensed join takes one word, never a particle chain (`Sir abdul van der Berg`), and a bound word that is also a particle reads as the bound word, reporting nothing (`Sheik Abu Bakar`) — 2026-08-22 #369. +- A given-name title licenses the join with one word to spare, read through the predicate H1 reads a title run with: `Sir abdul salam` → given 'abdul salam' (`Sheik` until 2026-10-27, when it became a title only inside a title phrase, decisions/H9), `Dr. abdul salam` → family 'salam'. The licensed join takes one word, never a particle chain (`Sir abdul van der Berg`), and a bound word that is also a particle reads as the bound word, reporting nothing (`Sir Abu Bakar`) — 2026-08-22 #369. - The join declines a word that reads as a suffix: `abdul Jr Smith Berg` → given 'abdul', `abdul Ph. D. Smith Berg` → suffix 'Ph. D.' — 2026-08-22 #401. -- S2's trailing run is read once before the join is tried, so the words to spare are the name pieces it leaves and the join moves no suffix reading (`abdul J. V` → given 'abdul J.', family 'V'; `Sheik abdul Ma` reports `suffix-or-name`) — 2026-10-07 #614. +- S2's trailing run is read once before the join is tried, so the words to spare are the name pieces it leaves and the join moves no suffix reading (`abdul J. V` → given 'abdul J.', family 'V'; `Sir abdul Ma` reports `suffix-or-name`) — 2026-10-07 #614. - After a family comma the pair is the given name whatever follows (`Berg, abdul V` → given 'abdul V'), a bare ambiguous acronym the run does not take is a name word (`abdul Ma Smith` → given 'abdul Ma'), and a title word inside the name joins like a name word (`abdul Sir Smith Berg`) — 2026-08-22 #425. - The join declines a title word a trailing rule reads as a title: after a family comma one H5's trailing run takes (`Berg, abdul Prof.` → title 'Prof.'), and past the second comma any title word — 2026-10-11 #637. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index d96df7f6..09e85295 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -218,6 +218,7 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest. Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. +- **2026-10-27 #752, phase 2j, `sheikh` and its spellings — THE NINTH AND LAST TITLE BATCH: `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` have dated `given` readings**, so each is a title only inside a title phrase; `shaykh` stays a title. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed` and `Cheikh Anta Diop` read the given name; a ruler written with the bare title does too (`Sheikh Mohammed bin Rashid`), its cost here for Derek to decide. With it every word item 12's measurement named has its reading. The measurement is decisions/H9's entry of this date. - **2026-10-27 #752, phase 2j, `prince` and `princess` (Derek) — THE EIGHTH TITLE BATCH: both have a dated `given` reading**, so each is a title only inside a title phrase. `Prince Fielder`, `Mr Prince Smith` and `Smith, Princess` read the given name; a royal written with the bare title does too (`Prince Harry`), the class accepted for `king`, `sultan` and `shaikh`, accepted by Derek on 2026-10-27. `His Royal Highness Prince Harry` and `Prince of Wales` keep the title. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `shaikh` (Derek) — THE SEVENTH TITLE BATCH: `shaikh` has a dated `given` reading**, so the style is a title only inside a title phrase. `Shaikh Ahmed Khan` and `Mr Shaikh Ahmed` read given 'Shaikh'; a ruler written with the bare title does too (`Shaikh Hamad bin Isa`), the class accepted for `king` and `sultan`, the larger share of rulers Derek accepted on 2026-10-27. `sheikh` keeps its own batch. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `sultana` and `sultan` (Derek) — THE FIFTH AND SIXTH TITLE BATCHES, IN ONE PR: both have dated `given` readings**, `sultan`'s on its tr.toml entry, so each is a title only inside a title phrase. `Sultana Begum` and `Sultan Khan` read the given name; a sultan written with the bare title does too (`Sultan Hassanal Bolkiah`), the cost accepted for `king`. Derek combined the two batches on 2026-10-26. The measurement is decisions/H9's entry of this date. diff --git a/docs/design/rules/H1-title-and-one-name-word.md b/docs/design/rules/H1-title-and-one-name-word.md index e14cab6b..7c32473d 100644 --- a/docs/design/rules/H1-title-and-one-name-word.md +++ b/docs/design/rules/H1-title-and-one-name-word.md @@ -54,7 +54,7 @@ - A listing record whose given field holds only a given-name title reads that title's reading, so `Smith, Sir` (a row with the given name missing) reads given `Smith` with the family empty, as `Sir Smith` does: the input names one word and one title, and the title is the only evidence there is. 1.4.0 read the word before the comma as the last name in these rows (`Elizabeth II, Queen`, `John III, Sir`, `Smith, Sir`) but for `Anne, Swami`, which it read as first `Anne`. An ambiguous particle no comma settled, or a bare one, is a word the order placed, and reads given. ``` "Smith, Sir" → family="" - "Sheik Abu Bakar" family-first → given="Abu Bakar" + "Sir Abu Bakar" family-first → given="Abu Bakar" "Sir de" family-first → given="de" "St Sir de Mesnil" family-first → given="de Mesnil" ``` diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 516d3854..4526e200 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -25,6 +25,7 @@ "Sultana Begum" → given="Sultana" "Mr Shaikh Ahmed" → given="Shaikh" "Ms Princess Smith" → given="Princess" + "Cheikh Anta Diop" → given="Cheikh" ``` - **A modifier in the leading run of titles before the first comma** has its phrase in a head title after it in that run, in `of` after it, or in the object of `of` or `the` there. A modifier outside that run, ending the name or after a comma, is read as any title is, save one that may be a given name, which is read everywhere as it is in that run. ``` @@ -129,6 +130,7 @@ "Sultan Hassanal Bolkiah" → given="Sultan" "Shaikh Hamad bin Isa" → given="Shaikh" "Prince Harry" → given="Prince" + "Sheikh Mohammed bin Rashid" → given="Sheikh" "His Majesty King Charles" → title="His Majesty King" "Prince of Wales" → title="Prince of Wales" ``` diff --git a/docs/design/rules/P5-bound-given-names.md b/docs/design/rules/P5-bound-given-names.md index 6a786b07..ffb2d730 100644 --- a/docs/design/rules/P5-bound-given-names.md +++ b/docs/design/rules/P5-bound-given-names.md @@ -10,10 +10,10 @@ "Abu Bakar" → given="Abu" · boundary "Dr. abdul salam" → family="salam" · boundary "Salam, abd Allah" → given="abd Allah" - "Sheik abdul salam" → given="abdul salam" - "Sheik abdul salam Jr" → given="abdul salam" + "Sir abdul salam" → given="abdul salam" + "Sir abdul salam Jr" → given="abdul salam" "Sir abdul van der Berg" → family="van der Berg" · boundary - "Sheik abdul Jr Smith" → given="abdul" · boundary + "Sir abdul Jr Smith" → given="abdul" · boundary ``` - **What there is to spare is what the name's trailing reading leaves**: the trailing suffix run (S2) and the trailing title run (H5), each read over what the other leaves until neither takes anything more, are read once, before the join is tried, and the name words they leave are the words to spare. A trailing roman numeral the run takes (S10), or a bare acronym it takes, or a trailing title word the run takes, is no word to spare; a bare ambiguous acronym is a name word wherever the peel does not take it. The join joins two name words into one and, read after the trailing run, changes no suffix reading: a word the run left as a name word stays one, whatever the join makes of the word in front of it. ``` @@ -73,7 +73,7 @@ - **A bound word that is also a particle** ("Abu Bakar Salim"): where the join fires, it is read as the bound word: the join outranks the leading-position reading (P4) and no fork is reported; where the reserve blocks the join, P4's reading and its fork stand. ``` - "Sheik Abu Bakar" → given="Abu Bakar" + "Sir Abu Bakar" → given="Abu Bakar" "Abu Bakar Salim" → given="Abu Bakar" ``` - **A word that is BOTH bound-given and suffix vocabulary** ("abd Allah Smith", "Smith, Abd"): position decides and both readings survive: leading, it is the bound word; trailing, it is the suffix (S2). In the given slot after a family comma the suffix reading wins. @@ -92,10 +92,10 @@ ``` - A given-name title plus a bound pair leaves the family empty, as H1 accepts for "Sir John" — the input names no family — under a family-first order too: the join is order-blind, and H1 moves the pair that order placed in the family back to the given name, as it does "Sir John" there. ``` - "Sheik abdul salam" → family="" - "Sheik abdul salam" family-first → given="abdul salam" - "Sheik abdul salam" family-first → family="" - "Sheik abdul salam" family-first-given-last → given="abdul salam" + "Sir abdul salam" → family="" + "Sir abdul salam" family-first → given="abdul salam" + "Sir abdul salam" family-first → family="" + "Sir abdul salam" family-first-given-last → given="abdul salam" ``` - The given part after a family comma reads the words behind a title by its own walk, and where that walk reads them otherwise than the comma-less name does, the join is declined and no title is read — a shape-only numeral (VI to X), or V before a third comma part, stays a name word there, and a title word that is also a particle is attached by P6 (`Berg, abdul Prof. VI` reads middle 'Prof. VI'; `Berg, abdul St.` reads family 'St. Berg'), each as the same name with an ordinary given word reads. diff --git a/docs/release_log.rst b/docs/release_log.rst index 9ba56a9f..46129ff2 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -18,6 +18,7 @@ Release Log - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752` - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). A v1 ``Constants().titles`` still lists ``King``, ``Mahdi`` and the other titles 2.6.0 listed, and removing one reads it as a name everywhere, but adding one there no longer makes it a title outside a phrase; that takes the ``Lexicon`` spelling. :issue:`752`, :issue:`853` + - **Fix the name Sheikh and its spellings reading as titles.** ``Sheikh Ahmed Khan``, ``Mr Sheikh Ahmed`` and ``Cheikh Anta Diop`` give first ``Sheikh`` and ``Cheikh``, where 1.4.0 through 2.6.0 gave title ``Sheikh``, ``Mr Sheikh`` and ``Cheikh``; ``Khan, Sheikh`` and the spellings ``Sheik``, ``Shekh`` and ``Shaik`` read the same. A ruler written with the bare title reads it as a first name too (``Sheikh Mohammed bin Rashid`` gives first ``Sheikh``), where a style keeps it (``His Highness Sheikh Mohammed``); ``Shaykh`` is still a title. :issue:`752`, :issue:`853` - **Fix the given names Prince and Princess reading as titles.** ``Prince Fielder`` and ``Mr Prince Smith`` give first ``Prince``, where 1.4.0 through 2.6.0 gave title ``Prince`` and ``Mr Prince``; ``Smith, Princess`` reads the same. A royal written with the bare title reads it as a first name too (``Prince Harry`` gives first ``Prince``, last ``Harry``), where a style keeps it (``His Royal Highness Prince Harry``), and so does ``of`` (``Prince of Wales``). :issue:`752`, :issue:`348`, :issue:`853` - **Fix the name Shaikh reading as a title.** ``Shaikh Ahmed Khan`` and ``Mr Shaikh Ahmed`` give first ``Shaikh``, where 1.4.0 through 2.6.0 gave title ``Shaikh`` and ``Mr Shaikh``; ``Khan, Shaikh`` reads the same. A ruler written with the bare title reads it as a first name too (``Shaikh Hamad bin Isa`` gives first ``Shaikh``), where a style keeps it (``His Highness Shaikh Hamad``); the spellings ``Sheikh`` and ``Shekh`` are still titles. :issue:`752`, :issue:`853` - **Fix the given names Sultan and Sultana reading as titles.** ``Sultan Khan`` and ``Sultana Begum`` give first ``Sultan`` and ``Sultana``, where 1.4.0 through 2.6.0 gave title ``Sultan`` and ``Sultana`` with no first name; ``Mr Sultan Khan`` and ``Khan, Sultan`` read the same. A sultan written with the bare title reads it as a first name too (``Sultan Hassanal Bolkiah`` gives first ``Sultan``), where a style keeps it (``His Majesty Sultan Hassanal Bolkiah``). :issue:`752`, :issue:`853` diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 73b1705b..34cf28d8 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -377,7 +377,7 @@ def _family_settled(state: ParseState, tokens: list[WorkToken], particle opening the name with a word behind it ('Sir de Mesnil'), asked by the fold's own predicate on the same piece. An ambiguous particle nothing settled, or a bare one with nothing to fold, is a - word the order placed ('Sir Van', 'Sheik Abu Bakar' and 'Sir de' + word the order placed ('Sir Van', 'Sir Abu Bakar' and 'Sir de' under FAMILY_FIRST; 'Sir von und zu', whose never-given 'zu' does not open it).""" if state.structure is Structure.FAMILY_COMMA: diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index ee1f1a99..0e047592 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -194,7 +194,6 @@ 'charge', 'chargé', 'chef', - 'cheikh', 'chemist', 'chief', 'chieftain', @@ -670,12 +669,8 @@ 'sgt', 'sgtmaj', 'sgtmajmc', - 'shaik', 'shayk', 'shaykh', - 'sheik', - 'sheikh', - 'shekh', 'sheriff', 'sig.na', 'sig.ra', @@ -957,6 +952,7 @@ TITLES_AMBIGUOUS: frozenset[str] = frozenset({ 'bard', 'buddha', + 'cheikh', 'dean', 'grace', 'gyani', @@ -967,7 +963,11 @@ 'marshal', 'prince', 'princess', + 'shaik', 'shaikh', + 'sheik', + 'sheikh', + 'shekh', 'siddha', 'sultan', 'sultana', diff --git a/nameparser/vocab/ar.toml b/nameparser/vocab/ar.toml index c6efc149..0e85ecf3 100644 --- a/nameparser/vocab/ar.toml +++ b/nameparser/vocab/ar.toml @@ -6,7 +6,9 @@ style = { expands = "one who has made the Hajj" } [[word]] text = "sheikh" -style = { addresses = "given", gender = "m" } +style = { addresses = "given", gender = "m", issue = 752, decided = 2026-10-27, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'His Highness Sheikh Mohammed' and 'Sheikh of Kuwait' read the title, 'Sheikh Ahmed Khan' and 'Mr Sheikh Ahmed' the given name, unreported, since no listed phrase holds it" } +given = { us_ssa = 436, wikidata = 21, as_of = 2026-10-27, note = "US SSA: male and female births added; Wikidata is a lower bound (Bangladesh 3, India 2); UK PSC filers put it first in the forename box 174 times and in the title box 19 (2026-10-27)" } +family = { us_census_2010 = 6462, wikidata = 60, as_of = 2026-10-27, note = "US Census rank 5,384, asian/PI 71.62%" } [[word]] text = "cheikh" @@ -14,7 +16,9 @@ style = { addresses = "given", gender = "m", same_as = "sheikh", expands = "the [[word]] text = "shaik" -style = { addresses = "given", gender = "m", same_as = "sheikh" } +style = { addresses = "given", gender = "m", same_as = "sheikh", issue = 752, decided = 2026-10-27, note = "a title only inside a title phrase, its given reading the clash (rules/H9): read as 'sheikh' is" } +given = { us_ssa = 10, as_of = 2026-10-27, note = "US SSA births; UK PSC filers put it first in the forename box 14 times and never in the title box (2026-10-27)" } +family = { us_census_2010 = 1053, wikidata = 14, as_of = 2026-10-27, note = "US Census asian/PI 93.73%; mostly a South Indian surname" } [[word]] text = "shaikh" @@ -32,11 +36,14 @@ style = { addresses = "given", gender = "m", same_as = "sheikh" } [[word]] text = "sheik" -style = { addresses = "given", gender = "m", same_as = "sheikh" } +style = { addresses = "given", gender = "m", same_as = "sheikh", issue = 752, decided = 2026-10-27, note = "a title only inside a title phrase, its given reading the clash (rules/H9): read as 'sheikh' is" } +given = { us_ssa = 39, as_of = 2026-10-27, note = "US SSA births; UK PSC filers put it first in the forename box 23 times and in the title box once (2026-10-27)" } +family = { us_census_2010 = 401, wikidata = 4, as_of = 2026-10-27 } [[word]] text = "shekh" -style = { addresses = "given", gender = "m", same_as = "sheikh" } +style = { addresses = "given", gender = "m", same_as = "sheikh", issue = 752, decided = 2026-10-27, note = "a title only inside a title phrase, its given reading the clash (rules/H9): read as 'sheikh' is" } +given = { note = "no US or Wikidata count; UK PSC filers put it first in the forename box 6 times and in the title box once, and 378 of Nepal's 2022 local candidates write it, its own romanization of the Devanagari (2026-10-27)" } [[word]] text = "الشيخ" diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 55ddf79f..48750301 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -22,7 +22,9 @@ salutation = { gender = "m", after_given = true } [[word]] text = "cheikh" -style = { addresses = "given", gender = "m", same_as = "sheikh", expands = "the French transliteration, 'Cheikh Anta Diop'" } +style = { addresses = "given", gender = "m", same_as = "sheikh", expands = "the French transliteration, 'Cheikh Anta Diop'", issue = 752, decided = 2026-10-27, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Cheikh Anta Diop' reads the given name" } +given = { us_ssa = 512, wikidata = 51, as_of = 2026-10-27, note = "Wikidata is a lower bound (Senegal 34, France 9); a common Senegalese given name; UK PSC filers put it first in the forename box 14 times and never in the title box (2026-10-27)" } +family = { us_census_2010 = 112, wikidata = 14, as_of = 2026-10-27 } [[word]] text = "me" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 2ee2a95b..45f8903f 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -10521,12 +10521,14 @@ def _check_cjk_shape_purity(self) -> None: "never match as a last-word key either -- `sir` is " "what this run is read by"), Case("title_run_licences_the_bound_join_by_its_last_title", - "Sir Sheikh abdul rahman", - {"title": "Sir Sheikh", "given": "abdul rahman"}, + "Dr. Sir abdul rahman", + {"title": "Dr. Sir", "given": "abdul rahman"}, classification="fix(#489)", notes="rules/P5's licence keys the same way H1 does, the " "invariant the #369 entry set: behind a given-name " - "title there is no family to spare"), + "title there is no family to spare. 1.4.0 read last " + "'rahman'. Written 'Sir Sheikh' until #752 phase 2j " + "made `sheikh` a title only inside a title phrase"), Case("title_run_does_not_address_by_a_non_given_name_title", "His Excellency Lord Duncan", {"title": "His Excellency Lord", "family": "Duncan"}, @@ -12623,10 +12625,60 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#752)", notes="H9's Accepted clause, the class accepted for 'king' " "(Derek, 2026-10-26); read title 'Sultan' before"), - Case("sultan_behind_sheikh_keeps_the_title", "Sheikh Sultan Khan", - {"title": "Sheikh Sultan", "family": "Khan"}, - notes="'sheikh' is a style, so it partners the word: master's " - "reading, left for sheikh's own batch"), + Case("sultan_behind_a_bare_leading_sheikh", "Sheikh Sultan Khan", + {"given": "Sheikh", "middle": "Sultan", "family": "Khan"}, + classification="fix(#752)", + notes="H9: a style read as a name word is no title there, so it " + "partners nothing, as for 'shaikh'; read title " + "'Sheikh Sultan' 1.4.0 through 2.6.0"), + Case("sultan_behind_a_styled_sheikh_keeps_the_title", + "His Highness Sheikh Sultan Khan", + {"title": "His Highness Sheikh Sultan", "family": "Khan"}, + notes="H9: behind a style 'sheikh' is a title, and a style " + "partners the contested word behind it"), + Case("given_name_title_sheikh_at_the_head", "Sheikh Ahmed Khan", + {"given": "Sheikh", "middle": "Ahmed", "family": "Khan"}, + classification="fix(#752)", + notes="H9; read title 'Sheikh', given 'Ahmed' 1.4.0 through " + "2.6.0 (#752 phase 2j)"), + Case("given_name_title_sheikh_after_a_salutation", "Mr Sheikh Ahmed", + {"title": "Mr", "given": "Sheikh", "family": "Ahmed"}, + classification="fix(#853)", + notes="#853's table: a salutation is no partner; read title " + "'Mr Sheikh' before"), + Case("given_name_title_sheikh_after_a_family_comma", "Khan, Sheikh", + {"given": "Sheikh", "family": "Khan"}, classification="fix(#752)", + notes="H9; read title 'Sheikh' before"), + Case("given_name_title_sheikh_before_a_family_comma", + "Sheikh Smith, Ahmed", + {"given": "Ahmed", "family": "Sheikh Smith"}, + notes="H6 peels only a title, so the surname keeps the word, as " + "1.4.0 read it; master read title 'Sheikh'"), + Case("sheikh_behind_a_style_keeps_the_title", + "His Highness Sheikh Ahmed Khan", + {"title": "His Highness Sheikh", "given": "Ahmed", "family": "Khan"}, + notes="H9: a style is a partner, as master and 1.4.0 read it"), + Case("ruler_with_a_bare_leading_sheikh", + "Sheikh Mohammed bin Rashid Al Maktoum", + {"given": "Sheikh", "middle": "Mohammed bin Rashid", + "family": "Al Maktoum"}, + classification="fix(#752)", + notes="H9's Accepted clause, the class 'king', 'sultan', " + "'shaikh' and 'prince' took; read title 'Sheikh' before"), + Case("given_name_title_cheikh_at_the_head", "Cheikh Anta Diop", + {"given": "Cheikh", "middle": "Anta", "family": "Diop"}, + classification="fix(#752)", + notes="H9: the French spelling, a common Senegalese given " + "name; read title 'Cheikh', given 'Anta' before"), + Case("given_name_title_shekh_at_the_head", "Shekh Abdul Rahim", + {"given": "Shekh", "middle": "Abdul", "family": "Rahim"}, + classification="fix(#752)", + notes="H9: the spelling Nepal's romanization writes; read " + "title 'Shekh', given 'Abdul' before"), + Case("shaykh_stays_a_title", "Shaykh Hamza Yusuf", + {"title": "Shaykh", "given": "Hamza", "family": "Yusuf"}, + notes="'shaykh' has no given reading: no filed data writes it " + "as a given name, so it is still a title everywhere"), Case("given_name_title_shaikh_at_the_head", "Shaikh Ahmed Khan", {"given": "Shaikh", "middle": "Ahmed", "family": "Khan"}, classification="fix(#752)", diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 7f480a23..39b0e054 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -852,9 +852,9 @@ def test_the_joined_pair_is_a_given_name_whatever_tag_the_word_carried() -> None # join takes it as v1 did -- but the conjunction merge derives a # `title` piece tag for "mr and mrs", and merge_pieces' tag union # would hand that tag to the joined pair, which assign then peels - # as a leading title: 'abdul Sheikh and Ahmad Bakar Smith' read - # title 'abdul Sheikh and Ahmad' on 2.0 and 2.1, and the shorter - # 'abdul Sheikh and Ahmad Bakar' would have too once the count's + # as a leading title: 'abdul Sir and Ahmad Bakar Smith' read + # title 'abdul Sir and Ahmad' on 2.0 and 2.1 (written with `Sheikh`, + # a title then), and the shorter 'abdul Sir and Ahmad Bakar' would have too once the count's # title exclusion went. The bound join drops the tag: the pair is # a given name. Found by the code review. out = _grouped("abdul mr and mrs Smith Berg") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index a8c4af83..d60faa75 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -524,14 +524,14 @@ def test_bound_given_name_that_is_also_a_particle() -> None: # reason. Through 2.1 it read correctly only because the title # displaced 'abu' out of the leading position and the prefix chain # claimed 'Bakar' -- a side effect of the #367 bug, not a rule. Now - # the bound given-name join reads it: "Sheik" is a given-name + # the bound given-name join reads it: "Sir" is a given-name # title, which licenses the join with one word to spare # (rules/P5), so the particle never enters into it -- and a # bound word read as the bound word is not a fork, so the # PARTICLE_OR_GIVEN report the chain used to emit is gone. The # same precedence, untitled, is what "Abu Bakar Salim" has always # had; where the reserve blocks the join, P4 and its report stand. - titled = parse("Sheik Abu Bakar") + titled = parse("Sir Abu Bakar") assert (titled.given, titled.ambiguities) == ("Abu Bakar", ()) assert parse("Abu Bakar Salim").ambiguities == () assert [a.kind for a in parse("Abu Bakar").ambiguities] == \ @@ -544,23 +544,25 @@ def test_a_given_name_title_licenses_the_bound_given_join() -> None: # particle -- now joins the same way, and for the same reason. A # plain title does not license it: "Dr." addresses by family, so # the second word stays the family name. - licensed = parse("Sheik abdul salam") + # `sheik` has been a title only inside a title phrase since #752 + # phase 2j, so `Sir` stands in for it here, reading as it did. + licensed = parse("Sir abdul salam") assert (licensed.title, licensed.given, licensed.family) == \ - ("Sheik", "abdul salam", "") + ("Sir", "abdul salam", "") plain = parse("Dr. abdul salam") assert (plain.title, plain.given, plain.family) == \ ("Dr.", "abdul", "salam") # and the pair's report. Until #614 a bound word read as the bound - # word was no fork, so 'Sheik abdul Ma' reported nothing where - # 'Sheik John Ma' reports its pick (the #369 precedent). S2's run is + # word was no fork, so 'Sir abdul Ma' reported nothing where + # 'Sir John Ma' reports its pick (the #369 precedent). S2's run is # read once now, before the join, and the read weighs 'Ma' as a # credential and declines it before the join makes it part of the # given pair -- so the pick reports in both (decisions/P5, # 2026-10-07) - for text in ("Sheik abdul Ma", "Sheik John Ma"): + for text in ("Sir abdul Ma", "Sir John Ma"): assert [a.kind for a in parse(text).ambiguities] == \ [AmbiguityKind.SUFFIX_OR_NAME], text - assert parse("Sheik abdul Ma").given == "abdul Ma" + assert parse("Sir abdul Ma").given == "abdul Ma" def test_the_bound_given_reserve_spares_the_family_assign_will_keep() -> None: @@ -666,13 +668,13 @@ def test_the_leading_piece_never_anchors_under_any_name_order() -> None: def test_a_joined_pair_is_never_peeled_as_a_title() -> None: - # The conjunction merge derives a title tag for 'Sheikh and Ahmad'; + # The conjunction merge derives a title tag for 'Sir and Ahmad'; # the bound join takes the piece (a mid-name title word is a name # word, as 1.4.0 read it) and the pair must stay the given name, # not inherit the tag and be peeled as a leading title. - n = parse("abdul Sheikh and Ahmad Bakar") + n = parse("abdul Sir and Ahmad Bakar") assert (n.title, n.given, n.family) == \ - ("", "abdul Sheikh and Ahmad", "Bakar") + ("", "abdul Sir and Ahmad", "Bakar") # the title-word class: name words to P5, as v1 read them -- # parity restored for the post-comma shape, never lost for the # main-walk one @@ -808,8 +810,8 @@ def test_every_bound_word_spares_the_family_before_every_numeral( @pytest.mark.parametrize("title", [ - "Sir", "Sheikh", "Pope", "الشيخ", "Dr.", "Mr.", "Mr. Sir", "Sir Dr.", - "Sir and Dame", "Mr. and Mrs.", "Sheik and Mrs"]) + "Sir", "Swami", "Pope", "الشيخ", "Dr.", "Mr.", "Mr. Sir", "Sir Dr.", + "Sir and Dame", "Mr. and Mrs.", "Swami and Mrs"]) def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( title: str) -> None: # The licence's one invariant, as a contract: P5 lifts the reserve diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 08f77b51..ec5ef8b2 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -86,6 +86,7 @@ "Carod y de Rovira i" "Chas. Smith" "Chas. Smith, Ed" +"Cheikh Anta Diop" "Chief Dean and Mary Smith" "Chief Justice John Roberts" "DOE, JANE MA VD PHD" @@ -737,19 +738,20 @@ "Sgt Marshal Smith" "Shaikh Hamad bin Isa" "Shaikh. Sultan Khan" -"Sheik Abu Bakar" -"Sheik abdul Jr Smith" -"Sheik abdul salam" -"Sheik abdul salam Jr" +"Sheikh Mohammed bin Rashid" "Shin Ae-ra" "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." "Silva Neto, Pedro" "Silva, Neto" +"Sir Abu Bakar" "Sir John" "Sir John Prof." "Sir Ph. D. Van Johnson" "Sir Smith, John" +"Sir abdul Jr Smith" +"Sir abdul salam" +"Sir abdul salam Jr" "Sir abdul van der Berg" "Sir de" "Sir de Mesnil" diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 2c904a52..1338ed3d 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -676,8 +676,8 @@ class GroupCounts: differ: Counter[tuple[str, str]] #: The words of the titles, and of the suffixes, of a person some of #: whose spellings read one and others none, folded by word_key, - #: counted once a person: two spellings reading two titles (Sheikh, - #: Shekh) with no untitled spelling beside them add nothing. + #: counted once a person: two spellings reading two titles (Dr., + #: Doctor) with no untitled spelling beside them add nothing. titles: Counter[str] suffixes: Counter[str] diff --git a/tools/differential/local_nepal.py b/tools/differential/local_nepal.py index 1daf4513..edd0901f 100644 --- a/tools/differential/local_nepal.py +++ b/tools/differential/local_nepal.py @@ -7,7 +7,8 @@ as filed and in the Commission's own romanization. It is the repository's first large population of South Asian names: romanized names with surnames written as initials (B.K., K.C., G.C.), honorific -words that are titles in front and names elsewhere (Shekh, Guru), Md., +words that are titles in front and names elsewhere (Guru; Shekh until +#752's phase 2j made it a title only in a title phrase), Md., Muhammad's abbreviation, which its period makes a title by shape, and Devanagari written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 460b5e44..ed0dcad7 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -512,6 +512,7 @@ {"name": "Chas. Smith", "given": "Chas.", "family": "Smith", "initials": "C. S."} {"name": "Chas. Smith, Ed", "given": "Chas.", "family": "Smith", "suffix": "Ed", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: name as given (declared)"], "initials": "C. S.", "capitalized_force": "Chas. Smith ED"} {"name": "Chattel, Frederik Jacobus van Rossum", "given": "Frederik", "middle": "Jacobus van Rossum", "family": "Chattel", "initials": "F. J. R. C."} +{"name": "Cheikh Anta Diop", "given": "Cheikh", "middle": "Anta", "family": "Diop", "initials": "C. A. D."} {"name": "Chemistry Jane Smith", "given": "Chemistry", "middle": "Jane", "family": "Smith", "initials": "C. J. S."} {"name": "Cherice J. (Johnson) Williams", "given": "Cherice", "middle": "J.", "family": "Williams", "nickname": "Johnson", "initials": "C. J. W."} {"name": "Chief Dean and Mary Smith", "title": "Chief Dean and Mary", "family": "Smith", "initials": "S."} @@ -3082,10 +3083,9 @@ {"name": "Shaikh. Sultan Khan", "title": "Shaikh.", "given": "Sultan", "family": "Khan", "initials": "S. K."} {"name": "Shanti Gaonkar", "given": "Shanti", "family": "Gaonkar", "initials": "S. G."} {"name": "Shehu Muhammed", "given": "Shehu", "family": "Muhammed", "initials": "S. M."} -{"name": "Sheik Abu Bakar", "title": "Sheik", "given": "Abu Bakar", "initials": "A. B."} -{"name": "Sheik abdul Jr Smith", "title": "Sheik", "given": "abdul", "middle": "Jr", "family": "Smith", "initials": "a. J. S.", "capitalized_force": "Sheik Abdul Jr Smith"} -{"name": "Sheik abdul salam", "title": "Sheik", "given": "abdul salam", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam"} -{"name": "Sheik abdul salam Jr", "title": "Sheik", "given": "abdul salam", "suffix": "Jr", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam Jr"} +{"name": "Sheik Abu Bakar", "given": "Sheik", "family": "Abu Bakar", "initials": "S. B.", "capitalized_force": "Sheik abu Bakar"} +{"name": "Sheik abdul salam", "given": "Sheik", "middle": "abdul", "family": "salam", "initials": "S. a. s.", "capitalized_force": "Sheik Abdul Salam"} +{"name": "Sheikh Mohammed bin Rashid", "given": "Sheikh", "middle": "Mohammed", "family": "bin Rashid", "initials": "S. M. R."} {"name": "Shemuʾel ben Ḥofni", "given": "Shemuʾel", "middle": "ben", "family": "Ḥofni", "initials": "S. b. Ḥ.", "capitalized_force": "Shemuʾel Ben Ḥofni"} {"name": "Shin Ae-ra", "given": "Shin", "family": "Ae-ra", "initials": "S. A.", "capitalized_force": "Shin Ae-Ra"} {"name": "Shirley Maclaine", "given": "Shirley", "family": "Maclaine", "initials": "S. M.", "capitalized_force": "Shirley MacLaine"} @@ -3130,6 +3130,7 @@ {"name": "Sindbad", "given": "Sindbad", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as family (position)"], "initials": "S."} {"name": "Sindbad", "order": "FAMILY_FIRST", "family": "Sindbad", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as given (position)"], "initials": "S."} {"name": "Sinibaldo Gutiérrez Mas", "given": "Sinibaldo", "middle": "Gutiérrez", "family": "Mas", "initials": "S. G. M."} +{"name": "Sir Abu Bakar", "title": "Sir", "given": "Abu Bakar", "initials": "A. B."} {"name": "Sir Bob Andrew Dole", "title": "Sir", "given": "Bob", "middle": "Andrew", "family": "Dole", "initials": "B. A. D."} {"name": "Sir Gerald", "title": "Sir", "given": "Gerald", "initials": "G."} {"name": "Sir John", "title": "Sir", "given": "John", "initials": "J."} @@ -3137,6 +3138,9 @@ {"name": "Sir Ph. D. Van Johnson", "title": "Sir", "given": "Van Johnson", "suffix": "Ph. D.", "initials": "V. J."} {"name": "Sir Smith, John", "title": "Sir", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Sir Van Johnson", "title": "Sir", "given": "Van", "family": "Johnson", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: particle as family (vocabulary)"], "initials": "V. J."} +{"name": "Sir abdul Jr Smith", "title": "Sir", "given": "abdul", "middle": "Jr", "family": "Smith", "initials": "a. J. S.", "capitalized_force": "Sir Abdul Jr Smith"} +{"name": "Sir abdul salam", "title": "Sir", "given": "abdul salam", "initials": "a. s.", "capitalized_force": "Sir Abdul Salam"} +{"name": "Sir abdul salam Jr", "title": "Sir", "given": "abdul salam", "suffix": "Jr", "initials": "a. s.", "capitalized_force": "Sir Abdul Salam Jr"} {"name": "Sir abdul van der Berg", "title": "Sir", "given": "abdul", "family": "van der Berg", "initials": "a. B.", "capitalized_force": "Sir Abdul van der Berg"} {"name": "Sir de", "title": "Sir", "given": "de", "initials": "d.", "capitalized_force": "Sir De"} {"name": "Sir de Mesnil", "title": "Sir", "family": "de Mesnil", "initials": "M."} From 86a82af66fb07ffda631ab70d48d12f56bba237c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 02:29:35 -0700 Subject: [PATCH 2/4] Fix the prose the review found stale, and record the bound-pair split (#752 phase 2j) decisions/H1's Standing line, AGENTS.md and tools/differential/README.md's nepal-candidates rows, the vocabulary-exclusions divergence line, a test comment and the given_name_titles docstring still used a sheikh spelling as a title in front; the Shekh case note misstated master's reading. decisions/H9 now records that a bound given name behind a bare leading spelling no longer joins forward. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions/H1-title-and-one-name-word.md | 2 +- docs/design/decisions/H9-title-phrase.md | 1 + docs/design/decisions/vocabulary-exclusions.md | 2 +- nameparser/_lexicon.py | 2 +- tests/v2/cases.py | 7 +++++-- tests/v2/test_locales.py | 7 ++++--- tools/differential/README.md | 2 +- 8 files changed, 15 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 152eeee4..39f5afd4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -257,7 +257,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Shekh` in front, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific words that were titles in front until #752's phase 2j made them titles only in a title phrase, such as `Shekh`, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. ## Gotchas diff --git a/docs/design/decisions/H1-title-and-one-name-word.md b/docs/design/decisions/H1-title-and-one-name-word.md index 9a341169..9fd71f75 100644 --- a/docs/design/decisions/H1-title-and-one-name-word.md +++ b/docs/design/decisions/H1-title-and-one-name-word.md @@ -8,7 +8,7 @@ - Where a run stands before the one name word, that run addresses; a run behind the word decides its field only when none stands before, and a run written around a nickname is one run: `Sir John Prof.` → given 'John', `Dr. Smith Sir.` → family 'Smith', `Smith Sir.` → given 'Smith', `Dr. 'Smitty' Sir John` → given 'John' — 2026-09-08 #489; 2026-09-09 #316. - `prince` and `princess` are given-name titles and `lord` and `lady` are not, decided rather than deferred, since the text does not say which rank a bearer holds: `His Royal Highness Prince Harry` → given 'Harry' (the bare `Prince Harry` reads given 'Prince' in the default order since 2026-10-27, `prince` being a title only inside a title phrase, decisions/H9), `Lord Byron` → family 'Byron', `His Excellency Lord Duncan` → family 'Duncan' — 2026-09-10 #519. - H1 decides the field of the one name word before a comma with no given name after it, and of the one name word of a name read family-first: `Elizabeth II, Queen` → given 'Elizabeth', `Smith, Sir` → given 'Smith' with the family empty, `Sir John` and `Pope Francis` given under every order — 2026-10-18 #720. -- A particle surname keeps the family where something settled it, a comma (`van der Berg, Sir`, `Van, Sir`) or P1's fold asked of the leading name piece (`Sir de Mesnil`); an ambiguous particle no comma settled reads given (`Sheik Abu Bakar`, `Sir Van` family-first), and so does a never-given one the fold does not take (`St Sir de Mesnil` under every order) — 2026-10-18 #720. +- A particle surname keeps the family where something settled it, a comma (`van der Berg, Sir`, `Van, Sir`) or P1's fold asked of the leading name piece (`Sir de Mesnil`); an ambiguous particle no comma settled reads given (`Sir Abu Bakar`, `Sir Van` family-first; `Sheik Abu Bakar` until 2026-10-27, decisions/H9), and so does a never-given one the fold does not take (`St Sir de Mesnil` under every order) — 2026-10-18 #720. Elsewhere: the graf class, a title word borne as a name but rarely as its first word shipping as a title (`Greve Anna` → title 'Greve', family 'Anna'), is decisions/salutation-titles' 2026-10-04 #606 bundle; the renunciate class, the given-name titles of traditions that drop the surname (`Swami Vivekananda` → given 'Vivekananda'), is decisions/indic-honorifics' 2026-09-06 #346/#344/#343 bundle. diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 431c5499..2bb8703c 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -81,5 +81,6 @@ - THE COUNTS, recorded on the words (`uv run python tools/name_usage.py Sheikh Shekh Sheik Shaykh Shaik Cheikh`, 2026-10-27): `sheikh` given US SSA 436, Wikidata 21, family Census 6,462 (rank 5,384), Wikidata 60; `sheik` given SSA 39, family Census 401, Wikidata 4; `shaik` given SSA 10, family Census 1,053 (93.73% asian/PI), Wikidata 14; `cheikh` given SSA 512, Wikidata 51 (Senegal 34), family Census 112, Wikidata 14; `shekh` no US or Wikidata count, its evidence the filers above and 378 of Nepal's local candidates. - THE COST, for Derek to decide: a ruler, cleric or leader written with the bare title in front reads it as the given name (`Sheikh Mohammed bin Rashid Al Maktoum` → given 'Sheikh', middle 'Mohammed bin Rashid', where 1.4.0 through 2.6.0 read title 'Sheikh', given 'Mohammed'), the class accepted for `king`, `sultan`, `shaikh` and `prince`. In jrc-names 174 persons have a spelling opening with a bare form of the word: by a reading of each, about 40 Gulf rulers and members of their houses, about 55 clerics and militia, tribal or political leaders, about 60 people whose own name opens with it (most of them Senegalese and Mauritanian `Cheikh`s, beside Bangladeshi and Malaysian bearers), and the rest unclear. In PSC 16 of the 244 moved records file the word in the title box (13 broken, 3 changed), while Qatari and Emirati ruling-house members there file it in the forename box. - MEASURED (2026-10-27, against master `79a4111f`, py3.11, every local check with `--base`). PSC (1,039,144 scored) 209 records fixed, 13 broken, every one the word filed in the title box (12 `Sheikh`, 1 `Sheik`), 22 changed, most now reading the given name and splitting the filer's middle box differently; ctgov-officials 10 moved, 8 fixes (4 `Aa Sheikh, CRED` whose credential master read as the given name, 4 opening with `Sheikh` or `Cheikh`) and 2 lower-case or credential-run names still wrong; fec-individuals 26, all fixes, every one `AA, SPELLING…`, 18 of which had read no family name; nepal-candidates 267, every one opening with `Shekh` (265) or `Sheikh` (2); jrc-names 571 spellings; tse-donors and ukec-individuals none. Spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE): jrc-names 37 persons now alike and 1 apart (spelled `Shaykh` beside `Sheikh` and `Cheikh`), and all 12 persons the `shaikh` entry above left apart are alike again; nepal-candidates 280 pairs now alike and 0 apart, the romanized `Shekh` now reading as its Devanagari does. The committed corpora move by two names in corpus_issues, #367's `Sheik Abu Bakar` (given 'Sheik', family 'Abu Bakar') and `Sheik abdul salam` (given 'Sheik', middle 'abdul', family 'salam'), beside the batch's own rule examples, where `Sir` now stands for `Sheik` (below); CLDR and GND agreement do not move. A v1 `Constants().titles` keeps all five with no edit (the v1 title list entry above). + - A BOUND GIVEN NAME BEHIND IT. Read as a name word, the word licenses no bound-given join (rules/P5's licence is a given-name TITLE's), and the bound word is not the name's first piece, so the pair it would have made splits: `Sheikh abdul Rahman` → given 'Sheikh', middle 'abdul', family 'Rahman', where master read title 'Sheikh', given 'abdul Rahman' (and 1.4.0 given 'abdul', last 'Rahman'); `Shekh Abdul Rahim` and #367's `Sheik abdul salam` read so, as `Prince abdul Rahman` has since the batch above. The reading follows from the word being a name, as `Ahmed abdul Rahman` reads, and is the bound-pair face of the cost below. - WORKED EXAMPLES MOVED. `Sheik` was rules/P5's given-name title licensing the bound join (`Sheik abdul salam` and its family-first and suffix rows) and rules/H1's `Sheik Abu Bakar`; `Sir` replaces it, reading as `Sheik` read on master in every one of them under all three orders. The title-run row `Sir Sheikh abdul rahman` is `Dr. Sir abdul rahman`. Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 68b85410..da4d9fd7 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -61,7 +61,7 @@ Excluded (TITLES): - The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, #660's shehu, and #789's wm (2026-10-22) — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error. - shehu's counts do not settle a graf-class weighing, and must not be read as if they did: its one leading-position figure, the Wikidata given-name count on its entry, is a lower bound (Shagari's own entry has no given-name statement), and its other counts are family-name bearers, a position C-i does not ask about. The evidence that Shehu leads a name is not a count. It is Shagari himself, and CLDR's Hausa sample data, whose six renderings carrying the word all file it as the given name (`grep '"given": "Shehu"' tools/differential/cldr_fields.jsonl`). It arrived with an early bulk import (1a51555f) and was never put to C-i before #660. Whoever re-argues it counts Hausa given-name use first; the accepted cost is the honorific use, `Shehu Abubakar` reading given Shehu as any untitled name does, and the reading belongs in an opt-in pack if anyone asks for it. malam is the opposite case: CLDR's Hausa data files Malam as the title (`grep '"title": "Malam"' tools/differential/cldr_fields.jsonl`), so its refusal is a decline on the counts, not a shehu-style removal. -- The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). +- The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. Since 2026-10-27 the Latin spellings but `shaykh` are titles only inside a title phrase (decisions/H9), so at the head of a name `Shekh Ahmed` and `शेख अहमद` both read the word as the given name, and the divergence stands only where a style, a modifier or `of` makes the Latin word a title. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). - The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. (Since 2026-10-26 `king` is a title only inside a title phrase, decisions/H9, #752 phase 2j: "Mary Jane King." reads family "King." and the surnames this paragraph names are pope, judge, bishop, baron and sheriff, rules/H5's example "Mary Jane Pope.".) Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. The bare prohibition has one exception since #835 (decisions/H8): a listed salutation ending the given part after a family comma is the title, the comma having marked the surname's place and the salutations borne as names being excluded. Excluded (Policy.script_orders defaults): Script.KATAKANA is deliberately absent — a pure-katakana token is predominantly a transcribed foreign name kept in its source order, so nothing defaults on it (rule W4's boundary). Noted 2026-08-15: of the three Script-keyed axes, this is the one with no force-a-decision guard (mechanisms.md#FORCE-A-DECISION-TABLE), so a new Script member silently gets no order. diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 3c72c95e..e6e68005 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -594,7 +594,7 @@ class Lexicon: #: list: :data:`~nameparser.config.titles.TITLES`. titles: frozenset[str] = frozenset() #: Titles whose single name word, before or after them, reads as a - #: GIVEN name ("sheikh", "sister", ...) rather than a family name. Full + #: GIVEN name ("sir", "sister", ...) rather than a family name. Full #: default list: :data:`~nameparser.config.titles.GIVEN_NAME_TITLES`. given_name_titles: frozenset[str] = frozenset() #: Post-nominal acronym suffixes, matched with or without periods diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 45f8903f..ea8e784a 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -12673,8 +12673,11 @@ def _check_cjk_shape_purity(self) -> None: Case("given_name_title_shekh_at_the_head", "Shekh Abdul Rahim", {"given": "Shekh", "middle": "Abdul", "family": "Rahim"}, classification="fix(#752)", - notes="H9: the spelling Nepal's romanization writes; read " - "title 'Shekh', given 'Abdul' before"), + notes="H9: the spelling Nepal's romanization writes. Read " + "title 'Shekh', given 'Abdul Rahim' (P5's licensed join) " + "2.2.0 through 2.6.0, and given 'Abdul', last 'Rahim' in " + "1.4.0; as a name word it licenses no join, so the bound " + "word is a middle name"), Case("shaykh_stays_a_title", "Shaykh Hamza Yusuf", {"title": "Shaykh", "given": "Hamza", "family": "Yusuf"}, notes="'shaykh' has no given reading: no filed data writes it " diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index b48d9cd6..1352ce27 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -1156,9 +1156,10 @@ def test_non_interference_all_packs_combined() -> None: ("أبو مازن", "given", "أبو"), ("أحمد أبو خليل", "family", "أبو خليل"), ("علي ابو خالد", "family", "ابو خالد"), - # "الشيخ" carries the GIVEN_NAME_TITLES semantics of its - # transliterated cousin 'sheikh': a single following name reads as - # given, not family. + # "الشيخ" is a given-name title: a single following name reads as + # given, not family. Its transliteration 'sheikh' is a title only + # inside a title phrase since #752 phase 2j, so 'Sheikh Mohammed' + # reads given 'Sheikh'; the article form is no given name. ("الشيخ محمد", "given", "محمد"), # Arabic honorifics precede the GIVEN name, so all ship in # given_name_titles like الشيخ: article forms (never given names) diff --git a/tools/differential/README.md b/tools/differential/README.md index 59b92dd5..29001895 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -284,7 +284,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | | `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. The one local corpus whose names carry their fields, scored by `local_agreement.py` | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | -| `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Shekh`, `Guru`) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | +| `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. From d4d3444a882589df81132d737440e97d3fb79f9a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 02:34:07 -0700 Subject: [PATCH 3/4] Point the bound-pair bullet by key, and finish the second review's prose fixes (#752 phase 2j) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions/H1-title-and-one-name-word.md | 2 +- docs/design/decisions/H9-title-phrase.md | 2 +- docs/design/decisions/vocabulary-exclusions.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 39f5afd4..1bc04967 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -257,7 +257,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific words that were titles in front until #752's phase 2j made them titles only in a title phrase, such as `Shekh`, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. ## Gotchas diff --git a/docs/design/decisions/H1-title-and-one-name-word.md b/docs/design/decisions/H1-title-and-one-name-word.md index 9fd71f75..a7fe57d2 100644 --- a/docs/design/decisions/H1-title-and-one-name-word.md +++ b/docs/design/decisions/H1-title-and-one-name-word.md @@ -8,7 +8,7 @@ - Where a run stands before the one name word, that run addresses; a run behind the word decides its field only when none stands before, and a run written around a nickname is one run: `Sir John Prof.` → given 'John', `Dr. Smith Sir.` → family 'Smith', `Smith Sir.` → given 'Smith', `Dr. 'Smitty' Sir John` → given 'John' — 2026-09-08 #489; 2026-09-09 #316. - `prince` and `princess` are given-name titles and `lord` and `lady` are not, decided rather than deferred, since the text does not say which rank a bearer holds: `His Royal Highness Prince Harry` → given 'Harry' (the bare `Prince Harry` reads given 'Prince' in the default order since 2026-10-27, `prince` being a title only inside a title phrase, decisions/H9), `Lord Byron` → family 'Byron', `His Excellency Lord Duncan` → family 'Duncan' — 2026-09-10 #519. - H1 decides the field of the one name word before a comma with no given name after it, and of the one name word of a name read family-first: `Elizabeth II, Queen` → given 'Elizabeth', `Smith, Sir` → given 'Smith' with the family empty, `Sir John` and `Pope Francis` given under every order — 2026-10-18 #720. -- A particle surname keeps the family where something settled it, a comma (`van der Berg, Sir`, `Van, Sir`) or P1's fold asked of the leading name piece (`Sir de Mesnil`); an ambiguous particle no comma settled reads given (`Sir Abu Bakar`, `Sir Van` family-first; `Sheik Abu Bakar` until 2026-10-27, decisions/H9), and so does a never-given one the fold does not take (`St Sir de Mesnil` under every order) — 2026-10-18 #720. +- A particle surname keeps the family where something settled it, a comma (`van der Berg, Sir`, `Van, Sir`) or P1's fold asked of the leading name piece (`Sir de Mesnil`); an ambiguous particle no comma settled reads given (`Sir Abu Bakar`, `Sir Van` family-first; `Sheik Abu Bakar` until 2026-10-27, decisions/H9, which under the family-first orders now reads the word as the family and the particle given by position), and so does a never-given one the fold does not take (`St Sir de Mesnil` under every order) — 2026-10-18 #720. Elsewhere: the graf class, a title word borne as a name but rarely as its first word shipping as a title (`Greve Anna` → title 'Greve', family 'Anna'), is decisions/salutation-titles' 2026-10-04 #606 bundle; the renunciate class, the given-name titles of traditions that drop the surname (`Swami Vivekananda` → given 'Vivekananda'), is decisions/indic-honorifics' 2026-09-06 #346/#344/#343 bundle. diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 2bb8703c..2ca7a3e2 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -81,6 +81,6 @@ - THE COUNTS, recorded on the words (`uv run python tools/name_usage.py Sheikh Shekh Sheik Shaykh Shaik Cheikh`, 2026-10-27): `sheikh` given US SSA 436, Wikidata 21, family Census 6,462 (rank 5,384), Wikidata 60; `sheik` given SSA 39, family Census 401, Wikidata 4; `shaik` given SSA 10, family Census 1,053 (93.73% asian/PI), Wikidata 14; `cheikh` given SSA 512, Wikidata 51 (Senegal 34), family Census 112, Wikidata 14; `shekh` no US or Wikidata count, its evidence the filers above and 378 of Nepal's local candidates. - THE COST, for Derek to decide: a ruler, cleric or leader written with the bare title in front reads it as the given name (`Sheikh Mohammed bin Rashid Al Maktoum` → given 'Sheikh', middle 'Mohammed bin Rashid', where 1.4.0 through 2.6.0 read title 'Sheikh', given 'Mohammed'), the class accepted for `king`, `sultan`, `shaikh` and `prince`. In jrc-names 174 persons have a spelling opening with a bare form of the word: by a reading of each, about 40 Gulf rulers and members of their houses, about 55 clerics and militia, tribal or political leaders, about 60 people whose own name opens with it (most of them Senegalese and Mauritanian `Cheikh`s, beside Bangladeshi and Malaysian bearers), and the rest unclear. In PSC 16 of the 244 moved records file the word in the title box (13 broken, 3 changed), while Qatari and Emirati ruling-house members there file it in the forename box. - MEASURED (2026-10-27, against master `79a4111f`, py3.11, every local check with `--base`). PSC (1,039,144 scored) 209 records fixed, 13 broken, every one the word filed in the title box (12 `Sheikh`, 1 `Sheik`), 22 changed, most now reading the given name and splitting the filer's middle box differently; ctgov-officials 10 moved, 8 fixes (4 `Aa Sheikh, CRED` whose credential master read as the given name, 4 opening with `Sheikh` or `Cheikh`) and 2 lower-case or credential-run names still wrong; fec-individuals 26, all fixes, every one `AA, SPELLING…`, 18 of which had read no family name; nepal-candidates 267, every one opening with `Shekh` (265) or `Sheikh` (2); jrc-names 571 spellings; tse-donors and ukec-individuals none. Spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE): jrc-names 37 persons now alike and 1 apart (spelled `Shaykh` beside `Sheikh` and `Cheikh`), and all 12 persons the `shaikh` entry above left apart are alike again; nepal-candidates 280 pairs now alike and 0 apart, the romanized `Shekh` now reading as its Devanagari does. The committed corpora move by two names in corpus_issues, #367's `Sheik Abu Bakar` (given 'Sheik', family 'Abu Bakar') and `Sheik abdul salam` (given 'Sheik', middle 'abdul', family 'salam'), beside the batch's own rule examples, where `Sir` now stands for `Sheik` (below); CLDR and GND agreement do not move. A v1 `Constants().titles` keeps all five with no edit (the v1 title list entry above). - - A BOUND GIVEN NAME BEHIND IT. Read as a name word, the word licenses no bound-given join (rules/P5's licence is a given-name TITLE's), and the bound word is not the name's first piece, so the pair it would have made splits: `Sheikh abdul Rahman` → given 'Sheikh', middle 'abdul', family 'Rahman', where master read title 'Sheikh', given 'abdul Rahman' (and 1.4.0 given 'abdul', last 'Rahman'); `Shekh Abdul Rahim` and #367's `Sheik abdul salam` read so, as `Prince abdul Rahman` has since the batch above. The reading follows from the word being a name, as `Ahmed abdul Rahman` reads, and is the bound-pair face of the cost below. + - A BOUND GIVEN NAME BEHIND IT. Read as a name word, the word licenses no bound-given join (rules/P5's licence is a given-name TITLE's), and the bound word is not the name's first piece, so the pair it would have made splits: `Sheikh abdul Rahman` → given 'Sheikh', middle 'abdul', family 'Rahman', where master read title 'Sheikh', given 'abdul Rahman' (and 1.4.0 given 'abdul', last 'Rahman'); `Shekh Abdul Rahim` and #367's `Sheik abdul salam` read so, as `Prince abdul Rahman` has since the 2026-10-27 `prince` and `princess` entry. The reading follows from the word being a name, as `Ahmed abdul Rahman` reads, and is the bound-pair face of this entry's THE COST bullet; a style in front keeps the join (`His Highness Sheikh abdul Rahman` → given 'abdul Rahman'). - WORKED EXAMPLES MOVED. `Sheik` was rules/P5's given-name title licensing the bound join (`Sheik abdul salam` and its family-first and suffix rows) and rules/H1's `Sheik Abu Bakar`; `Sir` replaces it, reading as `Sheik` read on master in every one of them under all three orders. The title-run row `Sir Sheikh abdul rahman` is `Dr. Sir abdul rahman`. Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index da4d9fd7..a486526d 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -61,7 +61,7 @@ Excluded (TITLES): - The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, #660's shehu, and #789's wm (2026-10-22) — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error. - shehu's counts do not settle a graf-class weighing, and must not be read as if they did: its one leading-position figure, the Wikidata given-name count on its entry, is a lower bound (Shagari's own entry has no given-name statement), and its other counts are family-name bearers, a position C-i does not ask about. The evidence that Shehu leads a name is not a count. It is Shagari himself, and CLDR's Hausa sample data, whose six renderings carrying the word all file it as the given name (`grep '"given": "Shehu"' tools/differential/cldr_fields.jsonl`). It arrived with an early bulk import (1a51555f) and was never put to C-i before #660. Whoever re-argues it counts Hausa given-name use first; the accepted cost is the honorific use, `Shehu Abubakar` reading given Shehu as any untitled name does, and the reading belongs in an opt-in pack if anyone asks for it. malam is the opposite case: CLDR's Hausa data files Malam as the title (`grep '"title": "Malam"' tools/differential/cldr_fields.jsonl`), so its refusal is a decline on the counts, not a shehu-style removal. -- The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. Since 2026-10-27 the Latin spellings but `shaykh` are titles only inside a title phrase (decisions/H9), so at the head of a name `Shekh Ahmed` and `शेख अहमद` both read the word as the given name, and the divergence stands only where a style, a modifier or `of` makes the Latin word a title. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). +- The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. Since 2026-10-27 the Latin spellings but `shaykh` are titles only inside a title phrase (decisions/H9), so at the head of a name `Shekh Ahmed` and `शेख अहमद` both read the word as the given name, and the divergence stands only where a style, a modifier or `of` makes the Latin word a title, or a period does at the front by rules/H2's shape (`Sheikh. Ahmed` reads title 'Sheikh.', `शेख. अहमद` given 'शेख.'). shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). - The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. (Since 2026-10-26 `king` is a title only inside a title phrase, decisions/H9, #752 phase 2j: "Mary Jane King." reads family "King." and the surnames this paragraph names are pope, judge, bishop, baron and sheriff, rules/H5's example "Mary Jane Pope.".) Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. The bare prohibition has one exception since #835 (decisions/H8): a listed salutation ending the given part after a family comma is the title, the comma having marked the surname's place and the salutations borne as names being excluded. Excluded (Policy.script_orders defaults): Script.KATAKANA is deliberately absent — a pure-katakana token is predominantly a transcribed foreign name kept in its source order, so nothing defaults on it (rule W4's boundary). Noted 2026-08-15: of the three Script-keyed axes, this is the one with no force-a-decision guard (mechanisms.md#FORCE-A-DECISION-TABLE), so a new Script member silently gets no order. From 8b64637e20242ab3a373e40d9096355d2b537e06 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 02:38:55 -0700 Subject: [PATCH 4/4] Record Derek's acceptance of the sheikh ruler cost, and decide #853's Md. question (#752 phase 2j, #853) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H9-title-phrase.md | 6 +++--- docs/design/decisions/indic-honorifics.md | 2 ++ docs/design/decisions/vocabulary-model.md | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 2ca7a3e2..b500493a 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -76,11 +76,11 @@ - THE COST, accepted by Derek on 2026-10-27, shown the share of royals below beside `king`'s, `sultan`'s and `shaikh`'s and that a style or `of` keeps the title: a royal written with the bare title in front reads it as the given name (`Prince Harry` → given 'Prince', family 'Harry', where 2.3.0 through 2.6.0 read title 'Prince', given 'Harry'), the class accepted for `king`, `sultan` and `shaikh`. In jrc-names 43 persons have a spelling opening with a bare `Prince` or `Princess` (one not followed by `of`): by a reading of each, about 33 are royals, 8 people whose given name is Prince, and 2 no person at all (a fictional character, a place). In PSC 12 of the 233 moved records file the word in the title box (10 broken, 2 changed). - MEASURED (2026-10-27, against master `ac81de9c`, py3.11, every local check with `--base`). PSC 205 records fixed, 10 broken, every one `Prince` (6) or `Princess` (4) filed in the title box, which the scorer now sets aside, 18 changed, most now reading the given name right and splitting a two-word surname box, 2 of them title-box filings; ctgov-officials 6 moved, all fixes: 4 open with the word, 3 of them before a credential comma, 1 glues an initial to it (`J.Prince Aa, CRED`), and 1 is `Aa Prince, CRED`, whose credential master read as the given name; fec-individuals 43, all fixes, every one `AA, PRINCE` or `AA, PRINCESS` in the listing form, 28 of which had read no family name; nepal-candidates 3, fixes; jrc-names 101 spellings moved, the royals above and `Archduchess of X Princess Y`, a title before the word that is no partner of it; tse-donors and ukec-individuals none; spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE) jrc-names 4 persons now alike and 0 apart, nepal-candidates 3 and 0. The committed corpora move by `Prince Charles` and `Prince Fielder` (corpus_issues, #348's examples) beside the batch's own rule examples; CLDR and GND agreement do not move. A v1 `Constants().titles` keeps both words with no edit (the v1 title list entry above). Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). -- **2026-10-27 #752, phase 2j, `sheikh` and its spellings — THE NINTH AND LAST TITLE BATCH.** `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` gain dated `given` readings and are titles only inside a title phrase, as `shaikh` became in the seventh batch; no listed phrase holds any of them, so nothing is reported. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed`, `Khan, Sheikh`, `Cheikh Anta Diop` and `Shekh Abdul Rahim` read the word as the given name, where 1.4.0 through 2.6.0 read the title. A style keeps it (`His Highness Sheikh Ahmed Khan`, as 1.4.0 read it), as does a modifier (`Grand Sheikh Ahmed`) or `of` (`Sheikh of Kuwait`), and the period-marked spelling at the front is a title by H2's shape (`Sheikh. Ahmed Khan`). The other sites read as for `shaikh`: before a family comma (`Sheikh Smith, Ahmed` → family 'Sheikh Smith', 1.4.0's reading), ending the name, alone after a full name and a comma, and past a second comma. A style read as a name word partners nothing (`Sheikh Sultan Khan` → given 'Sheikh', middle 'Sultan', where master read title 'Sheikh Sultan'), and behind a style it partners the contested word as before (`His Highness Sheikh Sultan Khan`), closing the NOT REACHED bullet of the 2026-10-26 `sultana` and `sultan` entry above. +- **2026-10-27 #752, phase 2j, `sheikh` and its spellings (Derek) — THE NINTH AND LAST TITLE BATCH.** `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` gain dated `given` readings and are titles only inside a title phrase, as `shaikh` became in the seventh batch; no listed phrase holds any of them, so nothing is reported. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed`, `Khan, Sheikh`, `Cheikh Anta Diop` and `Shekh Abdul Rahim` read the word as the given name, where 1.4.0 through 2.6.0 read the title. A style keeps it (`His Highness Sheikh Ahmed Khan`, as 1.4.0 read it), as does a modifier (`Grand Sheikh Ahmed`) or `of` (`Sheikh of Kuwait`), and the period-marked spelling at the front is a title by H2's shape (`Sheikh. Ahmed Khan`). The other sites read as for `shaikh`: before a family comma (`Sheikh Smith, Ahmed` → family 'Sheikh Smith', 1.4.0's reading), ending the name, alone after a full name and a comma, and past a second comma. A style read as a name word partners nothing (`Sheikh Sultan Khan` → given 'Sheikh', middle 'Sultan', where master read title 'Sheikh Sultan'), and behind a style it partners the contested word as before (`His Highness Sheikh Sultan Khan`), closing the NOT REACHED bullet of the 2026-10-26 `sultana` and `sultan` entry above. - WHICH SPELLINGS. Vetted by where UK PSC's filers put each (the first word of the forename box against any word of the title box, 2026-10-27): `sheikh` 174 against 19, `sheik` 23 against 1, `shekh` 6 against 1, `shaik` 14 against 0, `cheikh` 14 against 0. `shaykh`, the scholarly transliteration, stays a title everywhere: no filer writes it in front and no source counts it as a given name; the one jrc-names person it now holds apart from its other spellings (below) is the cost. - THE COUNTS, recorded on the words (`uv run python tools/name_usage.py Sheikh Shekh Sheik Shaykh Shaik Cheikh`, 2026-10-27): `sheikh` given US SSA 436, Wikidata 21, family Census 6,462 (rank 5,384), Wikidata 60; `sheik` given SSA 39, family Census 401, Wikidata 4; `shaik` given SSA 10, family Census 1,053 (93.73% asian/PI), Wikidata 14; `cheikh` given SSA 512, Wikidata 51 (Senegal 34), family Census 112, Wikidata 14; `shekh` no US or Wikidata count, its evidence the filers above and 378 of Nepal's local candidates. - - THE COST, for Derek to decide: a ruler, cleric or leader written with the bare title in front reads it as the given name (`Sheikh Mohammed bin Rashid Al Maktoum` → given 'Sheikh', middle 'Mohammed bin Rashid', where 1.4.0 through 2.6.0 read title 'Sheikh', given 'Mohammed'), the class accepted for `king`, `sultan`, `shaikh` and `prince`. In jrc-names 174 persons have a spelling opening with a bare form of the word: by a reading of each, about 40 Gulf rulers and members of their houses, about 55 clerics and militia, tribal or political leaders, about 60 people whose own name opens with it (most of them Senegalese and Mauritanian `Cheikh`s, beside Bangladeshi and Malaysian bearers), and the rest unclear. In PSC 16 of the 244 moved records file the word in the title box (13 broken, 3 changed), while Qatari and Emirati ruling-house members there file it in the forename box. + - THE COST, accepted by Derek on 2026-10-27 ("accept the ruler cost"), shown the share of rulers below beside the earlier batches' and the bound-pair split: a ruler, cleric or leader written with the bare title in front reads it as the given name (`Sheikh Mohammed bin Rashid Al Maktoum` → given 'Sheikh', middle 'Mohammed bin Rashid', where 1.4.0 through 2.6.0 read title 'Sheikh', given 'Mohammed'), the class accepted for `king`, `sultan`, `shaikh` and `prince`. In jrc-names 174 persons have a spelling opening with a bare form of the word: by a reading of each, about 40 Gulf rulers and members of their houses, about 55 clerics and militia, tribal or political leaders, about 60 people whose own name opens with it (most of them Senegalese and Mauritanian `Cheikh`s, beside Bangladeshi and Malaysian bearers), and the rest unclear. In PSC 16 of the 244 moved records file the word in the title box (13 broken, 3 changed), while Qatari and Emirati ruling-house members there file it in the forename box. - MEASURED (2026-10-27, against master `79a4111f`, py3.11, every local check with `--base`). PSC (1,039,144 scored) 209 records fixed, 13 broken, every one the word filed in the title box (12 `Sheikh`, 1 `Sheik`), 22 changed, most now reading the given name and splitting the filer's middle box differently; ctgov-officials 10 moved, 8 fixes (4 `Aa Sheikh, CRED` whose credential master read as the given name, 4 opening with `Sheikh` or `Cheikh`) and 2 lower-case or credential-run names still wrong; fec-individuals 26, all fixes, every one `AA, SPELLING…`, 18 of which had read no family name; nepal-candidates 267, every one opening with `Shekh` (265) or `Sheikh` (2); jrc-names 571 spellings; tse-donors and ukec-individuals none. Spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE): jrc-names 37 persons now alike and 1 apart (spelled `Shaykh` beside `Sheikh` and `Cheikh`), and all 12 persons the `shaikh` entry above left apart are alike again; nepal-candidates 280 pairs now alike and 0 apart, the romanized `Shekh` now reading as its Devanagari does. The committed corpora move by two names in corpus_issues, #367's `Sheik Abu Bakar` (given 'Sheik', family 'Abu Bakar') and `Sheik abdul salam` (given 'Sheik', middle 'abdul', family 'salam'), beside the batch's own rule examples, where `Sir` now stands for `Sheik` (below); CLDR and GND agreement do not move. A v1 `Constants().titles` keeps all five with no edit (the v1 title list entry above). - - A BOUND GIVEN NAME BEHIND IT. Read as a name word, the word licenses no bound-given join (rules/P5's licence is a given-name TITLE's), and the bound word is not the name's first piece, so the pair it would have made splits: `Sheikh abdul Rahman` → given 'Sheikh', middle 'abdul', family 'Rahman', where master read title 'Sheikh', given 'abdul Rahman' (and 1.4.0 given 'abdul', last 'Rahman'); `Shekh Abdul Rahim` and #367's `Sheik abdul salam` read so, as `Prince abdul Rahman` has since the 2026-10-27 `prince` and `princess` entry. The reading follows from the word being a name, as `Ahmed abdul Rahman` reads, and is the bound-pair face of this entry's THE COST bullet; a style in front keeps the join (`His Highness Sheikh abdul Rahman` → given 'abdul Rahman'). + - A BOUND GIVEN NAME BEHIND IT. Read as a name word, the word licenses no bound-given join (rules/P5's licence is a given-name TITLE's), and the bound word is not the name's first piece, so the pair it would have made splits: `Sheikh abdul Rahman` → given 'Sheikh', middle 'abdul', family 'Rahman', where master read title 'Sheikh', given 'abdul Rahman' (and 1.4.0 given 'abdul', last 'Rahman'); `Shekh Abdul Rahim` and #367's `Sheik abdul salam` read so, as `Prince abdul Rahman` has since the 2026-10-27 `prince` and `princess` entry. The reading follows from the word being a name, as `Ahmed abdul Rahman` reads, and is the bound-pair face of this entry's THE COST bullet, accepted with it; a style in front keeps the join (`His Highness Sheikh abdul Rahman` → given 'abdul Rahman'). - WORKED EXAMPLES MOVED. `Sheik` was rules/P5's given-name title licensing the bound join (`Sheik abdul salam` and its family-first and suffix rows) and rules/H1's `Sheik Abu Bakar`; `Sir` replaces it, reading as `Sheik` read on master in every one of them under all three orders. The title-run row `Sir Sheikh abdul rahman` is `Dr. Sir abdul rahman`. Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base `, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base ` (and `fec-individuals`, `tse-donors`, `ukec-individuals`, `nepal-candidates`, `jrc-names`). diff --git a/docs/design/decisions/indic-honorifics.md b/docs/design/decisions/indic-honorifics.md index 90e24d5e..7da4d362 100644 --- a/docs/design/decisions/indic-honorifics.md +++ b/docs/design/decisions/indic-honorifics.md @@ -15,6 +15,7 @@ Closes #346, #344 and #343 as one bundle: each CLASS is decided once, and each s - Title vocabulary beats the one-letter-plus-period initial shape, measured and pinned by rows and not a rule: `ড. মুহাম্মদ ইউনূস` reads title 'ড.' while `র. কে. নারায়ণ`, with no entry behind it, keeps its initials — 2026-09-06 #346. - An unlisted abugida honorific stays a name word, and rules/H2's executable example of it is `প্রকৌশলী. Sen`, so whoever ships প্রকৌশলী moves the example first; the Tier 4 candidates in the Indic scripts (transliterated military ranks, Advocate, Engineer, Marhum, the kinship forms, मां/माता/মা, the hafiz/syed/shahid forms) are skipped pending a corpus check and not declined — 2026-09-06 #346. - Latin `md` is a credential and no title: a leading `Md` is the given name it abbreviates (`Md Abdul Karim` → given 'Md', middle 'Abdul'; `Mr Md Karim Uddin` → title 'Mr', given 'Md', middle 'Karim'), at two accepted costs, a particle behind it starting the family (`Md Abu Sayed` → family 'Abu Sayed') and the given part of a family comma it opens, titles in front or not, reading as credentials (`Uddin, Md Karim` and `Uddin, Mr Md Karim` → suffix 'Md Karim'); the period-marked `Md.` is still a title by rules/H2's leading-abbreviation shape (`Md. Abdul Karim` → title 'Md.'), and the Bengali মোঃ/মো entries stay titles — 2026-10-23 #853. +- The period-marked `Md.` stays a title by rules/H2's leading-abbreviation shape, an accepted cost: `Md. Karim Uddin` reads title 'Md.', given 'Karim', and `Mr Md. Karim Uddin` title 'Mr Md.' — 2026-10-27 (Derek), #853. Elsewhere: the bundle's class-level refusals are recorded under decisions/vocabulary-exclusions, not here: the Latin forms of the Indic titles belonging to the opt-in packs of #345 (2026-09-06, #344), the native spelling shipping alone where a transliteration collides with real names (sri/shri, sant, pandit, mst), the ban on the spaced trailing set becoming glued honorifics, and the per-word title refusals (ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, শেখ/शेख, आचार्य, राजा/रानी), each an `excluded` entry (`uv run python tools/vocab/query.py --excluded title`). @@ -41,3 +42,4 @@ Open: #514 (whether the Tibetan prefix titles Tulku, Khenpo, Geshe, Kyabje and J ACCEPTED COSTS (Derek). (1) A particle behind `Md` starts the family: `Md Abu Sayed` reads given 'Md', family 'Abu Sayed', where the title reading left `Abu` the leading name word and read given 'Abu', family 'Sayed'. 18 of the 1,300 lose the family name the title reading gave them this way, each with `Abu`, `Al` or `Bin`, the filer's surname box holding the last word alone. It is the spelled-out name's reading (`Mohammad Abu Sayed` reads the same), so the gap is P2's and P5's for any given name before `abu`, not Md's. (2) The given part of a family comma that `Md` opens reads as a credential run (rules/C13), and so does one with only titles in front of it, the run starting behind titles alone: `Uddin, Md Karim` reads family 'Uddin', suffix 'Md Karim' with a `suffix-or-name` report, where the title reading gave given 'Karim', and `Uddin, Mr Md Karim` reads title 'Mr', suffix 'Md Karim', where it gave title 'Mr Md', given 'Karim'. 9 of the 10 FEC moves are this shape, FEC writing `FAMILY, GIVEN`. Marking `md` ambiguous (`may_be_name`) would fix it, measured: `Uddin, Md Karim` reads given 'Md', and every `John Smith MD` then reports `suffix-or-name`, so it was declined for the commonest credential there is. NOT CHANGED: the period-marked `Md.` is still a title by rules/H2's leading-abbreviation shape (37 of the 1,300 PSC names write it so), which is #853's question 3 and not this entry's; and the Bengali মোঃ/মো and মোসাঃ/মোছাঃ entries stay titles, no filed data reaching Bengali script. The case rows and rule examples that used `MD` as their title/suffix dual (rules/H3, H4, S7 and C13; decisions/H3, H4, H6, S7 and C9's Standing lines; rules/C9, C11, C12 and H6; the cases.py dual rows) are respelled with `RA`, which reads as `MD` did at 1.4.0, 2.3.0 and the parent; their dated log entries keep `MD` as written. RECIPE for the figures above. PSC: in `~/.cache/nameparser-tools/local/psc-individuals/raw.jsonl`, the records whose `forename` field, split on whitespace, opens with a word that is `md` once lowercased and stripped of edge periods (1,300); "alone" is a one-word forename, `Md.` one ending in a period; the title-box count is over every record, its `title` folded the same way; a given name matches the box where `given` and `forename` agree lowercased with whitespace runs collapsed; a family is lost where the comparator's `family` equals the `surname` field lowercased and the tree's does not. Local corpora moves: every record of each `corpus.jsonl` matching `(^|[\s,])md\.?([\s,]|$)` case-insensitively, a move being any difference in the seven name fields between the comparator and the tree; "title to suffix" (or "to given") is `md`, periods removed, a word of the comparator's `title` and of the tree's `suffix` (or `given`). +- 2026-10-27 (Derek), #853 — THE PERIOD-MARKED MD. STAYS A TITLE, AN ACCEPTED COST: #853's question 3, left open by the 2026-10-23 entry above, decided. `Mr Md. Karim Uddin` reads title 'Mr Md.', given 'Karim', family 'Uddin', and `Md. Karim Uddin` title 'Md.', as rules/H2's shape reads any unlisted period-marked word opening the given part; `md` is not given the yield a given name written short has (`Lexicon.given_name_abbreviations`, which `Ma.` takes since #789). Derek: "Md. is an accepted cost: record it as decided and close #853". The evidence he weighed: 37 of the 1,300 UK PSC filers who open the forename box with `Md` write the period (the 2026-10-23 entry's recipe), and in nepal-candidates 221 of the 313 romanized names opening with `Md` write it so, all 221 reading it as a title (Derek's comment on #853, 2026-10-24). With it, and #752 phase 2j's given readings (decisions/H9), every question #853 asked is answered, and the issue closes with the `sheikh` batch. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 09e85295..ba54d843 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -218,7 +218,7 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest. Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did. - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. -- **2026-10-27 #752, phase 2j, `sheikh` and its spellings — THE NINTH AND LAST TITLE BATCH: `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` have dated `given` readings**, so each is a title only inside a title phrase; `shaykh` stays a title. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed` and `Cheikh Anta Diop` read the given name; a ruler written with the bare title does too (`Sheikh Mohammed bin Rashid`), its cost here for Derek to decide. With it every word item 12's measurement named has its reading. The measurement is decisions/H9's entry of this date. +- **2026-10-27 #752, phase 2j, `sheikh` and its spellings (Derek) — THE NINTH AND LAST TITLE BATCH: `sheikh`, `sheik`, `shekh`, `shaik` and `cheikh` have dated `given` readings**, so each is a title only inside a title phrase; `shaykh` stays a title. `Sheikh Ahmed Khan`, `Mr Sheikh Ahmed` and `Cheikh Anta Diop` read the given name; a ruler written with the bare title does too (`Sheikh Mohammed bin Rashid`), the cost Derek accepted on 2026-10-27. With it every word item 12's measurement named has its reading. The measurement is decisions/H9's entry of this date. - **2026-10-27 #752, phase 2j, `prince` and `princess` (Derek) — THE EIGHTH TITLE BATCH: both have a dated `given` reading**, so each is a title only inside a title phrase. `Prince Fielder`, `Mr Prince Smith` and `Smith, Princess` read the given name; a royal written with the bare title does too (`Prince Harry`), the class accepted for `king`, `sultan` and `shaikh`, accepted by Derek on 2026-10-27. `His Royal Highness Prince Harry` and `Prince of Wales` keep the title. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `shaikh` (Derek) — THE SEVENTH TITLE BATCH: `shaikh` has a dated `given` reading**, so the style is a title only inside a title phrase. `Shaikh Ahmed Khan` and `Mr Shaikh Ahmed` read given 'Shaikh'; a ruler written with the bare title does too (`Shaikh Hamad bin Isa`), the class accepted for `king` and `sultan`, the larger share of rulers Derek accepted on 2026-10-27. `sheikh` keeps its own batch. The measurement is decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `sultana` and `sultan` (Derek) — THE FIFTH AND SIXTH TITLE BATCHES, IN ONE PR: both have dated `given` readings**, `sultan`'s on its tr.toml entry, so each is a title only inside a title phrase. `Sultana Begum` and `Sultan Khan` read the given name; a sultan written with the bare title does too (`Sultan Hassanal Bolkiah`), the cost accepted for `king`. Derek combined the two batches on 2026-10-26. The measurement is decisions/H9's entry of this date.