diff --git a/AGENTS.md b/AGENTS.md index 1748e5d6..a4aeca56 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -197,7 +197,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2 and re-measured 2026-10-25 with the key below, the earlier `dphil` example having become a shipped mask in #797: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. - `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute) -- `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` +- `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a word reading on a word with a name reading, `fils` and the agnomens, gated by the count, rules/S11, since 2026-10-25) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set with no v1 `Constants` attribute of its own, so the only v1 knob that reaches it is deleting the conjunction @@ -244,7 +244,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). -- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. +- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), `suffix_words_ambiguous = SUFFIX_WORDS_AMBIGUOUS ∩ suffix_words` with `agnomens` inside it (#752, likewise: deleting the suffix word turns S11's gate off, `test_snapshot_removing_a_suffix_word_turns_its_gate_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. - **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard. @@ -270,11 +270,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential or particle reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential or particle reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, and a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous` or `after_given_titles`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. diff --git a/docs/customize.rst b/docs/customize.rst index 270e3e68..ca0c92e9 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -105,13 +105,17 @@ Removing works the same way, and drops the word from recognition: A few fields mark a subset of another — ``given_name_titles`` and ``after_given_titles`` over ``titles``, ``particles_ambiguous`` over ``particles``, ``suffix_acronyms_ambiguous`` over ``suffix_acronyms``, -``conjunctions_ambiguous`` over ``conjunctions``, and -``honorific_tails`` over ``suffix_words``. Entries belong in the base -field too, so add to both and remove from the marker first. Three of -them enforce that — ``particles_ambiguous``, ``suffix_acronyms_ambiguous`` -and ``honorific_tails`` raise ``ValueError`` naming the orphans, because -an orphan in each of those does real harm rather than nothing. The -other three are deliberately unchecked because an orphan there is inert: +``conjunctions_ambiguous`` over ``conjunctions``, +``honorific_tails`` and ``suffix_words_ambiguous`` over +``suffix_words``, and ``agnomens`` over ``suffix_words_ambiguous``. +Entries belong in the base field too, so add to both and remove from +the marker first. Four of them enforce that — ``particles_ambiguous``, +``suffix_acronyms_ambiguous``, ``honorific_tails`` and +``suffix_words_ambiguous`` raise ``ValueError`` naming the orphans, +because an orphan in each of those does real harm rather than nothing. +The other four are deliberately unchecked because an orphan there is +inert, an ``agnomens`` entry being read only for a word +``suffix_words_ambiguous`` marks: ``given_name_titles`` matches a title run as one space-joined string, or by that run's last word, so a legitimate entry like ``"sir and dame"`` is no single word in ``titles``; and a ``conjunctions_ambiguous`` or @@ -303,12 +307,13 @@ Words that are also ordinary names Some vocabulary words are also ordinary name words: an acronym suffix that is also borne as a surname, a particle that doubles as a given -name, a connective letter that doubles as an initial. Three fields mark -them — ``suffix_acronyms_ambiguous``, ``particles_ambiguous`` and -``conjunctions_ambiguous``, one each for ``suffix_acronyms``, -``particles`` and ``conjunctions``. They add no vocabulary by -themselves; they narrow how an existing entry is read when it appears -alone, and the parse reports the fork as an ambiguity. +name, a connective letter that doubles as an initial. Four fields mark +them — ``suffix_acronyms_ambiguous``, ``suffix_words_ambiguous``, +``particles_ambiguous`` and ``conjunctions_ambiguous``, one each for +``suffix_acronyms``, ``suffix_words``, ``particles`` and +``conjunctions``. They add no vocabulary by themselves; they narrow how +an existing entry is read when it appears alone, and the parse reports +the fork as an ambiguity. Credentials that are also surnames ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -348,6 +353,39 @@ The full reading of a bare marked acronym, in order: - **In brackets**, ``"John Smith (BA)"`` falls through to nickname parsing. +Generation words that are also surnames +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``suffix_words_ambiguous`` does the same for a suffix word: French +``fils`` and the Brazilian agnomens ``Filho``, ``Neto`` and +``Sobrinho`` mark a generation, and each is a surname too. An agnomen, +written in the case of the name around it and listed in ``agnomens``, +is a suffix only where the name keeps a word for its family. The +writing of ``fils`` speaks as well: in a mixed-case name a Title-case +``Fils`` is the family name even with a word to spare: + +.. doctest:: + + >>> parse("Pedro Alves Neto").suffix + 'Neto' + >>> parse("Ana Neto").family + 'Neto' + >>> parse("Marie Fils").family + 'Fils' + +A caller whose data writes an agnomen as part of the surname takes it +out by removing the word from ``suffix_words`` and from the two +markers over it, together: an orphan in ``suffix_words_ambiguous`` +raises ``ValueError``, and one left in ``agnomens`` is simply never read: + +.. doctest:: + + >>> agnomen = {"neto"} + >>> kept = Lexicon.default().remove( + ... agnomens=agnomen, suffix_words_ambiguous=agnomen, suffix_words=agnomen) + >>> Parser(lexicon=kept).parse("Pedro Alves Neto").family + 'Neto' + Mark a word, or leave it out ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index f73a0000..2a7fb11d 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -55,6 +55,7 @@ Entry conventions: - [S8 — the credential's trailing run](S8-credentials-trailing-run.md) - [S9 — unlisted words that join the ambiguous class by shape](S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md) - [S10 — the trailing roman numeral](S10-trailing-roman-numeral.md) +- [S11 — a suffix word that is also a name](S11-suffix-word-that-is-also-a-name.md) ## Nicknames & quoted names (N) diff --git a/docs/design/decisions/S11-suffix-word-that-is-also-a-name.md b/docs/design/decisions/S11-suffix-word-that-is-also-a-name.md new file mode 100644 index 00000000..b751a291 --- /dev/null +++ b/docs/design/decisions/S11-suffix-word-that-is-also-a-name.md @@ -0,0 +1,23 @@ +# S11 — a suffix word that is also a name + +## Standing + +- A generation word with a `given` or `family` reading is gated as a bare ambiguous acronym is: the generation where the name has words to spare, the name otherwise, either reading reported (`Pedro Alves Neto` → suffix 'Neto', `Ana Neto` → family 'Neto') — 2026-10-25 #752. +- The Brazilian agnomens (`filho`, `filha`, `neto`, `neta`, `netto`, `sobrinho`, `sobrinha`) are generation readings of the `agnomen` kind, gated by the count alone, their writing never leaning; `fils` is a `generation` reading whose lower case in a mixed-case name is the generation and whose Title case leans to the name (`Marie Fils` → family 'Fils') — 2026-10-25 #752. +- A suffix word that is also a name makes no chunk claim inside a word with interior periods, which reads by its dotted shape as it did before the word was listed: `Pedro M.Filho` keeps family 'M.Filho', and `Pedro Alves M.Filho` reads suffix 'M.Filho' as on master — 2026-10-25 #752. + +- Only name words in front of a generation word that is also a name count toward its words to spare, and a suffix word there is no company for it: `Pedro Junior Filho` and `Pedro VI Filho` keep family 'Filho' — 2026-10-25 (Derek), #890. + +- The floor holds before a generation word's case: a lower-case `fils` in a mixed-case name is the generation only where the name keeps a family word, so `Dumas fils` and `Pedro Junior fils` read family 'fils' — 2026-10-25 (Derek), #890, THE FLOOR. + +Open: none in this file; #843 keeps the agnomens' abbreviations (`FH`, `FO`, `NT`, `SB`). + +## Log + +- 2026-10-25 #752 (its ACCEPTED suffix word in front of an agnomen superseded by 2026-10-25 (Derek), #890) — A GENERATION WORD THAT IS ALSO A NAME IS GATED BY THE COUNT, AND ITS KIND SAYS WHAT ITS WRITING MEANS. Carries out the generation half of the 2026-10-23 #752 entry in decisions/vocabulary-model (phase 2h), replacing decisions/generational-words' 2026-10-23 #843 decision to keep the agnomens family names. An agnomen is no part of the surname a family shares: it marks its bearer against a relative of the same name, as Júnior does, so it ends the name the way a generation does. But each one is a borne name as well, `Neto` a surname (US Census 1,645, Wikidata 276+) and a given name (Wikidata 12+), `Neta` a given name (US SSA 3,226 female births, mostly the Hebrew Neta), `Filho` and `Netto` surnames, counts from `tools/name_usage.py` on 2026-10-25 recorded on the words in `pt.toml` and `pt-BR.toml`, and a plain generation reading leaves a two-word name with no family name. So the reading is gated as an ambiguous acronym is, by the words to spare (rules/S4's count), through the same sites. Case is a cue for a generation word and none for an agnomen: French writes `Alexandre Dumas fils` with the generation in lower case and `Marie Fils` with her surname in Title case, where an agnomen is written in the case of the name around it (`Pedro Alves Neto`, `PEDRO ALVES NETO`). Capitals say nothing for either, a surname being often written in them. The kind carries the difference: an `agnomen` reading and a `generation` reading each beside a name reading derive `SUFFIX_WORDS_AMBIGUOUS`, and the agnomen kind `AGNOMENS` inside it. + WHY THE COUNT ALONE FOR THE AGNOMENS, MEASURED. The only spike run before the design was agreed carried S6's Title-case lean, which kept 98 mixed-case ctgov agnomens as names and moved nothing in PSC. The count-only reading was measured on origin/master 225f3d6f against that tree (`--base`, counts and shapes only; Python 3.11): PSC 23 records fixed, 20 broken, 8 changed, every one of them a name of three words or more, so no two-word name moves. The 23 fixed: the filer typed both surnames, the second the agnomen, in the surname box; the 20 broken: the filer typed only the agnomen as the surname, the surname before it in the middle box. Filers split on where the agnomen's companion goes, and the reading follows the larger half. Shown these figures, Derek decided on 2026-10-25: "keep the count-only agnomen gate". The committed corpora move 4 names: `Pedro Alves Filho` gains the agnomen as the suffix, `Marie Fils` gains a report, and CLDR's French sample and its family-first twin gain `fils` as the suffix (cldr_agreement: 4 fields newly agreeing, 1 still wrong but changed). tse-donors: 21,059 of 848,351 filed names move, all of them in capitals, 12,811 where the agnomen had been the whole family name and the word before it now is, 8,198 where the family held more, and 45 a report only; FEC 12, reports only; ctgov 110 entries, 4 a report only, most of the rest the agnomen joining the credentials after a comma (`Neto, MD`, as `Júnior, MD` reads). The spelling pairs: of the 405 tse filed/registry pairs that differ only by a trailing agnomen, master gives both spellings the same last family word in 1, the gate in all 405; over all 205,925 differing pairs, compared given and family word by word with accents folded and an initial matching its word, 604 go from apart to alike and 258 from alike to apart, against the lean spike's 604 and 304. The 258 are mostly pairs master read alike only because the agnomen was the family name in both spellings, and the 46 fewer are the ones the lean's case cue moved. + ACCEPTED. A word standing straight before the agnomen counts as a name word, as before an ambiguous acronym: an initial there is left as the family name (`Pedro J. Neto` → family 'J.', 568 tse names, the initial most often abbreviating the surname), a particle as a family name of its own (`Pedro de Neto` → family 'de', one tse name), and a given name that is a generation word (`PEDRO JUNIOR FILHO`) or dotted initials (`PEDRO C.M. FILHO`) join the agnomen in a suffix run that leaves no family name (5 tse names). A three-word name whose surname is an agnomen spelling loses it to the suffix, reported (`Ruth Ann Neta`). The particle chain's report on a member it takes into the name reaches an agnomen inside a name, `PEDRO DE JESUS NETO DE SOUSA` reading as before with a report, as `PEDRO DE JESUS MA DE SOUSA` does. + NOT DONE. The filed abbreviations of the agnomens read as the family name, where the registry spelling of the same donor now reads the agnomen as the suffix, so those pairs stay apart: `FH` 229, `NT` 119, `FO` 99, `SOBR` 19, `FILH` 16, `F` 15, `NET` 12, `SB` 11 pairs and fewer of others. Listing them is #843's, short capitals meeting initials and credentials. + Recompute: `uv run python tools/differential/snapshot.py --local tse-donors --base ` (and `ctgov-officials`, `fec-individuals`), `uv run python tools/differential/local_agreement.py psc-individuals --base `; the pair counts read tse-donors' records through `tools/differential/local_corpora.py` (`require("tse-donors")`, `read_records(..., text_only=True)`), keep the 205,925 whose filed `name` and `registry` differ, and parse both spellings in each tree, run from that tree's checkout; a pair is alike when its given names and its family names match word by word, each word folded (NFD, combining marks dropped, upper case, edge periods stripped) and a one-letter word matching any word it begins. The 405 are the pairs whose longer spelling, split on spaces in upper case, is the shorter one plus one trailing agnomen, compared on the family's last word. `local_corpora.py pairs tse-donors --agreement` reports in whole percents, which do not move for these counts. +- 2026-10-25 (Derek), #890 (its accepted lower-case `fils` superseded by 2026-10-25 (Derek), #890, THE FLOOR) — A SUFFIX WORD IN FRONT OF AN AGNOMEN IS NO WORD TO SPARE. SUPERSEDES: the ACCEPTED sentence of 2026-10-25 #752 on a generation word or dotted initials in front of an agnomen. Derek approved the fix before merge, on the language-packs session's finding: the entry above accepted that a generation word or credential straight before an agnomen joined it in the suffix, leaving no family name, as a cost of 5 tse names, but that count was of capitals only, and the shape is ordinary in mixed case (`Pedro Junior Filho`, `Pedro Júnior Neto`, `Pedro Jr. Filho`, each read with no family name). One word was counted twice: the count took `Junior` as the agnomen's word to spare and the peel then took `Junior` as a suffix; and an unambiguous suffix word in front made the agnomen a credential by rules/S7's company. A generation word that is also a name now counts only the name words in front of it, skipping what the trailing run takes there (a suffix word, a numeral of two or more letters read as the generation, which the review of the fix found still counted in `Pedro VI Filho`, a member of the ambiguous class its writing makes a suffix, `Pedro MA Filho`, and a bare member any of those carries in front of it, `Pedro Junior Ma Filho` and `Pedro Junior do Filho`, the second review's), and keeps no company, so `Pedro Junior Filho` reads given 'Pedro', middle 'Junior', family 'Filho' as master did, while `Pedro Alves Junior Filho` still reads suffix 'Junior Filho'; dotted initials in front read as master did too (`PEDRO C.M. FILHO` → middle 'C.M.', family 'FILHO'), and so does a credential (`Pedro PhD Neto` → family 'Neto'). The count asks a scan that stops at the first word the walk keeps, not the walk itself: asking the walk over the words in front was tried and is exponential on a run of agnomens, quadratic when memoized, and the scaling guard (`agnomens`, `carried_agnomens` in tests/v2/test_benchmark.py) refuses both. The lower-case `fils` of a mixed-case name is settled by its case and not the count, so `Pedro Junior fils` still reads suffix 'Junior fils' (rules/S11's Accepted). +- 2026-10-25 (Derek), #890, THE FLOOR — A NAME KEEPS ITS FAMILY WORD BEFORE A GENERATION WORD'S CASE IS READ. SUPERSEDES: the lower-case `fils` sentence of 2026-10-25 (Derek), #890, and the 2026-10-25 #752 entry's reading of `Dumas fils`. Derek: "apply the floor to fils too". In the gate the agreed design orders (decisions/vocabulary-model, 2026-10-23 #752), the floor, a name keeping a word to address, is the first step and the form cue the second, so case may settle a generation word against the name only where the floor already holds. A lower-case `fils` in a mixed-case name had been taken with no word to spare, as an acronym's capitals are, so `Dumas fils` read given 'Dumas' with no family name and `Pedro Junior fils` suffix 'Junior fils'. Both now read family 'fils', as master did; `Alexandre Dumas fils` and `Pedro Alves Junior fils` keep the suffix, `Marie Fils` her family name, and after a comma, where a family is already named, lower case still settles it (`Dumas, Alexandre fils` → suffix 'fils'). The same holds after a comma with no name before it, the review of the fix found: `, Dr. fils` and `(Pe), Dr. fils` had read suffix 'fils' with no family, the comma decision taking the lower case where no family is named, and read family 'fils' now, as master did; and its second review found the head's name-word count too generous for this, a joined salutation (`Mr. and Mrs., fils`) and a lone name word behind a given-name title (`Sir Alexandre, fils`) counting one where no family remains, so at the first slot after a comma a generation word's lower case adds nothing to rules/C4's count, which reads two name words before the comma first: `Dumas, fils` reads given 'fils', family 'Dumas', as master did, and `Alexandre Dumas, fils` suffix 'fils'. An acronym keeps its capitals' lean with no word to spare (rules/S6, `Jack MA`); an agnomen has no case to weigh, so no other gated word changes. Measured with `--base` against master 225f3d6f: no corpus name moves against the commit before. diff --git a/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md b/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md index 718db8f8..040cdac9 100644 --- a/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md +++ b/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md @@ -10,6 +10,8 @@ - `Freiherr von Berg MA` reads family 'von Berg', suffix 'MA', the lean ending P2's chain with no words to spare — 2026-09-15 (Derek). - The whole-PR review's five readings stand: an acronym the particle chain absorbs reports `suffix-or-name` (`John van der Berg Ma`); the CJK stage asks the post-comma run with the one-case fact; a by-shape credential in a tail segment raises no `comma-structure`; a word a caller lists as a surname is never caps-shaped; and a dotted listed member takes the lean — 2026-09-18, WHOLE-PR REVIEW ROUND. +Elsewhere: the case a generation word that is also a name reads (rules/S6's bullet on it) is decisions/S11's 2026-10-25 #752 entry. + Open: none in this file. ## Log diff --git a/docs/design/decisions/T5-written-shape-of-an-initial.md b/docs/design/decisions/T5-written-shape-of-an-initial.md index 6ae8140f..1df44051 100644 --- a/docs/design/decisions/T5-written-shape-of-an-initial.md +++ b/docs/design/decisions/T5-written-shape-of-an-initial.md @@ -4,6 +4,8 @@ - The written shape of an initial is its own rule, split from T4, which keeps the case contrast; the shape itself was decided in v1, narrowed to scripts that write initials by #320 (decisions/initials-repertoire) and asked of the composed spelling since #665 (decisions/nfd-shapes) — 2026-10-19. +Elsewhere: the one-letter generations recorded off by default (rules/T5's Accepted bullet on them) is decisions/vocabulary-model's 2026-10-25 #752, phase 2h entry. + Open: none in this file. ## Log diff --git a/docs/design/decisions/generational-words.md b/docs/design/decisions/generational-words.md index 9eaf3370..f050de94 100644 --- a/docs/design/decisions/generational-words.md +++ b/docs/design/decisions/generational-words.md @@ -10,14 +10,15 @@ Closes #661. 31 spelled-out generational words join SUFFIX_WORDS, and one predic - A refused word carries `excluded = { generation = … }` with its reason in the file of the language whose reading is refused (`uv run python tools/vocab/query.py --excluded generation`) — 2026-10-14 #661; 2026-10-14 #679. - The words ship spaced only, `二世` excepted, so a hyphen-glued Jr. is no suffix: `Теодор Рузвелт-младши` reads family 'Рузвелт-младши' — 2026-10-14 #661. - `二世` is a generation word that peels spaced and glued, the one exception to spaced-only shipping: `陳怡君二世` reads family '陳怡君', suffix '二世', and its `ja.toml` and `zh.toml` entries carry `issue = 671` and `attached` with `issue = 707` — 2026-10-21 #762. -- The generational words whose collisions are with OTHER languages written in the same script (fr `fils`, `père`; the Latin abbreviations `ml.` and `st.`) ship in no default, that being the case a locale pack exists for: `Alexandre Dumas fils` reads family 'fils' and `Jiří Dienstbier ml.` family 'ml.', and no fr or cs pack exists — 2026-10-14 #679. +- The generational words whose collisions are with OTHER languages written in the same script (fr `père`; the Latin abbreviations `ml.` and `st.`) ship in no default, that being the case a locale pack exists for: `Jiří Dienstbier ml.` reads family 'ml.', and no cs pack exists; cs `st.` is recorded as a reading off by default, for that pack — 2026-10-14 #679; 2026-10-25 #752. +- fr `fils` is a generation reading gated by its family reading, its case and the count (rules/S11): `Alexandre Dumas fils` reads suffix 'fils' and `Marie Fils` family 'Fils', both reported — 2026-10-25 #752. Elsewhere: the run start that lets any unambiguous suffix word of two or more letters begin the credential run, whatever its script (`Antonio Pavlović mlađi Jones` reads suffix 'mlađi Jones', as does `Pieter Bruegel stariji Jones` with 'stariji Jones'), is decisions/S8's 2026-10-14 (Derek), #679 entry, which retracted this file's #602 run-start bullet; the glued peel of 二世 is decisions/W2's 2026-10-17 #707 entry. - pt `júnior`, the accented spelling of `junior`, ships beside it with `same_as = "jr"` and inherits its vetting and its accepted collisions rather than being re-vetted under the two tests above (Junior is a borne surname, 2,146 US Census bearers, where `senior` was refused for 3,864): `José Silva Júnior` reads family 'Silva', suffix 'Júnior', as `José Silva Junior` does — 2026-10-23 (Derek), #843. -- The Brazilian agnomens `Filho`, `Neto`, `Netto`, `Sobrinho` and their feminines are not suffix vocabulary, and nothing refuses them either: `Pedro Alves Filho` reads family 'Filho', middle 'Alves', as 1.4.0 and 2.5.0 did — 2026-10-23 (Derek), #843. +- The Brazilian agnomens `Filho`, `Neto`, `Netto`, `Sobrinho` and their feminines are generation readings of the `agnomen` kind, gated by the count alone (rules/S11): `Pedro Alves Filho` reads family 'Alves', suffix 'Filho', and `Ana Neto` keeps family 'Neto', both reported — 2026-10-25 #752. -Open: #684 (locale packs for `Alexandre Dumas fils` and `Jiří Dienstbier ml.`, the generational words that collide outside their language); #843 (whether the Brazilian agnomens read as suffixes, as a family name that takes the word before it, or stay the family name, with their abbreviations FH, FO, NT and SB). +Open: #684 (locale packs for `Jiří Dienstbier ml.` and `st.`, the generational words that collide outside their language); #843 (the agnomens' abbreviations FH, FO, NT and SB). ## Log @@ -35,3 +36,5 @@ Open: #684 (locale packs for `Alexandre Dumas fils` and `Jiří Dienstbier ml.`, SUPERSEDES: 2026-10-14 #679's refusal of zh 二世 only; its other refusals stand. - **2026-10-23 #843 (Derek) — JÚNIOR SHIPS; THE AGNOMENS WAIT.** "Suffix, Júnior only for now." `júnior` joins `pt.toml` as a generation reading with `same_as = "jr"`: the standard Portuguese spelling, which `José Silva Júnior` read as family 'Júnior', middle 'Silva' in 1.4.0 and 2.5.0 (checked against both wheels) while `José Silva Junior` read suffix 'Junior'. Its collisions are `junior`'s, already accepted: a given name (Wikidata 37+ as Júnior, 43,182+ US SSA births as Junior), where the suffix claim does not act (`Júnior Silva` keeps given 'Júnior'), and family bearers (Wikidata 49+, US Census 2,146 as JUNIOR), which a suffix word costs its surname wherever it ends a name (`Pedro Júnior` reads given 'Pedro', suffix 'Júnior', as `Pedro Junior` does), counts from `tools/name_usage.py` on 2026-10-23. In the local corpus `tse-donors` (fetched 2026-10-22), 1,071 of its 854,531 records write it anywhere in the name (`tools/name_usage.py`), 991 distinct filed names end in it, and 983 records end in it, in either case, where the tax registry's name for the same donor ends `JUNIOR` (last words compared after `str.upper()`). The accented spelling reads as the unaccented one in all 120 pairs of a grid of 20 shapes (comma shapes, an empty head, a title, a particle, a trailing title, a nickname, a maiden clause, a credential and a name word behind it, both one-case spellings) under all three orders, composed and decomposed, comparing fields, ambiguity kinds and `initials()`; with `júnior` removed from the lexicon (`Lexicon.default().remove(suffix_words={'júnior'})`), 108 of the 120 pairs differ, and the 12 that do not hold Júnior where no suffix reading reaches it (`Júnior Silva`, and `Júnior, José`, before a family comma). Measured 2026-10-23 on `tse-donors` against the tree before this change (`snapshot.py --local tse-donors`, baseline recorded with `--tree`): 993 of 848,351 filed names move, 990 ending in Júnior and gaining it as the suffix out of the family name (3 of them two-word names, `RONALDO JÚNIOR`, left with no family as `RONALDO JUNIOR` is), and 3 with Júnior inside the name, where it starts a credential run (decisions/S8) that takes the surnames behind it, `JORGE ANTÔNIO JÚNIOR BARBOSA SILVA` reading family 'ANTÔNIO', suffix 'JÚNIOR BARBOSA SILVA' as the JUNIOR spelling already did; and `local_corpora.py pairs tse-donors --agreement` stops listing JÚNIOR / JUNIOR (866 accents-only pairs read with two surnames before, none after). The agnomens (distinct filed names ending in them: `Filho` 10,979, `Neto` 8,439, `Sobrinho` 955, `Netto` 432) stay family names for now. They are legally part of the surname, and `Neto` is a borne surname and given name alone (Wikidata 276+ family, 12+ given), which a suffix reading would leave with no family, as `Pedro Junior` has none. No exclusion records them, because nothing was refused: the question is open on #843. + +- **2026-10-25 #752 — THE AGNOMENS AND `fils` ARE GATED GENERATION READINGS (phase 2h).** SUPERSEDES: 2026-10-23 #843's agnomen half, which kept the agnomens family names, and 2026-10-14 #679's refusal of `fils` as pack-shaped. The 2026-10-23 #752 entry in decisions/vocabulary-model gave each generation reading a gate by its kind, and the record, with its measurements, is decisions/S11. `fils` collides inside French, a generation word and a French surname, so no French pack could settle it, which is the premise of #684 the design reversed: its `generation` reading now stands beside its `family` reading, and the gate reads the name around it. The agnomens are not part of the surname: an agnomen marks its bearer against a relative of the same name, as Júnior does, and its name readings (dated counts in `pt.toml` and `pt-BR.toml`, `netto` there being the spelling of Brazil's older orthography) keep a two-word name's family. cs `st.` keeps no exclusion: it is a `generation` reading written `default = false`, recorded for a Czech pack, since nothing in its writing tells it from Saint; hr, sk and sl keep theirs, as do `ml.` and `père`. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 9f6cf884..68b64625 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -22,11 +22,11 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. - The marker sets are explicit properties on readings (`may_be_initial`, `attached`, `addresses`, `after_given`, `cased`), and `after_given` stands on a salutation reading only (decisions/H8) — 2026-10-18 #737; 2026-10-24 #835. -- The two markers that record a clash with a name are derived from name readings: an acronym suffix reading beside a `given` or `family` reading is an ambiguous acronym, a particle reading beside a `given` reading a particle that may be given, wherever in the files each reading sits. The build refuses `may_be_name` and `may_be_given = true`, naming the reading to write; a particle whose word has no `given` reading still states `may_be_given = false`, never defaulted; a name reading may stand without a count, with a `note` saying why; and a count kept as evidence against marking such a word goes in the clashing reading's `note` (`ld`). `may_be_initial` is not derived — 2026-10-25 #752, A CLASH WITH A NAME IS A NAME READING. +- The two markers that record a clash with a name are derived from name readings: an acronym suffix reading beside a `given` or `family` reading is an ambiguous acronym, a particle reading beside a `given` reading a particle that may be given, wherever in the files each reading sits. The build refuses `may_be_name` and `may_be_given = true`, naming the reading to write; a particle whose word has no `given` reading still states `may_be_given = false`, never defaulted; a name reading may stand without a count, with a `note` saying why; and a count kept as evidence against marking such a word goes in the clashing reading's `note` (`ld`). `may_be_initial` is not derived — 2026-10-25 #752, A CLASH WITH A NAME IS A NAME READING. A third marker is derived the same way: a suffix word reading beside a `given` or `family` reading is a suffix word that may be a name (`SUFFIX_WORDS_AMBIGUOUS`), and an `agnomen` reading needs a name reading on its word — 2026-10-25 #752, phase 2h. - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. -- The names later phases will use (`default`, `prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. +- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `attaches` is phase 2's and `bare_only`, `leading_only`, `trailing_only` and `separators` phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. - Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762. - The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749. - The titles take ten kinds beside the generic one, `academic` and `occupation` divided by function rather than subject, and the title keep-outs are exclusions — 2026-10-19 #750, title kinds; 2026-10-19 #750, the rest of the titles. @@ -197,3 +197,9 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **Recompute:** `uv run python tools/differential/snapshot.py` on the parent of this change's merge with its code, and for each local corpus `snapshot.py --local --record --tree ` then `snapshot.py --local ` from this tree, back to back; `uv run python tools/differential/local_agreement.py psc-individuals` the same way, `--list N` printing the moved records locally. - **2026-10-25 #752 — phase 2c: a credential joined by a hyphen or a slash reads as one where every chunk counts as a suffix as written**, carrying out the listed half of item 5 of the 2026-10-23 #752 entry. `John Smith, MD/PhD` reads suffix 'MD/PhD' where it read given 'MD/PhD', family 'John Smith'; a bare member of the ambiguous class is no chunk the test counts (`John Smith, MA/PhD` keeps its reading). The record, with what was measured and what was left unread, is decisions/S3's entry of this date (rules/S3). - **2026-10-25 #752 — phase 2d: an unlisted word of capitals joined by a hyphen or a slash is the caps shape**, carrying out the unlisted half of item 5 of the 2026-10-23 #752 entry (#797's half A). `John Smith, PU-PH` reads suffix 'PU-PH', reported, where it read given 'PU-PH', family 'John Smith'; C6's case contrast gates it after a comma as it gates the unjoined word. The record is decisions/S9's entry of this date (rules/S9, rules/C6). + +- **2026-10-25 #752, phase 2h — GENERATION READINGS ARE GATED BY KIND, AND `default = false` HAS ITS MEANING.** Carries out item 9 of the 2026-10-23 #752 entry above and its reserved-names item. SUPERSEDES: in 2026-10-18 #737, `default` as a reserved name. Three changes, each one the design's: + - A suffix word reading beside a name reading derives `SUFFIX_WORDS_AMBIGUOUS` (`Lexicon.suffix_words_ambiguous`), gated as an ambiguous acronym is, by the words to spare, at the same sites; the kind says what the writing means. A new kind, `agnomen`, records the Brazilian agnomens (`filho`, `filha`, `neto`, `neta`, `sobrinho`, `sobrinha` in `pt.toml`, `netto` in `pt-BR.toml`), whose writing never leans and which `AGNOMENS` (`Lexicon.agnomens`) lists; build check 11 refuses an agnomen with no name reading, since one would take the family name from every two-word name it ends. `fils` takes a `generation` reading beside its family reading, its exclusion gone. No shipped suffix word had a name reading before, so nothing else moves. The rule is rules/S11 and the record, with what was measured and what was accepted, decisions/S11. + - `default = false` is a behavioral property of every category but `name`: the reading is held to the per-entry checks and compiles into no set and no marker, and checks 4, 5, 6 and 9 do not weigh it, so it may stand beside a live reading of the word in another file and beside another language's exclusion of it. `default = true` is refused, being every reading's unwritten value. The first three are the readings with no cue: Spanish `h` (*hijo*) and the German regnal ordinal on `i`, each written with the period that makes a one-letter word an initial (rules/T5), and Czech `st` (*starší*), whose `cs.toml` exclusion it replaces. No reading moves. + - The other reserved names keep their reservation and name the phase that needs them: `attaches` phase 2 (its front-of-the-name work), `prefer` and `strict` phase 3, and `bare_only`, `leading_only`, `trailing_only` and `separators` phase 4, the gate's slots carrying position and shape settling a bare letter (rules/P7). + Adding the two fields changes `Lexicon`'s pickle layout, as 2.6 did. diff --git a/docs/design/rules/C12-suffix-word-ending-the-name-before-the-comma.md b/docs/design/rules/C12-suffix-word-ending-the-name-before-the-comma.md index 785d9aee..4e40e5d3 100644 --- a/docs/design/rules/C12-suffix-word-ending-the-name-before-the-comma.md +++ b/docs/design/rules/C12-suffix-word-ending-the-name-before-the-comma.md @@ -41,6 +41,11 @@ **Except:** +- **A generation word that is also a name** (S11) is no evidence here, since it may be the surname the family comma names: the part before the comma ends in a name word. + ``` + "Silva Neto, Pedro" → family="Silva Neto" + "Pedro Alves Neto, AD" → given="AD" · boundary + ``` - **A word of it written as a name, or an initial, keeps the listing form.** ``` "Lloret Palol II, Josep Antoni" → given="Josep" · boundary diff --git a/docs/design/rules/C4-ambiguous-credential-class-at-the-comma.md b/docs/design/rules/C4-ambiguous-credential-class-at-the-comma.md index c6eae763..4aa88e16 100644 --- a/docs/design/rules/C4-ambiguous-credential-class-at-the-comma.md +++ b/docs/design/rules/C4-ambiguous-credential-class-at-the-comma.md @@ -4,6 +4,11 @@ **Then:** the count before the comma is of NAME words rather than of words, and two or more of them read the part after the comma as the credential run, whatever case the name is written in. +- **A generation word that is also a name (S11) is a word of this class here**, its count read the same way. + ``` + "Pedro Alves, Neto" → suffix="Neto" + "Silva, Neto" → given="Neto" · boundary + ``` - **The count goes FIRST and written case is asked only after it**: where two name words stand before the comma the part after it is the credential run however that part is written, and only where the count leaves the word a name — one name word before the comma — is the case read, capitals in a mixed-case name making it the credential there too (S6). ``` "John Smith, MA" → suffix="MA" diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 6b72b715..dc539456 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -148,6 +148,7 @@ Terms the rules use without defining them each time, each with the rule that def - [S8 — the credential's trailing run](S8-credentials-trailing-run.md) - [S9 — unlisted words that join the ambiguous class by shape](S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md) - [S10 — the trailing roman numeral](S10-trailing-roman-numeral.md) +- [S11 — a suffix word that is also a name](S11-suffix-word-that-is-also-a-name.md) ## [Nicknames & quoted names (N)](N-nicknames-and-quoted-names.md) diff --git a/docs/design/rules/S11-suffix-word-that-is-also-a-name.md b/docs/design/rules/S11-suffix-word-that-is-also-a-name.md new file mode 100644 index 00000000..58eaf02d --- /dev/null +++ b/docs/design/rules/S11-suffix-word-that-is-also-a-name.md @@ -0,0 +1,69 @@ +# S11 — a suffix word that is also a name + +**When:** a generation word that is also borne as a name stands where a suffix may stand, as in "Pedro Alves Neto" or "Alexandre Dumas fils". + +**Then:** it reads as the generation only where the name has words to spare, as a bare ambiguous acronym does (S4), and at the slots that report, either reading carries the ambiguity flag. + +- **The writing is read by S6**, and for a generation word such as French `fils` it speaks at the other end of the case: lower case in a name of mixed case is the generation, Title case leans to the name, capitals say nothing. The floor holds first: lower case settles it only where the name keeps a family word, so two name words keep the family, and at the first slot after a comma the count of name words before it decides alone (C4). An agnomen, the Portuguese `Filho`, `Neto`, `Sobrinho` and their feminines, is written in the case of the name around it, so its writing never speaks and the count decides alone. +- **Only name words in front of it count**: a word there that the trailing run takes, a suffix word, a roman numeral read as the generation (S10), a member of the ambiguous class its writing makes a suffix (S6) or one such a word carries in front of it (S7), is no word to spare, and a suffix word there is no company either, as it is for an acronym (S7). +- **After a comma** the word is read as a member of the ambiguous class is: by the count of the name words before the comma at the first slot after it (C4), and by its writing alone where it ends the given part after a family comma (S5). + +**Because:** an agnomen marks its bearer against a relative of the same name, as Junior does, so it ends the name the way a generation does and is not part of the surname the family shares. But `Neto`, `Fils` and `Neta` are borne as names too, and a name of two words has no word to spare for a generation. Case tells a French surname from its generation word, and it cannot tell an agnomen, which takes the case of whatever it is written beside. + +``` +"Pedro Alves Neto" → family="Alves" +"Pedro Alves Neto" → suffix="Neto" +"Pedro Alves Neto" → ambiguities=("suffix-or-name",) +"PEDRO DA SILVA FILHO" → suffix="FILHO" +"Pedro Alves Neto, PhD" → suffix="Neto, PhD" +"Pedro Alves Neto Júnior" → suffix="Neto Júnior" +"Alexandre Dumas fils" → suffix="fils" +"Alves, Pedro Neto" → suffix="Neto" +"Ana Neto" → family="Neto" · boundary +"Ana Neto" → ambiguities=("suffix-or-name",) · boundary +"PEDRO FILHO" → family="FILHO" · boundary +"Silva, Neto" → given="Neto" · boundary +"Marie Fils" → family="Fils" · boundary +"Pedro Junior Filho" → family="Filho" · boundary +", Dr. fils" → family="fils" · boundary +"Dumas, fils" → given="fils" · boundary +"PEDRO JUNIOR FILHO" → family="FILHO" · boundary +"Pedro VI Filho" → family="Filho" · boundary +"Pedro MA Filho" → family="Filho" · boundary +"Pedro Junior Ma Filho" → family="Filho" · boundary +"Pedro Alves Junior Filho" → suffix="Junior Filho" +``` + +**Except:** + +- **Glued behind an initial** with a period between, the word is part of that one word and is weighed only by the shape the joined word wears (S3), as it was before the word was listed: the name word it is with no word to spare, the dotted credential shape with words to spare. + ``` + "Pedro M.Filho" → family="M.Filho" + "Pedro Alves M.Filho" → suffix="M.Filho" + ``` + +**Precedence:** + +- **Written case in a mixed-case name** ("Marie Fils", "Dumas fils"): a generation word's Title case outranks the count, as an acronym's does (S6), while its lower case gives way to the floor, a name keeping its family word; an agnomen has no case to weigh. + ``` + "Jean Marie Fils" → family="Fils" + "Dumas fils" → family="fils" + "Pedro Junior fils" → family="fils" + "Pedro Alves NETO" → suffix="NETO" + ``` + +**Accepted:** + +- A name word standing straight before it counts as a name word whatever it is, so an initial there is left as the family name and a particle there as a family name of its own, as with an ambiguous acronym. An initial before an agnomen most often abbreviates the surname it follows. + ``` + "Pedro J. Neto" → family="J." + "Pedro de Neto" → family="de" + ``` +- A three-word name whose last word is a surname that is also an agnomen loses that surname to the suffix, reported, as `Ruth Ann Neta` does. A register whose filer typed the agnomen alone in the surname box disagrees with the reading, and one whose filer typed it after the surname before it agrees. + ``` + "Ruth Ann Neta" → suffix="Neta" + ``` + +--- +**Signal:** count: name words to spare beside a generation word that is also a name +**history:** decisions/S11 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md b/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md index a23d8642..5a0342b5 100644 --- a/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md +++ b/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md @@ -9,6 +9,7 @@ - **A word joining that class by its shape** is read by S9. - **A credential with more name text behind it** is read by S8. - **A roman numeral** is read by S10. +- **A generation word that is also a name** (`fils`, the agnomen `Neto`) is read by S11. **Because:** generational suffixes and credentials are recognized by vocabulary, and are written after the name they belong to. @@ -17,13 +18,15 @@ "Antonio Pavlović mlađi" → family="Pavlović" "Antonio Pavlović mlađi" → suffix="mlađi" "José Silva Júnior" → suffix="Júnior" -"Pedro Alves Filho" → family="Filho" · boundary +"Pedro Alves Filho" → suffix="Filho" +"Pedro Filho" → family="Filho" · boundary "Агриппина Младшая" → suffix="Младшая" "جورج بوش الابن" → suffix="الابن" "ロバート・ダウニー・ジュニア" → suffix="ジュニア" "佐藤恵子 国会議員" → suffix="国会議員" "陳子媚 二世" → suffix="二世" "Питер Брейгель мл." → suffix="мл." +"Alexandre Dumas fils" → suffix="fils" "Marie Fils" → family="Fils" · boundary "Jiří Dienstbier ml." → family="ml." · boundary "Pavlović, Antonio mlađi" → suffix="mlađi" @@ -64,5 +67,5 @@ ``` --- -**Signal:** position: a suffix word ending the name, never opening the string; hub of S3, S4, S5, S6, S7, S8, S9, S10 +**Signal:** position: a suffix word ending the name, never opening the string; hub of S3, S4, S5, S6, S7, S8, S9, S10, S11 **history:** decisions/S2 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py diff --git a/docs/design/rules/S5-given-parts-trailing-slot-after-a-family-comma.md b/docs/design/rules/S5-given-parts-trailing-slot-after-a-family-comma.md index a7e11465..7e39fc63 100644 --- a/docs/design/rules/S5-given-parts-trailing-slot-after-a-family-comma.md +++ b/docs/design/rules/S5-given-parts-trailing-slot-after-a-family-comma.md @@ -4,6 +4,11 @@ **Then:** after a family comma, a member of the ambiguous class (S4) ending the GIVEN part is read as the comma-less spelling reads a word ending the name, and the count is not what decides it there. +- **A generation word that is also a name (S11) is a member here too**: an agnomen's writing never keeps it a name, and a generation word's Title case does, as S6 reads it. + ``` + "Alves, Pedro Neto" → suffix="Neto" + "Alves, Pedro Fils" → middle="Fils" · boundary + ``` - **In the given part, the writing decides**: the comma has already named the family and the first name word after it is the given name, so the words to spare are there by construction and the count says nothing; a word that ENDS that part reads as the credential unless its WRITING says otherwise, and a word that does not end it is never asked. ``` "Doe, John MA" → suffix="MA" diff --git a/docs/design/rules/S6-case-signal-at-the-suffix-slot.md b/docs/design/rules/S6-case-signal-at-the-suffix-slot.md index 5e02befa..16245860 100644 --- a/docs/design/rules/S6-case-signal-at-the-suffix-slot.md +++ b/docs/design/rules/S6-case-signal-at-the-suffix-slot.md @@ -9,6 +9,15 @@ "JACK MA" → family="MA" · boundary ``` - **At the trailing slot of the given part** after a family comma, the comma has already settled the count, so the writing is the only evidence there is (S5). +- **A generation word that is also a name (S11) is written the other way at one end**: its lower case in a name of mixed case is the generation wherever the name keeps a family word, never with none to spare (S11), its capitals say nothing, a surname being often written in them, and its Title case leans to the name as an acronym's does. An agnomen's writing says nothing at all, since it takes the case of the name around it. + ``` + "Dumas, Alexandre fils" → suffix="fils" + "Dumas fils" → family="fils" · boundary + "Marie FILS" → family="FILS" + "Jean Marie Fils" → family="Fils" + "Dumas, Alexandre Fils" → middle="Fils" + "Pedro Alves Neto" → suffix="Neto" + ``` **Because:** in a name written in more than one case, the writer's capitals are evidence the count does not have: credentials are written in capitals and names are not. A name written in one case carries no such evidence. @@ -30,4 +39,4 @@ --- **Signal:** case contrast: capitals on a word of the ambiguous class in a name written in more than one case -**history:** decisions/S6 · **implemented:** nameparser/_pipeline/_vocab.py +**history:** decisions/S6 · **implemented:** nameparser/_pipeline/_vocab.py, nameparser/_render.py diff --git a/docs/design/rules/S7-company-of-an-unambiguous-credential.md b/docs/design/rules/S7-company-of-an-unambiguous-credential.md index 508fd628..0d36e731 100644 --- a/docs/design/rules/S7-company-of-an-unambiguous-credential.md +++ b/docs/design/rules/S7-company-of-an-unambiguous-credential.md @@ -28,6 +28,11 @@ **Except:** +- **A generation word that is also a name (S11) keeps no company**: a suffix word in front of it is no evidence, and its own count, which looks past that word, decides. + ``` + "Pedro PhD Neto" → family="Neto" · boundary + "Pedro Junior Filho" → family="Filho" · boundary + ``` - **A word of both the title and the suffix vocabulary standing in the given part's leading title run** is a title there and speaks for nothing, and no credential behind it speaks for a word of this class in that part: the title reading makes the next word the given name, and from there the part reads as any given part does, the given part's own company included. A part holding no such word, or only words whose capitals decide them, reads as it did. Anywhere else such a word speaks like any suffix word. ``` "Smith, Ms Ma" → given="Ma" · boundary diff --git a/docs/design/rules/T5-written-shape-of-an-initial.md b/docs/design/rules/T5-written-shape-of-an-initial.md index 29415e2a..8f0d1283 100644 --- a/docs/design/rules/T5-written-shape-of-an-initial.md +++ b/docs/design/rules/T5-written-shape-of-an-initial.md @@ -25,6 +25,11 @@ ``` "1. née Smith" → given="1." ``` +- A generation written as one letter and its period, the Spanish H. for hijo or a German regnal I., is that initial too, since nothing in its writing tells the two apart: the vocabulary records both readings off by default, for a language pack to turn on. + ``` + "Juan Pérez H." → family="H." + "Wilhelm I." → family="I." + ``` --- **Signal:** shape: a one-letter word's period, or a bare unaccented Latin capital diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index fd888d7a..0285c867 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -50,7 +50,7 @@ Phase 1 re-encoded the vocabulary and changed no parse. One TOML file per BCP 47 Two choices in phase 1 set up phase 2 and are worth knowing before reading it: - **The marker sets were explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings, recording past curation rather than evidence. Phase 2a (2026-10-25) derived the two that record a clash with a name, `may_be_name` and `may_be_given`, from name readings, each with dated counts or a `note` where no source counts the word (`te`, `tho` and `du`, once thought to have none, have SSA counts), keeping `may_be_initial` as a property (decisions/vocabulary-model). -- **Names a later phase will use are build errors now** (`default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. +- **Names a later phase will use are build errors now** (`default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. Phase 2h (2026-10-25) gave `default = false` its meaning, a reading recorded that compiles to nothing, and named the phase each of the others waits for. The reference for the format is nameparser/vocab/README.md; the decisions behind it, with what was declined and measured, are decisions/vocabulary-model's 2026-10-18 entry. @@ -110,7 +110,7 @@ A reading whose writing offers no cue is one the gate could only guess, so it is **The numerals ask one predicate.** `i` to `v` are vocabulary and `vi` to `x` are shape, and until 2f some rules asked only the vocabulary, so `John Smith III MD` and `John Smith VI MD` read differently (#784, #774). One predicate answers for every spelling the way the trailing numeral rule does (shape, the count for one letter, an initial in front, the period, the case), and every rule that asks either recognizer asks it. Listing `vi` to `x` instead was measured and declined: a middle initial `X` would read as a generation. -**`fils` and the agnomens show what kind-specific cues buy.** `fils` is gated by its case and the count, so `Alexandre Dumas fils` is to read suffix `fils` while `Marie Fils` keeps her family name, reported. The Brazilian agnomens are to be gated by the count alone: case would say nothing, and the plain generation reading breaks real surnames (`Ana Neto`) and given names. +**`fils` and the agnomens show what kind-specific cues buy.** `fils` is gated by its case and the count, so `Alexandre Dumas fils` reads suffix `fils` while `Marie Fils` keeps her family name, reported. The Brazilian agnomens are gated by the count alone: case would say nothing, and the plain generation reading breaks real surnames (`Ana Neto`) and given names. Phase 2h landed both on 2026-10-25, the floor holding before the case cue (`Dumas fils` keeps its family name, Derek) ([rules/S11](rules/S11-suffix-word-that-is-also-a-name.md), a suffix word that is also a name), with `default = false` for the three readings with no cue. **A credential's spelling is a form cue.** A credential joined by `-` or `/` is to read as one when every chunk is listed and none is ambiguous, since a hyphen also joins names (`Smith-Jones`); an unlisted word of capitals joined that way is to join the ambiguous class after a comma, gated by the name's case contrast. `John Smith, MD/PhD` read given `MD/PhD` until phase 2c (2026-10-25) read the listed half: a joined word whose every chunk counts as a suffix as written ([rules/S3](rules/S3-credentials-written-in-joined-chunks.md)). A credential split across a space (`Pharm. D.`, generalizing the `Ph. D.` merge) is to merge only where its first chunk has two or more letters, the merged piece a member of the ambiguous class, decided by the comma count and words to spare: an opening single letter is a middle initial (`SMITH, JOHN A. MS.` holds an initial and a salutation, not a credential `AMS`). #872 made the same judgment for the joined spelling (`A.B.`), in scripts with capitals and outside the slot right after a comma. The merged piece is also to start the credential's run (#827), once the numeral predicate answers the numeral before a credential; since [rules/S8](rules/S8-credentials-trailing-run.md) refuses a member of the ambiguous class as a run start, how the two combine is the split-credential PR's to state (today's `Ph. D.` merge is unambiguous, and S8 already covers it). diff --git a/docs/locales.rst b/docs/locales.rst index c4989248..33534b7e 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -309,12 +309,12 @@ The ``policy`` half is a patch, but the ``lexicon`` half is not. A pack's :class:`~nameparser.Lexicon` is a complete value in its own right and is validated on its own, before it is unioned onto the base — so a fragment listing a word in ``particles_ambiguous``, -``suffix_acronyms_ambiguous`` or ``honorific_tails`` must also list it -in the set that field marks (``particles``, ``suffix_acronyms``, -``suffix_words``). A fragment holding ``particles_ambiguous={"van"}`` +``suffix_acronyms_ambiguous``, ``honorific_tails`` or +``suffix_words_ambiguous`` must also list it in the set that field marks +(``particles``, ``suffix_acronyms``, ``suffix_words``). A fragment holding ``particles_ambiguous={"van"}`` without ``particles={"van"}`` raises ``ValueError`` before any union -happens. ``given_name_titles``, ``conjunctions_ambiguous`` and -``after_given_titles`` are not checked this way, and can mark a word only the base holds. ``zh`` is the +happens. ``given_name_titles``, ``conjunctions_ambiguous``, +``after_given_titles`` and ``agnomens`` are not checked this way, and can mark a word only the base holds. ``zh`` is the shipped worked example: its ``Lexicon(surnames=...)`` has to satisfy every ``Lexicon`` rule standing alone, before anything unions it onto the base. diff --git a/docs/migrate.rst b/docs/migrate.rst index e169c77b..aabce0b5 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -267,8 +267,8 @@ fields: Some ``Lexicon`` fields have no ``CONSTANTS`` attribute at all — ``conjunctions_ambiguous``, ``maiden_markers``, ``surnames``, -``honorific_tails`` and ``after_given_titles`` — so they are reached -through the 2.0 API only. +``honorific_tails``, ``after_given_titles``, ``suffix_words_ambiguous`` +and ``agnomens`` — so they are reached through the 2.0 API only. .. warning:: diff --git a/docs/release_log.rst b/docs/release_log.rst index 5f0823ab..e506e56c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -9,6 +9,8 @@ Release Log - **Fix roman numerals VI to X reading as names where III reads as a suffix.** ``John Smith VI MD`` gives suffix ``VI MD`` and ``Smith, John VI`` suffix ``VI``, as ``III`` does, where 1.4.0 through 2.6.0 gave last ``VI``, suffix ``MD`` and middle ``VI``. ``John Smith V, M.J.`` keeps suffix ``V`` and ``Georg VI.`` gives suffix ``VI.``, where 1.4.0 through 2.6.0 gave last ``John Smith V`` and ``VI.``. A one-letter numeral before a credential is still a name (``John Smith V PhD`` keeps last ``V``), and a Title-case ``Vi`` in a mixed-case name reads as a name (``Nguyen Van Vi`` gives last ``Van Vi``, where 2.2.0 through 2.6.0 gave suffix ``Vi``). :issue:`784`, :issue:`774` - **Fix a lone letter before a further comma reading as part of a credential.** ``Smith, John PhD I, Jr.`` gives middle ``I``, suffix ``PhD, Jr.``, as 1.4.0 through 2.3.0 did, where 2.4.0 through 2.6.0 gave suffix ``PhD I, Jr.``. :issue:`819` - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals reads as a credential too after a comma behind two or more name words written in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; a hyphenated name written in Title case keeps its reading even where both parts are listed credentials (``Kim, Ae-Ra``), though its capitals spelling does not (``KIM, AE-RA`` gives suffix ``AE-RA``). :issue:`752`, :issue:`797` + - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` + - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` * 2.6.0 - October 25, 2026 diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index 8a970ba1..725ed498 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1046,7 +1046,8 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: """ from nameparser.config.conjunctions import CONJUNCTIONS_AMBIGUOUS from nameparser.config.maiden_markers import MAIDEN_MARKERS - from nameparser.config.suffixes import GLUED_HONORIFICS + from nameparser.config.suffixes import ( + AGNOMENS, GLUED_HONORIFICS, SUFFIX_WORDS_AMBIGUOUS) from nameparser.config.surnames import KOREAN_SURNAMES from nameparser.config.titles import AFTER_GIVEN_TITLES # Every set field and every capitalization_exceptions key passes @@ -1216,6 +1217,23 @@ def matchable(field: str) -> frozenset[str]: # so the intersection is kept for `_snapshot() == # Lexicon.default()` legibility, as conjunctions_ambiguous'. after_given_titles=AFTER_GIVEN_TITLES & titles, + # likewise no v1 manager: a suffix word that is also a + # name is 2.7 behavior (#752), so it rides in the + # snapshot only. Intersect with the word set, as the + # tails are: Lexicon enforces the subset, and deleting + # the suffix word is the one v1 knob, which turns the + # reading off. agnomens is consulted only inside it, so + # its intersection is kept for legibility alone. + # A v1 caller who added such a word to suffix_acronyms + # had it read as a suffix through the acronym branch, + # so the word leaves the gate rather than raising at + # Lexicon's word-gate check (folded, as the two + # cross-set computations above are). + suffix_words_ambiguous=frozenset( + w for w in SUFFIX_WORDS_AMBIGUOUS & suffix_words + if _normalize(w) not in { + _normalize(a) for a in acronyms - ambiguous_acronyms}), + agnomens=AGNOMENS & SUFFIX_WORDS_AMBIGUOUS & suffix_words, # TupleManager is dict[str, object] (v1 parity: values were # never statically str-typed); every real entry is a str, # same assumption _DelimiterManager's sentinel lookup makes. diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 949d4d03..ee630961 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -26,6 +26,7 @@ "suffix_acronyms_ambiguous", "particles", "particles_ambiguous", "conjunctions", "conjunctions_ambiguous", "bound_given_names", "maiden_markers", "surnames", "honorific_tails", "after_given_titles", + "suffix_words_ambiguous", "agnomens", ) #: (marker, base, why) triples. Each marker QUALIFIES how entries of @@ -68,6 +69,8 @@ #: decides nothing, and AGENTS.md's invariants rule guards harm, not #: no-ops. after_given_titles -- the same: classify tags a word from it #: only where the word is in titles too, so an orphan is never read. +#: agnomens -- the same: classify reads it only for a word already in +#: suffix_words_ambiguous, so an orphan decides nothing. _SUBSET_FIELDS = ( ("particles_ambiguous", "particles", "an orphan emits a spurious particle-or-given ambiguity"), @@ -75,6 +78,8 @@ "an orphan silently becomes a period-gated suffix"), ("honorific_tails", "suffix_words", "an orphan splits the name and leaves the tail inside it"), + ("suffix_words_ambiguous", "suffix_words", + "an orphan silently becomes a suffix gated by the count"), ) @@ -699,6 +704,22 @@ class Lexicon: #: Full default list: #: :data:`~nameparser.config.titles.AFTER_GIVEN_TITLES`. after_given_titles: frozenset[str] = frozenset() + #: Subset of suffix_words that are also borne as names ("fils", + #: "neto"). Read as a suffix only where the name has words to + #: spare, as an ambiguous acronym is: "Pedro Alves Neto" ends in a + #: suffix, "Ana Neto" keeps her family name, and either reading is + #: reported (rules/S11). Written in Title case in a name of mixed + #: case it leans to the name even with words to spare, and in lower + #: case it is the suffix after a comma that names the family, + #: unless it is also in ``agnomens``. Full default list: + #: :data:`~nameparser.config.suffixes.SUFFIX_WORDS_AMBIGUOUS`. + suffix_words_ambiguous: frozenset[str] = frozenset() + #: Subset of suffix_words_ambiguous whose writing says nothing: + #: an agnomen ("filho", "neto") is written in the case of the name + #: around it, so only the words to spare decide (rules/S11). An + #: entry outside suffix_words_ambiguous is never consulted. Full + #: default list: :data:`~nameparser.config.suffixes.AGNOMENS`. + agnomens: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> #: "PhD"), laid over the word as written, so the one entry @@ -773,6 +794,22 @@ def __post_init__(self) -> None: f"period gate suffix_acronyms_ambiguous exists to impose" ) + # the mirror of the check above: suffix_as_written ORs the + # unambiguous acronym branch too, so a gated suffix word listed + # there is claimed as a suffix with no count ('Ana Neto' would + # lose her family name, rules/S11) + word_gate_bypassed = self.suffix_words_ambiguous & ( + self.suffix_acronyms - self.suffix_acronyms_ambiguous) + if word_gate_bypassed: + raise ValueError( + f"an ambiguous suffix word must not also be an unambiguous " + f"suffix acronym; in both: " + f"{', '.join(sorted(word_gate_bypassed))}. The acronym " + f"branch matches it outright, which bypasses the count " + f"suffix_words_ambiguous exists to impose; drop it from " + f"suffix_acronyms or from suffix_words_ambiguous" + ) + # -- constructors ---------------------------------------------------- @classmethod @@ -951,8 +988,8 @@ def _default_lexicon() -> Lexicon: from nameparser.config.maiden_markers import MAIDEN_MARKERS from nameparser.config.particles import NON_GIVEN_NAME_PARTICLES, PARTICLES from nameparser.config.suffixes import ( - GLUED_HONORIFICS, SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, - SUFFIX_WORDS, + AGNOMENS, GLUED_HONORIFICS, SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, + SUFFIX_WORDS, SUFFIX_WORDS_AMBIGUOUS, ) from nameparser.config.surnames import KOREAN_SURNAMES from nameparser.config.titles import ( @@ -985,6 +1022,8 @@ def _default_lexicon() -> Lexicon: surnames=KOREAN_SURNAMES, honorific_tails=GLUED_HONORIFICS, after_given_titles=AFTER_GIVEN_TITLES, + suffix_words_ambiguous=SUFFIX_WORDS_AMBIGUOUS, + agnomens=AGNOMENS, # pass canonical pair-tuples so this strictly-typed call site never # feeds a Mapping to the tuple-annotated field; __post_init__ # still tolerates a Mapping at runtime for interactive use diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index f86b98dc..f81f5c91 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -32,6 +32,7 @@ namespaced (unstable): "vocab:title", "vocab:given-title", "vocab:after-given-title", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", +"vocab:suffix-word-ambiguous", "vocab:agnomen", "vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", "vocab:maiden-marker-cont"; and, in a namespace of its own, "shape:acronym". @@ -62,8 +63,8 @@ from nameparser._lexicon import _normalize, _spells_an_initial from nameparser._policy import CapsSuffixes from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG, ParseState, PendingAmbiguity, - WorkToken, copy_with, + AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, SHAPE_ACRONYM_TAG, + ParseState, PendingAmbiguity, WorkToken, copy_with, ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( @@ -135,7 +136,16 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, tags.add("vocab:after-given-title") if suffix_as_written(n, token.text, lex): tags.add("vocab:suffix") - if n in lex.suffix_words: + if n in lex.suffix_words_ambiguous: + # rules/S11: "it reads as the generation only where the name has + # words to spare" -- the ambiguous class's tag, so every site + # that weighs a member weighs this one, and the cue tags that + # tell `_pieces.listed_lean` how its writing leans + tags.add(AMBIGUOUS_ACRONYM_TAG) + tags.add(AMBIGUOUS_WORD_TAG) + if n in lex.agnomens: + tags.add(AGNOMEN_TAG) + elif n in lex.suffix_words: tags.add("vocab:suffix-word") if n in lex.suffix_acronyms_ambiguous: tags.add(AMBIGUOUS_ACRONYM_TAG) diff --git a/nameparser/_pipeline/_comma.py b/nameparser/_pipeline/_comma.py index b3a9fb85..46e9c842 100644 --- a/nameparser/_pipeline/_comma.py +++ b/nameparser/_pipeline/_comma.py @@ -36,7 +36,8 @@ listed_lean, own_words, segment_suffix_reading, ) from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG, ParseState, PendingAmbiguity, + AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, SHAPE_ACRONYM_TAG, ParseState, + PendingAmbiguity, Structure, WorkToken, copy_with, ) from nameparser._pipeline._vocab import ( @@ -487,7 +488,7 @@ def decide(state: ParseState) -> ParseState: if maybe: reading = segment_suffix_reading( p1, pt1, tokens, pol.lenient_comma_suffixes, state.one_case, - anchored, absorbed, licensed) + anchored, absorbed, licensed, head_named=names >= 2) if not listing: if whole is None: whole = _whole_name(seg0, tokens) @@ -571,7 +572,10 @@ def decide(state: ParseState) -> ParseState: settled = (len(p1) > 1 and bool(members) and len(listed) == len(members) and all(listed_lean(tokens[i], state.one_case) - == "credential" for i in listed)) + == "credential" + and (names >= 2 + or AMBIGUOUS_WORD_TAG not in tokens[i].tags) + for i in listed)) member_set = frozenset(members) pairs = [k for k, p in enumerate(p1) if p[0] in member_set and (is_paired_initials(texts[p[0]]) diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 807779c8..25a931e1 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -54,7 +54,8 @@ def _suffix_shaped(content: str, lexicon: Lexicon) -> bool: veto -- v1 deliberately skipped it here.""" stripped = _normalize(content) acronym = stripped.replace(".", "") - return (stripped in lexicon.suffix_words + return ((stripped in lexicon.suffix_words + and stripped not in lexicon.suffix_words_ambiguous) or (acronym in lexicon.suffix_acronyms and acronym not in lexicon.suffix_acronyms_ambiguous) or content.endswith(".")) diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index d6fa3e76..25d48c38 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -58,8 +58,8 @@ from typing import NamedTuple from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, NAME_ROLES, SHAPE_ACRONYM_TAG, SUFFIX_OR_UNREAD, - WorkToken, + AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, NAME_ROLES, + SHAPE_ACRONYM_TAG, SUFFIX_OR_UNREAD, WorkToken, ) from nameparser._pipeline._vocab import ( _PERIOD_ABBREV, _ROMAN, Lean, ambiguous_lean, in_initialless_script, @@ -732,6 +732,7 @@ def segment_suffix_reading(pieces: Sequence[Sequence[int]], anchored: list[int] | None = None, absorbed: list[int] | None = None, licensed: Container[int] = (), + head_named: bool = True, # rules/S11 ) -> tuple[bool, ...] | None: """How each piece of a no-name segment reads: True a suffix, False a title. None when the segment holds a name word and so is not a @@ -888,8 +889,15 @@ def segment_suffix_reading(pieces: Sequence[Sequence[int]], continue member = (len(piece) == 1 and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags) + # rules/S11: "The floor holds first" -- at the first slot after + # a comma a generation word's writing adds nothing to the count + # C4 reads first, `head_named` being that count's two name + # words before the comma (', Dr. fils' and 'Sir Alexandre, fils' + # keep a family word) if member and listed_lean(tokens[piece[0]], one_case) \ - == "credential": + == "credential" and ( + head_named + or AMBIGUOUS_WORD_TAG not in tokens[piece[0]].tags): leading = False out.append(True) continue @@ -1051,6 +1059,12 @@ def listed_lean(token: WorkToken, one_case: bool | None) -> Lean | None: if (one_case is None or AMBIGUOUS_ACRONYM_TAG not in token.tags or SHAPE_ACRONYM_TAG in token.tags): return None + # rules/S11: "An agnomen, the Portuguese `Filho`, `Neto`, `Sobrinho` + # and their feminines, is written in the case of the name around it, + # so its writing never speaks" -- the cue is what the member is + if AMBIGUOUS_WORD_TAG in token.tags: + return ambiguous_lean(token.text, one_case, + "none" if AGNOMEN_TAG in token.tags else "word") return ambiguous_lean(token.text, one_case) @@ -1119,6 +1133,54 @@ def second_unit(rest: Sequence[int], units: Sequence[int]) -> int: return len(rest) +def _kept_in_front(rest: Sequence[int], front: int, + pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken], + one_case: bool | None) -> int: + """The position in `rest` of the last word the trailing walk keeps + at or before `front`, which stands in front of a generation word + that is also a name, so that word's count looks past what the walk + takes there (rules/S11, #890); -1 where it keeps none. Taken by + itself: a suffix piece, a numeral of two or more letters read as the + generation ('Pedro VI Filho'), a member of the ambiguous class its + writing makes a suffix ('Pedro MA Filho'). Taken when carried: a + bare member of the class behind any of those, the members between + passing the run on, as rules/S7's company reads it ('Pedro Junior + Ma Filho'). One pass, so a carried run costs a step a member.""" + members: int | None = None # where an uncarried run of members began + j = front + while j >= 0: + piece = pieces[rest[j]] + if is_suffix_piece(piece, ptags[rest[j]], tokens): + members = None # a carrier: the run behind it is taken + j -= 1 + continue + if len(piece) != 1: + break + tok = tokens[piece[0]] + stem = tok.text.rstrip(".") + if ((len(stem) > 1 and _ROMAN.match(stem) is not None + and reads_as_generation( + tok.text, tok.tags, + tokens[pieces[rest[j - 1]][0]].text if j >= 1 else None, + one_case)) + or (AMBIGUOUS_ACRONYM_TAG in tok.tags + and listed_lean(tok, one_case) == "credential")): + members = None + j -= 1 + continue + # a bare member that leans nowhere is taken only if something in + # front carries the run; an agnomen carries nothing and is kept + if (AMBIGUOUS_ACRONYM_TAG not in tok.tags + or AMBIGUOUS_WORD_TAG in tok.tags): + break + if members is None: + members = j + j -= 1 + return j if members is None else members + + def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], @@ -1269,7 +1331,23 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], lean = listed_lean(tokens[piece[0]], one_case) # peeling still leaves given + family, or the writing says # to peel anyway - if lean == "credential" or (lean is None and k - 2 >= second): + # rules/S11: "only where the name has words to spare" -- a + # generation word counts the NAME words in front of it, so + # what the walk takes there ('Pedro Junior Filho', 'Pedro VI + # Filho', 'Pedro Junior Ma Filho') is not counted as the word + # spared and then peeled as well (#890). One scan to the + # last word the walk keeps, so a run costs a step a word + front = k - 2 + if AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags: + front = _kept_in_front(rest, front, pieces, ptags, tokens, + one_case) + # rules/S11: "the floor holds first" -- a generation word's + # case may settle it as the suffix only where the name keeps + # a family word ('Pedro Junior fils' keeps family 'fils', + # Derek, #890); an acronym's capitals take it with none + gated = AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags + if (lean == "credential" and (not gated or front >= second) + or (lean is None and front >= second)): k -= 1 continue # #544: or an unambiguous credential stands IN FRONT of it @@ -1291,6 +1369,11 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], # The pass, once built, answers first: asked of every # member, the reach test would walk back through the run # each time, one look-behind per member. + # A generation word that is also a name has no company to + # keep (rules/S7's credentials): a suffix word in front of + # it is no evidence, its count above having looked past it + if AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags: + break if anchors is None and anchor_in_reach( rest[k - 2:0:-1], pieces, ptags, tokens): anchors = credential_anchors(rest, pieces, ptags, tokens) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index fbe60d32..ef96ec98 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -86,6 +86,15 @@ class WorkToken: #: reader ever finds. AMBIGUOUS_ACRONYM_TAG = "vocab:suffix-ambiguous" +#: Beside `AMBIGUOUS_ACRONYM_TAG` on a suffix WORD that is also a name +#: (`Lexicon.suffix_words_ambiguous`, rules/S11): the class is the +#: same, and so is every site that weighs it, but the writing leans the +#: other way -- lower case in a mixed-case name is the suffix, where an +#: acronym's capitals are (`_vocab.ambiguous_lean`). `AGNOMEN_TAG` +#: rides beside both on an agnomen, whose writing leans nowhere. +AMBIGUOUS_WORD_TAG = "vocab:suffix-word-ambiguous" +AGNOMEN_TAG = "vocab:agnomen" + #: EITHER way a token joins the ambiguous credential class -- the #: vocabulary's claim and the writing's. The two emitters that report #: a fork the peel called and declined ask exactly this: `_group`'s diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 221ccf08..340b917b 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -487,12 +487,18 @@ def is_one_case(texts: Sequence[str]) -> bool: #: reason; it has one caller shape and no wrapper to keep in step. Lean = Literal["credential", "name"] +#: How a member's writing leans, by what the member is (rules/S11): +#: an acronym's capitals are the credential, a suffix word's lower +#: case is the suffix, and an agnomen's writing says nothing. +Cue = Literal["acronym", "word", "none"] + # #289's credential lean, rules/S6: "Written case is evidence beside # the count (S4), and it speaks only in a name written in more than # one case" -- the WRITING evidence that decides ahead of the # words-to-spare count (history: decisions/S6). -def ambiguous_lean(text: str, one_case: bool) -> Lean | None: +def ambiguous_lean(text: str, one_case: bool, + cue: Cue = "acronym") -> Lean | None: """Which way the WRITING leans for a member of the ambiguous credential set: "credential", "name", or None for no lean at all. @@ -516,18 +522,35 @@ def ambiguous_lean(text: str, one_case: bool) -> Lean | None: so 'MA.' leans as 'MA' does -- the lean reads CASE, not periods, and the dotted spelling is the vocabulary's own question (suffix_as_written). + + `cue` is what the member is (rules/S11). A suffix WORD that is + also a name ("word", 'fils') leans the other way at one end: its + lower case in a mixed-case name is the suffix ("credential", the + suffix reading), its capitals say nothing (a surname is often + written so), and Title case leans to the name as an acronym's + does. An agnomen ("none", 'Neto') is written in the case of the + name around it, so its writing never leans. """ - if one_case: + if one_case or cue == "none": return None if text.upper() == text.lower(): # caseless: no contrast to read return None if text.isupper(): - return "credential" + return "credential" if cue == "acronym" else None if text.islower(): - return None + return "credential" if cue == "word" else None return "name" +def lexicon_cue(n: str, lexicon: Lexicon) -> Cue: + """The `ambiguous_lean` cue of the folded member `n`, read off the + lexicon by a caller that has no tags to read (`is_wholly_suffix`); + `_pieces.listed_lean` reads the same answer off classify's tags.""" + if n in lexicon.suffix_words_ambiguous: + return "none" if n in lexicon.agnomens else "word" + return "acronym" + + _DOTTED = re.compile(r"(?:" + _LETTER + r"\.)+") # #544: the CHUNKED spelling -- two or more runs of letters, each # closed by a period ('M.Eng.', 'L.Ac.'). Both callers compare the @@ -606,7 +629,8 @@ def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: return True if ((a in lexicon.suffix_acronyms and a not in lexicon.suffix_acronyms_ambiguous) - or n in lexicon.suffix_words): + or (n in lexicon.suffix_words + and n not in lexicon.suffix_words_ambiguous)): return True # the joiners asked in C first, so a word holding neither pays no # frame @@ -694,7 +718,8 @@ def is_suffix_lenient(text: str, lexicon: Lexicon) -> bool: bypassing the initial veto -- only safe in unambiguous positions (after a comma).""" n = _normalize(text) - return n in lexicon.suffix_words \ + return (n in lexicon.suffix_words + and n not in lexicon.suffix_words_ambiguous) \ or _is_suffix_strict_n(n, text, lexicon) @@ -783,7 +808,8 @@ def period_joined_vocab( ANY suffix chunk makes it a suffix. Chunk-level suffix membership is v1's is_suffix: bare ambiguous acronyms COUNT ('Msc.Ed.' derives via 'ed') -- the ambiguous period-gate applies to whole - tokens only. Returns "title", "suffix", "shape", or None. + tokens only -- while a suffix word that is also a name does not + ('M.Filho', rules/S11). Returns "title", "suffix", "shape", or None. "shape" is the third verdict and it is the absence of a claim worth acting on (#516): two or more chunks, EVERY one of them @@ -838,8 +864,13 @@ class by shape for the first time (measured regression, #516 chunks = [_normalize(c) for c in text.split(".") if c] if any(c in lexicon.titles for c in chunks): return "title" + # rules/S11: "the word is part of that one word and is weighed only + # by the shape the joined word wears" -- no chunk claim for a + # suffix word that is also a name ('M.Filho') matched = [c for c in chunks - if c in lexicon.suffix_acronyms or c in lexicon.suffix_words] + if c in lexicon.suffix_acronyms + or (c in lexicon.suffix_words + and c not in lexicon.suffix_words_ambiguous)] if matched and not all(len(c) == 1 and c.isascii() for c in matched): return "suffix" if (len(chunks) >= 2 and all(c.isalpha() for c in chunks) @@ -893,7 +924,9 @@ class at all). """ if "." in text: return False - return _normalize(text) in lexicon.suffix_acronyms_ambiguous + n = _normalize(text) + return (n in lexicon.suffix_acronyms_ambiguous + or n in lexicon.suffix_words_ambiguous) # #516's all-caps half, ONE PREDICATE for the shape test and its @@ -1410,7 +1443,8 @@ def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, def counts_as_suffix(text: str) -> bool: if (one_case is not None and ambiguous_class_member(text, lexicon) - and ambiguous_lean(text, one_case) == "credential"): + and ambiguous_lean(text, one_case, lexicon_cue( + _normalize(text), lexicon)) == "credential"): return True if ((is_suffix_lenient(text, lexicon) if lenient else is_suffix_strict(text, lexicon, letters_one_case)) diff --git a/nameparser/_render.py b/nameparser/_render.py index c7e1b002..62ab02f9 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -458,6 +458,15 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # whose tags it keeps on purpose, and keying this on `span is None` # overrode them -- `revise(middle='e-f')` repaired to 'e-F' where # the same words parsed gave 'E-F' (#463 review). + # rules/S6: "its lower case in a name of mixed case is the generation + # wherever the name keeps a family word" -- a generation word that is also a + # name, in the suffix role, repairs to the one writing that keeps + # that reading; Title case would lean it back to the name. An + # agnomen takes no case cue and title-cases as any word does. + if (role is Role.SUFFIX and normalized in lex.suffix_words_ambiguous + and normalized not in lex.agnomens + and normalized not in lex.capitalization_exceptions_map): + return word.lower() generation = role is Role.SUFFIX and "vocab:suffix" in tags if not generation and ( _lowers_as_particle(normalized, word, role, tags, lex) diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index beb85ead..13bfd8d6 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -1217,6 +1217,9 @@ 'eldri', 'esq', 'esquire', + 'filha', + 'filho', + 'fils', 'i', 'ii', 'iii', @@ -1235,10 +1238,15 @@ 'mlajši', 'mlađa', 'mlađi', + 'neta', + 'neto', + 'netto', 'nuorempi', 'ret', 'rinpoche', 'snr', + 'sobrinha', + 'sobrinho', 'sr', 'starejša', 'starejši', @@ -1417,6 +1425,27 @@ '씨', }) +SUFFIX_WORDS_AMBIGUOUS: frozenset[str] = frozenset({ + 'filha', + 'filho', + 'fils', + 'neta', + 'neto', + 'netto', + 'sobrinha', + 'sobrinho', +}) + +AGNOMENS: frozenset[str] = frozenset({ + 'filha', + 'filho', + 'neta', + 'neto', + 'netto', + 'sobrinha', + 'sobrinho', +}) + SUFFIX_ACRONYMS_AMBIGUOUS: frozenset[str] = frozenset({ 'aba', 'aca', diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index c030b5ad..396610b3 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -2,8 +2,9 @@ # nameparser/vocab/*.toml, classified by kind (generation, credential, # honorific, ...) and by language. A suffix reading is an acronym or a # word (`acronym`, true by default for a credential), so the two sets -# below are disjoint by construction; a name reading beside an acronym -# reading and `attached` on a word reading make the two marker subsets. +# below are disjoint by construction; a name reading beside a suffix +# reading, `attached` on a word reading and the agnomen kind make the +# marker subsets. # tools/vocab/build.py compiles them into _compiled.py and runs the # checks that were import-time asserts here (decisions/vocabulary-model). from nameparser.config import _compiled @@ -74,6 +75,24 @@ ships glued, above. Bare 선생 and 교수: they read as common nouns as readily as address terms, and only their -님 forms ship. +""" +SUFFIX_WORDS_AMBIGUOUS = _compiled.SUFFIX_WORDS_AMBIGUOUS +""" + +The subset of :data:`SUFFIX_WORDS` also borne as a name: a generation +word with a ``given`` or ``family`` reading, such as French ``fils`` and +the Brazilian agnomens (``filho``, ``neto``). Each is read as a suffix +only where the name keeps a word for its family (rules/S11): ``Alexandre +Dumas fils`` ends in a suffix and ``Marie Fils`` keeps her family name. + +""" +AGNOMENS = _compiled.AGNOMENS +""" + +The subset of :data:`SUFFIX_WORDS_AMBIGUOUS` whose writing says nothing +about the reading: an agnomen is written in the case of the name around +it, so only the words to spare decide (rules/S11). + """ SUFFIX_ACRONYMS_AMBIGUOUS = _compiled.SUFFIX_ACRONYMS_AMBIGUOUS """ @@ -138,6 +157,8 @@ __all__ = [ "SUFFIX_WORDS", "GLUED_HONORIFICS", + "SUFFIX_WORDS_AMBIGUOUS", + "AGNOMENS", "SUFFIX_ACRONYMS_AMBIGUOUS", "SUFFIX_ACRONYMS", "SUFFIX_NOT_ACRONYMS", diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 99dc2342..6b12c9a7 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -54,14 +54,12 @@ note = "Vietnamese Đỗ leads a great many surnames" ```toml [[word]] -text = "fils" -family = { us_census_2010 = 767, wikidata = 5, as_of = 2026-10-16 } -[word.excluded.generation] -reason = "pack-shaped: Marie Fils, 767 US Census bearers of Fils" -issue = [679, 684] +text = "père" +[word.excluded] +generation = { reason = "pack-shaped: collides outside French", issue = [679, 684] } ``` -`fils` ("son", the French generational word) is deliberately not a suffix. It has no vocabulary reading at all: a `family` reading recording the counts that kept it out, and an exclusion refusing the `generation` reading with the reason. Anyone who adds `generation = {}` to `fils` in any file gets a build error that quotes that reason. +`père` ("father", the French word for the Elder) is deliberately not a suffix. It has no vocabulary reading at all, only an exclusion refusing the `generation` reading with the reason. Anyone who adds `generation = {}` to `père` in any file gets a build error that quotes that reason. Its son, `fils`, was refused the same way until 2026-10-25; it now has the `generation` reading beside its `family` reading, which gates it by the name around it ([derived markers](#derived-markers)), so `Marie Fils` keeps her family name. ```toml [[word]] @@ -84,7 +82,7 @@ Two rules about the `text`: Every file but `und.toml` lists its words in one order, and the build refuses a file that does not (check 10), naming the entry to move and where it belongs: -1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`fils`, `duke`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else. +1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`père`, `duke`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else. 2. **Then by the word it is a spelling of**: a reading's `same_as`, or the word's own text when it has none. A head comes first and its spellings follow it, so `jr` is followed by `jnr` and `junior`, and `sr` by `snr`. A spelling that has spellings of its own heads a group inside its head's: in `ar.toml`, `el-` follows the `a…-` spellings of `al-` with its own assimilated forms after it, and `ul-` comes last. Only the file's own entries are followed, so a word that is a head in one file and a spelling in another is placed in each file by that file alone. 3. **Then by the word's text.** @@ -113,6 +111,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `modifier` | `title` | `TITLES` | | `suffix` | `suffix` | `SUFFIX_WORDS` | | `generation` | `suffix` | `SUFFIX_WORDS` | +| `agnomen` | `suffix` | `SUFFIX_WORDS`, `SUFFIX_WORDS_AMBIGUOUS`, `AGNOMENS` | | `credential` | `suffix` | `SUFFIX_ACRONYMS` | | `honorific` | `suffix` | `SUFFIX_WORDS` | | `professional_honorific` | `suffix` | `SUFFIX_WORDS` | @@ -135,6 +134,8 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or A suffix reading goes to `SUFFIX_ACRONYMS` when its `acronym` property is true and to `SUFFIX_WORDS` otherwise; the column shows where each kind goes with `acronym` unwritten. A word has at most one suffix kind (check 4). +`generation` is a word that numbers or ranks its bearer among relatives of the same name, written after the name: junior and senior and their translations, `fils`, the roman numerals the vocabulary lists. `agnomen` is the Brazilian and Portuguese kind of it (`filho`, `neto`, `sobrinho` and their feminines): it is written in the case of the name around it, so its writing settles nothing, and an agnomen reading needs a name reading on its word (check 11), which gates it by the count alone ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md), a suffix word that is also a name). The column shows an agnomen with that name reading. + The title kinds say what the title is: - `salutation`: a language's everyday form of address, its "Mr" and "Mrs" (`mr`, `mrs`, `mx`, `herr`, `señora`, `г-н`, `श्री`), and the English `master` for a boy; @@ -179,6 +180,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | | `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` is required where the word has no `given` reading in any file, and that absence, not the property, is what puts the word in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). `true` is refused, naming the `given` reading to write instead (check 11) | | `may_be_initial` | behavioral | `conjunction` | false | `CONJUNCTIONS_AMBIGUOUS`, on a cased single letter only: read as an initial in a name written in one case, as in `JOHN E SMITH` ([rules/P8](../../docs/design/rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells) | +| `default` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation` | true | `false` keeps the reading out of every compiled set, `SUFFIX_WORDS` included for a generation: recorded and checked as a reading, for a pack to turn on in phase 3, where its writing offers the default no cue (`h`, Spanish *hijo*, written `H.`, exactly as an initial is). An exclusion does not refuse it, and it cannot read two ways beside the word's live readings (checks 4 to 6); a name reading beside it marks nothing. `true` is refused: it is what every reading is unwritten | | `cased` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation` | the word as written | `CAPITALIZATION_EXCEPTIONS`: the case `capitalized()` gives the word (`PhD`), a mask over its letters and digits, never a replacement (#459) | | `note` | descriptive | every category, a `name` reading included | | free text: why the reading is as it is | | `issue` | descriptive | every category but `name` | | an issue number, or a list of them | @@ -195,14 +197,15 @@ Two of these need more than their row: ### Derived markers -Two sets mark a clash with a name, and both are derived: a word is in one when it carries the clashing reading and a name reading, in any of its files (check 11, since 2026-10-25, [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md)). Phase 1 wrote them as flags, `may_be_name` and `may_be_given = true`, which the build now refuses, naming the reading to write. +Three sets mark a clash with a name, and all three are derived: a word is in one when it carries the clashing reading and a name reading, in any of its files (check 11, since 2026-10-25, [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md)). Phase 1 wrote two of them as flags, `may_be_name` and `may_be_given = true`, which the build now refuses, naming the reading to write; the third, a suffix word that is also a name, arrived with phase 2h. | Marker | Readings | Compiled into | Effect | |---|---|---|---| +| possibly a name, a word | a `generation` or `agnomen` reading and a `given` or `family` reading; a name reading beside any other suffix word kind is refused (check 11), no gate reading that clash | `SUFFIX_WORDS_AMBIGUOUS` | a suffix word that is also a name (`fils`, `neto`) is read as a suffix only where the name has words to spare, as an ambiguous acronym is: `Alexandre Dumas fils` and `Pedro Alves Neto` end in a suffix, `Marie Fils` and `Ana Neto` in a family name, either way reported ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md)). A generation's Title case in a name of mixed case leans it to the name, and its lower case settles it as the suffix wherever the name keeps a family word; an agnomen's writing settles nothing | | possibly a name | an acronym `suffix` reading and a `given` or `family` reading | `SUFFIX_ACRONYMS_AMBIGUOUS` | an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | | possibly given | a `particle` reading and a `given` reading | kept out of `NON_GIVEN_NAME_PARTICLES` | a particle opening the name stays a given name there (`van Gogh`, given `van`) and the fork is reported, rather than starting the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold) | -A word reading of the suffix category (`jr`, `generation`) beside a name reading is no clash the parser weighs and marks nothing. The marker is the clash, not the counts: a count recorded for a word that should stay unmarked goes in the clashing reading's `note`, as `ld`'s does, since a name reading would mark it. `may_be_initial` is not derived: its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. +Before phase 2h (2026-10-25) a word reading of the suffix category beside a name reading marked nothing, and no shipped word had one. The marker is the clash, not the counts: a count recorded for a word that should stay unmarked goes in the clashing reading's `note`, as `ld`'s does, since a name reading would mark it. `may_be_initial` is not derived: its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. ### Name readings @@ -227,7 +230,7 @@ An exclusion records that a reading was considered and refused, with the reason, - **A refused reading** is a word-level table: `[word.excluded.generation]` with a `reason`, or `excluded = { generation = "the reason" }`. It refuses that kind in every file, and also every other kind that compiles into the same set, so a generic `suffix = {}` cannot put back a refused `generation`. A category name refuses the whole category (`excluded = { suffix = "..." }`). A `name` kind refuses only itself. - **A refused property** sits on a reading the word keeps: `[word.honorific.excluded]` with `attached = "the reason"`. It refuses any value of that property other than the one it takes unwritten, on any reading of the category; a refused `cased`, which belongs to the word, on any reading of the word. -The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `fils` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error. +The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `père` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error. To see what has been refused, and why: `uv run python tools/vocab/query.py --excluded generation`. @@ -254,14 +257,15 @@ These names will mean something in a later phase. Writing one now as a key, in a | Name | Phase | |---|---| -| `bare_only` | 2 | -| `leading_only` | 2 | -| `trailing_only` | 2 | | `attaches` | 2 | -| `separators` | 2 | -| `default` | 3 | | `prefer` | 3 | | `strict` | 3 | +| `bare_only` | 4 | +| `leading_only` | 4 | +| `trailing_only` | 4 | +| `separators` | 4 | + +Phase 2 gave `default` its meaning (2026-10-25, see [Properties](#properties)) and keeps `attaches` for its front-of-the-name work. The gate's slots carry position and shape already settles a bare letter ([rules/P7](../../docs/design/rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells), so `leading_only`, `trailing_only` and `bare_only` wait for phase 4's shapes, as `separators` waits for its glued words. Two property values are refused for the same reason, with a message saying a later phase: `addresses = "full"` and `"any"`, and `attached` with a `side` other than `"after"`. diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index 05900b56..110b386b 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -29,9 +29,7 @@ generation = { issue = 661 } [[word]] text = "st" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" -issue = [661, 679, 684] +generation = { default = false, expands = "starší, the Elder: Jiří Dienstbier st.", issue = [661, 679, 684, 752], note = "real and standard, but St. is Saint in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Czech pack turns it on" } [[word]] text = "starší" diff --git a/nameparser/vocab/de.toml b/nameparser/vocab/de.toml index b58a7276..c0af9f64 100644 --- a/nameparser/vocab/de.toml +++ b/nameparser/vocab/de.toml @@ -122,6 +122,10 @@ nobility = { gender = "m", expands = "imperial count" } text = "ritter" nobility = { gender = "m", expands = "knight" } +[[word]] +text = "i" +generation = { default = false, expands = "the regnal ordinal written with its period: Wilhelm I.", issue = 752, note = "the default reads I as a numeral only bare (mul.toml); written with its period it is exactly how an initial is written (rules/T5), so no default reads the ordinal; a German pack turns it on" } + [[word]] text = "mda" credential = { issue = 18, expands = "Mitglied des Abgeordnetenhauses" } diff --git a/nameparser/vocab/es.toml b/nameparser/vocab/es.toml index c231cbc0..423acc48 100644 --- a/nameparser/vocab/es.toml +++ b/nameparser/vocab/es.toml @@ -49,6 +49,10 @@ salutation = { gender = "f", same_as = "señoritas", after_given = true } text = "dra" academic = { gender = "f", expands = "Doctora, Doutora" } +[[word]] +text = "h" +generation = { default = false, expands = "hijo, the son: Juan Pérez H.", issue = 752, note = "written with its period, it is exactly how an initial is written (rules/T5), so no default reads it; a Spanish pack turns it on" } + [[word]] text = "de" preposition = { may_be_given = false, expands = "of" } diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 751ab981..0852dadd 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -95,10 +95,8 @@ modifier = { expands = "the second word of chargé d'affaires" } [[word]] text = "fils" +generation = { issue = [679, 752], expands = "son, the Younger: Alexandre Dumas fils", note = "gated by the family reading (rules/S11): Title case leans to the name, so Marie Fils keeps her family name, and lower case is the generation wherever the name keeps a family word" } family = { us_census_2010 = 767, wikidata = 5, as_of = 2026-10-16 } -[word.excluded.generation] -reason = "pack-shaped: Marie Fils, 767 US Census bearers of Fils" -issue = [679, 684] [[word]] text = "père" diff --git a/nameparser/vocab/pt-BR.toml b/nameparser/vocab/pt-BR.toml new file mode 100644 index 00000000..22c1bc10 --- /dev/null +++ b/nameparser/vocab/pt-BR.toml @@ -0,0 +1,6 @@ +schema = 1 + +[[word]] +text = "netto" +agnomen = { same_as = "neto", gender = "m", issue = [843, 752], expands = "the grandson, in the spelling of Brazil's older orthography" } +family = { us_census_2010 = 1243, wikidata = 30, as_of = 2026-10-25 } diff --git a/nameparser/vocab/pt.toml b/nameparser/vocab/pt.toml index 07a7c599..6976b2a5 100644 --- a/nameparser/vocab/pt.toml +++ b/nameparser/vocab/pt.toml @@ -31,6 +31,38 @@ same_as = "jr" issue = 843 note = "the standard Portuguese spelling of Junior: tse-donors (2026-10-22) has 991 distinct filed names ending in it, 983 records ending in it where the tax registry's name ends JUNIOR. Its collisions are junior's: a given name (Wikidata 37+ as Júnior, 2026-10-23) where the suffix claim does not act, and 49+ Wikidata family bearers" +[[word]] +text = "filha" +agnomen = { gender = "f", issue = [843, 752], expands = "the daughter", note = "the feminine of filho; tse-donors (2026-10-22) has 66 distinct filed names ending in it" } +family = { note = "no source counts it as a name; declared as filho's is, so that two words ending in it keep it as the family name" } + +[[word]] +text = "filho" +agnomen = { gender = "m", issue = [843, 752], expands = "the son", note = "an agnomen marks its bearer against a relative of the same name, as Júnior does, and is written in the case of the name around it, so only the count gates it (rules/S11); tse-donors (2026-10-22) has 10,973 distinct filed names ending in it" } +family = { us_census_2010 = 513, wikidata = 136, as_of = 2026-10-25 } + +[[word]] +text = "neta" +agnomen = { gender = "f", issue = [843, 752], expands = "the granddaughter", note = "the feminine of neto; tse-donors (2026-10-22) has 152 distinct filed names ending in it" } +given = { us_ssa = 3226, wikidata = 31, as_of = 2026-10-25, note = "female births only; mostly the Hebrew given name Neta" } +family = { us_census_2010 = 115, wikidata = 2, as_of = 2026-10-25 } + +[[word]] +text = "neto" +agnomen = { gender = "m", issue = [843, 752], expands = "the grandson", note = "an agnomen marks its bearer against a relative of the same name, as Júnior does, and is written in the case of the name around it, so only the count gates it (rules/S11); tse-donors (2026-10-22) has 8,434 distinct filed names ending in it" } +given = { wikidata = 12, as_of = 2026-10-25 } +family = { us_census_2010 = 1645, wikidata = 276, as_of = 2026-10-25 } + +[[word]] +text = "sobrinha" +agnomen = { gender = "f", issue = [843, 752], expands = "the niece", note = "the feminine of sobrinho; tse-donors (2026-10-22) has 5 distinct filed names ending in it" } +family = { note = "no source counts it as a name; declared as sobrinho's is, so that two words ending in it keep it as the family name" } + +[[word]] +text = "sobrinho" +agnomen = { gender = "m", issue = [843, 752], expands = "the nephew", note = "an agnomen marks its bearer against a relative of the same name, as Júnior does, and is written in the case of the name around it, so only the count gates it (rules/S11); tse-donors (2026-10-22) has 955 distinct filed names ending in it" } +family = { us_census_2010 = 166, wikidata = 18, as_of = 2026-10-25 } + [[word]] text = "da" preposition = { expands = "of the, fem. sg." } diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 51166464..e8394cc2 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1559,12 +1559,195 @@ def _check_cjk_shape_purity(self) -> None: notes="the Portuguese spelling reads as 'José Silva Junior' " "does. 1.4.0 and 2.5.0 read middle 'Silva', family " "'Júnior'"), - Case("an_agnomen_stays_the_family_name", "Pedro Alves Filho", - {"given": "Pedro", "middle": "Alves", "family": "Filho"}, + Case("an_agnomen_is_a_generational_suffix", "Pedro Alves Filho", + {"given": "Pedro", "family": "Alves", "suffix": "Filho"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="an agnomen marks its bearer against a relative of the same " + "name, as Junior does (rules/S11). 1.4.0 and 2.6.0 read " + "middle 'Alves', family 'Filho'"), + Case("an_agnomen_with_no_word_to_spare", "Ana Neto", + {"given": "Ana", "family": "Neto"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="Neto is a surname too, and a name of two words has none to " + "spare for the generation (rules/S11); the fork is reported"), + Case("an_agnomen_in_capitals_with_no_word_to_spare", "PEDRO FILHO", + {"given": "PEDRO", "family": "FILHO"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="a name written in one case says nothing, and an agnomen's " + "writing never does, so the count keeps the family"), + Case("an_agnomen_after_a_particle_surname", "PEDRO DA SILVA FILHO", + {"given": "PEDRO", "family": "DA SILVA", "suffix": "FILHO"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="the particle surname is the family the agnomen follows. " + "1.4.0 and 2.6.0 read family 'DA SILVA FILHO'"), + Case("an_agnomen_before_a_suffix_comma", "Pedro Alves Neto, PhD", + {"given": "Pedro", "family": "Alves", "suffix": "Neto, PhD"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="the agnomen joins the credentials after it, as 'José Silva " + "Júnior, PhD' does. 1.4.0 and 2.6.0 read middle 'Alves', " + "family 'Neto'"), + Case("an_agnomen_ending_the_given_part", "Alves, Pedro Neto", + {"given": "Pedro", "family": "Alves", "suffix": "Neto"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="the given part's last word is read by its writing alone " + "(rules/S5), and an agnomen's never keeps it a name. 1.4.0 " + "and 2.6.0 read middle 'Neto'"), + Case("an_agnomen_alone_after_a_family_comma", "Silva, Neto", + {"given": "Neto", "family": "Silva"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the first word after the comma is the given name, the count " + "deciding first (rules/C4); FEC holds NETO as a given name"), + Case("an_agnomen_in_capitals_in_a_mixed_case_name", "Pedro Alves NETO", + {"given": "Pedro", "family": "Alves", "suffix": "NETO"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="an agnomen's capitals say nothing, where an acronym's would " + "be the credential, so the count decides. 1.4.0 and 2.6.0 " + "read middle 'Alves', family 'NETO'"), + Case("an_agnomen_glued_behind_an_initial", "Pedro M.Filho", + {"given": "Pedro", "family": "M.Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="no chunk claim for a suffix word that is also a name " + "(rules/S11): the joined word is weighed by its dotted shape, " + "as it was"), + Case("a_generation_word_in_front_of_an_agnomen_is_no_word_to_spare", "Pedro Junior Filho", + {"given": "Pedro", "middle": "Junior", "family": "Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="only name words in front of an agnomen count toward its " + "words to spare, so the suffix word there spares nothing " + "and keeps no company (rules/S11, #890)"), + Case("an_accented_junior_in_front_of_an_agnomen", "Pedro Júnior Neto", + {"given": "Pedro", "middle": "Júnior", "family": "Neto"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="only name words in front of an agnomen count toward its " + "words to spare, so the suffix word there spares nothing " + "and keeps no company (rules/S11, #890)"), + Case("an_abbreviated_junior_in_front_of_an_agnomen", "Pedro Jr. Filho", + {"given": "Pedro", "middle": "Jr.", "family": "Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="only name words in front of an agnomen count toward its " + "words to spare, so the suffix word there spares nothing " + "and keeps no company (rules/S11, #890)"), + Case("a_generation_word_in_front_of_an_agnomen_before_a_comma", "Pedro Junior Filho, MD", + {"given": "Pedro", "middle": "Junior", "family": "Filho", "suffix": "MD"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="only name words in front of an agnomen count toward its " + "words to spare, so the suffix word there spares nothing " + "and keeps no company (rules/S11, #890)"), + Case("a_generation_word_in_front_of_an_agnomen_in_capitals", "PEDRO JUNIOR FILHO", + {"given": "PEDRO", "middle": "JUNIOR", "family": "FILHO"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="only name words in front of an agnomen count toward its " + "words to spare, so the suffix word there spares nothing " + "and keeps no company (rules/S11, #890)"), + Case("a_numeral_in_front_of_an_agnomen_is_no_word_to_spare", "Pedro VI Filho", + {"given": "Pedro", "middle": "VI", "family": "Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="what the trailing run would take in front of an agnomen " + "is no word to spare for it, a numeral the fork reads or " + "a member its writing makes a suffix (rules/S11, #890)"), + Case("a_credential_in_capitals_in_front_of_an_agnomen_is_no_word_to_spare", "Pedro MA Filho", + {"given": "Pedro", "middle": "MA", "family": "Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="what the trailing run would take in front of an agnomen " + "is no word to spare for it, a numeral the fork reads or " + "a member its writing makes a suffix (rules/S11, #890)"), + Case("a_member_carried_in_front_of_an_agnomen_is_no_word_to_spare", "Pedro Junior Ma Filho", + {"given": "Pedro", "middle": "Junior Ma", "family": "Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="a member a suffix word in front carries is what the " + "trailing run takes, so it spares no word for the agnomen " + "behind it (rules/S11, #890)"), + Case("a_particle_carried_in_front_of_an_agnomen", "Pedro Junior do Filho", + {"given": "Pedro", "middle": "Junior", "family": "do Filho"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="a member a suffix word in front carries is what the " + "trailing run takes, so it spares no word for the agnomen " + "behind it (rules/S11, #890)"), + Case("an_agnomen_behind_a_joined_surname", "Maria Gomes e Silva Filho", + {"given": "Maria", "family": "Gomes e Silva", "suffix": "Filho"}, + classification="fix(#843)", + ambiguities=("suffix-or-name",), + notes="a connective join in front is one name word, the " + "agnomen's word to spare (rules/S11, #890). 1.4.0 and " + "2.6.0 read family 'Filho'"), + Case("fils_after_a_name", "Alexandre Dumas fils", + {"given": "Alexandre", "family": "Dumas", "suffix": "fils"}, + classification="fix(#752)", + ambiguities=("suffix-or-name",), + notes="fils is a generation word with a family reading, gated " + "(rules/S11). 1.4.0 and 2.6.0 read middle 'Dumas', family " + "'fils'"), + Case("fils_in_title_case_is_a_name", "Marie Fils", + {"given": "Marie", "family": "Fils"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="Title case leans a generation word to the name (rules/S6); " + "the fork is reported"), + Case("fils_in_title_case_with_words_to_spare", "Jean Marie Fils", + {"given": "Jean", "middle": "Marie", "family": "Fils"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the lean holds with words to spare too"), + Case("fils_in_lower_case_with_no_word_to_spare", "Dumas fils", + {"given": "Dumas", "family": "fils"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the floor holds before the case cue: two name words keep " + "the family, lower case or not (rules/S11, Derek, #890)"), + Case("fils_in_lower_case_behind_a_suffix_word", "Pedro Junior fils", + {"given": "Pedro", "middle": "Junior", "family": "fils"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the suffix word in front is no word to spare, and the " + "floor holds before the case cue (rules/S11, #890)"), + Case("fils_in_lower_case_after_a_family_comma", "Dumas, Alexandre fils", + {"given": "Alexandre", "family": "Dumas", "suffix": "fils"}, + classification="fix(#752)", + ambiguities=("suffix-or-name",), + notes="the comma has named the family, so lower case settles the " + "generation (rules/S5, S6). 1.4.0 and 2.6.0 read middle " + "'fils'"), + Case("fils_in_lower_case_after_an_empty_head", ", Dr. fils", + {"title": "Dr.", "family": "fils"}, + classification="parity", + notes="no name before the comma, so no family is named and the " + "floor holds before the case cue (rules/S11, #890)"), + Case("fils_in_lower_case_after_one_name_word_and_a_comma", "Dumas, fils", + {"given": "fils", "family": "Dumas"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="at the first slot after a comma the count decides alone, " + "lower case adding nothing to it (rules/S11, C4, #890)"), + Case("fils_in_lower_case_after_a_joined_salutation", "Mr. and Mrs., fils", + {"given": "fils", "family": "Mr. and Mrs."}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the joined salutation is no family word for the case cue " + "to rely on (rules/S11, #890)"), + Case("a_one_letter_generation_is_off_by_default", "Juan Pérez H.", + {"given": "Juan", "middle": "Pérez", "family": "H."}, classification="parity", - notes="Filho, Neto and Sobrinho are not suffix vocabulary: " - "whether they read as suffixes is open on #843. The " - "contrast row for the one above"), + notes="Spanish H. (hijo) is written as an initial is (rules/T5), so " + "its generation reading is recorded off by default, for a " + "pack"), Case("a_feminine_generational_word_in_cyrillic", "Агриппина Младшая", {"given": "Агриппина", "suffix": "Младшая"}, classification="fix(#661)", diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index b8aa650c..a394d1c9 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -14,6 +14,7 @@ SPLIT_PAIR_OPENS, is_cased_paired_initials, is_paired_initials, is_single_letter_numeral, split_pair, is_suffix_lenient, is_suffix_strict, is_title_shaped, is_wholly_suffix, + lexicon_cue, maiden_marker_run, name_word_count, period_joined_vocab, written_as_a_name, resolve_script_set, single_script, suffix_as_written, ) @@ -542,12 +543,12 @@ def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: assert caps_shape_candidate("ZZQX", base, on, False) for field in _VOCAB_FIELDS: extra: dict[str, set[str]] = {field: {"zzqx"}} - if field == "honorific_tails": + if field in ("honorific_tails", "suffix_words_ambiguous"): # honorific_tails ⊆ suffix_words is a Lexicon invariant, # so this entry cannot be added alone -- which is also why # the field was already excluded transitively, and why it # joins the roster for completeness rather than for a - # behavior change + # behavior change (suffix_words_ambiguous the same) extra["suffix_words"] = {"zzqx"} elif field == "given_name_titles": extra["titles"] = {"zzqx"} # ⊆ titles in practice @@ -1026,3 +1027,27 @@ def test_comma_follows_marker_reads_past_what_tokenize_drops( word: str, expected: bool) -> None: assert comma_follows_marker(word) is expected + + +def test_the_lean_reads_the_cue_of_what_the_member_is() -> None: + # an acronym's capitals are the credential, a generation + # word's lower case is the suffix and its capitals say nothing, and + # an agnomen's writing never leans; Title case leans to the name + # for both that lean at all + lex = Lexicon.default() + assert [lexicon_cue(w, lex) for w in ("ma", "fils", "neto")] == [ + "acronym", "word", "none"] + assert [ambiguous_lean(t, False, "word") for t in ("fils", "FILS", "Fils")] == [ + "credential", None, "name"] + assert [ambiguous_lean(t, False, "none") for t in ("neto", "NETO", "Neto")] == [ + None, None, None] + + +def test_a_comma_part_reads_a_gated_word_by_its_cue() -> None: + # is_wholly_suffix has no tags to read, so it asks lexicon_cue: a + # lower-case fils in a mixed-case name counts as a suffix, a + # Title-case one and an agnomen in any case do not + lex, pol = Lexicon.default(), Policy() + assert is_wholly_suffix(["fils"], lex, pol, one_case=False) + assert not is_wholly_suffix(["Fils"], lex, pol, one_case=False) + assert not is_wholly_suffix(["Neto"], lex, pol, one_case=False) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index a6409cab..3027a834 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -4,7 +4,7 @@ "C1": {"words": 448, "then_words": 196, "because_sentences": 1, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 116}, "C2": {"words": 492, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 1, "accepted_words": 81}, "C3": {"words": 260, "then_words": 228, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 173}, - "C4": {"words": 498, "then_words": 371, "because_sentences": 1, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, + "C4": {"words": 520, "then_words": 393, "because_sentences": 1, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, "C5": {"words": 319, "then_words": 79, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 54}, "C6": {"words": 207, "then_words": 162, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "C7": {"words": 119, "then_words": 68, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, @@ -12,7 +12,7 @@ "C9": {"words": 298, "then_words": 219, "because_sentences": 1, "except": 0, "precedence": 1, "accepted": 1, "accepted_words": 48}, "C10": {"words": 377, "then_words": 225, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 5, "accepted_words": 311}, "C11": {"words": 372, "then_words": 247, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "C12": {"words": 283, "then_words": 107, "because_sentences": 2, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 59}, + "C12": {"words": 316, "then_words": 107, "because_sentences": 2, "except": 2, "precedence": 1, "accepted": 2, "accepted_words": 59}, "C13": {"words": 259, "then_words": 74, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 0, "accepted_words": 0}, "C14": {"words": 231, "then_words": 111, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, "D1": {"words": 73, "then_words": 26, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, @@ -53,20 +53,21 @@ "R4": {"words": 991, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, - "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, + "S2": {"words": 657, "then_words": 119, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, "S3": {"words": 928, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 4, "accepted": 6, "accepted_words": 334}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, - "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, - "S7": {"words": 570, "then_words": 388, "because_sentences": 1, "except": 1, "precedence": 2, "accepted": 1, "accepted_words": 147}, + "S5": {"words": 665, "then_words": 350, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, + "S6": {"words": 419, "then_words": 203, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, + "S7": {"words": 602, "then_words": 388, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 147}, "S8": {"words": 381, "then_words": 267, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S9": {"words": 463, "then_words": 262, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 4, "accepted_words": 176}, + "S11": {"words": 474, "then_words": 266, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T4": {"words": 334, "then_words": 236, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 1, "accepted_words": 56}, - "T5": {"words": 149, "then_words": 95, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 42}, + "T5": {"words": 149, "then_words": 95, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 89}, "W1": {"words": 181, "then_words": 105, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, "W2": {"words": 99, "then_words": 56, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 192}, "W3": {"words": 651, "then_words": 348, "because_sentences": 1, "except": 1, "precedence": 3, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index 89aab5ce..3ca873e9 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -292,6 +292,8 @@ def test_a_thousand_names_still_parse_in_reasonable_time( "maiden_clause": "nee MA ", # group: M2's view over the segment "link_run": "i Und ", # group: P3's both-sides walk (#397) "credential_run": "PhD Ma ", # pieces: S7's anchor pass (#544) + "agnomens": "Neto ", # pieces: a gated word scans its front (#890) + "carried_agnomens": "MA ma Neto ", # pieces: the scan through carried members } _BASE = 800 @@ -1004,6 +1006,15 @@ def test_a_segment_with_no_free_connective_skips_the_join() -> None: lambda n, k: (n.family == "SMITH VD " + "DE " * k + "MA" and n.suffix.split() == ["MA"] * (k - 1)), ), + # #890: a gated generation word looks past what the walk takes in + # front of it, here a run of members a credential carries; a rescan + # per member was 15.6x in `is_suffix_piece` calls for 4x the input + # (3e035c5b, measured by the PR review, py3.11) + "agnomen_front": ( + lambda k: "Pedro PhD " + "Ma " * k + "Filho", "is_suffix_piece", + lambda n, k: (n.middle.split() == ["PhD"] + ["Ma"] * k + and n.family == "Filho"), + ), } diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 5939a7ed..59e90608 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -239,9 +239,16 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 'Dr. Smith VI, John' row, which reports since, each claim under # the three orders; 1445 -> 1451 the same day: #752's two # joined-capitals rows that report (phase 2d), one suffix-or-name - # claim under each order. - assert claims == 1451, ( - f"the field sweep checks {claims} claims, recorded as 1451 on " + # claim under each order; 1451 -> 1493 the same day: #752's + # generation-word rows (phase 2h, fils and the agnomens), the claims + # their reports make under the three orders; 1493 -> 1508 the same + # day: #890's five rows with a suffix word in front of an agnomen; + # 1508 -> 1514 the same day: its numeral and capitals rows; 1514 + # -> 1517 the same day: its carried-member rows; 1517 -> 1520 the + # same day: its joined-surname row; 1520 -> 1523 the same day: the + # floor's fils rows; 1523 -> 1529 the same day: its comma rows. + assert claims == 1529, ( + f"the field sweep checks {claims} claims, recorded as 1529 on " f"2026-10-25") diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py index 9f4cebc3..35ae04ab 100644 --- a/tests/v2/test_config_shim.py +++ b/tests/v2/test_config_shim.py @@ -570,6 +570,35 @@ def test_snapshot_removing_a_title_turns_its_after_given_reading_off() -> None: "", "Jane", "Mrs", "Doe") +def test_snapshot_removing_a_suffix_word_turns_its_gate_off() -> None: + # The deciding case for suffix_words_ambiguous (#752, rules/S11), + # which has no v1 manager of its own: the snapshot intersects it with + # the v1 word set, and Lexicon holds the subset, so deleting the word + # turns the reading off rather than leaving an orphan that gates a + # word the caller removed; agnomens rides inside it + c = Constants() + name = HumanName("Pedro Alves Neto", constants=c) + assert (name.last, name.suffix) == ("Alves", "Neto") # baseline + c.suffix_not_acronyms.remove("neto") + lexicon, _, _ = c._snapshot() + assert "neto" not in lexicon.suffix_words_ambiguous + assert "neto" not in lexicon.agnomens + name = HumanName("Pedro Alves Neto", constants=c) + assert (name.middle, name.last, name.suffix) == ("Alves", "Neto", "") + + +def test_snapshot_lets_a_gated_word_v1_reads_as_an_acronym_out_of_the_gate() -> None: + # v1 read a word added to suffix_acronyms as a suffix through the + # acronym branch; the snapshot takes it out of suffix_words_ambiguous + # rather than raising at Lexicon's word-gate check, which a config + # v1 accepted must never do + c = Constants() + c.suffix_acronyms.add("neto") + lexicon, _, _ = c._snapshot() + assert "neto" not in lexicon.suffix_words_ambiguous + assert HumanName("Pedro Alves Neto", constants=c).suffix == "Neto" + + def test_snapshot_field_translation() -> None: c = Constants() lexicon, policy, defaults = c._snapshot() diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index cbbfc494..1b39015c 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -418,6 +418,14 @@ def test_suffix_ambiguous_must_be_subset_of_acronyms() -> None: Lexicon(suffix_acronyms_ambiguous=frozenset({"ma"})) +def test_an_ambiguous_suffix_word_may_not_be_an_unambiguous_acronym() -> None: + # the mirror of the gate-bypass raise: the unambiguous acronym + # branch claims a word outright, so a gated suffix word listed there + # loses its count ('Ana Neto' would read suffix 'Neto', rules/S11) + with pytest.raises(ValueError, match="bypasses the count"): + Lexicon.default().add(suffix_acronyms={"neto"}) + + def test_the_shipped_suffix_sets_are_disjoint() -> None: # the vocabulary build refuses the overlap (tools/vocab/build.py, # run over the shipped files by tests/v2/test_vocab.py), but over the @@ -550,7 +558,10 @@ def test_given_name_titles_is_not_constrained_against_titles() -> None: "suffix_acronyms_ambiguous", "suffix_acronyms"), (lambda w: Lexicon(honorific_tails=w), "honorific_tails", "suffix_words"), -], ids=["particles_ambiguous", "suffix_acronyms_ambiguous", "honorific_tails"]) + (lambda w: Lexicon(suffix_words_ambiguous=w), + "suffix_words_ambiguous", "suffix_words"), +], ids=["particles_ambiguous", "suffix_acronyms_ambiguous", "honorific_tails", + "suffix_words_ambiguous"]) def test_subset_error_names_the_fix( make: Callable[[frozenset[str]], Lexicon], marker: str, base: str ) -> None: @@ -804,11 +815,12 @@ def test_multiword_entry_warns_per_field(field: str) -> None: # Guard the whole family: every per-word field warns, not one. with pytest.warns(UserWarning, match="matched one word at a time"): if field in ("particles_ambiguous", "suffix_acronyms_ambiguous", - "honorific_tails"): + "honorific_tails", "suffix_words_ambiguous"): # subset fields need their base populated to construct base = {"particles_ambiguous": "particles", "suffix_acronyms_ambiguous": "suffix_acronyms", - "honorific_tails": "suffix_words"}[field] + "honorific_tails": "suffix_words", + "suffix_words_ambiguous": "suffix_words"}[field] Lexicon.empty().add(**{base: ["zqx zqy"], field: ["zqx zqy"]}) else: Lexicon.empty().add(**{field: ["zqx zqy"]}) diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index e1ac63ed..6cc3cd47 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -1160,7 +1160,7 @@ def _fix_invariants(**fields: frozenset[str]) -> dict[str, frozenset[str]]: Drawing dependent subsets directly (particles_ambiguous from whatever particles happened to be drawn) makes the strategy tree deep and mostly rejects; intersecting after the fact keeps every - draw usable and still reaches every shape. The five rules are + draw usable and still reaches every shape. The seven rules are Lexicon's own, restated here on purpose -- if one changes, this fails loudly rather than silently fuzzing a narrower space. """ @@ -1171,6 +1171,9 @@ def _fix_invariants(**fields: frozenset[str]) -> dict[str, frozenset[str]]: # or that later subtraction could re-orphan a tail this repair just # fixed fields["honorific_tails"] &= fields["suffix_words"] + fields["suffix_words_ambiguous"] &= fields["suffix_words"] + fields["suffix_words_ambiguous"] -= ( + fields["suffix_acronyms"] - fields["suffix_acronyms_ambiguous"]) fields["bound_given_names"] -= ( fields["particles"] - fields["particles_ambiguous"]) return fields diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 25e0fe93..669c94f2 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1866,3 +1866,17 @@ def test_a_marked_particle_lowercases_only_where_the_particle_arm_would( hyphened = Parser(lexicon=Lexicon.default().add(particles={"van-der"})) assert str(hyphened.capitalized(hyphened.parse("jan van-der"))) == \ "Jan Van-Der" + + +@pytest.mark.parametrize("text", [ + "JEAN DUPONT FILS", "jean dupont fils", "DUMAS, ALEXANDRE FILS", + "MARIE FILS", "pedro alves filho", "PEDRO ALVES NETO"]) +def test_repairing_a_generation_word_that_is_also_a_name_keeps_its_reading( + text: str) -> None: + # rules/S6 reads a generation word's case: Title case leans `fils` + # to the name, so a suffix `fils` repairs to lower case, the one + # writing that keeps the reading, and a family `Fils` to Title case; + # an agnomen's case says nothing and it title-cases as any word does + name = parse(text) + repaired = name.capitalized() + assert parse(str(repaired)).as_dict() == repaired.as_dict() diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 26d4e7dd..69113a00 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -88,6 +88,8 @@ def test_the_compiled_sets_are_the_default_lexicon(shipped: dict[str, str]) -> N maiden_markers=c["MAIDEN_MARKERS"], surnames=lex.surnames, honorific_tails=c["GLUED_HONORIFICS"], after_given_titles=c["AFTER_GIVEN_TITLES"], + suffix_words_ambiguous=c["SUFFIX_WORDS_AMBIGUOUS"], + agnomens=c["AGNOMENS"], capitalization_exceptions=tuple(c["CAPITALIZATION_EXCEPTIONS"].items()), ) assert rebuilt == lex @@ -142,11 +144,44 @@ def test_an_exclusion_in_one_file_refuses_the_reading_in_another( files = dict(shipped) # appended to an ordered file, out of place: the reason still comes # first, check 10 running last - files["en.toml"] += '\n[[word]]\ntext = "fils"\ngeneration = {}\n' - with pytest.raises(build.VocabError, match="767 US Census"): + files["en.toml"] += '\n[[word]]\ntext = "père"\ngeneration = {}\n' + with pytest.raises(build.VocabError, match="collides outside French"): _compile(files) +def test_a_reading_off_by_default_compiles_to_nothing() -> None: + # recorded and checked, for a pack to turn on (phase 3): it feeds no + # set, marks nothing beside a name reading, and so cannot clash with + # the word's live readings in another file (checks 4 and 5) or be + # refused by an exclusion, which keeps a reading out of the default + # where this one already is (check 6) + off = build.compile_files({"es.toml": _w('[[word]]\ntext = "zzq"\ngeneration = { default = false }\nfamily = { note = "x" }')}) + assert not any("zzq" in v for v in off.values()) + both = build.compile_files({ + "mul.toml": _w('[[word]]\ntext = "zzq"\ngeneration = {}'), + "de.toml": _w('[[word]]\ntext = "zzq"\ncredential = { default = false }'), + "hr.toml": _w('[[word]]\ntext = "zzr"\nexcluded = { generation = "x" }'), + "cs.toml": _w('[[word]]\ntext = "zzr"\ngeneration = { default = false }'), + }) + assert "zzq" in both["SUFFIX_WORDS"] and "zzq" not in both["SUFFIX_ACRONYMS"] + assert not any("zzr" in v for v in both.values()) + + +def test_a_suffix_word_beside_a_name_reading_is_gated_and_an_agnomen_by_the_count() -> None: + # the derived marker for a suffix word (rules/S11): a generation + # beside a name reading is gated with the case cue, an agnomen with + # none; a word with no name reading marks nothing + c = build.compile_files({"fr.toml": _w( + '[[word]]\ntext = "zzq"\ngeneration = {}\nfamily = { note = "x" }\n\n' + '[[word]]\ntext = "zzr"\ngeneration = {}'), + "pt.toml": _w('[[word]]\ntext = "zzs"\nagnomen = {}\ngiven = { note = "x" }')}) + assert {"zzq", "zzr", "zzs"} <= c["SUFFIX_WORDS"] + assert c["SUFFIX_WORDS_AMBIGUOUS"] == {"zzq", "zzs"} + assert c["AGNOMENS"] == {"zzs"} + # only the kinds a gate reads; another word kind beside a name + # reading is refused (check 11), never left ungated + + # ------------------------------------------------- one mutation per check OK = 'schema = 1\n\n[[word]]\ntext = "jr"\ngeneration = {}\n' @@ -160,11 +195,11 @@ def _w(body: str) -> str: ("no-schema", {"und.toml": '[[word]]\ntext = "jr"\ngeneration = {}\n'}, "needs `schema = 1`"), ("newer-schema", {"und.toml": OK.replace("schema = 1", "schema = 2")}, "newer than this compiler"), ("unknown-top-level-key", {"und.toml": "language = 'en'\n" + OK}, "unknown key 'language'"), - ("reserved-top-level-key", {"und.toml": "default = false\n" + OK}, "reserved for phase 3"), + ("reserved-top-level-key", {"und.toml": "prefer = ['pt']\n" + OK}, "reserved for phase 3"), ("file-name-not-a-tag", {"english.toml": OK}, "BCP 47 tag"), ("file-name-not-canonical-case", {"pt-br.toml": OK}, "BCP 47 tag"), ("file-with-no-entries", {"en.toml": OK, "und.toml": "schema = 1\n"}, "no \\[\\[word\\]\\] entries"), - ("reserved-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { bare_only = true }')}, "reserved for phase 2"), + ("reserved-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { bare_only = true }')}, "reserved for phase 4"), ("unknown-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { acronymm = true }')}, "unknown key 'acronymm'"), ("unknown-word-key", {"und.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\nlang = "en"')}, "unknown key 'lang'"), ("descriptive-key-on-the-word", {"und.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\nnote = "x"')}, "belongs on a reading"), @@ -238,6 +273,19 @@ def _w(body: str) -> str: ("particle-kind-without-may-be-given", {"nl.toml": _w('[[word]]\ntext = "van"\npreposition = {}')}, "states may_be_given"), ("flag-not-a-bool", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = "yes" }')}, "true or false"), ("issue-not-a-number", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { issue = "661" }')}, "issue number"), + # `default`: a reading is on unwritten, so only false is worth + # writing; and the four names phase 2 left reserved say which phase + ("default-true", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = true }')}, "write default = false"), + ("default-not-a-bool", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = "no" }')}, "write default = false"), + ("reserved-attaches", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military"] }')}, "reserved for phase 2"), + ("reserved-separators", {"th.toml": _w('[[word]]\ntext = "นาย"\nsalutation = { separators = [] }')}, "reserved for phase 4"), + # 11, the agnomen's half: an agnomen is gated by its name reading, + # so one without would take the family name from two-word names + ("name-reading-beside-an-honorific", {"ja.toml": _w('[[word]]\ntext = "zzt"\nhonorific = {}\nfamily = { note = "x" }')}, "gates nothing"), + ("name-reading-beside-a-generic-suffix-word", {"en.toml": _w('[[word]]\ntext = "zzt"\nsuffix = {}\ngiven = { note = "x" }')}, "gates nothing"), + ("agnomen-as-an-acronym", {"pt.toml": _w('[[word]]\ntext = "filho"\nagnomen = { acronym = true }\nfamily = { note = "x" }')}, "an agnomen is a word"), + ("agnomen-without-a-name-reading", {"pt.toml": _w('[[word]]\ntext = "filho"\nagnomen = {}')}, "an agnomen needs a `given` or `family` reading"), + ("agnomen-whose-name-reading-is-elsewhere-only-as-an-exclusion", {"pt.toml": _w('[[word]]\ntext = "filho"\nagnomen = {}\nexcluded = { family = "x" }')}, "an agnomen needs a `given` or `family` reading"), ("decided-not-a-date", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { decided = "2026-10-18" }')}, "TOML date"), # tomllib reads a date-time as a datetime, which isinstance calls a date ("decided-a-date-time", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { decided = 2026-10-18T09:00:00 }')}, "not a string or a date-time"), @@ -841,13 +889,16 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: """The compiled constants a one-word file holding a `kind` reading with `props` puts the word into. A particle that does not state may_be_given gets a given reading, as check 11 requires, so the bare - kind feeds only the sets its category does, and a bare name reading - gets the note a count-less one carries.""" + kind feeds only the sets its category does; an agnomen gets the + family reading check 11 requires of it, which gates it; and a bare + name reading gets the note a count-less one carries.""" if build.CATEGORY.get(kind) == "name" and props == "{}": props = '{ note = "probe" }' entry = f'{kind} = {props}' if build.CATEGORY.get(kind) == "particle" and "may_be_given" not in props: entry = 'given = { note = "probe" }\n' + entry + if kind == "agnomen": + entry = 'family = { note = "probe" }\n' + entry files = {"en.toml": _w(f'[[word]]\ntext = "{text}"\n{entry}')} with warnings.catch_warnings(): warnings.simplefilter("ignore", build.VocabWarning) # a name reading is inert @@ -865,6 +916,7 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: "may_be_initial": ("conjunction", "{ may_be_initial = true }", "q"), "cased": ("title", '{ cased = "ZzQ" }', "zzq"), "after_given": ("salutation", "{ after_given = true }", "zzq"), + "default": ("generation", "{ default = false }", "zzq"), } @@ -940,6 +992,7 @@ def test_the_readme_states_each_behavioral_propertys_unwritten_value() -> None: #: and without a given reading -- what moves is the marker's effect. _MARKER_PROBES = { "possibly a name": ("credential", "{}", "{}"), + "possibly a name, a word": ("generation", "{}", "{}"), "possibly given": ("preposition", "{ may_be_given = false }", "{}"), } diff --git a/tools/differential/cldr_agreement.json b/tools/differential/cldr_agreement.json index 20674499..f96e7807 100644 --- a/tools/differential/cldr_agreement.json +++ b/tools/differential/cldr_agreement.json @@ -3119,22 +3119,10 @@ "text": "Asmar" }, "fr/nativeFull/givenFirst": { - "family": { - "expected": "de Bouchart", - "got": "de Bouchart fils" - }, - "family_base": { - "expected": "Bouchart", - "got": "Bouchart fils" - }, "given_names": { "expected": "Jean-Nicolas Louis Marcel", "got": "M. Jean-Nicolas Louis Marcel" }, - "suffix": { - "expected": "fils", - "got": "" - }, "text": "M. Jean-Nicolas Louis Marcel de Bouchart fils", "title": { "expected": "M.", @@ -3156,11 +3144,7 @@ }, "given_names": { "expected": "Jean-Nicolas Louis Marcel", - "got": "de Bouchart Jean-Nicolas Louis Marcel fils" - }, - "suffix": { - "expected": "fils", - "got": "" + "got": "de Bouchart Jean-Nicolas Louis Marcel" }, "text": "M. de Bouchart Jean-Nicolas Louis Marcel fils", "title": { diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 1cb14d50..559d2f29 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -5,6 +5,7 @@ "'Smitty' Sir John" "(" ", Ana García López" +", Dr. fils" ", Jane" ", Jane Mrs" ", MD/PhD" @@ -16,9 +17,12 @@ "Abu Bakar" "Abu Bakar Salim" "Ahmad al-ʿAbbas" +"Alexandre Dumas fils" "Ali Ahmad Vali oglu" "Ali Ahmad Vali oglu Jr." "Ali Ahmad oglu" +"Alves, Pedro Fils" +"Alves, Pedro Neto" "Ana García" "Ana García (Anita) López" "Ana J.R. García" @@ -32,6 +36,7 @@ "Ana María de la Vega Rodríguez" "Ana María van der Berg" "Ana María-José López" +"Ana Neto" "Ana abdul Rahman López" "Andrew" "Andrew (Andy) Perkins" @@ -149,6 +154,10 @@ "Dr. nee Smith PhD Prof." "Dr., John" "Duke of Edinburgh" +"Dumas fils" +"Dumas, Alexandre Fils" +"Dumas, Alexandre fils" +"Dumas, fils" "Elizabeth II, Queen" "Eric H. Holder Jr. Attorney General" "Eric H. Holder, Jr., Attorney General" @@ -285,6 +294,7 @@ "Jane „JD Smith" "Jean 'JD' Smith" "Jean DUPONT" +"Jean Marie Fils" "Jean Pierre DUPONT" "Jean Pierre DUPONT-MORETTI" "Jiří Dienstbier ml." @@ -472,6 +482,7 @@ "Juan & Garcia" "Juan Carlos Pérez" "Juan McDonald" +"Juan Pérez H." "Juan Pérez de la Vega" "Juan Velasquez y Garcia" "Juan and Garcia" @@ -507,6 +518,7 @@ "Maria Kowalska z domu Nowak" "Maria Luisa y de la Cruz" "Maria López née García Pérez" +"Marie FILS" "Marie Fils" "Marquess of Bath" "Mary Beth Smith" @@ -537,8 +549,29 @@ "Nguyễn, Thị Vân" "Ortega y Gasset, Dr." "Ortega y Gasset, Ed" +"PEDRO DA SILVA FILHO" +"PEDRO FILHO" +"PEDRO JUNIOR FILHO" "Pavlović, Antonio mlađi" "Pedro Alves Filho" +"Pedro Alves Junior Filho" +"Pedro Alves M.Filho" +"Pedro Alves NETO" +"Pedro Alves Neto" +"Pedro Alves Neto Júnior" +"Pedro Alves Neto, AD" +"Pedro Alves Neto, PhD" +"Pedro Alves, Neto" +"Pedro Filho" +"Pedro J. Neto" +"Pedro Junior Filho" +"Pedro Junior Ma Filho" +"Pedro Junior fils" +"Pedro M.Filho" +"Pedro MA Filho" +"Pedro PhD Neto" +"Pedro VI Filho" +"Pedro de Neto" "Petrov Ivan Sergeevich" "Ph. D. Van Johnson" "Ph. D., John" @@ -561,6 +594,7 @@ "Rev. Sr. Smith, Mary" "Rinpoche" "Royce, Ed" +"Ruth Ann Neta" "SEÁN Ó MURCHÚ" "SHIRLEY MACLAINE" "SIG.RA MARIA ROSSI" @@ -579,6 +613,8 @@ "Shin Ae-ra" "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." +"Silva Neto, Pedro" +"Silva, Neto" "Sir John" "Sir John Prof." "Sir Ph. D. Van Johnson" @@ -688,6 +724,7 @@ "Vincent van Gogh van Beethoven" "Wang M.Eng." "Wang Ma PhD" +"Wilhelm I." "Xyz. John Smith" "Xyz. Smith, John" "Xyz. van Johnson" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index c4132974..3e02ed35 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -10,6 +10,7 @@ {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "김. 민. J. S."} {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} +{"name": ", Dr. fils", "title": "Dr.", "family": "fils", "initials": "f.", "capitalized_force": "Dr. Fils"} {"name": ", Jane", "given": "Jane", "initials": "J."} {"name": ", Jane Mrs", "title": "Mrs", "family": "Jane", "initials": "J."} {"name": ", John", "given": "John", "initials": "J."} @@ -119,6 +120,7 @@ {"name": "Alex Ben Johnson", "given": "Alex", "middle": "Ben", "family": "Johnson", "initials": "A. B. J."} {"name": "Alex van Johnson", "given": "Alex", "family": "van Johnson", "initials": "A. J."} {"name": "Alexander Parra Uribe", "given": "Alexander", "middle": "Parra", "family": "Uribe", "initials": "A. P. U."} +{"name": "Alexandre Dumas fils", "given": "Alexandre", "family": "Dumas", "suffix": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. D."} {"name": "Alexandru Ioan Popa", "given": "Alexandru", "middle": "Ioan", "family": "Popa", "initials": "A. I. P."} {"name": "Alexis", "given": "Alexis", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "A."} {"name": "Alfonso Reyes Echandía", "given": "Alfonso", "middle": "Reyes", "family": "Echandía", "initials": "A. R. E."} @@ -143,6 +145,8 @@ {"name": "Alonso de Cardenas", "given": "Alonso", "family": "de Cardenas", "initials": "A. C."} {"name": "Alquen, Gunter d'", "given": "Gunter", "family": "d' Alquen", "initials": "G. A."} {"name": "Alvensleben, Albrecht von", "given": "Albrecht", "family": "von Alvensleben", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "A. A."} +{"name": "Alves, Pedro Fils", "given": "Pedro", "middle": "Fils", "family": "Alves", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. F. A."} +{"name": "Alves, Pedro Neto", "given": "Pedro", "family": "Alves", "suffix": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} {"name": "Alzon, Emmanuel d'", "given": "Emmanuel", "family": "d' Alzon", "initials": "E. A."} {"name": "Amadi John Chinedu", "order": "FAMILY_FIRST", "given": "John", "middle": "Chinedu", "family": "Amadi", "initials": "J. C. A."} {"name": "Amadi Maazị Christopher Chukwudike Jr, MP", "order": "FAMILY_FIRST", "given": "Maazị", "middle": "Christopher Chukwudike", "family": "Amadi", "suffix": "Jr, MP", "initials": "M. C. C. A."} @@ -173,6 +177,7 @@ {"name": "Ana María de la Vega Rodríguez", "given": "Ana", "middle": "María", "family": "de la Vega Rodríguez", "initials": "A. M. V. R."} {"name": "Ana María van der Berg", "given": "Ana", "middle": "María", "family": "van der Berg", "initials": "A. M. B."} {"name": "Ana María-José López", "given": "Ana", "middle": "María-José", "family": "López", "initials": "A. M. L."} +{"name": "Ana Neto", "given": "Ana", "family": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. N."} {"name": "Ana Torres", "given": "Ana", "family": "Torres", "initials": "A. T."} {"name": "Ana abdul Rahman López", "given": "Ana", "middle": "abdul Rahman", "family": "López", "initials": "A. a. R. L.", "capitalized_force": "Ana Abdul Rahman López"} {"name": "Anan ben David", "given": "Anan", "middle": "ben", "family": "David", "initials": "A. b. D.", "capitalized_force": "Anan Ben David"} @@ -815,6 +820,10 @@ {"name": "Duke Snider", "given": "Duke", "family": "Snider", "initials": "D. S."} {"name": "Duke of Edinburgh", "given": "Duke of Edinburgh", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "D. E."} {"name": "Duke of Wellington", "given": "Duke of Wellington", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "D. W."} +{"name": "Dumas fils", "given": "Dumas", "family": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "D. f.", "capitalized_force": "Dumas Fils"} +{"name": "Dumas, Alexandre Fils", "given": "Alexandre", "middle": "Fils", "family": "Dumas", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. F. D."} +{"name": "Dumas, Alexandre fils", "given": "Alexandre", "family": "Dumas", "suffix": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. D."} +{"name": "Dumas, fils", "given": "fils", "family": "Dumas", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "f. D.", "capitalized_force": "Fils Dumas"} {"name": "Dumreicher, Johann von", "given": "Johann", "family": "von Dumreicher", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. D."} {"name": "Dut.ra Prof.ra Maria Giusepa Juana Teresa von Brühl González Domingo", "title": "Dut.ra Prof.ra", "given": "Maria", "middle": "Giusepa Juana Teresa", "family": "von Brühl González Domingo", "initials": "M. G. J. T. B. G. D.", "capitalized_force": "Dut.Ra Prof.Ra Maria Giusepa Juana Teresa von Brühl González Domingo"} {"name": "Dòmhnall Iain Tormod Caimbeul", "given": "Dòmhnall", "middle": "Iain Tormod", "family": "Caimbeul", "initials": "D. I. T. C."} @@ -1485,6 +1494,7 @@ {"name": "Jean DUPONT", "given": "Jean", "family": "DUPONT", "initials": "J. D.", "capitalized_force": "Jean Dupont"} {"name": "Jean Joseph Antoine Marie de Witte", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Jean Louis Charles d'Orléans-Longueville", "given": "Jean", "middle": "Louis Charles", "family": "d'Orléans-Longueville", "initials": "J. L. C. d.", "capitalized_force": "Jean Louis Charles D'Orléans-Longueville"} +{"name": "Jean Marie Fils", "given": "Jean", "middle": "Marie", "family": "Fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M. F."} {"name": "Jean Pierre DUPONT", "given": "Jean", "middle": "Pierre", "family": "DUPONT", "initials": "J. P. D.", "capitalized_force": "Jean Pierre Dupont"} {"name": "Jean Pierre DUPONT-MORETTI", "given": "Jean", "middle": "Pierre", "family": "DUPONT-MORETTI", "initials": "J. P. D.", "capitalized_force": "Jean Pierre Dupont-Moretti"} {"name": "Jean St Clair", "given": "Jean", "family": "St Clair", "initials": "J. C.", "capitalized_force": "Jean st Clair"} @@ -1902,6 +1912,7 @@ {"name": "Juan Garcia y Lopez y", "given": "Juan", "family": "Garcia y Lopez y", "initials": "J. G. L."} {"name": "Juan Harguindey Broussain", "given": "Juan", "middle": "Harguindey", "family": "Broussain", "initials": "J. H. B."} {"name": "Juan McDonald", "given": "Juan", "family": "McDonald", "initials": "J. M."} +{"name": "Juan Pérez H.", "given": "Juan", "middle": "Pérez", "family": "H.", "initials": "J. P. H."} {"name": "Juan Pérez de la Vega", "given": "Juan", "middle": "Pérez", "family": "de la Vega", "initials": "J. P. V."} {"name": "Juan Q. Velasquez y Garcia", "given": "Juan", "middle": "Q.", "family": "Velasquez y Garcia", "initials": "J. Q. V. G."} {"name": "Juan Q. Velasquez y Garcia III", "given": "Juan", "middle": "Q.", "family": "Velasquez y Garcia", "suffix": "III", "initials": "J. Q. V. G."} @@ -2205,8 +2216,8 @@ {"name": "Løseth, Kjetil Bjørn", "order": "FAMILY_FIRST", "given": "Kjetil", "middle": "Bjørn", "family": "Løseth", "initials": "K. B. L."} {"name": "Lúcio de Mendonça", "given": "Lúcio", "family": "de Mendonça", "initials": "L. M."} {"name": "Lžedmitrij II.", "given": "Lžedmitrij", "suffix": "II.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "L."} -{"name": "M. Jean-Nicolas Louis Marcel de Bouchart fils", "given": "M.", "middle": "Jean-Nicolas Louis Marcel", "family": "de Bouchart fils", "initials": "M. J. L. M. B. f.", "capitalized_force": "M. Jean-Nicolas Louis Marcel de Bouchart Fils"} -{"name": "M. de Bouchart Jean-Nicolas Louis Marcel fils", "order": "FAMILY_FIRST", "given": "de Bouchart Jean-Nicolas Louis Marcel fils", "family": "M.", "initials": "d. B. J. L. M. f. M.", "capitalized_force": "De Bouchart Jean-Nicolas Louis Marcel Fils M."} +{"name": "M. Jean-Nicolas Louis Marcel de Bouchart fils", "given": "M.", "middle": "Jean-Nicolas Louis Marcel", "family": "de Bouchart", "suffix": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. J. L. M. B."} +{"name": "M. de Bouchart Jean-Nicolas Louis Marcel fils", "order": "FAMILY_FIRST", "given": "de Bouchart Jean-Nicolas Louis Marcel", "family": "M.", "suffix": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "d. B. J. L. M. M.", "capitalized_force": "De Bouchart Jean-Nicolas Louis Marcel M. fils"} {"name": "MANƉELA Rolanɖ Alfɔns", "order": "FAMILY_FIRST", "given": "Rolanɖ", "middle": "Alfɔns", "family": "MANƉELA", "initials": "R. A. M.", "capitalized_force": "Rolanɖ Alfɔns Manɖela"} {"name": "MD - DO - DDS", "given": "MD", "middle": "-", "family": "DO -", "suffix": "DDS", "initials": "M. -. -.", "capitalized": "Md - do - DDS"} {"name": "MD Chirurgie dentară von der Ruhr Prof. dr. Eva-Eleonora Karl Hans Jr.", "order": "FAMILY_FIRST", "given": "Chirurgie", "middle": "dentară von der Ruhr Prof. dr. Eva-Eleonora Karl Hans", "family": "MD", "suffix": "Jr.", "initials": "C. d. R. P. d. E. K. H. M.", "capitalized_force": "Chirurgie Dentară von der Ruhr Prof. Dr. Eva-Eleonora Karl Hans Md Jr."} @@ -2307,7 +2318,8 @@ {"name": "Mariano Cuadra Medina", "given": "Mariano", "middle": "Cuadra", "family": "Medina", "initials": "M. C. M."} {"name": "Mariano Ospina Pérez", "given": "Mariano", "middle": "Ospina", "family": "Pérez", "initials": "M. O. P."} {"name": "Mariano Salvador de Maella", "given": "Mariano", "middle": "Salvador", "family": "de Maella", "initials": "M. S. M."} -{"name": "Marie Fils", "given": "Marie", "family": "Fils", "initials": "M. F."} +{"name": "Marie FILS", "given": "Marie", "family": "FILS", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. F.", "capitalized_force": "Marie Fils"} +{"name": "Marie Fils", "given": "Marie", "family": "Fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. F."} {"name": "Marie Laure de Noailles", "given": "Marie", "middle": "Laure", "family": "de Noailles", "initials": "M. L. N."} {"name": "Marie Louise Fernande de Montpensier", "given": "Marie", "middle": "Louise Fernande", "family": "de Montpensier", "initials": "M. L. F. M."} {"name": "Marie-Agnès Suzanne Gilot", "given": "Marie-Agnès", "middle": "Suzanne", "family": "Gilot", "initials": "M. S. G."} @@ -2597,6 +2609,9 @@ {"name": "Ozoliņš Pēteris", "order": "FAMILY_FIRST", "given": "Pēteris", "family": "Ozoliņš", "initials": "P. O."} {"name": "Ozoliņš Pēteris Kārlis", "order": "FAMILY_FIRST", "given": "Pēteris", "middle": "Kārlis", "family": "Ozoliņš", "initials": "P. K. O."} {"name": "P. V.", "given": "P.", "family": "V.", "initials": "P. V."} +{"name": "PEDRO DA SILVA FILHO", "given": "PEDRO", "family": "DA SILVA", "suffix": "FILHO", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. S.", "capitalized": "Pedro da Silva Filho"} +{"name": "PEDRO FILHO", "given": "PEDRO", "family": "FILHO", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. F.", "capitalized": "Pedro Filho"} +{"name": "PEDRO JUNIOR FILHO", "given": "PEDRO", "middle": "JUNIOR", "family": "FILHO", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. J. F.", "capitalized": "Pedro Junior Filho"} {"name": "Pablo Emilio Riveros Reina", "given": "Pablo", "middle": "Emilio Riveros", "family": "Reina", "initials": "P. E. R. R."} {"name": "Pablo Pérez Miguel Ángel Juan Antonio II", "order": "FAMILY_FIRST", "given": "Pérez", "middle": "Miguel Ángel Juan Antonio", "family": "Pablo", "suffix": "II", "initials": "P. M. Á. J. A. P."} {"name": "Pablo Pérez, Sr. Miguel Ángel Juan Antonio", "title": "Sr.", "given": "Miguel", "middle": "Ángel Juan Antonio", "family": "Pablo Pérez", "initials": "M. Á. J. A. P. P."} @@ -2614,13 +2629,31 @@ {"name": "Pavlović, Antonio mlađi", "given": "Antonio", "family": "Pavlović", "suffix": "mlađi", "initials": "A. P.", "capitalized_force": "Antonio Pavlović Mlađi"} {"name": "Pedro", "given": "Pedro", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "P."} {"name": "Pedro", "order": "FAMILY_FIRST", "family": "Pedro", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "P."} -{"name": "Pedro Alves Filho", "given": "Pedro", "middle": "Alves", "family": "Filho", "initials": "P. A. F."} +{"name": "Pedro Alves Filho", "given": "Pedro", "family": "Alves", "suffix": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} +{"name": "Pedro Alves Junior Filho", "given": "Pedro", "family": "Alves", "suffix": "Junior Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} +{"name": "Pedro Alves M.Filho", "given": "Pedro", "family": "Alves", "suffix": "M.Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A.", "capitalized_force": "Pedro Alves M.FILHO"} +{"name": "Pedro Alves NETO", "given": "Pedro", "family": "Alves", "suffix": "NETO", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A.", "capitalized_force": "Pedro Alves Neto"} +{"name": "Pedro Alves Neto", "given": "Pedro", "family": "Alves", "suffix": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} +{"name": "Pedro Alves Neto Júnior", "given": "Pedro", "family": "Alves", "suffix": "Neto Júnior", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} +{"name": "Pedro Alves Neto, AD", "given": "AD", "family": "Pedro Alves Neto", "initials": "A. P. A. N.", "capitalized_force": "Ad Pedro Alves Neto"} +{"name": "Pedro Alves Neto, PhD", "given": "Pedro", "family": "Alves", "suffix": "Neto, PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} +{"name": "Pedro Alves, Neto", "given": "Pedro", "family": "Alves", "suffix": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. A."} {"name": "Pedro Cañas Serrano", "given": "Pedro", "middle": "Cañas", "family": "Serrano", "initials": "P. C. S."} +{"name": "Pedro Filho", "given": "Pedro", "family": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. F."} {"name": "Pedro Galindo Galarza", "given": "Pedro", "middle": "Galindo", "family": "Galarza", "initials": "P. G. G."} {"name": "Pedro Infante Torrentera", "given": "Pedro", "middle": "Infante", "family": "Torrentera", "initials": "P. I. T."} +{"name": "Pedro J. Neto", "given": "Pedro", "family": "J.", "suffix": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. J."} {"name": "Pedro Jorba y Gassó", "given": "Pedro", "family": "Jorba y Gassó", "initials": "P. J. G."} +{"name": "Pedro Junior Filho", "given": "Pedro", "middle": "Junior", "family": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. J. F."} +{"name": "Pedro Junior Ma Filho", "given": "Pedro", "middle": "Junior Ma", "family": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. J. M. F."} +{"name": "Pedro Junior fils", "given": "Pedro", "middle": "Junior", "family": "fils", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. J. f.", "capitalized_force": "Pedro Junior Fils"} {"name": "Pedro López Ruiz H.", "given": "Pedro", "middle": "López Ruiz", "family": "H.", "initials": "P. L. R. H."} +{"name": "Pedro M.Filho", "given": "Pedro", "family": "M.Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. M."} +{"name": "Pedro MA Filho", "given": "Pedro", "middle": "MA", "family": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. M. F.", "capitalized_force": "Pedro Ma Filho"} {"name": "Pedro Morales Pino", "given": "Pedro", "middle": "Morales", "family": "Pino", "initials": "P. M. P."} +{"name": "Pedro PhD Neto", "given": "Pedro", "middle": "PhD", "family": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. P. N."} +{"name": "Pedro VI Filho", "given": "Pedro", "middle": "VI", "family": "Filho", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. V. F.", "capitalized_force": "Pedro Vi Filho"} +{"name": "Pedro de Neto", "given": "Pedro", "family": "de", "suffix": "Neto", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "P. d.", "capitalized_force": "Pedro De Neto"} {"name": "Pendeta Zaba", "given": "Pendeta", "family": "Zaba", "initials": "P. Z."} {"name": "Penkauskienė Sofija Rita", "order": "FAMILY_FIRST", "given": "Sofija", "middle": "Rita", "family": "Penkauskienė", "initials": "S. R. P."} {"name": "Penkauskienė, Sofija Rita", "given": "Sofija", "middle": "Rita", "family": "Penkauskienė", "initials": "S. R. P."} @@ -2816,6 +2849,7 @@ {"name": "Ruiz, Rosa María", "given": "Rosa", "middle": "María", "family": "Ruiz", "initials": "R. M. R."} {"name": "Rustemi, Ilir", "given": "Ilir", "family": "Rustemi", "initials": "I. R."} {"name": "Rustemi, Ilir I.", "order": "FAMILY_FIRST", "given": "Ilir", "middle": "I.", "family": "Rustemi", "initials": "I. I. R."} +{"name": "Ruth Ann Neta", "given": "Ruth", "family": "Ann", "suffix": "Neta", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "R. A."} {"name": "Ryan, Dermot", "given": "Dermot", "family": "Ryan", "initials": "D. R."} {"name": "Róbert Gunnarsson", "given": "Róbert", "family": "Gunnarsson", "initials": "R. G."} {"name": "Rómulo González Irigoyen", "given": "Rómulo", "middle": "González", "family": "Irigoyen", "initials": "R. G. I."} @@ -2944,9 +2978,11 @@ {"name": "Sigurður Ólafsson", "given": "Sigurður", "family": "Ólafsson", "initials": "S. Ó."} {"name": "Silva João Pedro", "order": "FAMILY_FIRST", "given": "João", "middle": "Pedro", "family": "Silva", "initials": "J. P. S."} {"name": "Silva Maria", "order": "FAMILY_FIRST", "given": "Maria", "family": "Silva", "initials": "M. S."} +{"name": "Silva Neto, Pedro", "given": "Pedro", "family": "Silva Neto", "initials": "P. S. N."} {"name": "Silva, José Pedro Eduardo da", "given": "José", "middle": "Pedro Eduardo", "family": "da Silva", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. P. E. S."} {"name": "Silva, João Pedro", "given": "João", "middle": "Pedro", "family": "Silva", "initials": "J. P. S."} {"name": "Silva, Maria", "given": "Maria", "family": "Silva", "initials": "M. S."} +{"name": "Silva, Neto", "given": "Neto", "family": "Silva", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "N. S."} {"name": "Silverio Cañada Acebal", "given": "Silverio", "middle": "Cañada", "family": "Acebal", "initials": "S. C. A."} {"name": "Silverio Franconetti y Aguilar", "given": "Silverio", "family": "Franconetti y Aguilar", "initials": "S. F. A."} {"name": "Silvije Strahimir Kranjčević", "given": "Silvije", "middle": "Strahimir", "family": "Kranjčević", "initials": "S. S. K."} @@ -3384,6 +3420,7 @@ {"name": "Wijaya, Maria", "given": "Maria", "family": "Wijaya", "initials": "M. W."} {"name": "Wijaya, Maria", "order": "FAMILY_FIRST", "given": "Maria", "family": "Wijaya", "initials": "M. W."} {"name": "Wildenbruch, Ernst von", "given": "Ernst", "family": "von Wildenbruch", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "E. W."} +{"name": "Wilhelm I.", "given": "Wilhelm", "family": "I.", "initials": "W. I."} {"name": "Wilhelm IV.", "given": "Wilhelm", "suffix": "IV.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "W."} {"name": "Wilkins, Charles", "given": "Charles", "family": "Wilkins", "initials": "C. W."} {"name": "Will De Groot", "given": "Will", "family": "De Groot", "initials": "W. G.", "capitalized_force": "Will de Groot"} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index a0f27dd0..5047e01f 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -57,17 +57,21 @@ 10. no same_as chain loops, in any file, and the words of every file but und.toml in canonical order (_order_key): by kind, then by the word a spelling is the same_as of, then by text; -11. a clash with a name is stated as a name reading, from which the two +11. a clash with a name is stated as a name reading, from which the markers are derived (decisions/vocabulary-model, 2026-10-25): `may_be_name` and `may_be_given = true` are refused, each naming - the reading to write, and a particle reading states + the reading to write, a particle reading states `may_be_given = false` exactly where its word has no `given` - reading in any file; + reading in any file, an agnomen reading has a name reading on + its word, which gates it (rules/S11), and no other suffix word + kind has one, since the gate reads only those two; numbered by subject, not by when they run: 1-3, 8, 7's case mask and 11's refusals run per entry as each file loads, and 4-7, 9, 11's -particle statement and 10 over the whole pool in check_pool, 6 first -and 10 last; +particle statement and agnomen and 10 over the whole pool in +check_pool, 6 first and 10 last. A reading written `default = false` +is held to 1-3, 7, 8 and 11 like any other and compiles to nothing, so +4, 5, 6 and 9 do not weigh it; plus the data invariants that were asserts: may_be_initial only on a cased single letter, an elided particle in both apostrophes (#700), and @@ -182,6 +186,7 @@ def _alnum(text: str) -> str: "modifier": "title", "suffix": "suffix", "generation": "suffix", + "agnomen": "suffix", "credential": "suffix", "honorific": "suffix", "professional_honorific": "suffix", @@ -219,12 +224,12 @@ def _alnum(text: str) -> str: #: Behavioral properties per category: each compiles into a field, and #: its allowed values are exactly what today's pipeline does. BEHAVIORAL: dict[str, frozenset[str]] = { - "title": frozenset({"addresses", "after_given", "cased"}), - "suffix": frozenset({"acronym", "attached", "cased"}), - "particle": frozenset({"cased"}), - "conjunction": frozenset({"may_be_initial", "cased"}), - "bound_given": frozenset({"cased"}), - "relation": frozenset({"cased"}), + "title": frozenset({"addresses", "after_given", "cased", "default"}), + "suffix": frozenset({"acronym", "attached", "cased", "default"}), + "particle": frozenset({"cased", "default"}), + "conjunction": frozenset({"may_be_initial", "cased", "default"}), + "bound_given": frozenset({"cased", "default"}), + "relation": frozenset({"cased", "default"}), } #: Stated properties per category: compiled to nothing, but held to the #: word's readings (check 11). A particle states `may_be_given = false` @@ -239,6 +244,10 @@ def _alnum(text: str) -> str: "may_be_name": "a `given` or `family` reading on the word", "may_be_given": "a `given` reading on the word", } +#: The suffix word kinds a name reading gates (rules/S11): the third +#: derived marker reads these, and a name reading beside another word +#: kind marks nothing, no gate reading that clash yet. +GATED_WORDS = frozenset({"generation", "agnomen"}) #: Descriptive properties: recorded, validated, compiled to nothing. DESCRIPTIVE = frozenset({"note", "issue", "decided", "expands", "same_as", "native", "gender"}) @@ -250,10 +259,13 @@ def _alnum(text: str) -> str: ATTACHED_KEYS = frozenset({"side", "issue", "decided", "note"}) #: Names a later phase will give meaning to. Writing one now is an error #: that says so, rather than "unknown", so nothing reads as working. +#: Phase 2 gives `default` its meaning and leaves `attaches` to its +#: front-of-the-name work; the gate's slots carry position, and shape +#: settles a bare letter (rules/P7), so the position and bare-letter +#: names wait with `separators` for phase 4's shapes and glued words. RESERVED = { - "default": 3, "prefer": 3, "strict": 3, - "bare_only": 2, "leading_only": 2, "trailing_only": 2, - "attaches": 2, "separators": 2, + "attaches": 2, "prefer": 3, "strict": 3, + "bare_only": 4, "leading_only": 4, "trailing_only": 4, "separators": 4, } _TAG = re.compile(r"[a-z]{2,3}(-[A-Z][a-z]{3})?(-([A-Z]{2}|[0-9]{3}))?") @@ -286,6 +298,14 @@ def acronym(self) -> bool: credential.""" return bool(self.props.get("acronym", self.kind == "credential")) + @property + def on_by_default(self) -> bool: + """Compiled into the default vocabulary: unwritten, true. A + reading written `default = false` is recorded and checked as a + reading but compiles to nothing, for a pack to turn on in + phase 3 (decisions/vocabulary-model).""" + return self.props.get("default", True) is not False + def flag(self, prop: str) -> bool: """A `may_be_*` flag or `after_given`: unwritten, false.""" return bool(self.props.get(prop, False)) @@ -347,7 +367,7 @@ class Pool: def _reserved_or_unknown(where: str, key: str, allowed: Iterable[str]) -> VocabError: if key in RESERVED: return VocabError(f"{where}: {key!r} is reserved for phase {RESERVED[key]}; " - f"phase 1 gives it no behavior") + f"it has no behavior yet") return VocabError(f"{where}: unknown key {key!r}; allowed: {sorted(allowed)}") @@ -765,6 +785,12 @@ def _check_reading(r: Reading, pool: Pool) -> None: for flag in ("acronym", "may_be_initial", "after_given"): if flag in p and not isinstance(p[flag], bool): raise VocabError(f"{where}: {flag} must be true or false") + if "default" in p and p["default"] is not False: + # every reading is on by default unwritten, so the one value + # worth writing is the one that turns it off + raise VocabError(f"{where}: default = {_shown(p['default'])}; write " + f"default = false for a reading a pack turns on, " + f"or leave it out") if r.category == "title": addresses = r.addresses if addresses in ("full", "any"): @@ -781,6 +807,12 @@ def _check_reading(r: Reading, pool: Pool) -> None: f"(rules/H8)") if r.category == "suffix": acronym = r.acronym + if r.kind == "agnomen" and acronym: + # an agnomen is a word whose writing never speaks (rules/S11); + # as an acronym it would compile into the credential class, + # its capitals leaning to the credential + raise VocabError(f"{where}: an agnomen is a word; acronym = " + f"true is refused") if acronym and "." in r.text: # suffix_as_written looks an acronym up with EVERY period # removed, so a dotted entry is never matched -- and it slips @@ -894,6 +926,14 @@ def check_pool(pool: Pool) -> None: by_text: dict[str, list[Reading]] = {} for r in pool.readings: by_text.setdefault(r.text, []).append(r) + # a reading written `default = false` compiles to nothing, so it + # cannot read two ways beside the word's other readings, and no + # exclusion refuses it: an exclusion keeps a reading out of the + # default, which is where this one already is (checks 4, 5, 6, 9). + # It is still a reading: check 3 has held its properties, and 7 and + # 11 read it with the rest + live: dict[str, list[Reading]] = { + t: [r for r in rs if r.on_by_default] for t, rs in by_text.items()} # 6: an excluded kind is absent from the whole pool, and an excluded # property never takes a value other than the one it has unwritten on @@ -905,7 +945,7 @@ def check_pool(pool: Pool) -> None: # by check 4 anyway, but without the reason), and `cased`, compiled # once per word whatever the reading, over every reading of the word for x in pool.exclusions: - for r in by_text.get(x.text, ()): + for r in live.get(x.text, ()): if x.prop is None and _refused_by(r, x.kind): raise VocabError(f"{r.where()}: {x.kind} is excluded for " f"{x.text!r} in {x.file}: {x.reason}") @@ -919,7 +959,7 @@ def check_pool(pool: Pool) -> None: f"{x.text!r} in {x.file}: {x.reason}") # 4: one suffix kind per word, across files - for text, rs in by_text.items(): + for text, rs in live.items(): kinds = sorted({r.kind for r in rs if r.category == "suffix"}) if len(kinds) > 1: raise VocabError(f"{text!r}: one suffix kind per word, got {kinds} " @@ -932,7 +972,7 @@ def check_pool(pool: Pool) -> None: # particle may take a different kind in each language (`de`, a # preposition in French and an article in Dutch), and two kinds # disagreeing on a property would compile silently to one of them - for text, rs in by_text.items(): + for text, rs in live.items(): first: dict[str, Reading] = {} for r in rs: if r.category == "name": @@ -982,10 +1022,36 @@ def check_pool(pool: Pool) -> None: f"under uncertainty, the given reading " f"(vocabulary-collisions C-i)") + # 11, the agnomen's half: an agnomen is gated by the count alone, and + # the gate is derived from a name reading, so one without a name + # reading would read as the suffix wherever it ends a name, taking + # the family name from every two-word name ('Maria Filha') + named = {t for t, rs in by_text.items() if any(r.category == "name" for r in rs)} + for r in pool.readings: + if r.kind == "agnomen" and r.text not in named: + raise VocabError(f"{r.where()}: an agnomen needs a `given` or " + f"`family` reading on its word, which gates it " + f"by the count (rules/S11); with dated counts " + f"from tools/name_usage.py or a `note` saying why") + + # 11, the other suffix words: the third marker reads only a + # generation's and an agnomen's clash with a name (GATED_WORDS), so a + # name reading beside another word kind would mark nothing and leave + # the word an ungated suffix, taking the family name from every + # two-word name it ends; refused rather than compiled to nothing + for r in pool.readings: + if (r.category == "suffix" and not r.acronym and r.on_by_default + and r.kind not in GATED_WORDS and r.text in named): + raise VocabError(f"{r.where()}: a name reading beside this " + f"reading gates nothing, the gate reading only " + f"{' and '.join(sorted(GATED_WORDS))} readings " + f"(rules/S11); classify the word as one, or " + f"keep the counts in this reading's `note`") + # 9: a bound given name that is also a particle must be one that may # be given -- the bound join reads it as a given name. Lexicon raises # on the same contradiction; this says which file and word - for text, rs in by_text.items(): + for text, rs in live.items(): if any(r.kind == "bound_given" for r in rs) and text not in given: for r in rs: if r.category == "particle": @@ -1026,6 +1092,8 @@ def _behavior(r: Reading, prop: str) -> object: return None if att is None else att.get("side") if prop == "may_be_initial" or prop == "after_given": return r.flag(prop) + if prop == "default": + return r.on_by_default return r.props.get(prop) @@ -1043,6 +1111,8 @@ def _behavior(r: Reading, prop: str) -> object: ("MAIDEN_MARKERS", "maiden_markers"), ("SUFFIX_WORDS", "suffixes"), ("GLUED_HONORIFICS", "suffixes"), + ("SUFFIX_WORDS_AMBIGUOUS", "suffixes"), + ("AGNOMENS", "suffixes"), ("SUFFIX_ACRONYMS_AMBIGUOUS", "suffixes"), ("SUFFIX_ACRONYMS", "suffixes"), ("CAPITALIZATION_EXCEPTIONS", "capitalization"), @@ -1063,6 +1133,8 @@ class Compiled(TypedDict): MAIDEN_MARKERS: frozenset[str] SUFFIX_WORDS: frozenset[str] GLUED_HONORIFICS: frozenset[str] + SUFFIX_WORDS_AMBIGUOUS: frozenset[str] + AGNOMENS: frozenset[str] SUFFIX_ACRONYMS_AMBIGUOUS: frozenset[str] SUFFIX_ACRONYMS: frozenset[str] CAPITALIZATION_EXCEPTIONS: dict[str, str] @@ -1092,11 +1164,15 @@ def compile_pool(pool: Pool) -> Compiled: markers: set[str] = set() words: set[str] = set() glued: set[str] = set() + ambiguous_words: set[str] = set() + agnomens: set[str] = set() ambiguous: set[str] = set() acronyms: set[str] = set() cased: dict[str, str] = {} for r in pool.readings: t, p = r.text, r.props + if not r.on_by_default: + continue if "cased" in p: cased[t] = p["cased"] if r.category == "title": @@ -1114,6 +1190,10 @@ def compile_pool(pool: Pool) -> Compiled: words.add(t) if "attached" in p: glued.add(t) + if t in named and r.kind in GATED_WORDS: + ambiguous_words.add(t) + if r.kind == "agnomen": + agnomens.add(t) elif r.category == "particle": particles.add(t) if t not in given: @@ -1138,6 +1218,8 @@ def compile_pool(pool: Pool) -> Compiled: MAIDEN_MARKERS=frozenset(markers), SUFFIX_WORDS=frozenset(words), GLUED_HONORIFICS=frozenset(glued), + SUFFIX_WORDS_AMBIGUOUS=frozenset(ambiguous_words), + AGNOMENS=frozenset(agnomens), SUFFIX_ACRONYMS_AMBIGUOUS=frozenset(ambiguous), SUFFIX_ACRONYMS=frozenset(acronyms), CAPITALIZATION_EXCEPTIONS=dict(sorted(cased.items())),