Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion docs/concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,8 @@ the parse records a ``particle-or-given`` ambiguity alongside its
answer. (Which name that piece then becomes is a separate question,
and ``name_order``'s: the given name under the default.) You can
inspect ``ambiguities`` to decide, case by case, whether your data
needs a second look.
needs a second look; each report also names the reading it did not
take (:ref:`reading-not-taken`).

An ambiguity records a *decision*, not a word. The same token in a
different position may present no fork at all: ``do`` is in the
Expand Down
14 changes: 14 additions & 0 deletions docs/design/decisions/A1-ambiguity-report.md

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
- Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`.
- Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19.
- Bearer counts are structured provenance (source and count), never used to compute a reading — 2026-10-15/16.
- Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading not taken is to surface as `Ambiguity.alternative` in 2.x and the reading taken as `Token.reading` in 3.0 — 2026-10-23 #752.
- Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading taken is to surface as `Token.reading` in 3.0 — 2026-10-23 #752. The reading not taken surfaces as `Ambiguity.alternative` since 2.7, without the vocabulary's languages (decisions/A1) — 2026-10-26 #752, phase 2m.
- Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f.
- A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e.
- A listed credential that is also a name, written in capitals in a name of mixed case, is settled as the credential by its writing: it reports nothing and starts the credential's run where no name word stands behind it with no family comma (2026-10-26, Derek), and keeps its other readings, the clash settled at the sites that choose a reading rather than by retagging the word (decisions/S6) — 2026-10-25 #752, phase 2g, #785.
Expand Down Expand Up @@ -230,3 +230,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR
- The words, with their counts on each entry: `benj`, `chas`, `edw`, `geo`, `jas`, `jno`, `richd`, `robt`, `saml`, `thos` and `wm` in en.toml (the last keeping its refused title, whose reason now names this reading), `jo` and `ma` in fil.toml. What moved, the vetting and the grid are decisions/H2's entry of this date. Recompute the list: `uv run python -c "from nameparser import Lexicon; print(sorted(Lexicon.default().given_name_abbreviations))"`.
Adding the field changes `Lexicon`'s pickle layout, as phases 2h and 2i did.
- **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry.
- **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry.
2 changes: 1 addition & 1 deletion docs/design/mechanisms.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ This catalog converts discovery into a one-time cost — it does not remove disc

**Problem shape.** An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary.

**Contract statement.** Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626). Segment's report on a part past the second comma names no field at all, since that part's words can land in two (#629).
**Contract statement.** Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626). Segment's report on a part past the second comma names no field at all, since that part's words can land in two (#629). The reading NOT taken is named at the same site (`PendingAmbiguity.alternative`, #752 phase 2m), and where its field is a later stage's to know, the emitter defers it to assign or assemble rather than guess it (`_state.Deferred`).

**How it works.** PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on
"the code got here" instead of "the outcome differed" once reported
Expand Down
1 change: 1 addition & 0 deletions docs/design/rules/A1-ambiguity-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"JOHN QUINCY SMITH I" → ambiguities=("suffix-or-name",)
```
- **A report that names the field its word was read into names the field the word holds in the result**, whatever rule moved it after the report was made.
- **A report of a fork carries the reading the parse did not take**: its kind, its source, and the field the word would land in with every other word read as the parse read it, save a reading no field could hold, a connective with no word before it to join.

**Because:** a caller can only act on doubt that is reported.

Expand Down
2 changes: 1 addition & 1 deletion docs/design/rules/P3-connectives.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@

---
**Signal:** neighbor: a connective between name words, a single-letter one in a three-word name staying a name word
**history:** decisions/P3 · **implemented:** nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py
**history:** decisions/P3 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py
2 changes: 1 addition & 1 deletion docs/design/vocabulary-model-rfc.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-i

### Readings on the parse

A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so `Nguyen Van Do`, which forks nothing, can still expose `Đỗ` without a report nobody would act on. A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` is planned for a 2.x minor, with the pickle guard widened to accept a field added later; `Token.reading` waits for 3.0, since it needs a per-word reading index compiled for parse time.
A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so `Nguyen Van Do`, which forks nothing, can still expose `Đỗ` without a report nobody would act on. A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` shipped in 2.7 (decisions/A1, 2026-10-26 #752): a `Reading` of a kind, the field the word would land in with every other word read as the parse read it, and a source, which is `vocabulary`, `declared`, `shape` or `position`, the fourth being the reading of an ordinary word nothing lists, which is what a word the credential's run took by position alone has. The pickle guard was widened to accept a field added later. A `languages` field, naming the files that hold a reading, was weighed and left out: it needs the per-word reading index, which costs about 0.4 ms of a 19.5 ms import and could not describe a pack's or a caller's lexicon, so it waits with `Token.reading` for 3.0.

### The phase 1 reserved names

Expand Down
9 changes: 9 additions & 0 deletions docs/modules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,15 @@ Results
.. autoclass:: nameparser.AmbiguityKind
:members:

.. autoclass:: nameparser.Reading
:members:

.. autoclass:: nameparser.ReadingKind
:members:

.. autoclass:: nameparser.ReadingSource
:members:

Configuration
~~~~~~~~~~~~~

Expand Down
4 changes: 4 additions & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Release Log

nameparser 2.7 is under development.

**Additions**

- **Add Ambiguity.alternative, the reading a report's parse did not take.** ``parse("JOHN SMITH MA").ambiguities[0].alternative`` is ``Reading('name' as 'family', 'declared')``: read as a name, ``MA`` would be the family name, a surname the vocabulary lists; ``John Smith PhD Jones`` gives source ``position`` for ``Jones``, a name only by where it stands. The new ``Reading``, ``ReadingKind`` and ``ReadingSource`` are exported from ``nameparser``. The field is not compared, so names and their reports compare as before, and a ``ParsedName`` pickled by 2.6 or earlier still loads, its reports with no alternative. :issue:`752`

**Behavior Changes**

- **Fix roman numerals VI to X reading as names where III reads as a suffix.** ``John Smith VI MD`` gives suffix ``VI MD`` and ``Smith, John VI`` suffix ``VI``, as ``III`` does, where 1.4.0 through 2.6.0 gave last ``VI``, suffix ``MD`` and middle ``VI``. ``John Smith V, M.J.`` keeps suffix ``V`` and ``Georg VI.`` gives suffix ``VI.``, where 1.4.0 through 2.6.0 gave last ``John Smith V`` and ``VI.``. A one-letter numeral before a credential is still a name (``John Smith V PhD`` keeps last ``V``), and a Title-case ``Vi`` in a mixed-case name reads as a name (``Nguyen Van Vi`` gives last ``Van Vi``, where 2.2.0 through 2.6.0 gave suffix ``Vi``). :issue:`784`, :issue:`774`
Expand Down
44 changes: 44 additions & 0 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1084,6 +1084,50 @@ nothing — periods make ``M.A.`` unambiguously a credential:
Most names report none. A non-empty ``ambiguities`` is a useful signal
for routing a record to human review instead of trusting it silently.

.. _reading-not-taken:

The reading it did not take
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since 2.7 a report of a fork also says what the other reading was.
``Ambiguity.alternative`` is a :class:`~nameparser.Reading` with three
parts:

- **kind** — what the word would have been read as: a ``name``, an
``initial``, a ``credential``, a ``generation``, a ``particle``, a
``title`` and so on (:class:`~nameparser.ReadingKind`).
- **role** — the field the word would have landed in, with every other
word of the name read as the parse read it.
- **source** — why that reading was there at all
(:class:`~nameparser.ReadingSource`): ``vocabulary`` for a listed word,
``declared`` for a word the vocabulary also lists as a name, ``shape``
for a roman numeral, dotted letters or a one-letter initial, and
``position`` for an ordinary
word nothing lists, which is a name only by where it stands.

The source is what tells two reports of the same kind apart. ``MA`` is
listed as a surname as well as a credential, so its name reading is
declared; ``Jones`` behind a credential is taken into the credential run
by position alone:

.. doctest::

>>> for text in ("JOHN SMITH MA", "John Smith PhD Jones", "Van Buren"):
... alt = parse(text).ambiguities[0].alternative
... print(f"{text}: {alt.kind.value} as {alt.role.value} ({alt.source.value})")
JOHN SMITH MA: name as family (declared)
John Smith PhD Jones: name as family (position)
Van Buren: particle as family (vocabulary)

A report of a shape rather than a fork — ``comma-structure``,
``unbalanced-delimiter``, ``segmentation`` — carries no alternative,
and neither does one whose other reading no field could hold: a
connective letter opening the name (``E SMITH JONES``) has no word
before it to join. Check for ``None`` before reading one.
The alternative is not compared: two names that differ only in it are
equal, as before 2.7, and a ``ParsedName`` pickled by an earlier
release loads with no alternative on its reports.

Tokens and spans
------------------

Expand Down
6 changes: 5 additions & 1 deletion nameparser/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
Ambiguity,
AmbiguityKind,
ParsedName,
Reading,
ReadingKind,
ReadingSource,
Role,
Segmentation,
Segmenter,
Expand All @@ -38,7 +41,8 @@
# v1 (compatibility layer)
"HumanName",
# v2 core
"Span", "Role", "Token", "Ambiguity", "AmbiguityKind", "ParsedName",
"Span", "Role", "Token", "Ambiguity", "AmbiguityKind", "Reading",
"ReadingKind", "ReadingSource", "ParsedName",
"STABLE_TAGS", "Segmentation", "Segmenter",
"Lexicon", "Policy", "PolicyPatch", "PatronymicRule", "Script", "UNSET",
"CapsSuffixes",
Expand Down
5 changes: 4 additions & 1 deletion nameparser/_lexicon.py
Original file line number Diff line number Diff line change
Expand Up @@ -925,7 +925,10 @@ def __setstate__(self, state: dict[str, object]) -> None:
# Lexicon field layout (version skew) -- silently loading it
# would defer the failure to some distant attribute read.
# Message kept in sync with _types._guarded_setstate by design
# (layering keeps this module import-free of _types).
# (layering keeps this module import-free of _types). Unlike
# that guard it admits no field added later (`ADDED_IN`, 2.7):
# every Lexicon field is still required, so a release adding one
# breaks older Lexicon pickles until this learns the marker.
expected = {f.name for f in dataclasses.fields(Lexicon)} - {"_cap_map"}
if set(state) != expected:
missing = ", ".join(sorted(expected - set(state))) or "none"
Expand Down
38 changes: 35 additions & 3 deletions nameparser/_pipeline/_assemble.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,41 @@
"""
from __future__ import annotations

from nameparser._pipeline._state import FIELD_READING, NAME_ROLES, ParseState
from nameparser._pipeline._state import (
FIELD_READING, NAME_ROLES, Deferred, ParseState, PendingAmbiguity,
)
from nameparser._types import (
Ambiguity, AmbiguityKind, ParsedName, Role, Token,
FOLDED_TAG, Ambiguity, AmbiguityKind, ParsedName, Reading, Role, Token,
)


def _alternative(pending: PendingAmbiguity, materialized: tuple[Token, ...],
final: dict[int, Token],
middle_as_family: bool) -> Reading | None:
"""The public Reading of a report's PendingReading, its field named
from the final roles where the emitter deferred it (O5's other
primary field) and folded as `middle_as_family` folds every middle
(rules/O6's neighbour, the policy's own fold)."""
alt = pending.alternative
if alt is None or not materialized:
return None
role = alt.role
if role is Deferred.OTHER_PRIMARY:
role = (Role.FAMILY if materialized[0].role is Role.GIVEN
else Role.GIVEN)
elif role is Deferred.BESIDE and alt.beside in final:
front = final[alt.beside]
role = (alt.unfolded
if FOLDED_TAG in front.tags and alt.unfolded is not None
else front.role)
elif isinstance(role, Deferred):
# assign found no name position to name it from
return None
if middle_as_family and role is Role.MIDDLE:
role = Role.FAMILY
return Reading(alt.kind, role, alt.source)


# rules/A2: "a name with no name content parses to the empty
# name — every field empty, false as a boolean — while ambiguities
# born from its punctuation survive on the empty result"
Expand Down Expand Up @@ -143,7 +172,10 @@ def assemble(state: ParseState) -> ParsedName:
continue
detail = (f"{detail}{FIELD_READING[materialized[0].role]}"
f"{pending.field_tail}")
ambiguities.append(Ambiguity(pending.kind, detail, materialized))
ambiguities.append(Ambiguity(
pending.kind, detail, materialized,
_alternative(pending, materialized, final,
state.policy.middle_as_family)))
return ParsedName(original=state.original,
tokens=tuple(final.values()),
ambiguities=tuple(ambiguities))
Loading
Loading