Repository navigation
chore(release): bump version to 6.2.42 (bundle: complete paging, per-user spec identity, NO_PASSWORD + helper_client, authoring can_edit/used_by, renderer) - #598
Merged
Conversation
…efore sending limitResultsByStatus filters on an element's header status, Egeria's ElementStatus (UNKNOWN, ACTIVE, INVALID, DELETED, OTHER on the build we run). A content, activity or deployment status there (DRAFT, APPROVED, UNDER_DEVELOPMENT, ...) reached Egeria and came back as a 400: seen 2026-11-13 17:32:01Z when a draft-spec preview passed limit_result_by_status ["DRAFT", ...] to GovernanceOfficer.find_governance_definitions. - pyegeria/models/element_status.py: ELEMENT_STATUSES, naming its source (ElementStatus.java @ odpi/egeria 3c8970b656), and check_element_statuses(), which upper-cases valid values and refuses the rest, saying what each one is, e.g. "DRAFT is a content status, not an element status; filter on contentStatus instead". - ResultsRequestBody.limit_results_by_status validates through it, so every find/filter body refuses before sending (surfacing as PyegeriaInvalidParameterException). - format_set_executor._normalize_report_params checks limit_result(s)_by_status too, so a report or preview run refuses before a client is built. - Report "Limit Result by Status" now offers the 5 element statuses (was 18 values of the old 5.x enum), via the Spec Editor API, then refresh_specs: templates, seed and generated report specs. generated_format_sets.json takes only that change (a full regeneration also rewrote every column with the Column model's newer fields). - Docstring sample bodies and the ListFilters doc no longer show ["ACTIVE", "DRAFT"]; docs/search-and-get-filters.md regenerated. - 8 tests in test_element_status_filter.py. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…warn on unheld statuses Follows the Architect's read of laz's valid metadata values (STATUS-VALID-VALUES-LAZ-2026-11-13): valid values are declared per property, not per type, and equal the Java enums, so the hint stays offline. - STATUS_PROPERTIES: contentStatus, deploymentStatus, activityStatus, membershipStatus, termRelationshipStatus, termAssignmentStatus, each with its enum's values (from open-metadata-framework .../openmetadata/enums at odpi/egeria 8de9572c4d, equal to laz's valid values and the enums at bae9bcb1) and the types that declare it. The deployment set now has REJECTED (the enum's value), not REJECTED_FOR_DEPLOYMENT. - A refusal names every status property the value belongs to and where it lives, and suggests only the properties carried by elements, e.g. "DRAFT is a contentStatus value (AuthoredReferenceable, DataAsset), or termRelationshipStatus on term relationships, not an element status; filter on contentStatus instead". A relationship-only value (VALIDATED) gets no "filter on" suggestion; a value of no status (APPROVED_CONCEPT, COMPLETE) says so. - UNKNOWN, INVALID and OTHER still pass (Egeria accepts them) but raise UnheldElementStatusWarning: every type's validElementStatusList is ACTIVE, DELETED, so they match nothing. - Tests: the new wording, relationship-only values, the warning, and a check that the offline tables equal the Java enums (skipped without a local odpi/egeria checkout). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…unheld-status warning - The test comparing the offline status tables with Egeria's Java enums now runs only when PYEGERIA_EGERIA_SRC names an odpi/egeria checkout, and fails in words if that path has no enums. It no longer reads a default local path, so a stale checkout can't fail an unrelated release run. - check_element_statuses' docstring: under -W error an UNKNOWN/INVALID/OTHER filter raises UnheldElementStatusWarning raw from the request-body validator, not as a ValidationError / PyegeriaInvalidParameterException. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
_extract_referenceable_properties returned create_time and updated_by but not update_time; only materialize_egeria_summary set it. A report column keyed update_time that went through populate_common_columns therefore rendered blank on every row, which reads as "never updated". Add update_time beside create_time. A new test uses a real-shape Egeria 6 element and checks that every version column (version, created_by, create_time, updated_by, update_time) is filled, and that the extractor and materialize_egeria_summary agree on all of them. The isc_register DICT/TABLE/REPORT golden hashes are re-pinned: the only change in those outputs is Update Time going from null to the element's real updateTime. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Step 1 of the null-vs-empty pager (BRIEF-PYEGERIA-PAGER-NULL-VS-EMPTY- 2026-11-14). A pager only; no caller changes, and no change to what any public method returns. Egeria's paging contract: page until the server returns null; an empty list means filtering removed every row of that page, so carry on. startFrom is a pre-filter offset, so the pager always advances by the page size it asked for, never by the rows returned. On some paths (metadata-expert by-search-conditions, raw capture 2026-11-13) Egeria also sends null for a filtered page, so each call site declares an end_policy: - trusted: null is the end (end="confirmed"); - suspect (default): on null, probe the next probe_pages pages; rows there resume paging, nothing there ends with end="not_confirmed". PageResult(items, end, note, pages_read, positions_read), with end one of confirmed / not_confirmed / capped / stalled; anything but confirmed is "at least N" (count_text). The Architect's changes: the fetch page size is separate from the rows wanted (want=N, fetch_page_size default 100, items cut to want); the empty-run cap is counted in positions (max_empty_rows, default 2000), so the fetch size doesn't change behaviour. One state machine serves page_all and async_page_all. Fetch errors propagate. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Read-back of 82ca106: default_key only read `guid` (elementHeader, relationshipHeader or the row), so the metadata-expert shape (OpenMetadataElement.elementGUID, OpenMetadataRelationship. relationshipGUID), which is the suspect path the pager exists for, fell back to id(item). Each page is decoded afresh, so the same element read twice was two objects: no dedupe, and the stall guard never fired (the Architect's repro: 4 items, confirmed, where 2 / stalled was right). default_key now reads guid, elementGUID or relationshipGUID from the header or the row; a row with none (or an empty one) is keyed by its JSON content, never by object identity (identity only for a row that can't be dumped). Tests json.loads every page, for the Egeria 6 header shape and the metadata-expert element and relationship shapes: the same rows twice stall at 2, overlapping pages dedupe. Six of the new tests fail on 82ca106. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
The Architect's ruling: no public un-collapse of [] now. Callers that
read "No elements found" as "missing" (exists and heal guards) would see
a false "present". So the pager gets its end signal underneath the
public methods, and every public return value stays as it was.
- _elements_noting_page_cap(response, default, key="elements") now also
records LIST_REQUEST_RECORD["page"] = "null" (key absent or null),
"empty" ([]) or "rows", only while a caller is recording. key= lets
the relationship reads use it.
- Collapse sites now routed through it, so they record too. Each read
is the same as before:
- core: _async_get_relationships_with_property_value
(relationships). The other core collapsing helpers (name, by
property value, related-with-property-value, the three status
filters) already went through it.
- omvs: metadata_expert._async_find_metadata_elements (the B5 path);
classification_explorer _async_get_relationships,
_async_get_relationships_with_property_value,
_async_find_root_elements; collection_manager
_async_find_digital_products, _async_get_digital_products_by_category;
project_manager _async_get_projects_by_classification_properties,
_async_get_project_team, _async_get_project_hierarchy.
- page_all / async_page_all run each fetch inside _end_signal and read
the record: a collapsed [] is a filtered page, a recorded null is
null, and with no record the value is read as it is. An outer
recording (exec_report_spec's cap/depth) gets the inner record merged
up, so it still sees the last list request.
- Read-back nit: the key= docstring says a caller paging GUID-less rows
that can legitimately repeat (aggregates) must pass its own key.
Tests (test_paging_end_signal.py), at three representative sites
(core _async_get_name_request, metadata_expert find_metadata_elements,
classification_explorer get_relationships), for an absent key, an
explicit null, [] and rows:
- the record says null / null / empty / rows;
- the public return is the same with and without a recording;
- the returns are pinned: [] reads "No elements found", and an absent
key reads as each site always answered it. The same pins pass on
54d4774's source.
The pager carries on through a collapsed [] on find_metadata_elements,
through both the async method and the sync wrapper on the background
loop.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Ruling (A) on step 3 of the null-vs-empty pager, with its three
conditions. Preview pages fully; the executor pages only when the first
page shows nothing, so a filtered first page no longer reads as "found
nothing", and tile runs pay no extra request.
Executor (exec_report_spec and the async run_report path; shared
helpers):
- Every find run carries result["end"]:
- "confirmed" only after a null on a trusted path;
- "first_page" when the first page had rows and wasn't paged on (new
fifth end value; structured only, no words added to rendered
output);
- otherwise the pager's end (not_confirmed / capped / stalled).
Anything but confirmed means "at least N".
- A first page that shows nothing is paged on through page_all from
start + size, for list formats (DICT/JSON/ALL/TABLE) whose find takes
start_from and page_size, unless it was a trusted null.
- A text format (REPORT/MD/LIST/FORM/HTML/MERMAID/GRAPH) isn't paged.
An empty answer that isn't confirmed says exactly "nothing in the
first N positions; can't rule out more" (N = positions read), in
result["notes"]. Never "found nothing" unless the end is confirmed.
- page_to=N (new keyword on exec_report_spec, _async_run_report and the
impls) pages a list format fully up to N rows, fetching 100 at a time.
- end_policy comes from paging.TRUSTED_END_METHODS (today only
find_authored_elements, from the Architect's capture). Every other
method is suspect.
- One list request per fetch: each page is one call of the spec's own
find method, so the record describes that page (step-2 caveat).
Preview (authoring/preview.py):
- _run passes page_to=max_rows.
- _cap carries end. A not-confirmed empty uses the executor's words,
never "ran, and found nothing". Rows with an unreached end are capped
with "there may be more".
Pager (core/paging.py): start_from offset; "first_page" in ENDS;
TRUSTED_END_METHODS and end_policy_for().
Goldens: 30 of the 31 R1 golden hashes are re-pinned. Each differs only
by the added "end": "first_page" key, checked by removing the key and
re-hashing against the stored golden. digital_products/MD is a recorded
error string and is unchanged.
Tests (test_paging_executor.py, through the real executor with a fake
find):
- a filtered first page is paged on (suspect → not_confirmed with an
"at least" note; trusted → confirmed);
- a trusted null is a confirmed empty;
- a suspect null probes, then gives the exact words with N=500;
- the B5 shape is found by a probe;
- rows on the first page → first_page, one request;
- text formats give the exact words with N=100, one request;
- page_to;
- preview's _cap and _run;
- condition 3: every golden case's end is pinned to first_page, with
exactly one request each.
Consumers that show totals or "nothing" and must read end (a consumer
task, not done here):
- Portal (egeria-workspaces-fs, PyegeriaWebHandler):
- local-dashboards.html:1130 (tile render, kind "empty" → "No
results.");
- type-explorer.html:2980 (execResult.kind === 'empty');
- report_specs_handler.py:719, report_spec_restricted.py:289 (exec
callers that pass results on);
- mcp_server.py:445 (run_report).
- EA (trellis/packages/egeria-advisor):
- advisor/report_pipeline.py:715 and :783 (kind "empty" handling);
- agents/report_spec_agent.py:197;
- agents/outcome_reporter.py:284.
Left for step 3b (after step 4): the executor completing a partly
filtered short page for list formats, which needs live re-captured
fixtures with the startFrom>0 requests.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…rds in preview Architect read-back of 68f95bb, two REQUIRED changes: 1. Cost. After a first page that shows nothing, the executor now pages on at max(tile size, DEFAULT_FETCH_PAGE_SIZE=100) for at most AFTER_EMPTY_MAX_PAGES=10 pages (want stays the tile size, start_from stays start+size). An all-filtered tile at Portal's pageSize 25 now costs 1 + 10 = 11 requests (was 81); a suspect null costs 1 + 4 = 5. When the page limit stops it, end is "capped" and the words are still "nothing in the first N positions; can't rule out more" (N = 1025 there). Tests pin both request sequences and that paging on at 100 still returns at most the tile's 25 rows. 2. Wording. preview's note no longer formats the raw end value: it says "{n} rows shown; Egeria didn't confirm there are no more". end stays in the structured field only. No other note formats end. Consumer note (not this branch): freshstart's twin files need the same `end` handling as the Portal consumers listed in 68f95bb. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Architect's step-4 ruling (A), 2026-11-13, five conditions. overview_metrics (the
11th loop: two scans) follows in its own commit under the overview_metrics
ruling (ii).
Pager (pyegeria/core/paging.py):
- PageResult.end_reason: trusted_null, null_then_empty_probes, empty_run_cap,
want, max_pages or stalled. reached_end is True only for the first two: a read
may be acted on as the end. complete (confirmed) is still the only state that
allows a total or "none". The empty-run cap is a limit everywhere.
- PageResult.trace(), log_end(method, paged): one structured line (method,
positions_read, end, end_reason). at_least_note(paged, what, count): the
"Showing at least N <what>: <why>." words.
- stall_pages (default 1): export_dcat's laz asset-domain search returns
overlapping short pages, so it allows 2. The very same page twice is always a
stall.
Every loop advances by page_size, never len(page), and pages as suspect
(TRUSTED_END_METHODS is still only find_authored_elements). Return types are
unchanged; any new channels are optional keyword arguments or extra result keys.
Per-loop table. "B5 path" means metadata-expert by-search-conditions
(find_metadata_elements), where Egeria sends null for a filtered page.
| Loop | Find method | B5 path? | Kind | On a not-confirmed end |
|---|---|---|---|---|
| authoring/drafts.async_list | classification_explorer _async_find_elements_by_property_value (by-property-value-search) | no | display | notes= gets "Showing at least N <kind> drafts"; prune_drafts' result carries read_notes |
| authoring/publication.async_read_history | metadata_expert _async_get_element_history | no (1) | WRITE (publish) | acts on null_then_empty_probes; refuses on capped/stalled/empty_run_cap; trace logged and in _resolve's history_read; fetch 100 |
| view/dashboards._dashboard_elements | classification_explorer _async_get_elements (by type) | no | display | DashboardSheetListing.notes gets "Showing at least N dashboard sheets" |
| view/saved_query._read_pages | get_collection_members (cached); the stored request's method, usually find_metadata_elements (live) | live: YES | display | capped=True plus the note, never nothing_found unless confirmed; the subtitle says "Egeria didn't confirm there are no more" |
| view/saved_query._all_related (delete links) | get_related_elements (SmartQuery) | no | delete | proceeds on null_then_empty_probes; result note "removed N SmartQuery link(s); Egeria didn't confirm there were no more"; refuses on any limit |
| view/export_dcat._paged | get_collection_members / find_in_asset_domain | no | display | note "Showing at least N ..."; "No dataset exported" says "in what was read; Egeria didn't confirm there are no more" |
| view/egeria_folders._members | get_collection_members | no | display | walk/tree record unconfirmed reads (entry end_confirmed False plus a tree note); max_pages still raises FolderTooLarge |
| md_processing dashboard_sheet._related / _resolve_report | _async_get_related_elements / core _async_get_elements_by_property_value | no | write-adjacent (sheet writer) | acted on when reached_end; a limit adds a writer note and uses what was read |
| md_processing processors._parent_links_of_type | metadata_expert _async_get_all_related_elements | no (2) | WRITE (Dr.Egeria parent link) | acts on null_then_empty_probes; raises _ParentLinksIncomplete on any limit (not added); trace logged and in the related result's message; fetch 100 |
| commands/tech/list_element_register | get_elements | no | display | counts shown as "≥N" with the at-least note |
(1) Element history: OpenMetadataStoreRESTServices.getMetadataElementHistory
(omf-metadata-server, :1139) -> OpenMetadataAPIGenericHandler.getBeanHistory
(generic-handlers, :10222). One read check on the element (:10241), then every
version the repository returned is converted (:10261-10273). Null (:10276) only
when the repository returned none, so there is no per-entry filtering that
could empty a page.
(2) Related elements: MetadataElementHandler.getRelatedMetadataElements
(omf-metadata-server handlers, :468) returns null only when the repository
returned no relationships (:550-552); filtering builds `results` (:573) and
returns it, possibly empty, as [] (:607).
Accepted provisionally by the Architect (overview ruling, condition 7); to be
checked at read-back.
Write paths on the B5 path: none, so nothing here waits for a capture.
saved_query's live mode is on B5, but it is display-only.
Tests:
- new test_paging_step4_loops.py: end_reason/reached_end/trace for each reason;
stall_pages; history and parent links acting on null_then_empty_probes with
the trace; history and parent links refusing on empty_run_cap; drafts and
folders "at least".
- re-pinned to the new contract (advance by page_size, 3 probes past a null,
"at least" instead of totals): d14 published versions, collection members
paging, dashboard folders, list_element_register (with an "at least" test), authoring api,
parent links, saved_query analytic/delete (plus a delete empty-run-cap
refusal), export_dcat.
- Two replay fixtures (dcat, catalog_health_scratch) were captured before the
probes existed, so they stop at the null. They answer a page past a RECORDED
null of the same call with that null, and list it in `synthesised`. On a
suspect path that can only end a read not_confirmed, never confirmed. They
should be replaced by re-captures that include the probe pages.
Not here: the Portal/EA consumers of these notes and flags (named in the
step 3 commit), and the overview_metrics loop (next commit).
Micro-tests: 3866 passed, 0 failed (full run, this commit).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
The 11th loop, overview_metrics' two scans (_all_relationships over
get_relationships, and _all_elements over find_metadata_elements, the B5 path),
follows the Architect's overview_metrics ruling (ii), 2026-11-14.
How a scan ended decides what its figures say:
- confirmed: unchanged. Both methods are suspect today; end_policy comes from
end_policy_for(), so a capture-backed promotion to TRUSTED_END_METHODS turns
this path on with no other change.
- null_then_empty_probes: the figure is computed over what was read and
marked. result["unconfirmed"][figure] = {unconfirmed: true, positions_read,
reason: "Egeria didn't confirm there are no more", words}:
- counts: "at least N";
- distinct counts: "N found; Egeria didn't confirm there are no more";
- maps/lists of counts: "each count is at least what is shown; ...";
- a total minus a scan count (orphanCount): "at most N; ..."
(not covered by the ruling's words; flagged for the read-back);
- shares from one read: "X% of the N read; ..." (the machinery and a test
are here, but no overview figure is such a share today);
- shares of a separate count (groundedAssetPct, contextualisedPct): not
measured, with the reason, also in groundedAssetNote. The ruling requires
numerator and denominator from the same read.
- capped / stalled / empty_run_cap: as before. None plus a not-measured note
carrying the lower bound (at_least): never 0, never a bare number.
The marking is done by _marks_unconfirmed({figure: kind}) on the 11 figure
functions that use a scan: governed_coverage, ownership_coverage,
certifications_summary, orphan_glossary_terms, semantic_grounding,
ai_ready_assets, drl_readiness_gates, feedback_summary,
contextualised_coverage, solution_component_realisation and
active_contributors. Unconfirmed scans are also collected by
recording_unconfirmed(), so a figure that is a bare int (count_relationships)
or a list (karma_leaderboard, engagement_series) still carries it. The
executor puts them on the analytic output as "unconfirmed", and a chart's
subtitle says "at least what was read: Egeria didn't confirm there are no
more". Unconfirmed is a separate channel from "unmeasured", so the executor
doesn't withhold these figures as not measured.
Rows without a GUID are each their own row, as the old loops counted them
(the pager's content key would have merged equal ones).
BLOCKER, not in this repo: ruling (ii) condition 4, the Portal renderer
(egeria-workspaces-fs, egeria-overview.html and its freshstart twin). It must
show an unconfirmed figure differently from a confirmed one, with a marker
plus result["unconfirmed"][key]["words"] or the output's "unconfirmed" list.
Until it does, the Overview shows these numbers unmarked. That is how they
showed before this branch, but they're no longer total-correct by contract.
For the Portal builder.
Capture (condition 6): ~/.re-gate/CAPTURE-PAGER-STEP4-OVERVIEW-NULLS-2026-11-14.py,
read-only. It runs the overview's 3 find bodies and 9 relationship types at
pageSize 25 and 500, going 10 pages past the first null. If rows follow a
null, that method's figures revert to (i) (not measured).
Tests:
- the 24 overview tests now assert the flag and the words, not only the
numbers; their fakes send Egeria's null past the end ([] is a filtered
page);
- new test_overview_unconfirmed.py: count / percent / distinct x confirmed
(via TRUSTED_END_METHODS) / null_then_empty_probes / capped; a share of a
separate count not stated; the executor carrying unconfirmed scans.
- the other scan tests (capped_scans_not_counts, capped_count_not_measured,
rs_w2_1a declarations, native_counts_dashboard): fakes end at null; the
orphan/lineage/coverage cases assert the marks and the probe requests.
Micro-tests: 3877 passed, 0 failed (full run, this commit).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
The laz capture (~/.re-gate/CAPTURE-PAGER-STEP4-FINDINGS-2026-11-14.md,
read-only, as erinoverview) showed mid-set nulls on find_metadata_elements and
get_relationships at pageSize 25 (runs of up to 3) and no [] in 457 requests.
The ruling replaces overview condition 6.
- pyegeria.core.paging.NULL_NOT_END_METHODS = {find_metadata_elements,
get_relationships}, citing the capture. On them null_then_empty_probes is
not an end: PageResult.null_not_end makes reached_end False, as for a
limit. paging_for(method) gives page_all its arguments: suspect,
probe_pages=NULL_NOT_END_PROBE_PAGES (6; the longest run was 3) and
null_not_end. end_policy_for never trusts them, and a module assert keeps
them out of TRUSTED_END_METHODS (promotion is closed until Egeria returns
[] for a filtered page).
- overview_metrics' two scans page with paging_for(...). DEFAULT_CAP stays
500.
- Lower-bound figures stay (ii), marked unconfirmed: "at least N",
"N found; ...", "each count is at least what is shown".
- Whole-set figures are not measured on any end but confirmed:
- shares (as before);
- orphanCount, now kind "whole_set", which replaces "at most N"
(supersedes read-back R1).
Tests:
- test_overview_unconfirmed.py:
- orphanCount not measured;
- a NULL_NOT_END read gives reached_end False, with 6 probes (and another
suspect method still 3);
- the capture's run of 3 mid-set nulls is read through;
- NULL_NOT_END methods are never trusted;
- no write path (publication history, parent links, saved_query link
delete, draft prune/list, dashboard sheet writer reads) reads from a
NULL_NOT_END method, checked by source.
- The confirmed-figure cases simulate a trusted end at paging_for, since
promotion is closed.
- Existing scan tests are re-pinned: 6 probes, positions_read, and
orphanCount None.
Micro-tests: 3882 passed, 0 failed (full run, this commit).
Still open (read-back):
- re-capture the dcat and catalog_health_scratch replay fixtures with probe
pages before step 6 closes;
- the Portal renderer marker (B1, egeria-workspaces-fs);
- step 5's new item: the viewer-population count's one-page len(find)
fallback treats a short page as the total.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
pyegeria pager (null vs empty), steps 1-4: page_all with end reasons; an empty page is not the end, null is; NULL_NOT_END_METHODS for find_metadata_elements and get_relationships (laz capture 2026-11-14); overview figures marked unconfirmed or not measured. Architect read-backs PASS (step 4 at 7379bd8). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…count pages
These are the two step 5 items from the Architect's step-4 read-back. The
"exists?" three-way answer follows separately, after a scoping question.
1. Every page_all / async_page_all call takes its paging from
**paging_for(method):
- format_set_executor's plan (it took only end_policy_for, so 3 probes and
no null_not_end);
- saved_query (cached: get_collection_members; live: the stored method,
e.g. find_metadata_elements, now NULL_NOT_END with 6 probes);
- the 10 step-4 loops.
test_paging_step5.py checks this by AST: no pager call sets end_policy,
probe_pages or null_not_end itself.
2. The viewer-population count's fallback (_element_count) used to read one
find page, so a short page was taken as the total. It now pages through
_all_elements: find_metadata_elements, NULL_NOT_END.
- The count it returns is a lower bound, recorded as unconfirmed ("at
least N").
- A limit (stall, empty-run cap, MAX_SCAN_PAGES) is not measured, with
its lower bound ("the find fallback stopped before the end, so there
are at least N").
- A failed query is not measured.
Downstream:
- asset_composition count cells: a lower bound is "measured" with
unconfirmed: true and reason "at least N: ...", never "nothing found".
The derived "Other" cell (parent minus children) is not measured when
any input is unconfirmed: it needs every count whole.
- metric_trend points: how "find-unconfirmed", exact False, display "≥N".
- Viewer totals in composites (termTotal, groundedAssetTotal,
contextualised assetTotal, solutionComponentsTotal) are marked "at least
N" only when their own count was unconfirmed ("count_of" kind). A
confirmed native total beside an unconfirmed scan is unchanged; the
first cut marked it too, and a test caught it.
- The executor's JSON path gives a scalar unconfirmed count a
display "≥N".
Tests re-pinned to the new contract: 6 probes on find_metadata_elements, the
viewer fallback paging, "≥N" points, and [] forever being the empty-run cap
rather than a zero. Covered: saved_query analytic, overview_metrics,
capped_count_not_measured, w2 trend points, issue_123 strict counts.
Micro-tests: 3884 passed, 0 failed (full run, this commit).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…cessors, list_element_register The three call sites left out of 7f6a534's staging (commands/ and md_processing/ weren't added). Same change: their page_all calls take **paging_for(method) (get_related_elements, get_elements_by_property_value, get_all_related_elements, get_elements). 7f6a534's full run (3884 passed) already included these files from the working tree; together the two commits are what was tested. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
A transport, auth or timeout error could make Create-or-Update create a duplicate.
resolve_element_guid folded lookup errors into "not found" in two places:
(a) core __async_get_guid__ swallowed any error from the qualifiedName unique-name
read (`except Exception: pass`) and fell back to the displayName read. A
"not found" there then answered for an element whose read had failed.
(b) resolve_element_guid logged any other lookup error at debug level and returned
None, which fetch_as_is and step 4a read as "absent", so the command created.
Fix (pager step 5, the Architect's exists-lookups ruling, item 6):
- __async_get_guid__ catches only PyegeriaNotFoundException on the qualifiedName
read. With strict_errors=True (new, default False, so other callers keep their
behaviour) it raises any other error. resolve_element_guid passes it in both
passes.
- resolve_element_guid records an error that isn't Egeria's not-found as unknown
(_note_lookup_unknown: "couldn't tell whether 'X' exists in Egeria
(TimeoutError)"). Egeria's not-found is still "absent". With no Egeria client
(a local-only store) there's nothing to be unknown.
- execute() blocks a command whose target wasn't found while any lookup is
unknown, before it writes: "Execution blocked: couldn't tell whether ... so
nothing was created or changed. Try again."
Tests: test_resolve_guid_unknown.py. A timeout, a connection error and a 401
each block the Create; Egeria's not-found still creates. Two fakes are updated:
the governance type-resolution fake takes **kw, and the forward-reference fake
answers "No elements found" explicitly (it relied on a missing method's
AttributeError being swallowed as "not found").
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ame; no error is "absent"
This implements the Architect's exists-lookups ruling (2026-11-14).
1. NULL_NOT_END_METHODS gains get_elements_by_property_value,
find_elements_by_property_value and _async_get_elements_by_property_value.
Classification-explorer by-property-value is
OpenMetadataClient.getMetadataElementsByPropertyValue, which is
findMetadataElements(...): the server method the capture caught.
Display callers (drafts.async_list) say "at least".
2. pyegeria.core.paging.exists / async_exists(fetch, match, method=...)
returns an Exists: "found" (a match was read), "absent" (reached_end, so
never on a NULL_NOT_END method) or "unknown".
- It takes **paging_for(method), max_pages <= EXISTS_MAX_PAGES (10),
stops at the first page with a match, and matches exactly client-side.
- Exists.words() gives the read's trace.
- For one decisive "found" only; never to prove uniqueness.
3. Publish (publication._find_exact) no longer pages by-property-value to
choose create versus update.
- A qualifiedName target uses the unique-name read
(classification_manager._async_get_element_by_unique_name):
- found: update (refused if the element under that qualifiedName is
another type);
- Egeria's not-found (the string, or PyegeriaNotFoundException):
absent, create;
- any other error: unknown, refused with the words, nothing planned.
- A target named by another property is refused: "Give the qualified
name: Egeria can't confirm there's only one Report with displayName
...".
- Draft kinds this changes (for the owner):
- Report drafts with no qualified_name (target_of resolves them by
displayName) must now give one.
- Spec drafts (ReportSpec by its qualified name) and sheet drafts
(Dashboard::<name>) are unchanged.
4. _async_get_guid_for_name keeps its return. ServerClient gains
_async_lookup_guid_for_name, which returns an Exists and still raises on
two or more matches.
The 13 callers of get_guid_for_name / _async_get_guid_for_name:
| caller | kind | change |
|---|---|---|
| core _async_add_journal_entry: the NoteLog lookup | WRITE GUARD (not found, so create a NoteLog) | moved to three-way: unknown raises, no NoteLog created |
| core _async_add_journal_entry: the associated element lookup | write input (not found, so raise) | moved to three-way: the unknown words say why |
| omvs security_officer __get_platform_guid__ | write input; "found" is decisive; not found raises | unchanged |
| omvs data_discovery _async_resolve_guid | name-to-GUID for reads and annotations; "found" is decisive | unchanged |
| core get_guid_for_name / classification_explorer get_guid_for_name (sync wrappers) | API | unchanged |
| core utils guid_or_none (docstring example only) | doc | unchanged |
| commands/cat list_terms, list_deployed_catalogs, list_deployed_database_schemas | CLI display | unchanged |
| commands/ops x_engine_actions (2), monitor_engine_status (commented out) | CLI ops display | unchanged |
5. Dr.Egeria resolve_element_guid: the error-folding fix is its own commit,
d060303 (just before this one), as ruled.
6. md_processing dashboard_sheet._resolve_report follows the Architect's
(a)+(b) ruling:
- (a) A ref with "::" uses the unique-name read:
- found: link;
- Egeria's not-found: the display-name path;
- any other error: no link, with a note naming the error class.
- (b) A display-name ref goes through **paging_for
("get_elements_by_property_value", NULL_NOT_END), with exact displayName
equality among Reports:
- 1 match, read at null_then_empty_probes: link, with the note "Linked the
Report named X that Egeria returned ('<its qualifiedName>'); Egeria
can't confirm it's the only one; give the Report's qualified name to be
sure.";
- 1 match at a limit: no link, with a note;
- 2 or more: no link, with a note listing up to 5 qualifiedNames;
- 0 or unknown: no link, with "No Report named X was found in what Egeria
returned".
- The placement-level note no longer says "was not found in Egeria".
- Notes reach the author through writer.notes: the command's warnings and
its "Egeria" lines.
- Follow-up (for the Architect's list, authoring phase 2): authoring tools
write the Report's qualifiedName into new placements, so the note dies
away. Once the #9453 version gate ships, a single match on a gated
platform is unique and the note drops.
7. drafts.async_prune_drafts is left as is: it deletes only drafts it read.
The write-path guard test says so in a comment, and
test_prune_decides_only_from_rows_read asserts it: a draft not in the read
is never deleted, and an empty read deletes nothing. The guard now also
covers publication._find_exact.
Tests:
- test_paging_step5_exists.py:
- exists found/absent/unknown, NULL_NOT_END never absent, the max_pages
bound, exact matching;
- the three-way guid lookup, and two matches raising;
- the journal entry never creating on unknown and using a found NoteLog;
- (the resolve_element_guid tests are in d060303).
- d14: unique-name found / not-found (string and exception) / error; unknown
creates nothing; another type under the qualifiedName refused; a
displayName-only Report refused.
- dashboard sheets: QN found, QN not-found then display, QN error; display 1
at null_then_empty_probes (link plus the QN note), 1 at a limit (no link),
2+ (QNs listed), 0 (the words), unknown (no link).
- prune decides only from rows read.
- Fakes updated: authoring_api (unique-name read).
Micro-tests: 3911 passed, 0 failed (full run, d060303 + this commit together).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
publication._find_exact read any unique-name answer that wasn't an element as "absent", so a list, another string, or a dict with neither elementHeader nor element meant a create. Now only result == NO_ELEMENTS_FOUND (or PyegeriaNotFoundException) is absent. Anything else is refused: "Egeria's answer to the unique-name read of <type> <value> wasn't an element or 'not found' (<type>); nothing was planned." Test: [] / a list / another string / an unrelated dict / None are each refused, and nothing is read or planned. dashboard_sheet (a) has the same shape but falls through to the display-name path, so it is left as is (per the read-back). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
pyegeria pager step 5 (dfeb411..da8d036): every page_all through paging_for; the viewer-count fallback pages; by-property-value methods are NULL_NOT_END; exists-lookups found/absent/unknown; publish uses the unique-name read and refuses non-QN Report drafts; dashboard_sheet links a single display-name match with a note. Includes d060303: resolve_element_guid no longer folds transport/auth/timeout errors into "absent" (Create-or-Update could create a duplicate). Architect read-backs PASS. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
… (pager step 6 prep)
The fixture predates the pager's probe pages, so its replay answers pages past a
recorded null with that null (marked 'synthesised'). This script re-captures it:
it runs async_load_held('sheet', 'catalog-health') through a recording proxy that
refuses any non-read method (_async_get_/_async_find_/get_/find_ only) and writes
the fixture in its existing shape (sub, method, args, kwargs, result), keeping its
_about and document.
It must run where the fixture's document was processed (the round-trip probe's
scratch Egeria); laz's sheet differs. Smoke-tested offline against the existing
replay: 66 calls (with probes), 6 placements. Not yet run against a live Egeria.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ted on fixed platforms)
This implements the Architect's step 6 ruling (2026-11-14). The evidence is the
post-#9453 capture on laz (5deda38d,
~/.re-gate/RESULTS-EGERIA-UPGRADE-5deda38d-AND-POST-9453-CAPTURE-2026-11-14.md):
- find_metadata_elements and by-property-value are fixed: [] for filtered
pages, and no rows after any null.
- get_relationships by type is not fixed: nulls mid-set, rows after, never [].
pyegeria.core.paging:
- GATED_TRUSTED_METHODS = {find_metadata_elements,
get_elements_by_property_value, find_elements_by_property_value,
_async_get_elements_by_property_value}. These are trusted on a gated
platform only, otherwise NULL_NOT_END as before. get_relationships stays
NULL_NOT_END on every platform (asserted at import).
- gate_from_about(about) promotes a platform when:
- the version is a release later than 6.3-SNAPSHOT (6.3 or newer, not a
SNAPSHOT), or
- the version is "6.3-SNAPSHOT" with a build time at or after
2026-11-14T19:52:00Z (the #9453 merge).
Anything else is not promoted: an older version or time, another SNAPSHOT
(e.g. 6.4-SNAPSHOT), no time, a time with no timezone, or an unparsable
answer.
- platform_promoted / async_platform_promoted(client) read /api/about
best-effort through the client's own get_platform_origin, or a subclient's
for EgeriaTech:
- cached per platform URL for GATE_TTL_SECONDS (3600);
- never raises and never blocks a read: a failed read means not promoted;
- PYEGERIA_TRUSTED_END=off forces not promoted, and there is no force-on.
Known limit: a 6.3-SNAPSHOT built after 19:52Z from code without #9453
passes the time test; the override covers it.
- paging_for(method, client=None) / async_paging_for(method, client) ask the
platform only when the method is gated. exists/async_exists take `client`.
Wired with the client that makes the read:
- overview_metrics._all_elements (find_metadata_elements);
- saved_query live (the stored method);
- format_set_executor's plan (the find's bound client; the async path awaits
async_paging_for, so the event loop isn't blocked);
- drafts.async_list;
- dashboard_sheet._resolve_report;
- core _async_lookup_guid_for_name.
On a promoted platform:
- a confirmed end gives plain counts; shares and orphanCount are measured (no
unconfirmed scan);
- a confirmed empty read may say "none";
- dashboard_sheet's single display-name match is linked with no note, and a
confirmed 0 says "There is no Report named X in Egeria";
- exists-lookups may answer absent;
- publish keeps the unique-name read.
On any other platform everything is as in steps 4/5.
Tests: test_paging_step6_gate.py.
- gate_from_about: laz's real 5deda38d answer promoted; 002a3ef1's earlier
time not; exactly at the merge; releases above and below; 6.4-SNAPSHOT;
no time; a naive time; junk.
- An about-read failing, and the override.
- Each gated method trusted only when promoted; get_relationships still
NULL_NOT_END while promoted.
- A promoted find reading to a confirmed end, and a promoted exists answering
absent.
- One about-read per platform within the TTL, re-read after it; no about-read
for a client that never pages a gated method.
- A promoted overview scan plain; a promoted sheet: one match linked with no
note, and "There is no Report named ...".
Micro-tests: 3945 passed, 0 failed (full run, this commit).
For the owner (from the ruling): ask Mandy for a git commit id in /api/about
or a "paging contract" capability flag, which would replace the time test; and
the B5 follow-up that classification-explorer relationships-by-type still
returns null for a filtered page mid-set.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…marked; publish visibility note These are the remaining code items on step 6's list (the step 5 read-backs and the Portal B1 read-back). 1. A referenced element (a parent, a member, an anchor scope) whose lookup failed with an error other than Egeria's not-found is never reported "not found". Dr.Egeria's reference validation now says "Couldn't tell whether referenced element 'X' for attribute 'A' exists in Egeria (the lookup failed); nothing was written." This covers both the single-value branch and the list branch; a list with some unknown and some not-found items names each kind. _lookup_was_unknown(name) reads the step 5 record. The command still fails validation, as before. test_dr_egeria_extraction's bare-object client now gets these words (its lookup errors), and its assertion is updated. 2. drl_readiness_gates marks byModality as "counts", as the Portal B1 read-back asked (pyegeria is the one source; the Portal's interim mark can go). Finding for the read-back: drl_readiness_gates and ai_ready_assets take their Assets from ONE find page (_find_or_none) and treat a short page as the whole set; only a full page is refused. That's the same shape as the viewer count fixed in step 5 part 1. Not changed here; a candidate for a later step. 3. pyegeria.authoring.publication's docstring now describes the step 5 lookup (the unique-name read, not a page of 25), and adds the visibility note: getBeanByUniqueName filters by what the publisher may see, so an element zoned away from them reads as "not found" and its publish plans a create. Whether that create then fails on the duplicate qualifiedName or makes a twin isn't checked here. Tests: - test_resolve_guid_unknown: a Create whose Parent ID lookup times out fails with the "couldn't tell" words, never "not found" or "does not exist", and creates nothing; - overview: the byModality mark. Micro-tests: 3946 passed, 0 failed (full run, this commit). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
… capture failure) The catalog-health re-capture on scratch (2026-11-14) failed with load_held saying "Egeria holds no dashboard sheet 'catalog-health' ... that you can see", although every Dr.Egeria command had just succeeded. Cause: - The capture ran against view server "qs-view-server". The scratch platform runs "view-server": all 313 requests in process.log went to /servers/view-server/. The qs-view-server name came from this script's own docstring example. - Egeria answers a request to a server that isn't running with relatedHTTPCode 404 (OMAG-MULTI-TENANT-404-001). pyegeria raised every relatedHTTPCode 404 as PyegeriaNotFoundException, and every not-found handler (find_element_by_qualified_name here, then load_held) read it as "the element isn't there". - So a wrong server name became an absent element: absence-as-answer on a read path. Fix: - _async_make_request raises OMAG-MULTI-TENANT-404-* (the server, or a service on it, not available) as PyegeriaAPIException, not PyegeriaNotFoundException. An element's own not-found is unchanged. load_held, publish, the Dr.Egeria resolver and the sheet writer now get an error, not "absent". - capture_catalog_health_fixture.py requires EGERIA_VIEW_SERVER (scratch: view-server; laz: qs-view-server) and its example says view-server. Tests (test_server_not_available_is_not_absent.py): 404-001 and 404-002 are API errors, not not-found; an element not-found is still not-found; find_element_by_qualified_name on a wrong server raises (so load_held never says "holds no"). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…bout-read; a 5 s timeout Step 6 read-back of b5250e4: - R1: a SNAPSHOT whose (major, minor) is later than (6, 3), e.g. 6.4-SNAPSHOT, is promoted: #9453 merged on main before the 6.3 line, so every later SNAPSHOT contains it. Without this, laz would drop back to "at least" the day Egeria bumps main. The time check stays for exactly "6.3-SNAPSHOT"; 6.2-SNAPSHOT is never promoted. - Advisory 1: a failed about-read is cached for GATE_FAILED_TTL_SECONDS (60) and a parsed one for GATE_TTL_SECONDS (3600), so a platform restart doesn't turn trust off for an hour. - Advisory 2: the about-read times out after GATE_ABOUT_TIMEOUT_SECONDS (5): asyncio.wait_for on the async path, a worker thread on the sync path. A slow read is "not promoted". Tests: 6.4-SNAPSHOT (with or without a time) and 7.0.1-SNAPSHOT promoted; 6.2-SNAPSHOT not; a failed read retried after the short TTL (promoted once the platform answers); a slow read (sync and async) not promoted. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
laz's 2026-11-11 20:27Z DB reset re-catalogued the clinical-trial files under new qualified names (CSVFile::egeria-workspaces:/deployments/coco-data-lake/research/ clinical-trials/...), so the old exact DataLake::PROJ-CT-TBDF::... search matched nothing in the 2026-11-14 re-capture (4 calls, 0 rows). PROJ-CT-TBDF_00002 finds the folder and its 10 CSVFiles (the Coordinator's check). This only affects the capture script; the fixture and its tests are re-pinned once it's re-run. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…t find (the one-page finding)
Both read Assets with one find page (_find_or_none) and treated a short page as
every Asset; only a full page was refused. (On laz, with 14k Assets, they always
hit the 500 cap.) Both now read through _all_elements (_all_assets), which pages
with **paging_for("find_metadata_elements", mgr):
- on a #9453-gated platform, a confirmed read gives plain totals;
- otherwise the rows are recorded unconfirmed: total, aiReadyCount, and
byModality are "at least";
- a limit (stall, empty-run cap, MAX_SCAN_PAGES) is not measured, with its
lower bound, and capped True;
- a failed read is None, as before.
total and byModality are marked from the Asset read only (new "counts_of" kind,
like count_of), so a confirmed Asset total isn't marked because the DataFlow
scan (get_relationships, never trusted) is unconfirmed. aiReadyCount depends on
both reads, so it is marked if either is unconfirmed. pyegeria computes no share
of these; any consumer's share of aiReadyCount/total is unconfirmed unless both
are unmarked.
SWEEP of single-page finds whose length becomes a count (overview_metrics and
pyegeria/view):
- ai_ready_assets, drl_readiness_gates: FIXED here.
- overview_metrics _element_count (the viewer-count fallback): fixed in step 5
part 1 (7f6a534).
- overview_metrics _rows_updated_before (stale_assets): intentional. It reads by
offset, one row or one window at a time, against a native total, and a hidden
row is handled as unknown; no length is a count.
- overview_metrics _one_page_or_none and _find: no callers left (dead code; left
as is, candidates for removal).
- egeria_spec_cache.fetch_spec_for_call (_find_specs_by_label): an exists /
uniqueness lookup, not a count. Its len(found) > 1 refusal is "two or more
read". Not changed; a candidate for the exists-lookup pattern.
- structured_table count: counts a row's own nested list, not a find. N/A.
- Also found, not a count: base_report_formats.load_egeria_report_specs pages
find_collections and stops on the first empty page, the step-4 loop pattern
the 11-loop survey missed. Not changed here; flagged for a follow-up.
Tests:
- a limited (repeating) Asset read measures nothing, with its bound;
- a short first page then a later page counts both (2, "at least 2" on
an ungated platform);
- on a gated platform the Asset total is plain while aiReadyCount stays marked
(DataFlow);
- drl's lineage-failure test pages its fake.
Micro-tests: 3957 passed, 0 failed (full run, with the docs commit below).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…(read from source) The visibility note said whether a zoned-away element's publish fails or makes a twin wasn't checked. Read at Egeria 8de9572c4d: - qualifiedName is a unique property (Referenceable, OpenMetadataTypesArchive1_2, setUnique(true)). - OpenMetadataAPIGenericHandler.createBeanInRepository -> validateNewEntityRequest (:1807) -> validateUniqueProperties (:1750) -> validateUniqueProperty (:1638) reads entities by that value as localServerUserId (a lineage read, Memento excluded), so the caller's zones don't hide the existing element, and throws throwUniqueNameInUse (:1718/:1728). So the publish fails; it doesn't make a twin or update the element. Caveat in the text: this is the generic-handler create path; not every OMVS create has been traced to it. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…esised-null rule The fixtures are re-captured from dd195d6 (~/.re-gate/FIXTURES-RECAPTURE-dd195d6e-2026-11-14/, owner-approved, read-only): - DCAT: laz 5deda38d as erinoverview, scripts/capture_dcat_fixtures.py. - catalog-health: scratch Egeria 5deda38d (view-server) as garygeeke, after processing the fixture's document there once, scripts/capture_catalog_health_fixture.py. Both now record the pager's probe pages, so both replays lose their "answer a page past a recorded null with that null" rule (_probe_after_recorded_null, _past_recorded_null, `synthesised`). Every page a test reads is now a recorded one. DCAT: - portal_dashboards, org_structure_products, coco_data_hub: replaced; their tests pass unchanged. - clinical_trial_files: replaced, scope "PROJ-CT-TBDF_00002" (dd195d6). The old exact DataLake::PROJ-CT-TBDF::... search was stale since laz's 11-11 20:27Z DB reset re-catalogued the files under new qualified names. Since the reset the files are neither zoned nor licensed and have no description, so re-pinned: - 11 datasets (10 CSVFiles plus their FileFolder); - the absence notes "for 11 datasets"; - the validator findings now name the 10 missing dct:description values (a real DCAT-AP gap); - the seed spec's file name is PROJ-CT-TBDF_00002-dcat.<ext>; - the repeated-page test counts the page it repeats. - clinical_trial_files_licensed_2026_11_11 (new name for the pre-reset capture): the only fixture with zoned (NON_PUBLIC), licensed CSVFiles with a distribution; no such shape is on laz now. Its element reads replay as captured. Its search predates the probes, so the one test using it (test_distribution_licence_and_non_public_for_zoned_files) answers the search with an explicit, labelled fake: the recorded first page, then null. This is a test-local fake, not a replay rule; flagged for the Architect. - not_visible_report: KEPT as captured 2026-11-11. The re-capture got a 404 (OMAG-REPOSITORY-HANDLER-404-007), because the zone-refused Report's GUID is gone since the reset, so it no longer records the 403 it exists for. It's one unpaged read, so the old capture is still exactly what it was. A new zone-refused element would be needed to re-capture it. catalog-health: - catalog_health_scratch.json: replaced (66 calls with probes, 6 placements). Its _about now names scratch 5deda38d (was 757a5df0), the view server, the persona and the capture script. - catalog_health_reports_scratch.json (new): the Reports' own reads (unpaged unique-name and displayName reads, 10 calls) split out of the 2026-11-12 capture with its original provenance. The new capture ran load_held("sheet") only, and those reads need no probes. Found while re-pinning (not changed, for the follow-up list): lifecycle._report_element (load_held("report")) reads by displayName with one page of 25 on a NULL_NOT_END method and says "holds no Report" on 0 matches, the exists-lookup shape. Micro-tests: 3958 passed, 0 failed (full run, with the next commit). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
item 2 (c9ed48e + 70343e0): _finish_fetch never writes a fetched spec to any manifest unless its visibility is "public"; zoned or unreadable visibility stays in-process, logged. Conflicts with items 3/4 in _finish_fetch and the tail of test_rs_egeria_tier.py resolved per the builder's trial merge. Architect read-back PASS. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com> # Conflicts: # pyegeria/view/egeria_spec_cache.py # tests/micro-tests/test_rs_egeria_tier.py
… (P2, owner decision C) The Architect's ruling (b), 2026-11-14 late. Read as the user, nothing cached: - yes: your own private draft (zoned exactly [userId]), or a held spec, Report or sheet you created (versions.createdBy) or own (Ownership names your user id); - no: a built-in spec (not in Egeria), or nothing in Egeria you can read; - unknown: everything else, e.g. a role-based steward, an owner named by a GUID, a display-name read with no confirmed end, or any failed read. A failed read is never "no". Tests: own draft yes, created yes, owned yes, built-in no, unreadable no, someone else's unknown, a GUID-named owner unknown, a failed read unknown, a Report on an older platform unknown, bad arguments refused. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…is slow P1 read-backs (Architect, sd-dcat): - egeria-table.js's header now says its home is pyegeria's package data: edit it here first, then copy it out unchanged to the Portal and trellis-ui, never the other way round, until they serve pyegeria's copy. The record (egeria-table.js.source.json) has the new sha256 and lines, and says the only difference from the Portal origin commit is that header paragraph. - test_renderer_is_in_the_wheel is marked slow (it builds a wheel; about 5 s). The "slow" marker is registered in the micro-tests conftest; deselect with -m "not slow". - New test: the header names the file's home and the rule. Tests: 6 in test_renderer_package_data.py (5 with -m "not slow"). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ts by type is gated (P3)
DESIGN-P3-AUTHORING-USED-BY-2026-11-14, APPROVED by the Architect's ruling (c), 2026-11-14 late:
- a spec's Reports: get_elements("Report") through the pager (2000 at most), matched on
additionalProperties.reportSpec (label, ReportSpec:: or ReportType::); via "reportSpec";
- a spec's or a sheet's consumer folders: CollectionMembership at end 2, kept to the consumer folder prefixes;
via "CollectionMembership";
- a Report's sheets: DataSetContent at end 2 (Dashboard -> Report), one item per sheet; via "DataSetContent".
No shared consumer index and no cache. complete only when every read is confirmed. A failed or unconfirmed read
goes in notes, never "used by none". An unresolvable Report or sheet refuses with answer absent or unknown.
paging: get_elements / _async_get_elements are added to NULL_NOT_END and GATED_TRUSTED. elements/by-type is
ClassificationExplorerRESTServices.getRootElementsByType -> StewardshipManagementHandler -> findMetadataElements,
the same server method as the gated find paths. dashboards._dashboard_elements passes its client to the gate.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…test and the register's 6 probes Two tests pinned the old behaviour: the NULL_NOT_END set listed its members explicitly, and element_register's read by type expected 3 probes past a null. It now probes 6, as every findMetadataElements path does on a platform without #9453. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…he shell's EGERIA_USER test_rs_egeria_tier failed 30/103 with EGERIA_USER unset or naming someone else (Coordinator, release/next 296abea). Not only the tests: a read outside spec_identity (listing, describing, report_spec_source) used EGERIA_USER / settings.User_Profile.user_name, while a private manifest is bound to the syncing client's credential. A CLI or the MCP server signed in as anyone other than the configured user never loaded its own private cache. current_spec_identity now: spec_identity if set; else the one credential this process's view-server clients use (recorded at BaseServerClient construction and on sync/fetch, pyegeria.core._client_identities); else, with more than one, None (a multi-user process loads no private cache outside a request, logged once); else, with no client, the configured user. client_credential_identity falls back to server_name when a client has no view_server. Tests: a per-test reset of the noted credentials (conftest); exec_report_spec tests name the fake's user; new tests run with EGERIA_USER unset, matching and different, a real client, a multi-user process and no client. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…settings default (READBACK U1)
_reader builds a ClassificationExplorer for a caller with no unique-name read of its own. It passed
user_pwd=getattr(client, "user_pwd", None), and the constructor turns None into the settings default
password. So a client whose password was cleared (Studio, EA) gave the explorer the default, and a 401
auto-retry could sign in again as that default user (the 10-28 re-sign shape, around their scrub).
(a) With a caller token: the explorer gets the token, bare (get_token returns the "Bearer ..." header
value, which gave "Bearer Bearer ..."), and user_pwd is set to None after construction. With no
token: only a password the caller itself holds (and its own user_id). With neither: unknown, no
call to Egeria.
(b) A caller without real view_server/platform_url strings (a fake, a non-client, a MagicMock): unknown,
no client built. This ends the Portal fake's https://test:9443 call.
(c) The explorer is closed after the read, sync or async, including when the read fails.
Tests are hermetic: EGERIA_USER and the password vars are cleared, the settings default is a canary,
and httpx send fails the test. 13 of the 15 new tests fail on the old code.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
P3-1: the artifact is resolved first, through Egeria's exact unique-name read (pyegeria.core.unique_name). "absent" comes only from Egeria's specific not-found; anything else raises AuthoringRefused(answer="unknown"). An unresolvable spec, Report or sheet raises with .answer, and nothing is scanned. A built-in spec Egeria doesn't hold still lists its Reports, with a note. A Report: unique-name read, then the display-name lookup. Any error there that isn't an AuthoringRefused is "unknown". P3-2: spec_label_of strips one ReportSpec:: or ReportType:: prefix. ReportType:: is marked as the legacy loader's form, to go with it in pyegeria 7. The match is then exact and case-sensitive. P3-3: complete and notes still come only from each read's PageResult (complete, at_least_note). A forbidden read is a failed read in the notes, never "not used". Tested. P3-4: the fixtures clear EGERIA_* identity vars. A test runs with the shell user unset, the same as the fake's, and another user, with the same answer. The two-users test uses two clients and no spec_identity. via is the closed set VIA_KEYS (exported, with spec_label_of), and the docstring says to call it on demand. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…; tests hermetic
Architect read-back (2026-11-15) of the EGERIA_USER test dependence:
T2 (bind from the client): every tier read made with a client in hand runs
under that client's credential (new esc.as_client):
- ServerClient._generate_formatted_output (the in-method path every OMVS
manager formats through: resolve, then the provenance footer);
- RegisteredInfo's three spec selections and SolutionArchitect's
design-pattern selection (they select a spec outside it);
- sync_report_specs, fetch_spec_for_call and async_fetch_spec_for_call
(the fetch write-through and its registry reads).
exec_report_spec, the async runner and the dashboard resolver already did.
The client-less fallback stays only for reads with no client.
T3 (one configured user): config.configured_user_id() is EGERIA_USER, then
the settings' user, then the quickstart default. resolve_egeria_credentials
(which builds a CLI's client) and current_spec_identity's no-client fallback
both use it, so a CLI with no configuration binds and loads as the same user
(before, the binding got the demo default and the fallback got None).
T1 (hermetic): the tier fixture clears EGERIA_USER; new tests run the
in-method path and the fetch write-through with EGERIA_USER unset, matching
and different, in a multi-user process; MyEgeria's on-mount test runs as the
test backend's user, not the shell's (it failed under another EGERIA_USER on
296abea too).
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
… user changes nothing READBACK-STUDIO-FIXES 2026-11-15 §1 gate rule: no test may pass only because the developer's shell names a user. The answer is the client's user's, with EGERIA_USER unset, set to the same user, or set to another. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…word Owner ruling 2026-11-14, "Opt-out in 6.x" (Architect). user_pwd=None keeps its 6.x meaning (it takes settings.User_Profile.user_pwd; CLIs pass args.password and rely on it). A token-only caller (Portal, Studio, EA acting for a signed-in user) passes NO_PASSWORD instead (pyegeria.core._no_password): - the client holds user_pwd None and use_default_password False (BaseServerClient, BasePlatformClient, EgeriaTech, Egeria, EgeriaConfig; the 16 OMVS classes that store user_pwd as given read NO_PASSWORD as None); - EgeriaTech, Egeria and EgeriaConfig carry it to every lazily built sub-client (carried_password), and so do pyegeria's helper clients: the spec cache's CollectionManager and the executor's fallback client; - so the default password is never used, and an expired token fails as a 401 instead of the one auto-retry signing in again as the demo user. unique_name._reader is fixed separately (sd-dcat U1, f53bef8); once both are in, it should build its explorer with carried_password (NO_PASSWORD when the caller has a token). Tests: a real EgeriaTech with a demo default configured plus NO_PASSWORD has no password on the client, any sub-client, any client class built directly, or a helper client; the token-expiry retry raises 401 with no re-mint; None still renews with the default (control). Release notes: New + pyegeria 7 plan. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ntity there Architect, delta read-back of 3984046: the seen-count (step 3) can be skipped. The Portal builds clients as its service user and then swaps user_id to the token's subject (egeria_auth.use_token), so the one identity recorded is the service user's, and every identity-less read resolved to it until a second identity was seen. - declare_multi_user() / PYEGERIA_MULTI_USER=1 (pyegeria.core._client_identities, re-exported from egeria_spec_cache): once declared, current_spec_identity is None outside spec_identity (steps 2 and 4 are skipped). The seen-count stays as a backstop for an undeclared server; the docstring says so. - as_client also notes the client's current identity, so a swapped client counts towards the backstop once it is used. - A fetched copy is never kept for, or served to, no identity. - pyegeria's MCP server declares itself multi-user at startup (it acts for every caller and syncs a shared cache). Tests: declared (call and env) + built as the service user then swapped gives None outside spec_identity and the signed-in user inside as_client; undeclared, the swap is caught once the client is used; an undeclared CLI with --userid X reads as X; the footer with two users' fetched copies serves each only their own, declared or not; the MCP server declares itself. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…itect C1) A parent built with user_pwd=None (so the configured default filled it) and then cleared (user_pwd = None: the Portal's use_token, Studio's and EA's scrub) still handed the default password to every sub-client it built lazily, because carried_password passed its None through and the child's constructor filled it again. carried_password now returns NO_PASSWORD when the parent opted out OR holds no password, for sub-clients and pyegeria's helper clients alike. A CLI's client already holds the configured password, so CLIs are unaffected. Tests (the Architect's probe): EgeriaTech built with None and then scrubbed, with a demo default configured, gives no password on any lazily built sub-client or helper client, and the token-expiry retry raises 401 with no re-mint; a CLI parent's password is still carried. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…se carried_password (C2) The Architect's C2 on U1, using fix/no-password-opt-out 7dd6441, merged in (ed7087e3): - caller has a token: ClassificationExplorer(..., user_pwd=NO_PASSWORD). It holds no password from construction (use_default_password False). Before, it held the settings default until it was cleared afterwards. - no token: carried_password(client). With the caller's NO_PASSWORD opt-out, or no password of its own, the answer is unknown and no client is built. Tests: the explorer never holds the settings-default canary, even right after construction, sync or async, with a token or with the caller's own password. A caller that opted out with no token is unknown. A real CollectionManager built with NO_PASSWORD and a token carries no password into the explorer. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…nd no password
_collection_client built its CollectionManager with set_bearer_token(
client.get_token()), but get_token returns the Authorization header value
("Bearer x"), so the helper sent "Bearer Bearer x". It also copied the
caller's password even when it shares the caller's token. Now, the U1 shape:
with a token, the token stripped of any "Bearer " prefix and NO_PASSWORD;
without one, only a password the caller itself holds (carried_password),
never the configured default.
Tests: a stored token with and without the prefix reaches the helper as
exactly "Bearer <token>", with no password; without a token the helper takes
the caller's own password, and none from a scrubbed caller.
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
can_edit(kind, ref, client) -> yes | no | unknown, ruling (b). Architect read-back PASS 2026-11-15. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
used_by(kind, label, client), live as the user; get_elements by type gated. Architect read-back PASS 2026-11-15. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com> # Conflicts: # docs/release-notes/next.md # pyegeria/authoring/__init__.py
…o fix/no-password-opt-out
…(Architect C3) The probe: a parent built with NO_PASSWORD whose token arrives through set_bearer_token (which writes only the headers, never self.token). Helpers that copied self.user_pwd raw and token=self.token got NO token and the configured default password, and signed in again as the demo user: ClassificationExplorer._collection_client, AutomatedCuration. _governance_engine_client, DataDiscovery._asset_maker_client, and (found by the same probe) format_set_executor's fallback client. pyegeria.core._helper_client.helper_client(cls, parent, **kw) now builds all of them, plus egeria_spec_cache._collection_client (532a3c5 folded in), unique_name._reader (sd-dcat's C2 bb4c456, merged in at 3af534b, its unknown-without-token-or-password rule kept) and the lazily built sub-clients of EgeriaTech, Egeria and EgeriaConfig: - the parent's BARE token from its Authorization header (class get_token, so EgeriaConfig's __getattr__ isn't re-entered), else its token, set once; - user_pwd: carried_password(parent), except NO_PASSWORD under a token the parent didn't mint for itself (never renewed, so a password could only re-sign); - _token_minted_for copied only when the parent minted its own token. Tests (test_helper_client.py): the probe and the scrubbed-parent shape over every helper; the expiry retry raises 401 with no sign-in; Bearer-Bearer never; a self-minted token keeps the parent's password and renewal; no token carries only the parent's own password; EgeriaConfig doesn't recurse; and a sweep that fails on any constructor called with another client's user_pwd/token outside helper_client. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…wapped atomically; spec_identity(None) is no one The race at 8dd9e08 (egeria_tier ~1328-1360, one shared _TIER slot): - a reload set raw_key and checked for the new identity before its slow disk load, while specs still held the previous identity's; - a concurrent request for the new identity passed the freshness check and got the previous user's tier, private specs included; - egeria_tier_info then read _TIER["meta"], possibly another identity's. (1) _TIERS: an OrderedDict LRU (TIER_CACHE_SIZE 64) of frozen _TierEntry (raw_key, key, stamp, checked, specs, meta, skipped, dir), keyed by the full raw key, including the identity. A reload builds a new entry and swaps it in with one assignment under _TIER_LOCK. A freshness re-check replaces the entry too (dataclasses.replace); nothing is changed field by field. The identity is resolved once per read. (2) egeria_tier_info reads the caller's own entry. _known_names (the line-638 name read) only peeks at it, because it runs during a load and must not trigger one. (3) The _SPEC_IDENTITY ContextVar defaults to an _UNSET sentinel. spec_identity(None), or an empty identity, is explicitly no one, with no fallback, declared multi-user or not. (4) tests/micro-tests/test_spec_tier_per_identity.py: - a read for b during b's gated slow reload gets b's own tier (fails on 8dd9e08); - four threads alternating identities never cross specs or meta; - tier_info and _known_names read the caller's entry, and _known_names never loads; - entries are replaced whole; - LRU eviction; - explicit None is no one, declared and undeclared, and nested blocks restore it. 6 of the 9 fail on 8dd9e08. The two authoring _state() snapshots read _TIERS instead of the removed _TIER (before, getattr would have silently returned {}). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…own get_token d139c66 looked get_token up on the class for every parent, to keep EgeriaConfig's __getattr__ from building a sub-client to answer (and recursing). That missed a get_token held on the instance (a test's MagicMock caller: test_reader_falls_back_to_classification_manager_then_a_ new_explorer). Only a facade (it holds a _subclient_map) is now asked through its class; any other parent through its own get_token. unique_name._caller_token is removed: helper_client.bare_token replaces it. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
… into release/next The EGERIA tier is cached per identity, built whole and swapped atomically; spec_identity(None) is no one. Architect read-back PASS 2026-11-15. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com> # Conflicts: # docs/release-notes/next.md
…dd133a) into release/next NO_PASSWORD sentinel (7dd6441), C1 carried_password (071bcac), spec-cache helper (532a3c5), U1 + C2 unique-name reader (merged in 3af534b), C3 one helper_client for every delegate client (d139c66, 8dd133a). Architect read-back PASS 2026-11-15. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com> # Conflicts: # docs/release-notes/next.md # pyegeria/core/_base_server_client.py
AuthoringRefused.answer, can_edit, used_by and the get_elements paging note had landed under Deprecations because release/next had no New section until the password opt-out added one. Moved only; no wording changed. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…880f99) into release/next The table renderer ships as pyegeria package data (D3). Architect read-back PASS 2026-11-15 01:43Z. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Bundle: complete paging (pager steps 1-6), spec-cache identity per user (T1-T4, R-A multi-user declaration), NO_PASSWORD opt-out with one helper_client for delegate clients (C1-C3, U1/C2), MCP server credential fix, authoring can_edit (P2) and used_by (P3), MyEgeria fixes, egeria-table renderer package data (P1), load_egeria_report_specs deprecation (removal in pyegeria 7). Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
The T4 tests that construct a real CollectionManager ran its check_connection against https://localhost:9443. They passed on a machine with a platform up and failed in CI (odpi#598, 3.13). The tier fixture now stubs BaseServerClient.check_connection, as other micro-tests do. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundled release 6.2.42, from the laz integration branch. Every change below was reviewed by the Architect and tested on laz. Each commit is signed, with DCO.
What's in it
Paging: a short or empty page is not the end
pyegeria.core.paging.page_all/paging_for, steps 1–6. Every loop that pages now pages through these.get_relationshipsis never trusted.Report-spec cache: one identity per user
EGERIA_USER. Paths that hold a client read as that client (as_client). There is one shared configured-user resolution.declare_multi_user()/PYEGERIA_MULTI_USER=1: a server that acts for many users gets no fallback identity. The MCP server declares itself.spec_identity(None)means "explicitly no one".Passwords: never one the caller didn't give
NO_PASSWORDopts out of the configured default password.user_pwd=Nonekeeps its 6.x meaning.carried_password: a sub-client never holds a password its parent doesn't.pyegeria.core._helper_client.helper_client: the only way pyegeria builds a client that acts for another client. It takes the bare token, set once, and never the default password. A sweep test enforces it.Authoring
AuthoringRefused.answer: absent, unknown or refused.can_edit(kind, ref, client): yes, no or unknown, read as the user.used_by(kind, label, client): live, as the user, with a closedVIA_KEYS.Also
egeria-table.jsships as package data (pyegeria/view/static/).update_timefilled from the element header.Deprecation
load_egeria_report_specs(legacyReportType::) is deprecated, with a warning logged once. It's removed in pyegeria 7, expected early January 2027. pyegeria's own callers no longer use it.Planned for pyegeria 7
See
docs/release-notes/next.mdfor the full notes.Tests
EGERIA_USERandEGERIA_USER_PASSWORDunset, and again withEGERIA_USER=someoneelse. 4308 passed, 0 failed, 5 skipped both ways; the renderer wheel test ran and passed.my_egeria/at top level,serve_my_profile,pyegeria-mcpandpyegeria/view/static/egeria-table.js.Signed-off-by: Dan Wolfson dan.wolfson@pdr-associates.com
🤖 Generated with Claude Code