Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
bf45d29
Refuse a non-ElementStatus value in limitResultsByStatus, in words, b…
dwolfson Nov 13, 2026
8d4ec55
Element-status refusals name the status property and where it lives; …
dwolfson Nov 13, 2026
05828fc
Element-status drift guard is opt-in; note -W error behaviour of the …
dwolfson Nov 13, 2026
8a1ef92
Merge fix/element-status-filter into release/next (6.2.42)
dwolfson Nov 13, 2026
98d71ef
Fill the update_time report column from the element header
dwolfson Nov 13, 2026
3e2f717
Merge fix/update-time-column into release/next
dwolfson Nov 13, 2026
82ca106
Add pyegeria.core.paging.page_all: an empty page is not the end
dwolfson Nov 13, 2026
54d4774
Pager: key rows by elementGUID/relationshipGUID too, and by content
dwolfson Nov 13, 2026
a9685bf
Pager step 2: record what Egeria sent before [] is collapsed
dwolfson Nov 13, 2026
68f95bb
Pager step 3: report runs carry end; page past an empty first page
dwolfson Nov 13, 2026
1f60d93
Pager step 3 fix: bound paging on after an empty first page; plain wo…
dwolfson Nov 13, 2026
f0d95bb
Pager step 4 (part 1): 10 of the 11 loops onto page_all, ruling (A)
dwolfson Nov 14, 2026
d0f1159
Pager step 4 (part 2): overview_metrics scans onto page_all, ruling (ii)
dwolfson Nov 14, 2026
7379bd8
Pager step 4 (part 3): the Architect's ruling after the capture
dwolfson Nov 14, 2026
ad17748
Merge fix/pager-null-vs-empty into release/next
dwolfson Nov 14, 2026
7f6a534
Pager step 5 (part 1): every page_all through paging_for; the viewer …
dwolfson Nov 14, 2026
dfeb411
Pager step 5 (part 1, completion): paging_for in dashboard_sheet, pro…
dwolfson Nov 14, 2026
d060303
Dr.Egeria: a lookup error is never "absent" (resolve_element_guid fix)
dwolfson Nov 14, 2026
128a969
Pager step 5 (part 2): exists-lookups, three-way; publish by unique n…
dwolfson Nov 14, 2026
da8d036
Publish: only Egeria's "not found" is absent (step 5 read-back R1)
dwolfson Nov 14, 2026
45add51
Merge fix/pager-null-vs-empty step 5 into release/next
dwolfson Nov 14, 2026
5b8ecd4
Add a read-only capture script for the catalog_health_scratch fixture…
dwolfson Nov 14, 2026
b5250e4
Pager step 6: the #9453 version gate (find and by-property-value trus…
dwolfson Nov 14, 2026
923b030
Pager step 6: unknown references say "couldn't tell"; drl byModality …
dwolfson Nov 14, 2026
e2ec66c
A "server not available" 404 is never "not found" (the catalog-health…
dwolfson Nov 14, 2026
6e875b1
Pager step 6 gate: later SNAPSHOTs promoted; short TTL for a failed a…
dwolfson Nov 14, 2026
dd195d6
DCAT fixture capture: clinical_trial_files searches PROJ-CT-TBDF_00002
dwolfson Nov 14, 2026
1447619
Pager step 6: ai_ready_assets and drl_readiness_gates page their Asse…
dwolfson Nov 14, 2026
3d11b77
Publication docs: Egeria refuses a duplicate qualifiedName on create …
dwolfson Nov 14, 2026
4a89864
Re-pin the replay fixtures on the dd195d6e re-captures; drop the synt…
dwolfson Nov 14, 2026
2c9464d
Pager step 6 gate: close the about-read worker thread's session and loop
dwolfson Nov 14, 2026
85f5bc1
load_egeria_report_specs: page through the pager; only a confirmed en…
dwolfson Nov 14, 2026
5e453f7
lifecycle._report_element: a Report by display name, three-way (never…
dwolfson Nov 14, 2026
57c3784
load_egeria_report_specs: a load status beside the bool; the CLI says…
dwolfson Nov 14, 2026
7d6c522
load_egeria_report_specs: deprecated (legacy ReportType::); retry bac…
dwolfson Nov 14, 2026
3dbc380
load_egeria_report_specs: removal in pyegeria 7 (early January 2027);…
dwolfson Nov 14, 2026
d941438
Move pyegeria's own callers off the deprecated legacy report-spec loader
dwolfson Nov 14, 2026
891dd6f
load_egeria_report_specs: the deprecation is also logged once at WARN…
dwolfson Nov 14, 2026
950d5a7
Merge fix/pager-null-vs-empty step 6 into release/next
dwolfson Nov 14, 2026
0b3efdd
fix(mcp_server): sign in as the configured identity, never a hard-cod…
dwolfson Nov 14, 2026
72447ab
Merge chore/off-legacy-report-spec-loader + fix/mcp-server-no-hardcod…
dwolfson Nov 14, 2026
1287cd2
fix(my_egeria): follow Egeria's paging contract; write guards are thr…
dwolfson Nov 14, 2026
0527621
paging: the by-name reads are findMetadataElements underneath (NULL_N…
dwolfson Nov 14, 2026
47ece6f
fix(my_egeria): display words key on a confirmed end; QN guards use t…
dwolfson Nov 14, 2026
46536f7
Merge fix/myegeria-paging-exists into release/next
dwolfson Nov 14, 2026
697f2b0
feat(view): egeria-table.js ships as pyegeria package data (D3)
dwolfson Nov 14, 2026
7d7cccd
view: exact qualified-name reads use Egeria's unique-name read, three…
dwolfson Nov 14, 2026
1a8c147
spec cache: the label lookup is paged with paging_for; "not found" on…
dwolfson Nov 14, 2026
b949d40
mcp_server: the startup spec sync writes a shared cache (never a zone…
dwolfson Nov 14, 2026
36f326a
Merge fix/collections-by-name-dispositions into release/next
dwolfson Nov 14, 2026
8f8560b
spec cache: a pinned spec is pruned only when its element is gone; a …
dwolfson Nov 14, 2026
3581806
Merge fix/pinned-spec-prune-three-way into release/next
dwolfson Nov 14, 2026
5e38d48
authoring: AuthoringRefused.answer = absent | unknown | refused, so c…
dwolfson Nov 14, 2026
c9ed48e
spec cache: a fetched zoned spec is never written to a disk manifest …
dwolfson Nov 14, 2026
ab34ed6
Merge feat/authoring-refused-answer into release/next
dwolfson Nov 14, 2026
b09ae69
Merge mcp_server shared spec sync (b949d409) into release/next
dwolfson Nov 14, 2026
6c5008a
spec cache: a private manifest is bound to its sync identity (ruling …
dwolfson Nov 14, 2026
70343e0
test: item-2 fetch tests read visibility through egeria_tier_info, no…
dwolfson Nov 14, 2026
75e9d65
spec cache: a fetched spec is served only to the identity that fetche…
dwolfson Nov 14, 2026
8cb444c
Merge spec-cache identity items 3 + 4 into release/next
dwolfson Nov 14, 2026
296abea
Merge spec-cache identity item 2 into release/next
dwolfson Nov 14, 2026
3ad04a7
authoring: can_edit(kind, ref, client) -> yes | no | unknown with why…
dwolfson Nov 14, 2026
e880f99
chore(view): egeria-table.js says where it is edited; the wheel test …
dwolfson Nov 14, 2026
58557b6
authoring: used_by(kind, label, client), live as the user; get_elemen…
dwolfson Nov 14, 2026
282a330
tests: get_elements by type is NULL_NOT_END now (P3): the method-set …
dwolfson Nov 15, 2026
3984046
spec cache: the default identity is the process's own client, never t…
dwolfson Nov 15, 2026
f53bef8
unique_name: the explorer it builds is the caller exactly, never the …
dwolfson Nov 15, 2026
14366ac
authoring.used_by: P3-1..P3-4 (the Architect's approval, 2026-11-15)
dwolfson Nov 15, 2026
9d3d39f
spec cache T1-T3: paths with a client bind to it; one configured user…
dwolfson Nov 15, 2026
be6af07
tests: can_edit fixtures are hermetic (EGERIA_* cleared); the shell's…
dwolfson Nov 15, 2026
7dd6441
clients: user_pwd=NO_PASSWORD opts out of the configured default pass…
dwolfson Nov 15, 2026
8dd9e08
spec cache R-A: a multi-user process declares itself; no fallback ide…
dwolfson Nov 15, 2026
586f3d3
Merge fix/no-password-opt-out 7dd64417 into the U1 branch (for C2)
dwolfson Nov 15, 2026
071bcac
clients: a sub-client never holds a password its parent doesn't (Arch…
dwolfson Nov 15, 2026
bb4c456
unique_name: the explorer is built with NO_PASSWORD under a token, el…
dwolfson Nov 15, 2026
532a3c5
spec cache: its helper client gets the caller's token once-prefixed a…
dwolfson Nov 15, 2026
73d9b40
Merge spec-identity T1-T3 + R-A (fix/spec-identity-hermetic 8dd9e087)…
dwolfson Nov 15, 2026
b7da68a
Merge P2 can_edit (feat/authoring-can-edit be6af07c) into release/next
dwolfson Nov 15, 2026
a7eb344
Merge P3 used_by (feat/authoring-used-by 14366ac7) into release/next
dwolfson Nov 15, 2026
3af534b
Merge remote-tracking branch 'origin/fix/unique-name-no-password' int…
dwolfson Nov 15, 2026
d139c66
clients: one helper_client builds every client that acts for another …
dwolfson Nov 15, 2026
50dea06
spec cache T4: the EGERIA tier is cached per identity, built whole, s…
dwolfson Nov 15, 2026
8dd133a
helper_client: ask a facade's class for its token, anything else its …
dwolfson Nov 15, 2026
f634ca4
Merge T4 per-identity tier cache (fix/spec-identity-hermetic 50dea06e…
dwolfson Nov 15, 2026
de87cd1
Merge the password opt-out, C1-C3 and U1/C2 (fix/no-password-opt-out …
dwolfson Nov 15, 2026
2fcb350
docs(release-notes): file the new authoring and paging entries under New
dwolfson Nov 15, 2026
68e1024
Merge P1 egeria-table.js package data (feat/egeria-table-package-data…
dwolfson Nov 15, 2026
171910c
chore(release): bump version to 6.2.42
dwolfson Nov 15, 2026
12e9354
test(spec cache): tier tests never reach a platform
dwolfson Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,6 @@
recursive-include tests *.py *.ini *.ipynb *.md *.txt
recursive-include pyegeria/view/schemas *.json
recursive-include pyegeria/view/seed *.json
recursive-include pyegeria/view/static *.js *.json
recursive-include sample-data/templates/basic *.md
recursive-include sample-data/templates/advanced *.md
125 changes: 67 additions & 58 deletions commands/cat/load_report_specs.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@
SPDX-License-Identifier: Apache-2.0
Copyright Contributors to the ODPi Egeria project.

Refresh the in-process report-spec registry from Egeria.

Reads ReportType collections and their QuestionSpec folders from Egeria and
merges question_specs into the runtime registry so that find_report_specs_by_*
and the MCP find_report_specs tool return Egeria-aware results.
DEPRECATED command name, kept until pyegeria 7: it now runs the spec cache's sync
(pyegeria.view.egeria_spec_cache.sync_report_specs, the same as `pyegeria report-specs sync`), which
loads the report specs held in Egeria into the cache the registry's EGERIA tier reads. It used to run
the legacy ReportType:: loader (load_egeria_report_specs), deprecated and removed in pyegeria 7.

Usage (standalone):
load_report_specs [--force] [--ttl <seconds>]
Expand All @@ -28,12 +27,6 @@

from pyegeria.core.config import settings
from pyegeria.egeria_tech_client import EgeriaTech
from pyegeria.view.base_report_formats import (
load_egeria_report_specs,
report_spec_list,
_EGERIA_SPECS_LOADED_AT,
_EGERIA_SPECS_CACHE_TTL,
)

EGERIA_USER = os.environ.get("EGERIA_USER", "erinoverview")
EGERIA_USER_PASSWORD = os.environ.get("EGERIA_USER_PASSWORD", "secret")
Expand All @@ -52,60 +45,76 @@ def _run(
jupyter: bool = settings.Environment.egeria_jupyter,
) -> None:
console = Console(width=width, force_terminal=not jupyter)

from pyegeria.view import base_report_formats as _brf
last = getattr(_brf, "_EGERIA_SPECS_LOADED_AT", None)
effective_ttl = ttl_seconds if ttl_seconds is not None else _EGERIA_SPECS_CACHE_TTL

if not force and last is not None:
from datetime import datetime, timezone
age = (datetime.now(timezone.utc) - last).total_seconds()
if age < effective_ttl:
console.print(
f"[green]Registry is fresh[/green] (loaded {age:.0f}s ago, TTL={effective_ttl}s). "
f"Use --force to reload anyway."
)
return

# The owner's direction (2026-11-14): our own callers move off the deprecated legacy ReportType:: loader
# (load_egeria_report_specs, removed in pyegeria 7). This command now runs the spec cache's sync, the
# system of record for Egeria-held report specs, exactly as `pyegeria report-specs sync` does.
console.print(DEPRECATION_WORDS)
if force or ttl_seconds is not None:
console.print("[dim]--force and --ttl applied to the legacy loader's cache and are ignored now: a sync "
"reads Egeria each time.[/dim]")
from pyegeria.view.egeria_spec_cache import SyncAlreadyRunning, sync_report_specs
client = EgeriaTech(view_server, view_url, user_id=user, user_pwd=user_pass)
client.create_egeria_bearer_token(user, user_pass)

before = set(report_spec_list())
start = time.perf_counter()
updated = load_egeria_report_specs(client, force=force, ttl_seconds=ttl_seconds)
elapsed = time.perf_counter() - start

client.close_session()

after = set(report_spec_list())
new_specs = sorted(after - before)

if updated:
console.print(
f"[green]Registry updated[/green] from Egeria in {elapsed:.2f}s. "
f"{len(new_specs)} new spec(s) added."
)
if new_specs:
for s in new_specs:
console.print(f" + {s}")
else:
console.print(
f"[yellow]No update performed[/yellow] — Egeria returned no ReportType collections "
f"(or registry was already fresh)."
)
try:
client.create_egeria_bearer_token(user, user_pass)
report = sync_report_specs(client)
except SyncAlreadyRunning as exc:
console.print(f"[yellow]{exc}[/yellow]")
return
finally:
try:
client.close_session()
except Exception: # noqa: BLE001
pass
console.print(report.render())
console.print(f"[dim]({time.perf_counter() - start:.2f}s)[/dim]")


#: Printed by load-report-specs / load_report_specs since it moved to the spec cache.
DEPRECATION_WORDS = ("[yellow]load-report-specs is deprecated[/yellow]: it ran the legacy ReportType:: loader, "
"which will be removed in pyegeria 7 (expected early January 2027). It now runs the spec "
"cache's sync (`pyegeria report-specs sync`), which loads Egeria-held report specs.")


#: An empty result from the legacy ReportType:: loader, confirmed or not (the legacy-loader ruling, 2026-11-14).
NO_LEGACY_WORDS = "No legacy ReportType collections found (Egeria-held specs load through the spec cache)."


def load_status_words(status: dict, elapsed: float, new_specs: list) -> str:
"""Words for how the deprecated load_egeria_report_specs ended (85f5bc15 read-back R1): one sentence per
status, never "no ReportType collections" unless Egeria confirmed the end. Kept for anyone still calling the
legacy loader until pyegeria 7; this command itself now runs the spec cache's sync."""
kind, count = status.get("status"), status.get("count") or 0
retry_at, error = status.get("retry_at"), status.get("error")
if kind in ("loaded", "registered_unconfirmed") and not count:
# the legacy-loader ruling: never imply specs are missing (they load through the spec cache)
return NO_LEGACY_WORDS
if kind == "loaded":
return (f"[green]Registry updated[/green] from Egeria in {elapsed:.2f}s: {count} ReportType(s) read, "
f"{len(new_specs)} new spec(s) added.")
if kind == "registered_unconfirmed":
return (f"[yellow]Registered {count} spec(s)[/yellow] in {elapsed:.2f}s; Egeria didn't confirm there "
f"are no more; will re-read after {retry_at}.")
if kind == "retry_window":
return (f"[yellow]Not re-read[/yellow]: the last read wasn't confirmed complete; next read after {retry_at} "
f"(use --force).")
if kind == "fresh":
return "[green]Registry is fresh[/green]: the last confirmed load is within its TTL (use --force)."
if kind == "failed":
return f"[red]Not loaded[/red]: Egeria couldn't be queried ({error}); nothing was marked loaded."
return f"[yellow]Load ended without a status[/yellow] ({elapsed:.2f}s)."


# ── Click command (for hey_egeria) ────────────────────────────────────────────

@click.command("load-report-specs")
@click.option("--force", is_flag=True, default=False,
help="Bypass the TTL cache and reload from Egeria unconditionally.")
help="Ignored (it applied to the legacy loader's cache); a sync reads Egeria each time.")
@click.option("--ttl", "ttl_seconds", default=None, type=int,
help="Override cache TTL for this call only (seconds). "
f"Default: PYEGERIA_REPORT_SPECS_CACHE_TTL env var or {_EGERIA_SPECS_CACHE_TTL}s.")
help="Ignored (it applied to the legacy loader's cache).")
@click.pass_context
def load_report_specs_cmd(ctx, force: bool, ttl_seconds: int | None) -> None:
"""Refresh the report-spec registry from Egeria (ReportTypes and QuestionSpecs)."""
"""DEPRECATED: syncs the report specs held in Egeria into the spec cache (use `pyegeria report-specs sync`)."""
c = ctx.obj
_run(
view_server=c.view_server,
Expand All @@ -121,13 +130,13 @@ def load_report_specs_cmd(ctx, force: bool, ttl_seconds: int | None) -> None:

def main() -> None:
parser = argparse.ArgumentParser(
description="Refresh the report-spec registry from Egeria."
description="DEPRECATED: sync the report specs held in Egeria into the spec cache "
"(use `pyegeria report-specs sync`)."
)
parser.add_argument("--force", action="store_true",
help="Bypass the TTL cache and reload unconditionally.")
help="Ignored (it applied to the legacy loader's cache).")
parser.add_argument("--ttl", dest="ttl_seconds", type=int, default=None,
help=f"Override cache TTL (seconds). "
f"Default: PYEGERIA_REPORT_SPECS_CACHE_TTL env var or {_EGERIA_SPECS_CACHE_TTL}s.")
help="Ignored (it applied to the legacy loader's cache).")
parser.add_argument("--server", dest="server",
default=app_config.egeria_view_server,
help="Egeria view server name")
Expand Down
45 changes: 24 additions & 21 deletions commands/tech/list_element_register.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,23 +75,23 @@ def _display_name(properties: dict, qualified_name: str) -> str:
return qualified_name or "---"


def _fetch_all_elements(c_client: EgeriaTech, om_type: str, page_size: int) -> list[dict]:
"""Page through get_elements until an empty page comes back.

Per Egeria's own paging contract, a short page does not mean the last
page -- only an empty result does -- so this keeps asking until that
happens rather than stopping the moment a page comes back under
page_size.
def _fetch_all_elements(c_client: EgeriaTech, om_type: str, page_size: int,
notes: list[str] | None = None) -> list[dict]:
"""Every element of `om_type`, read through pyegeria's pager (pager step 4; a suspect path).

Per Egeria's paging contract a short page is not the last page and an empty one is a filtered page, not
the end: only null ends the read, and on this path even null isn't proof (the pager probes past it).
When the end isn't confirmed, `notes` (if given) gets "Showing at least N elements: ...", so the
register's counts are never shown as totals.
"""
all_elements: list[dict] = []
start_from = 0
while True:
page = c_client.get_elements(om_type, start_from=start_from, page_size=page_size, output_format="JSON")
if not isinstance(page, list) or len(page) == 0:
break
all_elements.extend(page)
start_from += page_size
return all_elements
from pyegeria.core.paging import at_least_note, log_end, page_all, paging_for
paged = page_all(lambda start, size: c_client.get_elements(om_type, start_from=start, page_size=size,
output_format="JSON"),
fetch_page_size=page_size, **paging_for("get_elements"))
log_end("list_element_register._fetch_all_elements", paged)
if notes is not None and (words := at_least_note(paged, "elements")):
notes.append(words)
return [e for e in paged.items if isinstance(e, dict)]


def element_register(
Expand All @@ -115,7 +115,9 @@ def element_register(
sys.exit(1)

try:
raw_elements = _fetch_all_elements(c_client, om_type, page_size)
read_notes: list[str] = []
raw_elements = _fetch_all_elements(c_client, om_type, page_size, read_notes)
at_least = "≥" if read_notes else "" # pager step 4: counts are "at least" unless the end was confirmed

rows = []
for element in raw_elements:
Expand Down Expand Up @@ -154,9 +156,9 @@ def element_register(

stats = Table.grid(padding=(0, 3))
stats.add_row(
Text(str(len(rows)), style="bold white"),
Text(str(len(groups)), style="bold white"),
Text(str(len(distinct_classifications)), style="bold white"),
Text(f"{at_least}{len(rows)}", style="bold white"),
Text(f"{at_least}{len(groups)}", style="bold white"),
Text(f"{at_least}{len(distinct_classifications)}", style="bold white"),
)
stats.add_row(
Text("ELEMENTS", style="dim"),
Expand All @@ -169,7 +171,8 @@ def element_register(
header_lines.append(f"Search: '{search}'")
if classification:
header_lines.append(f"Classification filter: '{classification}'")
header_lines.append(f"{len(rows)} of {len(raw_elements)} shown · {time.asctime()}")
header_lines.append(f"{len(rows)} of {at_least}{len(raw_elements)} shown · {time.asctime()}")
header_lines.extend(read_notes)

panel = Panel(
Group(stats, Padding(Text("\n".join(header_lines), style="dim"), (1, 0, 0, 0))),
Expand Down
8 changes: 8 additions & 0 deletions docs/design/report_spec_migration_design.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Design: Migrating Report Specs to Egeria

> **Superseded loader (2026-11-14).** `load_egeria_report_specs()`, the loader this document describes, is the
> legacy path for the May-2026 `ReportType::` collections. It is **deprecated and will be removed in pyegeria 7
> (expected early January 2027)**. Egeria-held report specs are `ReportSpec::` elements, and they load through
> the spec cache: `pyegeria.view.egeria_spec_cache.sync_report_specs` (`pyegeria report-specs sync`), the system of
> record. It reads both markers and serves the registry's EGERIA tier, the consumer index and the authoring API.
> pyegeria's own callers have moved: the MCP server's startup and the `load_report_specs` / `hey_egeria ...
> load-report-specs` command now run the sync. The rest of this document is the May-2026 design, kept as history.

**Status:** Phase 1 (§6) shipped and complete — see "Migration status" below.
Phase 2 (`formats`/`action`/`target_type`/`family`) not started.
**Date:** 2026-05-17
Expand Down
Loading
Loading