Skip to content

chore(release): bump version to 6.2.41 (bundle: authoring API, D14 versions, MetadataExpert, additionalProperties merge) - #595

Merged
dwolfson merged 27 commits into
odpi:mainfrom
dwolfson:bundle/6.2.41
Nov 13, 2026
Merged

dwolfson merged 27 commits into
odpi:mainfrom
dwolfson:bundle/6.2.41

Conversation

@dwolfson

@dwolfson dwolfson commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Bundled release for pyegeria 6.2.41. Each branch below was reviewed by itself, then integrated and run locally on laz, before this bundle.

What's in it

  • additionalProperties merge (fix/merge-additional-properties, 447f3e6). A first Create sends the document's additionalProperties without reading the element first, and Updates merge key by key. This is needed before specs can be promoted or re-imported safely.
  • Raw element shape (fix/raw-element-shape, 58aebee). Raw metadata-expert elements are read in view shape, and the asset-domain search matches everything.
  • Dr.Egeria round-trip probe gate (from probe/release-gate 3e80e35). The release-gate probe files.
  • Authoring API, phase 1 (feat/authoring-api, 0b8304f): pyegeria.authoring for report specs, Reports and dashboard sheets. It covers drafts, validate, preview-as-user, diff and publish, with MCP tools.
    • Drafts are private to their author (zone [userId]) and pruned after 30 days.
    • The prune takes only unfiled private drafts, and its zone check fails closed.
  • Published versions are immutable (feat/authoring-d14-published-versions, c685741). Each publish stamps publishedVersion, publishedAt and publishedBy.
    • Pins name a publication, resolved from the element's history.
    • Before an update, the exact target must resolve; otherwise the update is refused.
    • Apply re-checks the version held in the plan.
  • Pin history cap (fix/authoring-pin-history-cap, 4328c5f). History is paged until an empty page; if it's cut short, the pin is refused. AUTHORING_KEYS can be imported from pyegeria.authoring.
  • Preview test without a mocked executor (test/authoring-preview-unmocked, 4780292). Tests only.
  • MetadataExpert command family (metadata-expert-family, a9f0905 + 01562c1).
    • All 16 commands are non-idempotent and not allowed from chat.
    • PURGE and HARD_DELETE need the GUID echoed back.
    • Property values are coerced to the typedef's declared type.
    • Delete Type Def, Update Type Def and Delete Enum Def require the name to be echoed back (Confirm Type Def Name / Confirm Enum Def Name).
  • Type Def GUID check (metadata-expert-family, 3a94922): a supplied type or enum GUID must belong to the echoed name; otherwise nothing is written.
  • Version bump to 6.2.41.

Checks

  • Read-backs: each branch was read back separately before integration.
  • Micro-tests: Python 3.13 tests/micro-tests on this head: 3721 tests, 0 failures, 0 errors, 4 skipped.
  • Probe gate: passed on the raw-element-shape fix, identical to the 002a3ef1 baseline.

Follow-ups (6.2.42)

  • Validation should check that a spec's find method fits its target type, and use the strict attribute check.

Signed-off-by: Dan Wolfson dan.wolfson@pdr-associates.com

🤖 Generated with Claude Code

dwolfson and others added 27 commits November 12, 2026 18:05
Egeria replaces a map property whole, even with mergeUpdate true. A Create
that found its element (and became an Update), or an Update with Merge
Update true, sent only the keys its document gave, so every other key was
removed: on laz a SavedQuery's refresh keys were wiped, and a CDO Report's
search keys nearly were.

- AsyncBaseCommandProcessor._prepare_additional_properties runs once before
  apply_changes() on an update of an existing element. When the update sends
  additionalProperties (the document gives Additional Properties, or the
  processor writes keys of its own: WRITES_ADDITIONAL_PROPERTIES, set on
  ReportProcessor), it reads the stored map and lays the document's keys
  over it; the merged map becomes the value every body builder reads.
- A replace update (Merge Update false) still replaces, with a warning that
  names the stored keys it removes.
- A merge update whose stored map can't be read (element read without its
  properties and a re-read by GUID fails) is refused in words
  (AdditionalPropertiesUnreadable), never sent as a replacement.
- upsert_dashboard_asset merges with the existing Dashboard's map too, and
  its unchanged check compares only the keys given.

A Dr.Egeria document has no way to clear one key (a JSON null is parsed to
the string "null"); Merge Update false is the way to remove keys.

Tests use the laz SavedQuery refresh keys and a 10-key CDO Report.
3.13 micro suite: 3,465 passed. Probe gate (88cf76b, PostgreSQL scratch,
757a5df0): PASS, identical to the baseline.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
The Report fixture is now the stored additionalProperties of
CocoPharma::Report::CDODashboard::MyNinetyDayPlanProgress as read on laz on
2026-11-12 (version 1, LIST): 10 string-valued keys, startFrom included, no
metadataElementTypeName. A Create Report block writing only reportSpec,
outputFormat TABLE and searchString sends all 10 keys with only outputFormat
changed.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…thout reading

A Create Saved Query / Create Report with Additional Properties and no as-is
element sends the document's map as given (plus the Report's own parameter
keys), never reads a stored element, and is neither merged nor refused
(Architect read-back suggestion, 2026-11-12).

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
16 commands wrapping pyegeria/omvs/metadata_expert.py: Create/Update/Delete
Type Def, Create/Delete Enum Def, Count Metadata Elements, Publish Metadata
Element, and Create/Update/Delete Instance, Link/Unlink Instances, Update
Relationship, Classify/Reclassify/Declassify Instance.

Adds processors, dispatcher registration, "Count" in STANDARD_VERBS,
compact spec, generated report specs, Advanced templates and unit tests.

Claude-Session: https://claude.ai/code/session_015pdMuS33mgrngrEv8FTfH1
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
One typed authoring path for report specs, Reports and dashboard sheets
(BRIEF-PYEGERIA-AUTHORING-API-PHASE1-2026-11-13), grown from
report_spec_publish (Stage 1a). It never authenticates: every Egeria read
uses the caller's client, every write the caller's runner, as the user.

- models: SpecDraft / ReportDraft / SheetDraft (Draft union by kind),
  ValidationResult, Diff, AuthoringGates.
- drafts: new_draft, new_from (prototype or template, derived_from recorded),
  DraftStore with EgeriaDraftStore (D16: the element itself, zoned private to
  the author, written as the author, prune_drafts(30) as the author, promotion
  clears the zone; zone name a parameter) and LocalDraftStore (tests, offline).
- serialize: deterministic Dr.Egeria markdown; never a Parent ID; spec rows
  shared with build_publish_document; every Report parameter explicit.
- validate: structural, semantic (target type and column attributes from the
  Properties-class catalog, load-time spec resolution, parameter names and
  types, placements resolve as the user), then Dr.Egeria validate via a
  runner. Never raises; says what it did not check.
- preview: runs the draft's own FormatSet as the caller via the executor's run
  dict, never registered; capped and said; sheets resolved without fetching.
- diff: breaking = removed column/placement, changed parameter type, changed
  target type (and a changed analytic routine).
- lifecycle: load_held, history, publish_plan / withdraw_plan for all three
  kinds (consumer folders checked; restricted publish and built-ins refused in
  words), apply (run_document: confirm + digest).
- versions (D14): a pinned spec version resolved as the user from the spec
  element's history, never through the fetched-name map.
- D7: reportParameters and the camelCase copy are both written (switch kept,
  PYEGERIA_REPORT_CAMEL_PARAMS, default both).
- runner: in-process Dr.Egeria runner as the client's user.
- mcp_tools: authoring_* tools registered on pyegeria's MCP server;
  authoring_apply needs confirmed + digest + an as-user credential and refuses
  the server's service account.

Tests: 59 new (CDO map round trip, Stage 1a golden, catalog-health sheet
read-back replayed from a scratch capture, D7 consistency, D14, D16, MCP
refusals, no Parent ID, preview side effects).

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ateTime

Per ANSWER-DRAFT-PRIVATE-ZONE-2026-11-13 (code-verified on the build laz runs:
validateZoneAccess grants a zone named for a user to that user only; the
search pushdown excludes other users' drafts):

- EgeriaDraftStore zones each draft [<author userId>] and nothing else (no
  marker zone: a control defined for it later would expose every draft).
- Drafts are marked on the element: a spec's contentStatus DRAFT, a Report's
  additionalProperties authoringState: draft (a sheet has no carrier yet).
- A user id that is a GovernanceZone's name, or one of controlled_zones, is
  refused in words.
- prune_drafts removes only marked drafts, by createTime, as the author;
  unmarked or undated elements are reported, never removed.
- Promotion: Declassify Zone Membership, contentStatus ACTIVE /
  authoringState: published. Drafts are never filed into consumer folders.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…yped properties, tests)

- Mark all 16 commands idempotent:false and chat_allowed:false in the compact
  spec; the loader now passes both flags through. Validate output says the
  command is not idempotent.
- PURGE / HARD_DELETE on Delete Instance and Unlink Instances are refused
  unless the new Confirm Delete GUID repeats the GUID; descriptions now say
  PURGE needs a prior soft delete.
- Property values are coerced to the primitive type the instance type's
  definition (and its super types) declares, instead of guessed from the text:
  "007" stays a string. Map, array and enum properties, undeclared properties
  and dict/list values are refused with a message.
- Processor-level tests through the real dispatcher: routing, SOFT_DELETE
  default, purge echo, Replace Properties, refusals, flags.

Claude-Session: https://claude.ai/code/session_015pdMuS33mgrngrEv8FTfH1
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…in search

- pyegeria.core.element_shape.as_view_element: a raw OpenMetadataElement
  (elementGUID, elementProperties.propertyValueMap, classifications list) as
  the view-bean shape every reader expects (elementHeader with Egeria 6's
  named classification slots -- anchor, memento, zoneMembership, ... -- and
  flat, typed properties). A view bean is returned unchanged.
- AsyncBaseCommandProcessor.fetch_element: its metadata-expert fallback
  returned the raw element as is, so the guid, qualifiedName (the ISSUE-59
  duplicate check) and the Memento check read as absent, and the merge saw
  an unknown stored map. It now converts the element.
- AssetCatalog.find_in_asset_domain: "*", None or "" is sent as ".*".
  findAssetsInDomain requires a searchString (OPEN-METADATA-400-022), and
  the shared find helper turned "*" into null.

Tests use real elements read from laz (read only, 002a3ef1): the live
openlineage.events Topic and its archived (Memento) twin.
3.13 micro suite: 3,472 passed.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Files from probe/release-gate 3e80e35 (re-pinned to Egeria 002a3ef1, #9448,
baseline 002a3ef1, known gaps trimmed to Project Scope). The probe branch
carries its own history from 18178918, so its files are taken as they stand
at 3e80e35 rather than merging the tip; none of them was changed on
release/next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…one check fails closed

Read-back 2026-11-13 F1, F2, F6:
- A sheet has no draft marker, so it is no longer treated as marked. Prune
  deletes a sheet only when its zones are exactly [userId], its qualified
  name is Dashboard::..., and it belongs to no collection (a filed sheet is
  published even if its Declassify step didn't run). When its memberships
  can't be read it is kept. Every kind now needs zones == [userId] exactly.
- Age by last update (falling back to creation), not creation.
- `removed` lists only real deletes; a delete that found nothing goes to
  `not_deleted`; each kept-but-old element gets a reason in `kept_reasons`.
- The GovernanceZone name check refuses when the lookup errors (only a
  not-found answer passes), and compares names case-insensitively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…pdate can't do

Read-back 2026-11-13 F3, F4, F4b:
- F3: reportSpecVersion is recorded and honoured by validate/preview, but
  nothing that runs a placed Report reads it. publish_plan adds a step
  saying so, validate lists it in not_applied, preview notes it. The stale
  "a pinned version is refused" line in validate's doc is corrected.
- F4: a sheet update adds and updates placements in place; it can't remove
  or reorder them. The plan says so instead of "in order", and diff's
  removed-placement and order changes carry the same words in `why`.
- F4b: mcp_tools documents that the local draft owner is unauthenticated
  (fine for a per-user stdio server, not a shared one).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…y tile included

Read-back 2026-11-13 point 1 / F5. The scratch replay has 6 placements;
laz's sheet has 7 (the saved-query tile, LOCAL_DASHBOARDS_SAVED_QUERY_TILE).
The new fixture is copied unchanged from the Portal's real-shape capture
(egeria-workspaces PyegeriaWebHandler/tests/fixtures/
local_dashboards_real_shape_2026_11_12.json, read-only from laz): the sheet
in reader shape and its 6 Report elements. Tests: 7 placements in order,
serialize/parse round trip, the Saved Query Result Report's cached-count
analytic parameters, and each Report's round trip through the serializer
(only the common find-parameter defaults are added).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…blished version

Owner decision 2026-11-13 (BRIEF-PYEGERIA-AUTHORING-D14-PUBLISHED-VERSIONS).
A new version under a stable identity, not a new element per version:

- Publication stamp: every publish writes publishedVersion (held stamp + 1),
  publishedAt and publishedBy into the element's own Create/Update block, so
  stamp and content land in one Egeria version. publishedAt is set when the
  plan runs; the digest leaves it out. Sheets: Create Dashboard Sheet now
  passes the document's Additional Properties to the Dashboard.
- Exact target or refuse: async_publish_plan resolves the element by exact
  qualifiedName (a Report without one by exact display name), as the user,
  page 25 (B5). A failed read, two matches or a full page refuses; publish_plan
  without a resolved target refuses.
- No silent overwrite: async_apply re-resolves the target and refuses when its
  GUID or publishedVersion changed since the plan.
- Pins name publications: reportSpecVersion resolves to the publication entry
  of that publishedVersion; missing or unversioned is refused in words. The
  entry is the OLDEST one stamped N (the publish write), not the newest: an
  edit outside publishing keeps the stored map, so it carries the stamp too.
- Edited since publication is a note in the plan, validate (not_applied) and
  preview, never an error; a zone-only change doesn't count.
- An element without a stamp is unversioned: its first publish stamps 1.
  No backfill writes.
- Promotion stamps in the Create/Update, before the Declassify.
- Nothing in pyegeria.authoring deletes a published element: draft prune keeps
  it and the draft store's delete refuses.
- The stamp, marker and pin keys are bookkeeping, never Report parameters
  (dashboard_resolver.report_from_element, report_draft_from_element).
- MCP: a publish plan now needs as_user (it reads Egeria as the user).

Tests: test_authoring_d14_published_versions.py (22, raw history in the laz
OpenMetadataElement shape); existing authoring tests plan with a resolved
target, and the Stage 1a golden is compared less the stamp row.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…ort AUTHORING_KEYS

D14 read-back follow-up 1. The oldest entry stamped N is published version N
only if the whole history was read: a capped read could hide the publication
and leave a later edit (still stamped N) as the oldest returned.

- publication.async_read_history pages the element history (100 a page) until
  Egeria answers an empty page (a short page isn't the last), up to 50 pages.
  Running out of pages, or a failed read, returns the reason instead.
- A pin on such a history is refused in words ("history is too long to read
  whole"); publish planning keeps going but says the edited-since check
  wasn't done.
- AUTHORING_KEYS and STAMP_KEYS are exported from pyegeria.authoring, a
  stable public path for the Portal's by-name report path.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…etch or tier state

Read-back 2026-11-13 F7. The existing preview test fakes exec_report_spec, so
it only proves preview.py registers nothing. These run pyegeria's own executor
(exec_report_spec -> _run_analytic_function) on the one-spec dict preview
builds. The routine is a registered analytic function in an injected module
that declares no Egeria client, so no server is needed, and EgeriaTech fails
the test if anything builds a client. Checked: the registry, _FETCHED and _TIER
are unchanged, the draft label is never registered, the routine ran once per
preview with the caller's page size, and the 40 rows came back capped to 5
and said so.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
A shallow copy shares the inner per-tier entries, so a mutation inside one
would compare equal before and after (read-back of 86af9e2).

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Delete Type Def and Update Type Def are refused unless Confirm Type Def Name
repeats Type Def Name exactly; Delete Enum Def likewise with Confirm Enum Def
Name. The check runs before any call to Egeria, the refusal says the change is
platform-wide and nothing was changed, and validate mode reports whether the
echo is present and matching without calling Egeria. Create Type Def and
Create Enum Def are additive and need no echo.

Claude-Session: https://claude.ai/code/session_015pdMuS33mgrngrEv8FTfH1
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Delete Type Def, Update Type Def and Delete Enum Def now read the definition
behind a supplied Type Def GUID / Enum Def GUID (a read only: Egeria has no
read-by-GUID, so the definitions are listed and matched) and refuse, with no
write, unless its name equals the Confirm ... Name echo exactly. The refusal
names both the GUID's actual name and the echo; an unresolvable GUID is
refused too. Validate mode reports the mismatch and writes nothing. Without
this, a wrong GUID with the right echo changed a different type platform-wide.

Claude-Session: https://claude.ai/code/session_015pdMuS33mgrngrEv8FTfH1
Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
…t belong to the echoed name

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
@dwolfson
dwolfson merged commit 5ccce39 into odpi:main Nov 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant