Skip to content
Merged
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ All notable changes to `@harperfast/agent-memory` are documented here. The forma

## [Unreleased]

### Added

- `scanMemories({ after?, includeInactive? })` reads every memory as stored, embedding included, oldest `createdAt`
first, for a trusted plugin building derived data (a first build, or a rebuild after it was not loaded). It
resumes after a given row without skipping or repeating rows that share a timestamp.
- `registerBootstrapSection({ name, render })` adds a section to `GET /Memory/bootstrap`, after `pinned` and
`needsReview` and before `recent`, rendered for each caller within the budget those sections left. Its items come
back as `memories` entries with a `section` and no `id`, so existing clients show them as stored data without a
change.
- `registerSearchAnnotator({ name, annotate })` adds per-hit data under its name to search results with `q`
(`GET /Memory/?q=`, MCP `recall`). It cannot change which hits come back or their order.
- A section or annotator that throws is left out and logged, and the request still succeeds. All three reach the
running plugin from another app on the same Harper, like `registerMemorySink`.

## [0.1.4] - 2026-11-11

### Added
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@ update it in the same PR when a rule changes. Follow the harper-engineering-guid
Harper DLC for every PR.

- Plugin shape mirrors `@harperfast/oauth`: `config.yaml` declares `pluginModule` + `graphqlSchema`.
- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer.
- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer, except `scanMemories`,
the unguarded read for trusted in-process plugins that no route reaches.
- Scope enforcement is pushed into queries as conditions, never applied as a post-filter on top-k.
80 changes: 69 additions & 11 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ allowlists, idempotent `PUT` create/revise with `If-Match` revision precondition
| Customer-content flag and quarantine (`customerContentCheck` option) | implemented (`src/quarantine.ts`) |
| Supersession backfill: candidates for an existing memory, links between existing memories, created-order cursor | implemented (`src/lifecycle.ts`) |
| Package exports reaching the plugin from another app (Extension points, Across apps) | implemented (`src/runtime.ts`) |
| `scanMemories`, bootstrap sections and search annotators for derived plugins (Extension points) | implemented (`src/extensions.ts`) |

## Data model

Expand Down Expand Up @@ -67,7 +68,8 @@ database).
- `MemoryWatch` β€” a principal's time-bounded interest in specific memories (live updates).

See `schema/memory.graphql`. The table is not `@export`ed: every read and write goes through the
`Memory` resource so the rules below have exactly one enforcement point.
`Memory` resource so the rules below have exactly one enforcement point. The one exception is `scanMemories`
(Extension points), an unguarded read for trusted in-process plugins that no route reaches.

Server-owned fields: `ownerId`, `status`, `revision`, `author`, `lastEditor`, `supersededBy`,
`autoSupersedes`, `evidenceMemoryIds`, `needsReview`, `needsReviewSince`, `confirmed`, `contentFlag`,
Expand Down Expand Up @@ -492,7 +494,8 @@ lease, since a batch writes up to 100 rows and flags their citers inside one tra
`contentFlagged = true`, the review queue of a cleared reviewer (an uncleared caller sees only its own). Results include `supersededBy` when `includeSuperseded` is set,
and `needsReview` under the Provenance visibility rule; withdrawn memories never appear (except to a `super_user` passing `includeWithdrawn=true`, below). The vector lane has no distance cutoff, so it contributes the
nearest readable memories whatever their distance. Passing `watch=true` records a `MemoryWatch` (with
the default 1 h TTL) for every memory id returned in the result.
the default 1 h TTL) for every memory id returned in the result. A search with `q` (REST, or MCP `recall`) also
carries each registered search annotator's value for a hit (Extension points); a listing without `q` carries none.
- **Full-text query boundary** (observed locally on Harper 5.3.2, native fulltext 0.5.0):
`Array.from({ length: 65 }, (_, i) => 'w' + i).join(' ')` is only 249 characters but is rejected;
64 distinct analyzed terms succeed. Native `any` reports
Expand Down Expand Up @@ -562,10 +565,10 @@ true` puts `@embed`'s HNSW index on the native plane). Harper 5.3 makes a query
`principal.id`, newest pin first, whose target memory is active, unexpired and readable by the caller), then the caller's own
memories that need review (section `needsReview`: up to 20 the caller may see, newest flag first,
through the indexed `needsReviewSince`, from at most the 100 newest flags, so hidden flags cannot make a
bootstrap scan without bound), then the caller's recent
bootstrap scan without bound), then the sections other plugins registered (Extension points), then the caller's recent
memories, then memories relevant to `q`, deduplicated and packed into a character budget (title +
content; default 8000). Expired memories (`validUntil <= now`) are excluded from every section.
A memory that does not fit is skipped rather than ending the pack. Passing `watch=true` records a
A memory or contributed item that does not fit is skipped rather than ending the pack. Passing `watch=true` records a
`MemoryWatch` (default 1 h TTL) for every memory id returned across all sections. Used at session start.
- Ranking may add recency decay and retrieval/citation counts later; decay is a ranking function only,
never deletion.
Expand Down Expand Up @@ -673,6 +676,54 @@ as REST:
sink registered or unregistered during a dispatch takes effect from the next one. An idempotent
`PUT` retry calls no sink. harper-eye uses it for the interim Reflex source-row handoff until memory is a
Reflex lane.
- `scanMemories({ after?, includeInactive? })` β€” an async iterable of `Memory` rows as stored (the view's fields with
their stored types, plus `embedding` and `embeddingModel`), oldest `createdAt` first, then id. By default only
`active` rows unexpired when the scan starts; `includeInactive: true` yields every row, superseded, withdrawn and
expired ones included. `after: { createdAt, id }`, the last row a caller saw, resumes strictly after it with the
`?order=created` tie-break, so rows sharing a `createdAt` are neither skipped nor repeated. It takes no principal
and applies no scope rule: it is the one read outside the guarded routes, for trusted in-process plugin code
building derived data, and no route reaches it. It reads in pages of 100 (`SCAN_PAGE`) with a context of its own
(`authorize: false`), so a consumer that pauses between rows holds no read open. It is a walk, not a snapshot
or a change feed: a row revised, withdrawn or expiring behind the cursor is not revisited, a row created with an
earlier `createdAt` (an import keeping its source time) lands behind it, and expiry makes no sink call. A derived
store therefore stays current from a sink registered before its walk starts, compares `revision` before applying
a scanned row over sink-driven state, and after a gap in which its sink did not run walks again from the start
with `includeInactive`.
- `registerBootstrapSection({ name, render })` returns an unregister function. `render({ principal, q, budgetChars,
context })` (sync or async) returns `{ text }`, an array of `{ title?, content }` items in priority order, or
nothing. Sections render after `pinned` and `needsReview` have packed, so `budgetChars` is the budget they left;
sections render concurrently, so each is told the same remainder. Their items pack, in registration order,
between `needsReview` and `recent`, each counted like a memory (title + content) and skipped when it does not
fit, never truncated; `{ text }` is one item, and agent-memory measures it. A packed item is an element of
`memories` that agent-memory builds as exactly `{ title, content, section: <name> }`. Clients frame it as stored
data like any memory. It carries no `id`, so clients never watch, cache or deduplicate it as a memory and
`watch=true` watches memory ids only; and no `author`, `scopes` or `revision`, so contributed text cannot borrow
a principal's attribution. A client that charges framing per entry (the Claude Code plugin does) can omit an
item that filled `budgetChars`; several smaller items in priority order lose only their tail. A section that
throws, rejects, or returns anything else is omitted and logged at most once a minute per name; bootstrap still
answers. `name` matches `^[A-Za-z][A-Za-z0-9_-]{0,63}$`, is not a built-in section, and is unique in the domain;
otherwise registration throws a `TypeError`. Bootstrap has no MCP tool.
- `registerSearchAnnotator({ name, annotate })` returns an unregister function. `annotate({ principal, hits,
context })` (sync or async) returns a `Map` from memory id to a JSON value, or nothing; each hit whose id the
map holds with a value other than `undefined` gains it as `hit[name]`. It runs for searches with `q` (REST and
MCP `recall`), not for listings (the `order=created` walk feeds Copy between instances, whose import rejects
unknown fields) or bootstrap's `relevant` search. `hits` is a frozen copy of the result list holding the result
objects themselves, which annotators must not change; agent-memory merges by position in its own list, so an
annotator cannot add, remove or reorder hits. One that throws, rejects, or returns something without a `get`
method is skipped and logged at most once a minute per name, and the search still answers. `name` follows the
section rule and is not a field a hit carries (the view's fields and `score`), checked by the registering copy:
a plugin of a later version that adds a hit field of that name sees it overwritten, so pick a name specific to
the plugin. With no annotator registered, results go out untouched.
- **Contributed output is the contributor's to authorize.** Sections and annotators get the resolved principal,
and agent-memory does not filter what they return. `canReadLabels` is the label test only: ownership, withdrawal
and the customer-content quarantine (`contentFlag`; clearance is `superUser` or the `customer:*` read wildcard in
`principal.read`) are rule 1's other parts, and `scanMemories` yields flagged and private rows. Text derived from
several memories is shown only to a reader of all of them, checked against the sources' current state: a
rescope, flag or withdrawal reaches the contributor through its sink before its derived text is rebuilt, so it
must stop serving that text then, not after the rebuild.
- Contributors (sections, annotators, `scanMemories` callers) are trusted in-process code, as sinks are:
agent-memory contains their failures, a throw or a rejection, but does not defend against the values they
return or the objects they are handed. They are awaited with no time bound, so a slow one delays the response.
- `setPrincipalResolver(fn)` β€” see Identity.
- `resolvePrincipal(context)` β€” the caller behind a request context, resolved exactly as the plugin's own routes
resolve it (custom resolver, else the Harper user and its `ScopeGrant` rows, read on every call). It resolves
Expand Down Expand Up @@ -700,19 +751,26 @@ as REST:
loaders. All live apps on a worker share that pair. A deploy-validation candidate loads into its own
`Resources`, and an app with branched databases has its own `databases`, so neither reaches the live plugin.
Within a domain:
- Sinks and the principal resolver live in the domain, so every copy registers where the plugin dispatches,
whatever order the apps load in. The last `setPrincipalResolver` from any app wins.
- `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels }` (the plugin's own functions)
once `configure()` succeeds, and never for an `isTransientValidation` scope. The exported `resolvePrincipal`
calls it when present, else resolves locally. `canReadLabels` is answered by the copy that compiled the
- Sinks, the principal resolver, bootstrap sections and search annotators live in the domain, so every copy
registers where the plugin dispatches, whatever order the apps load in. The last `setPrincipalResolver` from
any app wins. The section and annotator lists are added on first use (`??=`), since an older copy may have
created the domain object without them.
- `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels, scanMemories }` (the plugin's own
functions) once `configure()` succeeds, and never for an `isTransientValidation` scope. The exported
`resolvePrincipal` and `scanMemories` call it when present, else run locally. A runtime without `scanMemories`
is 0.1.4 or earlier, which never renders sections or runs annotators: registering one beside it warns once
per runtime (`warnIfOldRuntime`). A registration made before any plugin published turns the domain's `runtime`
into an accessor whose setter runs the same check (`noteExtensionRegistered`), since an older plugin publishes
with a plain assignment; reads and writes of `runtime` behave as before. A scan beside an older runtime runs
locally. `canReadLabels` is answered by the copy that compiled the
principal (`compiledHere`), so a principal resolved before the plugin published keeps working. A different
major version warns once and still delegates. The plugin's own routes call their functions directly; only
`src/index.ts` forwards.
- A second plugin instance in a domain throws before mounting: one resolver and one sink list cannot serve two
hosts' registrations. Each worker decides independently which instance loaded first. The check,
`configure()` and the publish run with no `await` between them.
- The anchor, the domain object, `MemoryRuntime` and the sink and resolver call shapes are a protocol between
package versions. They only gain members; an incompatible change needs a new anchor key.
- The anchor, the domain object, `MemoryRuntime` and the sink, resolver, section and annotator call shapes are a
protocol between package versions. They only gain members; an incompatible change needs a new anchor key.
- A copy whose `harper` has no `resources` was loaded without an app scope (`moduleLoader` `native` or
`compartment`, or `dependencyLoader: native`). It keeps a private domain, as every copy did before, so two
such copies in different apps never meet. Mixed with a scoped plugin, whichever side comes second warns
Expand Down
43 changes: 40 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Shared, permissioned, live memory for agents and people, as a Harper plugin.
- Customer-content flag: a `global` write that names a configured customer, or matches a configured host pattern, is committed but readable only by its owner and `customer:*` readers until a reviewer rescopes or confirms it; an optional model check runs after commit
- Live updates: Level 1 real-time subscriptions on `/Memory` and `/Memory/<id>` (SSE, WebSocket) with delivery-time read checks, and Level 2 watches (`POST /Memory/watch`, `watch=true` on search/bootstrap) delivered with own `needsReview` flags on `/MemoryEvents`
- MCP surface (`mcp: true`): `remember`, `revise`, `recall`, `get`, `withdraw`, `pin`, `unpin`, `confirm`, `confirm-scope`, `watch` tools and `memory:///{id}` resource
- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, and `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants
- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants, and `scanMemories`, `registerBootstrapSection` and `registerSearchAnnotator` let a plugin build derived data over memories and add it to bootstrap and search results

Status: v0 (0.x releases may change contracts; see [CHANGELOG.md](CHANGELOG.md)). Design and contracts:
[DESIGN.md](DESIGN.md).
Expand Down Expand Up @@ -172,14 +172,50 @@ const principal = await resolvePrincipal(context);
const visible = records.filter((record) => canReadLabels(principal, record.scopes));
```

### Derived data: scan, bootstrap sections, search annotations

A plugin that builds something over memories (topic summaries, say) reads them with `scanMemories`, stays current
with a sink, and serves its output beside memories:

```js
import {
canReadLabels,
registerBootstrapSection,
registerSearchAnnotator,
scanMemories,
} from '@harperfast/agent-memory';

// Every active, unexpired memory as stored (embedding included), oldest first. Pass the last row's
// { createdAt, id } as `after` to resume; `includeInactive: true` also yields superseded, withdrawn and expired rows.
for await (const memory of scanMemories({ after: savedCursor })) await index(memory);

// Packed into GET /Memory/bootstrap after pinned and needsReview, before recent, within the caller's budget.
registerBootstrapSection({
name: 'orientation',
render: async ({ principal, q, budgetChars, context }) =>
(await topicsFor(principal, budgetChars)).map((topic) => ({ title: topic.title, content: topic.summary })),
});

// Adds hit.summaries to GET /Memory/?q= and MCP recall results; cannot add, drop or reorder hits.
registerSearchAnnotator({
name: 'summaries',
annotate: async ({ principal, hits, context }) => new Map(await summariesFor(principal, hits)),
});
```

The scan takes no principal: it is for trusted plugin code. Sections and annotators receive the caller's principal,
and agent-memory shows what they return as is, so they must show a caller only what that caller may read: all of
the memories a summary draws on, judged by their current scopes and content flags. A section or annotator that
throws is left out and logged; the request still succeeds. Contract: [DESIGN.md](DESIGN.md#extension-points).

`read` and `write` list the effective grant patterns: `user:<id>`, exact labels and namespace wildcards, with
role and user grants merged and patterns that grant nothing dropped. A super_user's lists do not enumerate its
access; check `superUser`, or let `canReadLabels` do it. `canReadLabels` is the label rule only (any label
matches), and accepts only a principal `resolvePrincipal` returned. Contract: [DESIGN.md](DESIGN.md#extension-points).

### From another app

These four functions also work in an app that imports the package without listing the plugin, while
These functions also work in an app that imports the package without listing the plugin, while
agent-memory runs as its own app on the same Harper, for example so it can be upgraded separately. That app's
copy of the package registers with the running plugin and asks it to resolve principals. Its sinks run inside the
plugin's lifecycle transaction, and its authorization matches the plugin's routes even when the two versions
Expand All @@ -193,7 +229,8 @@ differ. This needs:
- Exactly one app listing the plugin in its `config.yaml`. A second one fails to load. Which of the two loads
first can differ from worker to worker, so remove one.
- A version of this package newer than 0.1.3 in the importing app. 0.1.3 and earlier keep registrations to
themselves.
themselves. Bootstrap sections and search annotators also need a running plugin newer than 0.1.4; beside 0.1.4
they are never used, and registering one logs a warning.

To move the plugin out of a host app:

Expand Down
Loading
Loading