Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ All notable changes to `@harperfast/agent-memory` are documented here. The forma
- `scanMemories({ after?, includeInactive? })` reads every memory as stored, embedding included, oldest `createdAt`
first, for a trusted plugin building derived data (a first build, or a rebuild after it was not loaded). It
resumes after a given row without skipping or repeating rows that share a timestamp.
- `getMemories(ids)` reads memories by id as stored, embedding included, in every status and in the order asked, for
a trusted plugin that re-reads members it tracks instead of copying their text and vectors. Unknown ids are left
out. It reaches the running plugin from another app, like `scanMemories`.
- `registerBootstrapSection({ name, render })` adds a section to `GET /Memory/bootstrap`, after `pinned` and
`needsReview` and before `recent`, rendered for each caller within the budget those sections left. Its items come
back as `memories` entries with a `section` and no `id`, so existing clients show them as stored data without a
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ update it in the same PR when a rule changes. Follow the harper-engineering-guid
Harper DLC for every PR.

- Plugin shape mirrors `@harperfast/oauth`: `config.yaml` declares `pluginModule` + `graphqlSchema`.
- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer, except `scanMemories`,
the unguarded read for trusted in-process plugins that no route reaches.
- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer, except `scanMemories` and `getMemories`,
the unguarded reads for trusted in-process plugins that no route reaches.
- Scope enforcement is pushed into queries as conditions, never applied as a post-filter on top-k.
26 changes: 18 additions & 8 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ allowlists, idempotent `PUT` create/revise with `If-Match` revision precondition
| Customer-content flag and quarantine (`customerContentCheck` option) | implemented (`src/quarantine.ts`) |
| Supersession backfill: candidates for an existing memory, links between existing memories, created-order cursor | implemented (`src/lifecycle.ts`) |
| Package exports reaching the plugin from another app (Extension points, Across apps) | implemented (`src/runtime.ts`) |
| `scanMemories`, bootstrap sections and search annotators for derived plugins (Extension points) | implemented (`src/extensions.ts`) |
| `scanMemories`, `getMemories`, bootstrap sections and search annotators for derived plugins (Extension points) | implemented (`src/extensions.ts`) |

## Data model

Expand Down Expand Up @@ -68,8 +68,8 @@ database).
- `MemoryWatch` β€” a principal's time-bounded interest in specific memories (live updates).

See `schema/memory.graphql`. The table is not `@export`ed: every read and write goes through the
`Memory` resource so the rules below have exactly one enforcement point. The one exception is `scanMemories`
(Extension points), an unguarded read for trusted in-process plugins that no route reaches.
`Memory` resource so the rules below have exactly one enforcement point. The exceptions are `scanMemories` and
`getMemories` (Extension points), unguarded reads for trusted in-process plugins that no route reaches.

Server-owned fields: `ownerId`, `status`, `revision`, `author`, `lastEditor`, `supersededBy`,
`autoSupersedes`, `evidenceMemoryIds`, `needsReview`, `needsReviewSince`, `confirmed`, `contentFlag`,
Expand Down Expand Up @@ -689,6 +689,16 @@ as REST:
store therefore stays current from a sink registered before its walk starts, compares `revision` before applying
a scanned row over sink-driven state, and after a gap in which its sink did not run walks again from the start
with `includeInactive`.
- `getMemories(ids)` β€” a promise of the `Memory` rows with those ids, shaped as `scanMemories` yields them (stored
fields, `embedding` and `embeddingModel` included) and in every status, as `scanMemories({ includeInactive: true })`:
the caller applies eligibility. Rows follow the order of `ids`, a repeated id once, and an id with no row is left
out. It reads in queries of at most 100 ids (`SCAN_PAGE`), not a get per id, takes no principal and applies no
scope rule, like `scanMemories`. A derived plugin that re-reads members it already tracks calls it, instead of
keeping a second copy of memory text and vectors that withdrawal, `contentFlag` and rescoping would also have to
clear. Anything but an array of strings rejects with a `TypeError`. It returns every row at once, embeddings included, so pass a cluster's
ids, not a whole walk's. A failed read rejects the whole call, never a partial list. It reads each row as it
is when its query runs: rows are not a snapshot of one moment, so a plugin building output from several rows fences
its publication against sink invalidations and rechecks eligibility.
- `registerBootstrapSection({ name, render })` returns an unregister function. `render({ principal, q, budgetChars,
context })` (sync or async) returns `{ text }`, an array of `{ title?, content }` items in priority order, or
nothing. Sections render after `pinned` and `needsReview` have packed, so `budgetChars` is the budget they left;
Expand Down Expand Up @@ -717,11 +727,11 @@ context })` (sync or async) returns a `Map` from memory id to a JSON value, or n
- **Contributed output is the contributor's to authorize.** Sections and annotators get the resolved principal,
and agent-memory does not filter what they return. `canReadLabels` is the label test only: ownership, withdrawal
and the customer-content quarantine (`contentFlag`; clearance is `superUser` or the `customer:*` read wildcard in
`principal.read`) are rule 1's other parts, and `scanMemories` yields flagged and private rows. Text derived from
`principal.read`) are rule 1's other parts, and `scanMemories` and `getMemories` yield flagged and private rows. Text derived from
several memories is shown only to a reader of all of them, checked against the sources' current state: a
rescope, flag or withdrawal reaches the contributor through its sink before its derived text is rebuilt, so it
must stop serving that text then, not after the rebuild.
- Contributors (sections, annotators, `scanMemories` callers) are trusted in-process code, as sinks are:
- Contributors (sections, annotators, `scanMemories` and `getMemories` callers) are trusted in-process code, as sinks are:
agent-memory contains their failures, a throw or a rejection, but does not defend against the values they
return or the objects they are handed. They are awaited with no time bound, so a slow one delays the response.
- `setPrincipalResolver(fn)` β€” see Identity.
Expand Down Expand Up @@ -755,13 +765,13 @@ context })` (sync or async) returns a `Map` from memory id to a JSON value, or n
registers where the plugin dispatches, whatever order the apps load in. The last `setPrincipalResolver` from
any app wins. The section and annotator lists are added on first use (`??=`), since an older copy may have
created the domain object without them.
- `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels, scanMemories }` (the plugin's own
- `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels, scanMemories, getMemories }` (the plugin's own
functions) once `configure()` succeeds, and never for an `isTransientValidation` scope. The exported
`resolvePrincipal` and `scanMemories` call it when present, else run locally. A runtime without `scanMemories`
`resolvePrincipal`, `scanMemories` and `getMemories` call it when present, else run locally. A runtime without `scanMemories`
is 0.1.4 or earlier, which never renders sections or runs annotators: registering one beside it warns once
per runtime (`warnIfOldRuntime`). A registration made before any plugin published turns the domain's `runtime`
into an accessor whose setter runs the same check (`noteExtensionRegistered`), since an older plugin publishes
with a plain assignment; reads and writes of `runtime` behave as before. A scan beside an older runtime runs
with a plain assignment; reads and writes of `runtime` behave as before. A scan or `getMemories` beside an older runtime runs
locally. `canReadLabels` is answered by the copy that compiled the
principal (`compiledHere`), so a principal resolved before the plugin published keeps working. A different
major version warns once and still delegates. The plugin's own routes call their functions directly; only
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Shared, permissioned, live memory for agents and people, as a Harper plugin.
- Customer-content flag: a `global` write that names a configured customer, or matches a configured host pattern, is committed but readable only by its owner and `customer:*` readers until a reviewer rescopes or confirms it; an optional model check runs after commit
- Live updates: Level 1 real-time subscriptions on `/Memory` and `/Memory/<id>` (SSE, WebSocket) with delivery-time read checks, and Level 2 watches (`POST /Memory/watch`, `watch=true` on search/bootstrap) delivered with own `needsReview` flags on `/MemoryEvents`
- MCP surface (`mcp: true`): `remember`, `revise`, `recall`, `get`, `withdraw`, `pin`, `unpin`, `confirm`, `confirm-scope`, `watch` tools and `memory:///{id}` resource
- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants, and `scanMemories`, `registerBootstrapSection` and `registerSearchAnnotator` let a plugin build derived data over memories and add it to bootstrap and search results
- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants, and `scanMemories`, `getMemories`, `registerBootstrapSection` and `registerSearchAnnotator` let a plugin build derived data over memories and add it to bootstrap and search results

Status: v0 (0.x releases may change contracts; see [CHANGELOG.md](CHANGELOG.md)). Design and contracts:
[DESIGN.md](DESIGN.md).
Expand Down Expand Up @@ -180,6 +180,7 @@ with a sink, and serves its output beside memories:
```js
import {
canReadLabels,
getMemories,
registerBootstrapSection,
registerSearchAnnotator,
scanMemories,
Expand All @@ -189,6 +190,9 @@ import {
// { createdAt, id } as `after` to resume; `includeInactive: true` also yields superseded, withdrawn and expired rows.
for await (const memory of scanMemories({ after: savedCursor })) await index(memory);

// The rows for ids you already track, in that order, in every status; ids with no row are left out.
const members = await getMemories(cluster.memoryIds);

// Packed into GET /Memory/bootstrap after pinned and needsReview, before recent, within the caller's budget.
registerBootstrapSection({
name: 'orientation',
Expand Down
4 changes: 3 additions & 1 deletion integrationTests/cross-app.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -226,12 +226,14 @@ for (const moduleLoader of ['vm-current-context', 'vm']) {
);
});

test("the consumer's scanMemories reads every memory through the plugin and resumes after a row", async () => {
test("the consumer's scanMemories reads every memory through the plugin, resumes after a row and fetches by id", async () => {
const probe = await call('alice', 'GET', 'ScanProbe/');
strictEqual(probe.status, 200, JSON.stringify(probe.body));
ok(probe.body.ids.length >= 2, JSON.stringify(probe.body));
deepStrictEqual(probe.body.resumed, probe.body.ids.slice(1));
strictEqual(probe.body.embedded, true);
deepStrictEqual(probe.body.fetched, [probe.body.ids.at(-1), probe.body.ids[0]]);
strictEqual(probe.body.fetchedEmbedded, true);
});
},
);
Expand Down
4 changes: 4 additions & 0 deletions integrationTests/fixtures/memory-consumer/resources.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import {
canReadLabels,
getMemories,
registerBootstrapSection,
registerMemorySink,
registerSearchAnnotator,
Expand Down Expand Up @@ -52,8 +53,11 @@ export class ScanProbe extends Resource {
for await (const row of scanMemories({ after: { createdAt: rows[0].createdAt, id: rows[0].id } })) {
resumed.push(row.id);
}
const fetched = await getMemories([rows.at(-1).id, 'mem-absent', rows[0].id]);
return {
ids: rows.map((row) => row.id),
fetched: fetched.map((row) => row.id),
fetchedEmbedded: fetched.every((row) => row.embedding?.length > 0 && row.embeddingModel === 'default'),
resumed,
embedded: rows.every((row) => row.embedding?.length > 0 && row.embeddingModel === 'default'),
};
Expand Down
19 changes: 18 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { resolvePrincipal as resolveHere } from './principal.ts';
import { MEMORY_RESOURCES, MemoryMcpResource } from './resource.ts';
import { domain, foreignRuntime, publishRuntime, type MemoryRuntime } from './runtime.ts';
import { canReadLabels as canReadLabelsHere, compiledHere, type Principal } from './scopes.ts';
import { scanMemories as scanHere, type ScanOptions } from './search.ts';
import { getMemories as getHere, scanMemories as scanHere, type ScanOptions } from './search.ts';
import { applySettings, parseSettings } from './settings.ts';
import { PACKAGE_VERSION } from './version.ts';

Expand All @@ -29,6 +29,7 @@ const RUNTIME: MemoryRuntime = Object.freeze({
resolvePrincipal: resolveHere,
canReadLabels: canReadLabelsHere,
scanMemories: scanHere,
getMemories: getHere,
});

/**
Expand Down Expand Up @@ -69,6 +70,22 @@ export function scanMemories(options?: ScanOptions): AsyncIterable<Record<string
return runtime?.scanMemories ? runtime.scanMemories(options) : scanHere(options);
}

/**
* The memories with these ids, as `scanMemories({ includeInactive: true })` yields them: every status, stored fields
* with `embedding` and `embeddingModel`. Rows come back in the order of `ids` (a repeated id once); ids with no row
* are left out. For trusted in-process code: it takes no principal and applies no scope rule. Read by the running
* plugin, even one in another app (DESIGN.md, Extension points).
*/
export function getMemories(ids: readonly string[]): Promise<Record<string, unknown>[]> {
let runtime: MemoryRuntime | undefined;
try {
runtime = foreignRuntime(RUNTIME);
} catch (error) {
return Promise.reject(error);
}
return runtime?.getMemories ? runtime.getMemories(ids) : getHere(ids);
}

function routePathFor(basePath: string, resource: { path: string }): string {
return resource.path.replace(/^\/Memory(?=\/|$|(?:Events|Mcp)\b)/, basePath);
}
Expand Down
1 change: 1 addition & 0 deletions src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ export interface MemoryRuntime {
canReadLabels(principal: Principal, labels: readonly string[] | null | undefined): boolean;
// Absent from a runtime published by 0.1.4 or earlier, which also never renders sections or runs annotators.
scanMemories?(options?: ScanOptions): AsyncIterable<Record<string, unknown>>;
getMemories?(ids: readonly string[]): Promise<Record<string, unknown>[]>;
}

export interface MemoryDomain {
Expand Down
24 changes: 24 additions & 0 deletions src/search.ts
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,30 @@ export async function* scanMemories(options: ScanOptions = {}): AsyncGenerator<R
}
}

export async function getMemories(ids: readonly string[]): Promise<Record<string, unknown>[]> {
if (!Array.isArray(ids) || !ids.every((id) => typeof id === 'string')) {
throw new TypeError('getMemories: ids must be an array of strings');
}
const wanted = [...new Set(ids)];
const Memory = memoryTable();
const found = new Map<string, Record<string, unknown>>();
for (let start = 0; start < wanted.length; start += SCAN_PAGE) {
const page = wanted.slice(start, start + SCAN_PAGE);
const rows = await collect(
Memory.search(
{
conditions: [anyOf(page.map((id) => ({ attribute: 'id', comparator: 'equals', value: id })))],
select: SCAN_FIELDS,
limit: page.length,
},
{ authorize: false },
),
);
for (const row of rows) found.set(row.id, row);
}
return wanted.map((id) => found.get(id)).filter((row) => row !== undefined);
}

function filterConditions(params: SearchParams, principal: Principal, now = new Date()): Condition[] {
const statuses = params.includeSuperseded ? [STATUS_ACTIVE, STATUS_SUPERSEDED] : [STATUS_ACTIVE];
if (params.includeWithdrawn) statuses.push(STATUS_WITHDRAWN);
Expand Down
28 changes: 28 additions & 0 deletions test/cross-copy.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as plugin from '../dist/index.js';
import { putMemory, withdrawMemory } from '../dist/lifecycle.js';
import { resolvePrincipal as pluginResolvePrincipal } from '../dist/principal.js';
import { MemoryMcpResource, MemoryResource } from '../dist/resource.js';
import { domain } from '../dist/runtime.js';
import { canReadLabels as pluginCanReadLabels } from '../dist/scopes.js';
import { bootstrapMemories, scanMemories as pluginScanMemories } from '../dist/search.js';

Expand Down Expand Up @@ -258,6 +259,33 @@ describe('a second copy of the package', () => {
resumed,
rows.slice(1).map((row) => row.id),
);
const published = domain().runtime;
const forwarded = [];
domain().runtime = {
...published,
getMemories: (ids) => (forwarded.push(ids), published.getMemories(ids)),
};
let fetched;
try {
fetched = await copy.getMemories([id, idFor('absent')]);
} finally {
domain().runtime = published;
}
assert.deepEqual(forwarded, [[id, idFor('absent')]]);
assert.deepEqual(
fetched.map((row) => row.id),
[id],
);
assert.ok(Array.isArray(fetched[0].embedding));
domain().runtime = { ...published, getMemories: undefined };
try {
assert.deepEqual(
(await copy.getMemories([id])).map((row) => row.id),
[id],
);
} finally {
domain().runtime = published;
}

unregisterSection();
unregisterAnnotator();
Expand Down
Loading
Loading