diff --git a/CHANGELOG.md b/CHANGELOG.md index 93f5b9e..eb6eb23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ All notable changes to `@harperfast/agent-memory` are documented here. The forma - `scanMemories({ after?, includeInactive? })` reads every memory as stored, embedding included, oldest `createdAt` first, for a trusted plugin building derived data (a first build, or a rebuild after it was not loaded). It resumes after a given row without skipping or repeating rows that share a timestamp. +- `getMemories(ids)` reads memories by id as stored, embedding included, in every status and in the order asked, for + a trusted plugin that re-reads members it tracks instead of copying their text and vectors. Unknown ids are left + out. It reaches the running plugin from another app, like `scanMemories`. - `registerBootstrapSection({ name, render })` adds a section to `GET /Memory/bootstrap`, after `pinned` and `needsReview` and before `recent`, rendered for each caller within the budget those sections left. Its items come back as `memories` entries with a `section` and no `id`, so existing clients show them as stored data without a diff --git a/CLAUDE.md b/CLAUDE.md index 33fc256..9789f27 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,6 +5,6 @@ update it in the same PR when a rule changes. Follow the harper-engineering-guid Harper DLC for every PR. - Plugin shape mirrors `@harperfast/oauth`: `config.yaml` declares `pluginModule` + `graphqlSchema`. -- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer, except `scanMemories`, - the unguarded read for trusted in-process plugins that no route reaches. +- The `Memory` table is never `@export`ed; the `Memory` resource is the only reader/writer, except `scanMemories` and `getMemories`, + the unguarded reads for trusted in-process plugins that no route reaches. - Scope enforcement is pushed into queries as conditions, never applied as a post-filter on top-k. diff --git a/DESIGN.md b/DESIGN.md index 31d58cf..36d3981 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -38,7 +38,7 @@ allowlists, idempotent `PUT` create/revise with `If-Match` revision precondition | Customer-content flag and quarantine (`customerContentCheck` option) | implemented (`src/quarantine.ts`) | | Supersession backfill: candidates for an existing memory, links between existing memories, created-order cursor | implemented (`src/lifecycle.ts`) | | Package exports reaching the plugin from another app (Extension points, Across apps) | implemented (`src/runtime.ts`) | -| `scanMemories`, bootstrap sections and search annotators for derived plugins (Extension points) | implemented (`src/extensions.ts`) | +| `scanMemories`, `getMemories`, bootstrap sections and search annotators for derived plugins (Extension points) | implemented (`src/extensions.ts`) | ## Data model @@ -68,8 +68,8 @@ database). - `MemoryWatch` — a principal's time-bounded interest in specific memories (live updates). See `schema/memory.graphql`. The table is not `@export`ed: every read and write goes through the -`Memory` resource so the rules below have exactly one enforcement point. The one exception is `scanMemories` -(Extension points), an unguarded read for trusted in-process plugins that no route reaches. +`Memory` resource so the rules below have exactly one enforcement point. The exceptions are `scanMemories` and +`getMemories` (Extension points), unguarded reads for trusted in-process plugins that no route reaches. Server-owned fields: `ownerId`, `status`, `revision`, `author`, `lastEditor`, `supersededBy`, `autoSupersedes`, `evidenceMemoryIds`, `needsReview`, `needsReviewSince`, `confirmed`, `contentFlag`, @@ -689,6 +689,16 @@ as REST: store therefore stays current from a sink registered before its walk starts, compares `revision` before applying a scanned row over sink-driven state, and after a gap in which its sink did not run walks again from the start with `includeInactive`. +- `getMemories(ids)` — a promise of the `Memory` rows with those ids, shaped as `scanMemories` yields them (stored + fields, `embedding` and `embeddingModel` included) and in every status, as `scanMemories({ includeInactive: true })`: + the caller applies eligibility. Rows follow the order of `ids`, a repeated id once, and an id with no row is left + out. It reads in queries of at most 100 ids (`SCAN_PAGE`), not a get per id, takes no principal and applies no + scope rule, like `scanMemories`. A derived plugin that re-reads members it already tracks calls it, instead of + keeping a second copy of memory text and vectors that withdrawal, `contentFlag` and rescoping would also have to + clear. Anything but an array of strings rejects with a `TypeError`. It returns every row at once, embeddings included, so pass a cluster's + ids, not a whole walk's. A failed read rejects the whole call, never a partial list. It reads each row as it + is when its query runs: rows are not a snapshot of one moment, so a plugin building output from several rows fences + its publication against sink invalidations and rechecks eligibility. - `registerBootstrapSection({ name, render })` returns an unregister function. `render({ principal, q, budgetChars, context })` (sync or async) returns `{ text }`, an array of `{ title?, content }` items in priority order, or nothing. Sections render after `pinned` and `needsReview` have packed, so `budgetChars` is the budget they left; @@ -717,11 +727,11 @@ context })` (sync or async) returns a `Map` from memory id to a JSON value, or n - **Contributed output is the contributor's to authorize.** Sections and annotators get the resolved principal, and agent-memory does not filter what they return. `canReadLabels` is the label test only: ownership, withdrawal and the customer-content quarantine (`contentFlag`; clearance is `superUser` or the `customer:*` read wildcard in - `principal.read`) are rule 1's other parts, and `scanMemories` yields flagged and private rows. Text derived from + `principal.read`) are rule 1's other parts, and `scanMemories` and `getMemories` yield flagged and private rows. Text derived from several memories is shown only to a reader of all of them, checked against the sources' current state: a rescope, flag or withdrawal reaches the contributor through its sink before its derived text is rebuilt, so it must stop serving that text then, not after the rebuild. -- Contributors (sections, annotators, `scanMemories` callers) are trusted in-process code, as sinks are: +- Contributors (sections, annotators, `scanMemories` and `getMemories` callers) are trusted in-process code, as sinks are: agent-memory contains their failures, a throw or a rejection, but does not defend against the values they return or the objects they are handed. They are awaited with no time bound, so a slow one delays the response. - `setPrincipalResolver(fn)` — see Identity. @@ -755,13 +765,13 @@ context })` (sync or async) returns a `Map` from memory id to a JSON value, or n registers where the plugin dispatches, whatever order the apps load in. The last `setPrincipalResolver` from any app wins. The section and annotator lists are added on first use (`??=`), since an older copy may have created the domain object without them. - - `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels, scanMemories }` (the plugin's own + - `handleApplication` publishes `{ version, resolvePrincipal, canReadLabels, scanMemories, getMemories }` (the plugin's own functions) once `configure()` succeeds, and never for an `isTransientValidation` scope. The exported - `resolvePrincipal` and `scanMemories` call it when present, else run locally. A runtime without `scanMemories` + `resolvePrincipal`, `scanMemories` and `getMemories` call it when present, else run locally. A runtime without `scanMemories` is 0.1.4 or earlier, which never renders sections or runs annotators: registering one beside it warns once per runtime (`warnIfOldRuntime`). A registration made before any plugin published turns the domain's `runtime` into an accessor whose setter runs the same check (`noteExtensionRegistered`), since an older plugin publishes - with a plain assignment; reads and writes of `runtime` behave as before. A scan beside an older runtime runs + with a plain assignment; reads and writes of `runtime` behave as before. A scan or `getMemories` beside an older runtime runs locally. `canReadLabels` is answered by the copy that compiled the principal (`compiledHere`), so a principal resolved before the plugin published keeps working. A different major version warns once and still delegates. The plugin's own routes call their functions directly; only diff --git a/README.md b/README.md index 57e3e3e..6bc827d 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Shared, permissioned, live memory for agents and people, as a Harper plugin. - Customer-content flag: a `global` write that names a configured customer, or matches a configured host pattern, is committed but readable only by its owner and `customer:*` readers until a reviewer rescopes or confirms it; an optional model check runs after commit - Live updates: Level 1 real-time subscriptions on `/Memory` and `/Memory/` (SSE, WebSocket) with delivery-time read checks, and Level 2 watches (`POST /Memory/watch`, `watch=true` on search/bootstrap) delivered with own `needsReview` flags on `/MemoryEvents` - MCP surface (`mcp: true`): `remember`, `revise`, `recall`, `get`, `withdraw`, `pin`, `unpin`, `confirm`, `confirm-scope`, `watch` tools and `memory:///{id}` resource -- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants, and `scanMemories`, `registerBootstrapSection` and `registerSearchAnnotator` let a plugin build derived data over memories and add it to bootstrap and search results +- Extension points: `registerMemorySink` runs inside the lifecycle transaction for apps that mirror memories elsewhere, `setPrincipalResolver` maps app-defined identities, `resolvePrincipal` / `canReadLabels` let other components authorize with the same identity and grants, and `scanMemories`, `getMemories`, `registerBootstrapSection` and `registerSearchAnnotator` let a plugin build derived data over memories and add it to bootstrap and search results Status: v0 (0.x releases may change contracts; see [CHANGELOG.md](CHANGELOG.md)). Design and contracts: [DESIGN.md](DESIGN.md). @@ -180,6 +180,7 @@ with a sink, and serves its output beside memories: ```js import { canReadLabels, + getMemories, registerBootstrapSection, registerSearchAnnotator, scanMemories, @@ -189,6 +190,9 @@ import { // { createdAt, id } as `after` to resume; `includeInactive: true` also yields superseded, withdrawn and expired rows. for await (const memory of scanMemories({ after: savedCursor })) await index(memory); +// The rows for ids you already track, in that order, in every status; ids with no row are left out. +const members = await getMemories(cluster.memoryIds); + // Packed into GET /Memory/bootstrap after pinned and needsReview, before recent, within the caller's budget. registerBootstrapSection({ name: 'orientation', diff --git a/integrationTests/cross-app.test.ts b/integrationTests/cross-app.test.ts index fd10a07..9aadc62 100644 --- a/integrationTests/cross-app.test.ts +++ b/integrationTests/cross-app.test.ts @@ -226,12 +226,14 @@ for (const moduleLoader of ['vm-current-context', 'vm']) { ); }); - test("the consumer's scanMemories reads every memory through the plugin and resumes after a row", async () => { + test("the consumer's scanMemories reads every memory through the plugin, resumes after a row and fetches by id", async () => { const probe = await call('alice', 'GET', 'ScanProbe/'); strictEqual(probe.status, 200, JSON.stringify(probe.body)); ok(probe.body.ids.length >= 2, JSON.stringify(probe.body)); deepStrictEqual(probe.body.resumed, probe.body.ids.slice(1)); strictEqual(probe.body.embedded, true); + deepStrictEqual(probe.body.fetched, [probe.body.ids.at(-1), probe.body.ids[0]]); + strictEqual(probe.body.fetchedEmbedded, true); }); }, ); diff --git a/integrationTests/fixtures/memory-consumer/resources.js b/integrationTests/fixtures/memory-consumer/resources.js index 33ea026..62e6db8 100644 --- a/integrationTests/fixtures/memory-consumer/resources.js +++ b/integrationTests/fixtures/memory-consumer/resources.js @@ -1,5 +1,6 @@ import { canReadLabels, + getMemories, registerBootstrapSection, registerMemorySink, registerSearchAnnotator, @@ -52,8 +53,11 @@ export class ScanProbe extends Resource { for await (const row of scanMemories({ after: { createdAt: rows[0].createdAt, id: rows[0].id } })) { resumed.push(row.id); } + const fetched = await getMemories([rows.at(-1).id, 'mem-absent', rows[0].id]); return { ids: rows.map((row) => row.id), + fetched: fetched.map((row) => row.id), + fetchedEmbedded: fetched.every((row) => row.embedding?.length > 0 && row.embeddingModel === 'default'), resumed, embedded: rows.every((row) => row.embedding?.length > 0 && row.embeddingModel === 'default'), }; diff --git a/src/index.ts b/src/index.ts index 9f5904c..bee380b 100644 --- a/src/index.ts +++ b/src/index.ts @@ -4,7 +4,7 @@ import { resolvePrincipal as resolveHere } from './principal.ts'; import { MEMORY_RESOURCES, MemoryMcpResource } from './resource.ts'; import { domain, foreignRuntime, publishRuntime, type MemoryRuntime } from './runtime.ts'; import { canReadLabels as canReadLabelsHere, compiledHere, type Principal } from './scopes.ts'; -import { scanMemories as scanHere, type ScanOptions } from './search.ts'; +import { getMemories as getHere, scanMemories as scanHere, type ScanOptions } from './search.ts'; import { applySettings, parseSettings } from './settings.ts'; import { PACKAGE_VERSION } from './version.ts'; @@ -29,6 +29,7 @@ const RUNTIME: MemoryRuntime = Object.freeze({ resolvePrincipal: resolveHere, canReadLabels: canReadLabelsHere, scanMemories: scanHere, + getMemories: getHere, }); /** @@ -69,6 +70,22 @@ export function scanMemories(options?: ScanOptions): AsyncIterable[]> { + let runtime: MemoryRuntime | undefined; + try { + runtime = foreignRuntime(RUNTIME); + } catch (error) { + return Promise.reject(error); + } + return runtime?.getMemories ? runtime.getMemories(ids) : getHere(ids); +} + function routePathFor(basePath: string, resource: { path: string }): string { return resource.path.replace(/^\/Memory(?=\/|$|(?:Events|Mcp)\b)/, basePath); } diff --git a/src/runtime.ts b/src/runtime.ts index 5500b7a..89ad427 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -16,6 +16,7 @@ export interface MemoryRuntime { canReadLabels(principal: Principal, labels: readonly string[] | null | undefined): boolean; // Absent from a runtime published by 0.1.4 or earlier, which also never renders sections or runs annotators. scanMemories?(options?: ScanOptions): AsyncIterable>; + getMemories?(ids: readonly string[]): Promise[]>; } export interface MemoryDomain { diff --git a/src/search.ts b/src/search.ts index c05857a..681fca2 100644 --- a/src/search.ts +++ b/src/search.ts @@ -222,6 +222,30 @@ export async function* scanMemories(options: ScanOptions = {}): AsyncGenerator[]> { + if (!Array.isArray(ids) || !ids.every((id) => typeof id === 'string')) { + throw new TypeError('getMemories: ids must be an array of strings'); + } + const wanted = [...new Set(ids)]; + const Memory = memoryTable(); + const found = new Map>(); + for (let start = 0; start < wanted.length; start += SCAN_PAGE) { + const page = wanted.slice(start, start + SCAN_PAGE); + const rows = await collect( + Memory.search( + { + conditions: [anyOf(page.map((id) => ({ attribute: 'id', comparator: 'equals', value: id })))], + select: SCAN_FIELDS, + limit: page.length, + }, + { authorize: false }, + ), + ); + for (const row of rows) found.set(row.id, row); + } + return wanted.map((id) => found.get(id)).filter((row) => row !== undefined); +} + function filterConditions(params: SearchParams, principal: Principal, now = new Date()): Condition[] { const statuses = params.includeSuperseded ? [STATUS_ACTIVE, STATUS_SUPERSEDED] : [STATUS_ACTIVE]; if (params.includeWithdrawn) statuses.push(STATUS_WITHDRAWN); diff --git a/test/cross-copy.test.js b/test/cross-copy.test.js index 40ebcf7..eb32ceb 100644 --- a/test/cross-copy.test.js +++ b/test/cross-copy.test.js @@ -14,6 +14,7 @@ import * as plugin from '../dist/index.js'; import { putMemory, withdrawMemory } from '../dist/lifecycle.js'; import { resolvePrincipal as pluginResolvePrincipal } from '../dist/principal.js'; import { MemoryMcpResource, MemoryResource } from '../dist/resource.js'; +import { domain } from '../dist/runtime.js'; import { canReadLabels as pluginCanReadLabels } from '../dist/scopes.js'; import { bootstrapMemories, scanMemories as pluginScanMemories } from '../dist/search.js'; @@ -258,6 +259,33 @@ describe('a second copy of the package', () => { resumed, rows.slice(1).map((row) => row.id), ); + const published = domain().runtime; + const forwarded = []; + domain().runtime = { + ...published, + getMemories: (ids) => (forwarded.push(ids), published.getMemories(ids)), + }; + let fetched; + try { + fetched = await copy.getMemories([id, idFor('absent')]); + } finally { + domain().runtime = published; + } + assert.deepEqual(forwarded, [[id, idFor('absent')]]); + assert.deepEqual( + fetched.map((row) => row.id), + [id], + ); + assert.ok(Array.isArray(fetched[0].embedding)); + domain().runtime = { ...published, getMemories: undefined }; + try { + assert.deepEqual( + (await copy.getMemories([id])).map((row) => row.id), + [id], + ); + } finally { + domain().runtime = published; + } unregisterSection(); unregisterAnnotator(); diff --git a/test/extensions.test.js b/test/extensions.test.js index f6836fb..280855d 100644 --- a/test/extensions.test.js +++ b/test/extensions.test.js @@ -9,7 +9,7 @@ import { loadBootstrapCache, loadSessionState } from '../claude-plugin/lib/cache import { resolveConfig } from '../claude-plugin/lib/config.mjs'; import { handleSessionStart } from '../claude-plugin/lib/hooks.mjs'; import { annotateSearchHits } from '../dist/extensions.js'; -import { registerBootstrapSection, registerSearchAnnotator, scanMemories } from '../dist/index.js'; +import { getMemories, registerBootstrapSection, registerSearchAnnotator, scanMemories } from '../dist/index.js'; import { pinMemory, putMemory, withdrawMemory } from '../dist/lifecycle.js'; import { MemoryBootstrapResource, MemoryMcpResource, MemoryResource } from '../dist/resource.js'; import { domain } from '../dist/runtime.js'; @@ -49,6 +49,96 @@ afterEach(() => { logger.error = originalError; }); +describe('getMemories', () => { + const seedRow = (id, overrides = {}) => + __test.stores.get('Memory').set(id, { + id, + title: id, + content: `content of ${id}`, + scopes: ['global', 'team:a'], + ownerId: 'alice', + status: 'active', + revision: 1, + embedding: [id.length, 1], + embeddingModel: 'default', + validUntil: null, + createdAt: new Date(Date.UTC(2026, 0, 1)), + updatedAt: new Date(Date.UTC(2026, 0, 1)), + ...overrides, + }); + + it('returns rows in the order asked, with the vector, its model and the scopes, and leaves out unknown ids', async () => { + for (const id of ['mem-a', 'mem-bb', 'mem-ccc']) seedRow(id); + const rows = await getMemories(['mem-ccc', 'mem-missing', 'mem-a', 'mem-ccc']); + assert.deepEqual( + rows.map((row) => row.id), + ['mem-ccc', 'mem-a'], + ); + assert.deepEqual(rows[0].embedding, [7, 1]); + assert.equal(rows[0].embeddingModel, 'default'); + assert.deepEqual(rows[0].scopes, ['global', 'team:a']); + assert.ok(rows[0].createdAt instanceof Date); + }); + + it('returns withdrawn, superseded, expired, flagged and review-pending rows', async () => { + seedRow('mem-withdrawn', { status: 'withdrawn' }); + seedRow('mem-superseded', { status: 'superseded', supersededBy: idFor('next') }); + seedRow('mem-expired', { validUntil: new Date('2020-01-01T00:00:00Z') }); + seedRow('mem-flagged', { + needsReview: true, + contentFlag: { kind: 'possible-customer-detail', since: '2026-01-01T00:00:00.000Z', source: 'pattern' }, + }); + const rows = await getMemories(['mem-withdrawn', 'mem-superseded', 'mem-expired', 'mem-flagged']); + assert.deepEqual( + rows.map((row) => row.status), + ['withdrawn', 'superseded', 'active', 'active'], + ); + assert.equal(rows[3].needsReview, true); + assert.equal(rows[3].contentFlag.kind, 'possible-customer-detail'); + assert.equal(rows[2].validUntil.getUTCFullYear(), 2020); + }); + + it('reads in queries of at most 100 ids, keeping the order across them', async () => { + const ids = Array.from({ length: 230 }, (_, i) => `mem-${String(i).padStart(4, '0')}`); + for (const id of ids) seedRow(id); + const asked = [...ids].reverse(); + __test.searchLog.length = 0; + assert.deepEqual( + (await getMemories(asked)).map((row) => row.id), + asked, + ); + const queries = __test.searchLog.filter((entry) => entry.table === 'Memory'); + assert.deepEqual( + queries.map((entry) => entry.query.limit), + [100, 100, 30], + ); + }); + + it('rejects the whole call when a later query fails, rather than returning the earlier rows', async () => { + const ids = Array.from({ length: 150 }, (_, i) => `mem-${String(i).padStart(4, '0')}`); + for (const id of ids) seedRow(id); + const Memory = databases.memory.Memory; + const search = Memory.search; + let calls = 0; + Memory.search = function (...args) { + if (++calls === 2) throw new Error('read failed'); + return search.apply(this, args); + }; + try { + await assert.rejects(getMemories(ids), /read failed/); + } finally { + Memory.search = search; + } + }); + + it('answers an empty list without a query, and rejects ids that are not an array of strings', async () => { + __test.searchLog.length = 0; + assert.deepEqual(await getMemories([]), []); + assert.equal(__test.searchLog.length, 0); + for (const ids of [['mem-a', 7], 'mem-a', 7, new Set(['mem-a'])]) await assert.rejects(getMemories(ids), TypeError); + }); +}); + describe('scanMemories', () => { // Rows written straight to the store, so createdAt can be shared the way an import preserving source time does. const seed = (count, createdAtFor, overridesFor = () => ({})) => {