Skip to content

Claude Code plugin: record why prompt-time recall injected nothing, and skip prompts with no topic - #57

Merged
kriszyp merged 5 commits into
mainfrom
fix/plugin-recall-silent-misses
Nov 17, 2026
Merged

kriszyp merged 5 commits into
mainfrom
fix/plugin-recall-silent-misses

Conversation

@kriszyp

@kriszyp kriszyp commented Nov 17, 2026 •

Copy link
Copy Markdown
Member
On Utanas, a tally board of returning dogs shows a column of padlocks: why some came back empty.

⊙ Problem

Prompt-time recall printed nothing on any failure (missing credentials, a 401, a timeout), so a miss looked like "no relevant memories".

On the dispatch fleet every dev-agent run missed. Each run gets a private HOME, which hides ~/.config/agent-memory/env, so the plugin found a URL but no token. And a prompt like 1, 2, 3 recalled an unrelated memory.

💡 Solution

  • Each session start and prompt logs its outcome to diagnostics.jsonl (plus latency and hit count when a request ran), never prompt text or memory content; bin/diagnostics.mjs summarizes it.
  • A partial configuration gets a session-start notice naming what is missing.
  • Prompts with no distinctive term skip the search; change notices still arrive.
  • A failed state or cache write no longer drops the memories the server returned.

Fleet recall comes back with dispatch task dispatch-dev-prompt-recall-subject, which passes AGENT_MEMORY_ENV_FILE to runs.

🤔 Should one short term (fix CI, PR 2368) trigger a search?

No: hasDistinctiveTerm needs one non-conversational term of 4+ characters (2 in CJK scripts). MCP recall still serves such prompts; retuning is one function.

🤔 Should a never-configured install log every prompt?

Yes: when env vars are lost (a GUI launch, a harness), it is the only record of why nothing happened. The alternative logs only half-found configurations.

⚖️ Alternatives

  • Server-side similarity cutoff — new API needing per-model calibration we have no data for.
  • Threshold on the returned score — reciprocal rank fusion is rank only: any top hit scores at least 1/61.
  • Two content terms — also blocks single identifiers (lmdb, ECONNREFUSED).

Framing-Verdict: better-alternative-exists — resolved: dispatch has its own task; one-term gate adopted.

🤔 Should the plugin find the token itself when HOME is replaced?

No: the passwd home would bypass dispatch's deliberate private HOME, and temp-HOME tests would read a developer's real token. The harness passes AGENT_MEMORY_ENV_FILE instead.

🔧 Changes

One line per hook run

diagnostics.mjs:53

{ ts: '2026-11-17T06:12:03.511Z', event: 'recall', session: '3f9c…', // same key as the state file
  outcome: 'unconfigured', reason: 'missing_token' }
{ ts: '…', event: 'recall', session: '3f9c…', outcome: 'injected', ms: 312, hits: 3, injected: 1, notices: 0 }
{ ts: '…', event: 'recall', session: '3f9c…', outcome: 'timeout', ms: 1501, injected: 0, notices: 0 }

persist keeps the injection when a write fails

hooks.mjs:81

-saveSessionState(config, sessionId, state)       // ✗ threw: hook.mjs printed {}
-saveBootstrapCache(config, memories)
+persist(entry, () => saveSessionState(config, sessionId, state))   // failure → entry.stateError
+persist(entry, () => saveBootstrapCache(config, memories))
 return hookOutput('SessionStart', formatted.text)  // the injection still goes out

Files

Product and architecture tour: outcomes, credentials, the search gate, persistence

Product and architecture tour

Why a recall injected nothing

What does a reader of the log learn about a prompt that got no memories?

Before After
Every recall failure printed {}: unconfigured, 401, timeout, network error, bad JSON, corrupt state, or a thrown write. Each hook run records one outcome: injected, empty, deduped, over_budget, http_<status>, timeout, error (with a reason), skipped, disabled or unconfigured.

Lines hold only allowlisted fields: never prompt text, the query, memory ids or content, URLs or credentials.
Logging is best-effort: a hook killed at its 3 s timeout, a full disk, or two writers rotating at once can lose a line.

  • The log sits in the cache dir, not the per-endpoint dir, so a run with no endpoint still logs.
  • It rotates to diagnostics.jsonl.1 at 256 KiB. It is opened O_NOFOLLOW | O_NONBLOCK, blocking planted symlinks or FIFOs.
  • node <plugin root>/bin/diagnostics.mjs prints counts by outcome and latency percentiles; --tail N and --json add detail.

Credentials under a replaced HOME

What does a dispatch run see now, before the dispatch task lands?

Where the hooks look for a token

The plugin reads only the environment, AGENT_MEMORY_ENV_FILE, or the env file under HOME. It now reports partial configurations.

flowchart LR
    Env[Token in environment] -->|no| File[AGENT_MEMORY_ENV_FILE or HOME env file]
    File -->|found| Ok[Hooks run]
    File -->|missing| Half{URL or env file set?}
    Half -->|yes| Notice[Session notice and unconfigured log line]
    Half -->|no| Quiet[Plugin not set up: log line only]
Loading

A set HOME is never bypassed for the passwd home; a harness that replaces HOME passes AGENT_MEMORY_ENV_FILE.

Prompts with no distinctive term

Which prompts still search?

Prompt Searches? Why
1, 2, 3, yes, abc, PR 2368 no no letter term of 4 or more characters
please continue, sounds good no conversational words only
lmdb, ECONNREFUSED, db migration yes one distinctive term is enough
修复, 복제 yes 2 characters in a script with short words

The gate skips only GET /Memory/?q=. Queued change notices (withdrawn, superseded, flagged) still drain and inject.

Writes that fail no longer drop an injection

What happens when state.json or the offline cache cannot be written?

  • The cost is a possible repeat of the same memory next turn, since the dedupe record was not saved.
  • A bootstrap body cut off by the deadline or the network now falls back to the offline cache, like any other outage.
  • The follower spawn's asynchronous error is now handled; unhandled, it ended the hook as an uncaught exception.

✅ Verification

  • npm test 522/522 on Node 22 and 26, npm run test:integration 81/81, format clean. A 65-term w0 w1 … probe now leads with a real word (memory.test.ts).
  • Fails on base: the reproduction test runs hook.mjs with a private HOME and the token in a host env file. Base prints {} with no trace; here it shows the notice and logs unconfigured/missing_token. Base also injects for 1, 2, 3 and drops an injection on a failed state write.
  • 24 hook sessions started at once against a 300 ms server inject 24 of 24 (table below).
Hook wall time, base vs this branch (real hook.mjs processes, all sessions launched at once, 300 ms mock server, median of 3 runs)

New tests cover each outcome (timeout via a hung server and a cut-off body), the size cap and rotation, symlink refusal, the gate with notices, and write failures.

Sessions Prompt Base p50 / p95 ms Branch p50 / p95 ms
12 normal 475 / 528 473 / 518
24 normal 603 / 700 586 / 680
24 1, 2, 3 637 / 718 (searches) 189 / 289 (skips)
24 unconfigured 82 / 105 88 / 112

Misses appear only when server latency nears the 1500 ms recall timeout (24 of 24 injected at 1200 ms, 0 at 1450 ms). Host: 12 cores, load about 1 before each run, turbo on, not under bench-run.

Docs: the plugin README is the user-facing doc; HarperFast/documentation has no agent-memory pages.

🤖 Generated by Claude Opus 5.5 with Claude Code; posted via @kriszyp.

Related PRs: none found
Complexity: medium

Deferred findings

Origin — the dispatch brief this PR was written from

Find and fix why the Claude plugin's prompt-time recall (UserPromptSubmit, claude-plugin/lib/hooks.mjs handleUserPromptSubmit) intermittently injects nothing, and make such misses observable. Every failure path returns {} silently today (401/403, non-OK status, timeout, fetch error, corrupt session state), so a miss looks identical to 'no relevant memories'. Step 1: add a bounded, append-only diagnostics log under the plugin cacheDir (one JSON line per recall: timestamp, session id hash, outcome = injected|empty|deduped|http_|timeout|error, latency ms, hit count; no prompt text, no memory content, no credentials), rotated or size-capped, plus a way to read it (e.g. a small script or a documented path). Step 2: use it plus a reproduction to find the root cause of the intermittent misses described in context and fix it. Step 3: when the query is too short or generic to be meaningful (e.g. a prompt like '1, 2, 3'), recall should inject nothing rather than the nearest unrelated memory: apply a minimum relevance score or minimum query content, whichever matches how the server scores results.

Acceptance

Unit tests for each recall outcome being logged (including timeout and http error), for the log size cap, and for the short/generic-query case injecting nothing. A reproduction (test or script) of the intermittent miss that fails before the fix and passes after, or, if the root cause is server latency, a measured latency breakdown in the PR with the chosen fix (e.g. timeout/budget change, query shortening, or server-side change) justified by the numbers. Existing unit and integration tests pass.

Dispatch: task agent-memory-recall-silent-misses · queued by kris-session · ran by claude/opus/xhigh · worker kzyp-xps-1

Ready-For-Promotion: 8ce60a2 urgency=normal late=folded

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,gemini-carbon,cursor-grok; late=gemini-carbon:adjudicated,cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=5; full=2 @ 8ce60a2

Review-Attention: read ~6m @ 8ce60a2

kriszyp and others added 4 commits November 16, 2026 23:08
…pts, keep injections when state writes fail

Prompt-time recall returned {} on every failure path, so a miss looked like "no relevant memories". On the
dispatch fleet every run missed because a run's private HOME hides ~/.config/agent-memory/env: the plugin
found AGENT_MEMORY_URL but no token and silently did nothing.

- Each SessionStart and UserPromptSubmit appends one line to <cacheDir>/diagnostics.jsonl (outcome, reason,
  latency, counts; no prompt text, memory content, URLs or credentials), rotated at 256 KiB;
  bin/diagnostics.mjs summarizes it.
- A partial configuration (URL without token, or the reverse) gets a SessionStart notice naming what is missing
  and AGENT_MEMORY_ENV_FILE.
- The recall search is skipped when the query has no distinctive term (1, 2, 3 / yes / please continue);
  queued change notices are still delivered. eval-recall applies the same gate.
- State and offline-cache writes are best-effort in both hooks, and the follower spawn's async error is
  handled, so neither can discard an injection the server returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: agent-memory-recall-silent-misses
… hardened log reader

- Bootstrap: a body cut off by the deadline or the network falls back to the offline cache like any other
  outage, and is labeled timeout or network (with its errno) instead of bad_json; hits that do not fit the
  budget log over_budget, as recall does.
- The partial-config notice falls back to a short sentence when the budget cannot hold the full one, and an
  explicitly set AGENT_MEMORY_ENV_FILE that yields nothing counts as partial configuration.
- Diagnostics: open before mkdir, one fstat, no descriptor leak; the reader refuses symlinks and FIFOs like the
  writer; summaries use Maps so odd event names cannot break them.
- Tests: permission-based write-failure test skips under root; DESIGN.md wording narrowed to what the code does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: agent-memory-recall-silent-misses
…maining silent paths

- The partial-config notice names the remedy for the reason: an env file for a missing URL or token, the
  https/loopback rule for a rejected URL, a single-line token for a rejected token.
- The reader reads at most the cap's worth from the end of each file.
- SessionStart with an unsupported source and the MCP headers helper without a credential each log a line;
  stateError keeps every distinct code; eval-recall's text report prints the skipped count.
- DESIGN.md: `injected` means returned to hook.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: agent-memory-recall-silent-misses
…deadline

AbortSignal.timeout's timer is unref'd, so with no socket open Node 22's test runner saw an empty event loop
and cancelled the rest of the file. Node 24 and later passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: agent-memory-recall-silent-misses
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: agent-memory-recall-silent-misses
@kriszyp
kriszyp marked this pull request as ready for review November 17, 2026 14:27
@kriszyp
kriszyp merged commit 73fb014 into main Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant