perf: cut per-pass main-thread work on long sessions (#834) by iceteaSA · Pull Request #842 · cortexkit/magic-context

iceteaSA · 2026-11-02T18:26:39Z

Approved issue: #834 is open and not yet triaged. The live results are in #834 (comment #834 (comment)). Close this if you prefer another approach to any of the changes; each is in its own commits.

What changes

Changes that cut per-pass main-thread work on long sessions, and the fixes the review found. The branch keeps the review history as separate commits; the final behaviour is:

Native input copy (OpenCode 2), fix(v2) commits. The context hook copied the whole native history on every pass so that capture could record it for the idle host fold and the compaction hook. idleFoldCanArm(cacheTtl) (in v2/fold/idle.ts, shared with HostFoldIdle.idle) is false only for a lifetime that never expires. For such a session the copy is made only to make the capture current, and at most once for each model and set of effort controls:
- NativeFoldReplay.captureCurrent is true while the last successful capture of the session in this process has the same model and effort controls. Each control is placed by the last host message id before it and the count of messages without an id after that one, not by the coordinates a capture binds on its copy. Such a pass skips the copy.
- copyWanted / settleCopy: when a copy did not give a current capture (the copy threw, the capture did not run, for example because a checkpoint was served, or it failed), the same inputs are not copied again until the served head changes.
- noteSkippedCapture: a skipped pass whose capture is not current, or whose served head (also when no head is recognised) or boundary changed, marks the capture unproven, so a host fold refuses it and uses the ordinary history path until a pass captures again.
- Each pass logs v2 native input: copied|skipped reason=….
Idle host fold for subagents, fix(v2). A subagent has no history, so the fold's summary fell back to an empty history block (v2 host fold writes an empty summary for sessions with no history (subagents), and OpenCode 2.2.0 then refuses the session #838). The idle fold is armed only for a session whose stored mode is primary.
Signed-thinking fit snapshot, perf(transform) commits. takeSignedThinkingFitSnapshot gets the pass's context limit, the last measured input, and the estimated size of the messages that request did not include (the newest assistant message and everything after it). It skips the copy when the limit minus both is more than SIGNED_FIT_SNAPSHOT_HEADROOM_TOKENS (250,000). An unknown limit or input keeps the copy. A held pass is always inside an assistant tool loop. A held pass that overflows without a copy is refused, which is the existing path for a failed copy.
Entry digest cache, perf(lkg) commits. LkgEntryDigestCache measures the session's need first, then gives shares in increasing order of need: each session gets the smaller of its need, the per-session cap, and an even split of what is left. Candidates are held only up to the largest share a session can receive. The limits are 768 MiB total, 192 MiB per session and 64 sessions.

Why

On sessions of 12,000 to 17,000 messages, three paths were the largest magic-context main-thread costs: a full encode and revive of the history on every pass, a full structuredClone on every held signed-thinking pass, and the re-hashing of every message outside a 16 MiB even share of the digest cache. #834 has the before and after profiles and the step data behind the 250K margin.

Measured on the final version: each long session with a never-expiring lifetime copied its native input once after each plugin load and skipped on every later pass (for example 1 copy and 71 skips over 100 minutes); sampling profiles show 0 samples in the copy path.

Tests

New tests, each checked to fail without its change unless noted:

src/v2/hooks/native-copy-ttl.test.ts: a never-expiring session stops copying once its lifetime is stored; an expiring session copies on every pass; a new effort control or a model change copies again; an effort control after an instruction row without an id does not make every pass copy; a copy that throws is not tried on every pass; a session whose capture cannot become current copies once and then skips, with the log lines; a manual compaction after the capture is marked unproven still gets the session's history as its summary. src/v2/fold/capture-skip.test.ts covers captureCurrent, noteSkippedCapture, copyWanted and settleCopy, including a refused host fold and its recovery after the next capture, a retry after the served head changes, and a pass with no recognised head. src/v2/fold/idle.test.ts covers idleFoldCanArm. src/v2/hooks/fold-contention-review.test.ts now uses cache_ttl: "5m", because it reaches the capture path. The existing fold-review-regressions.test.ts (repeated instructions and effort markers across a native fold) passes; an intermediate version of this change failed it, which led to the bounded copy.
src/v2/hooks/event-loop.test.ts: a subagent, and a session whose mode is not stored, never arms the idle fold (Expected: 0, Received: 1 without the change); the existing primary-session test still arms.
src/hooks/magic-context/signed-thinking-fit-snapshot.test.ts: the copy is skipped far from the window and kept near it, with unknown usage, or with a large unmeasured tail; through the transform, a held pass far below a 1M window makes no copy.
src/hooks/magic-context/lkg-entry-cache-work.test.ts: a long session among many small ones keeps all it needs; a small session's unused budget goes to the long sessions; more than 16 sessions do not evict a long session; a session over the cap keeps correct digests and a leading share. The candidate bound only lowers transient memory, and no test observes that.

bun run typecheck and biome check on the changed files are clean. Suites run on 0.49.35 plus these changes at 36971f5 (0.49.35 is the release the changes run on live). The last two commits, c53baad and e411e27, change only OpenCode 2 files under src/v2/. At e411e27 they were checked with the nine test files that cover them (native-copy-ttl, capture-skip, fold-review-regressions, event-loop, idle, fold-contention-review, native-replay, native-replay-review, native-replay-v3): 76 pass, 1 fail. The failing test, v2 never arms host folding for a subagent or a session whose mode is not stored, fails the same way at c53baad when these nine files run in one process, and passes when its file runs alone:

packages/plugin fast suite, on 36971f5: 9,793 pass, 103 skip, 11 fail. 5 fail the same way without the changes in our checkout (a writer-admission test and 4 bundle tests that need built outputs). 6 are long tests that failed under load; their files pass when run alone, with and without the changes.
packages/pi-plugin suite, on 36971f5: 3,806 pass, 53 skip, 20 fail. 16 are in 3 files (pi-historian-crash, issue-686-image-budget, dreamer/pi-session-api) that fail the same way without the changes in our checkout. The other 4 (two tests, and two load errors between tests that also stopped 46 tests from running) are in 4 files that pass when run alone, with and without the changes.

Harness coverage (required)

OpenCode 1: covered — the snapshot and digest cache changes apply (shared transform and digest cache), tested by the packages/plugin suite; the native copy and idle fold changes are not applicable (OpenCode 2 only)
OpenCode 2: covered — packages/plugin suite, and run live on an OpenCode 2.2.5 service
Pi: covered — the digest cache change applies through noteEntry, tested by the packages/pi-plugin suite; the other changes are not applicable, as the surface does not exist there (pi-plugin/src/native-replay-pi.ts, which the advisory names, rewrites captured native items and has no per-pass copy or capture)
OMP: NOT covered — the digest cache change applies through the same Pi package code; the Pi suite includes OMP cases, but no OMP host was run
Rust module: not applicable — surface does not exist there

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues remain.

Summary

Cuts repeated history copying and hashing on long sessions.

Never-expiring OpenCode 2 sessions skip unchanged native-history copies.
Held signed-thinking passes skip snapshots when the context window is far away.
Long sessions keep more reusable message digests in the shared cache.
Only sessions stored as primary arm the idle host fold.

_{Reviews (5) · Last reviewed commit: "perf(v2): take the capture inputs key on..." · Reviewed by Greptile}

The context hook encoded and revived the whole native history on every pass to capture it for the idle host fold. A session whose cache lifetime is "never" can never arm that fold, and a missing capture is restored from the source store, so the copy was pure main-thread cost on every pass of such sessions (cortexkit#834).

…window A held signed-thinking pass cloned the whole history on every pass so a fit retry could restart from it. The retry is needed only when one step takes the request over the window. Skip the copy when the last measured input is more than 250K tokens below the context limit; a held pass that still overflows is refused, as when the copy fails (cortexkit#834, cortexkit#798).

The entry digest cache split its budget evenly over up to 16 sessions, so many small child sessions cut each long session to a share it could not use, and the rest of its messages were hashed on every pass. Give shares in increasing order of need, and raise the limits to 768 MiB in total, 192 MiB per session and 64 sessions (cortexkit#834).

github-actions · 2026-11-02T18:26:55Z

Waiting for design-approved on #834.

This PR needs a linked issue with the design-approved label before it can be reviewed or merged. Link the approved issue with Approved issue: #<issue> (or Refs #<issue>) in the description. A maintainer will apply the design-approved label on the issue, or trivial on the pull request when there is genuinely no design to agree. The issue stays open until the change ships; maintainers take care of it then.

cubic-dev-ai

All reported issues were addressed across 10 files

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

Entries past the largest share a session can receive are counted as need but not kept until the share is known, so a long or cold history does not hold every flattened message until the end of the pass.

iceteaSA · 2026-11-02T19:31:44Z

Moving this to draft while I rework the first change. Status of each review item:

Native input copy, v2/hooks/context.ts (Greptile P1): valid, and wider than the effort controls. When the copy is skipped, capture is skipped too. capture also records two things that the source rows cannot rebuild: the effort controls, and the served history head that the compaction hook later reads as the summary (nativeFoldReplay.baseline). After a skip, a host compaction of a "never" session could use controls and a summary from an older capture. A more durable fix is in progress: a pass skips the copy only while the model, the effort controls (each with the row before it) and the served head are the same as at the last capture of that session. Any change, an invalidation, or a new process runs the full copy and capture as before. Tests will cover an effort change and a head change after a skipped pass.
Digest candidates, hooks/magic-context/lkg-slot.ts (cubic P2, Greptile P2): valid. Candidates are now held only up to the largest share a session can receive. Entries past it count toward the need but are not kept. A new test covers a session over the per-session cap. This is committed locally and goes up with the rework.
Snapshot headroom, hooks/magic-context/signed-thinking-fit-snapshot.ts (cubic P2): I will answer in that thread.

I will mark this ready for review again when the rework and its tests are in.

A skipped native copy also skipped the capture, which records the effort controls and the served head that a host compaction later uses. Skip only while the model, the effort controls (with the row before each) and the served head and boundary match the last successful capture of the session in this process. A pass whose head changed after a skip marks the stored capture unproven, so a host fold falls back until the next pass copies and captures again.

The 250K headroom was measured, not enforced. Also count the estimated size of the messages the last measured request did not include (the newest assistant message and everything after it), and keep the copy when the window minus both is within the headroom. A held pass is always inside an assistant tool loop, so no new user content arrives in it.

A subagent has no history of its own, so the idle fold's compaction summary falls back to an empty history block and replaces the subagent's conversation (cortexkit#838). Arm the fold only for a session whose stored mode is primary; a subagent or a session whose mode is not yet stored is not armed.

…e unproven The previous change copied whenever the last capture was not current. A capture never becomes current in a process where the session's capture does not run, for example after a host checkpoint, so such a session copied the whole history on every pass and discarded it. A session whose cache lifetime cannot arm the idle fold now never copies. When its last capture is not current for the pass (no record, another model or effort controls) or the served head or boundary changed, the capture is marked unproven, so a host fold refuses it and uses the ordinary history path. Each pass logs whether it copied the native input and why.

Never copying for a never-expiring session left it without any capture in a new process, so a native host fold refused the capture and lost the coordinates of repeated instructions and effort markers. A never-expiring session now copies when its capture is not current, unless the last copy with the same model and effort controls did not give a current capture (the capture did not run, for example because a checkpoint was served, or it failed). Such a pass skips the copy and marks the capture unproven. The copies are bounded to one for each change of those inputs, and the capture stays current wherever it runs.

github-actions · 2026-11-03T00:58:13Z

Harness coverage advisory: these conventional twins were not touched:

packages/plugin/src/v2/fold/native-replay.ts → packages/pi-plugin/src/native-replay-pi.ts

Cover every shipped harness or explain why the surface does not exist there. See CONTRIBUTING.md. This is advisory only; no draft state is changed.

cubic-dev-ai

All reported issues were addressed across 13 files

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

…new head The capture inputs key placed each effort control by the source coordinate a capture binds on its copy. A host draft has none bound, so a session with an effort control after an instruction row without an id never matched its capture and copied the whole history on every pass. The key now places each control by the last host id before it and the count of rows without an id since then. A copy that gave no current capture is retried once the served head changes, so a failure does not block the capture until the model or the effort controls change. A skipped pass with no recognised served head marks the capture unproven. The headroom constant moves above the snapshot function's doc comment.

cubic-dev-ai

All reported issues were addressed across 5 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

A copy that threw left no failed-copy record, so the next pass tried the same failing copy again. Every attempted copy is now settled, with the capture inputs key taken from the draft before the transform. The tests for a session whose capture cannot become current now stop the capture itself after a fold and a new plugin process, in place of forcing the current check to false.

cubic-dev-ai

All reported issues were addressed across 4 files (changes from recent commits).

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

View guided diff | Re-trigger cubic}

The skip decision walked the history for the capture inputs key up to four times per pass: once for the settle key, once in captureCurrent, and twice more in copyWanted when the capture was not current. A pass now computes the key once, before the transform, and hands it to captureCurrentFor and copyWanted, so a steady-state pass does one walk instead of two.

Tehan added 3 commits November 2, 2026 18:20

iceteaSA mentioned this pull request Nov 2, 2026

OpenCode 2: each pass copies the whole live history two or three times (native-fold capture and adaptPayload), also when no host fold can start #834

Open

cubic-dev-ai Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread packages/plugin/src/hooks/magic-context/signed-thinking-fit-snapshot.ts Resolved

Comment thread packages/plugin/src/hooks/magic-context/lkg-slot.ts Resolved

greptile-apps Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread packages/plugin/src/v2/hooks/context.ts Resolved

Comment thread packages/plugin/src/hooks/magic-context/lkg-slot.ts Resolved

perf(lkg): hold digest candidates only up to the largest share

611eefd

Entries past the largest share a session can receive are counted as need but not kept until the share is known, so a long or cold history does not hold every flattened message until the end of the pass.

iceteaSA marked this pull request as draft November 2, 2026 19:31

Tehan added 5 commits November 2, 2026 20:03

iceteaSA marked this pull request as ready for review November 3, 2026 00:58

cubic-dev-ai Bot reviewed Nov 3, 2026

View reviewed changes

greptile-apps Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/plugin/src/v2/fold/native-replay.ts Resolved

cubic-dev-ai Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/plugin/src/v2/hooks/context.ts Resolved

cubic-dev-ai Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/plugin/src/v2/hooks/context.ts Resolved

Conversation

iceteaSA commented Nov 2, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Why

Tests

Harness coverage (required)

Confidence Score: 5/5

Summary

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iceteaSA commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

iceteaSA commented Nov 2, 2026 •

edited by greptile-apps Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading