Repository navigation
feat: host-driven runs, leases and adapters for v1.10.0 - #63
Merged
Merged
Conversation
Postgres Migrate now holds a cortex advisory lock for the whole call, so replicas that boot together take turns. Before, all but one failed with "migration lock is held by another process". The key is -918477416930547652, the FNV-1a 64-bit hash of the Migrate symbol, so it never meets grove's key 1. Migrate takes and releases it on one dedicated connection, and it covers grove's tracking-table creation, the migration group, the rescope and the session backfill. The group runs through grove's pgmigrate executor now, which retries key 1 for up to five minutes, and the old fail-fast executor is gone. A waiting replica polls pg_try_advisory_lock. We tried blocking in pg_advisory_lock first and eight replicas hung: each blocked call keeps a snapshot open, and the holder's CREATE INDEX CONCURRENTLY waits for every older snapshot. Other grove users on the database never take the cortex lock. On a fresh database one of them can win the CREATE of grove_migrations and leave cortex with 42P07 or 42710, so Migrate retries that up to five times. Tests cover eight replicas on a fresh database (run 12 times), cortex alongside plain grove migrators of another group, a held cortex lock, a held key 1 and a cancelled wait. The Postgres store page explains the lock and what a host running the group in its own pass has to do.
The replica tests passed with the cortex lock deleted, because grove's key 1 already serializes the migration group. What the lock really protects is the rescope and the default-session backfill, which run after grove lets go of key 1. A new test seeds pre-v1.8.0 agents and conversation rows and starts eight replicas with one Rescoper that takes 100ms a call. It checks that each legacy identifier is rescoped once, nothing is left unscoped, and each agent ends up with one default session holding all of its rows. With withMigrateLock reduced to fn(ctx) it failed 5 runs out of 5, every replica but the first hitting a rescope collision. Migrate also fails fast on a pool of one now. Grove's driver doesn't report the pool size, so it waits up to five seconds for a second connection and then says it needs two, where before it hung until the context ended. The unlock gets its own five-second timeout, tests cover the lock coming back when the work errors or panics, and the docs name driver.WithPoolSize as what sizes the pool.
…n config engine.WithRunConcurrency(n) caps how many runs execute at once in a process. Zero, the default, is no limit. A run that finds every slot taken waits for one only as long as its ctx allows, then returns an error that matches ctx.Err() and writes no run row. RunAgent, StreamAgent, RunPrepared, StreamPrepared, Resume, ResumeStream, ResolveCheckpoint, the message bus's resumes and RecoverAgentReply are all counted. Streams hold their slot until the channel closes. A run started on the ctx of a run that already holds a slot shares it, so a tool that calls RunAgent can't deadlock on a cap of 1. Task worker runs and prepared orchestrations stay outside the cap. Stop now drains inside the lifecycle lock before it touches task services. New runs get cortex.ErrEngineStopping and Stop waits for the ones in flight. If its ctx ends first, Stop signals task services and returns the deadline error without stopping the sweeper or the bus and without OnShutdown; the runs keep executing and record their own result, and a second Stop finishes the job. Start admits runs again. The Forge extension now applies its config: the default_* keys and shutdown_timeout become the engine's cortex.Config, run_concurrency goes through WithRunConcurrency, and Stop is bounded by shutdown_timeout. The option order is now the precedence rule. The discovered store and the nexus, shield, weave and sentinel adapters go in before your own options, so WithStore and WithEngineOption win over them. WithoutAutoAdapters skips the discovery. Both DefaultConfig() values of RunConcurrency drop from 4 to 0. Verified with GOWORK=off go test -race on engine and extension (the new admission and config tests ran five times clean), the root suite, scopespy and every sub-module, and golangci-lint on root and extension.
Before this change, a peer reply that landed during Stop's drain was lost. The bus claimed or sealed the waiting ask and then called the resume, which admission refused, so the asker sat paused behind a consumed claim until its suspension expired. A directive that arrived during the drain was worse: the asker got "could not answer: cortex: engine is stopping" as a Failure reply and read it as its peer's answer. The bus now has an Admitter, and the engine gives it run admission. Drain asks before it claims each delivery, and the ask paths ask before they claim or seal, so while the engine stops nothing new is claimed. Queued rows wait for another replica's dispatcher, or for this one after Start. A reply sent from outside a run during the drain is queued as a delivery and leaves the ask alone. The expired-ask sweep ends its pass quietly. Work the bus admitted before the drain keeps its ticket, so the reply resume after a consumed claim runs through stopping and Stop waits for it. If a run is still refused after its delivery was claimed, the row goes back to queued with no reply, and nobody gets a Failure for ErrEngineStopping. Background agent-reply recovery treats ErrEngineStopping as a retry for a later pass and logs it at debug. Review minors fixed here: a nested run can no longer join a parent that is releasing (the released flag is guarded by the admission lock); Config() and UpdateConfig() report the cap the engine applies; WithStore stops the extension from looking for a default grove database; the docs and comments list plan replanning among the runs the cap doesn't count. Verified with GOWORK=off go test -race on engine and a2a (the drain tests three times over), the root suite, scopespy, every sub-module and make lint.
…pause
An authorizer can now return cortex.ApprovalRequired{Reason, TTL}. It still
matches ErrRequiresApproval, so existing hosts keep working, and the engine
puts the reason on the pending call (PendingCall.Note) and on the checkpoint.
External tools take engine.ExternalTTL, and a run waiting on another agent
follows its ask's reply-by plus WithAgentReplyGrace (5 minutes), so the bus
times the ask out before the engine fails the run. The suspension expires at
the earliest deadline among its calls. A negative approval TTL denies the call.
WithSuspensionTTL(0) now means no default deadline. The sweeper runs whenever
there is a store, because a call can carry a deadline of its own.
plugin.RunSuspended fires once per pause, for every reason, after the flip to
paused and before the step's questions go out to peers.
Verified with GOWORK=off: go test -race for the root, a2a, suspension, engine
(912s), store/sqlite, api and extension; make fmt and make lint report 0
issues in every module.
A paused run used to resume under whoever resumed it, so after an approval the authorizer was asked about the approver. The engine now encodes the caller's principal with a PrincipalCodec when it creates a run, stores the bytes on cortex_runs.principal, and decodes them before every claim: Resume, ResumeStream, ResolveCheckpoint, the messaging bus and RecoverAgentReply. A revoked principal fails the run. Any other decode error returns before the claim and leaves the run paused. You need a codec once you install a tool authorizer: engine.New returns ErrPrincipalCodecRequired otherwise. ResolveCheckpoint records the decider in Decision.DecidedByPrincipal. Migration 20261101000001 adds the column on Postgres and SQLite. Mongo needs no DDL. Verified with -race on root, engine, store/sqlite, store/postgres and store/mongo (containers, no skips), plus api, extension and sentinel. make lint reports 0 issues in every module.
… responses Follow-ups to the principal work in 76b9fb4. JSONPrincipalCodec now reports stored bytes that can't decode into T (corrupt, or written before T changed) as ErrPrincipalRevoked, since no retry fixes them. Other decode errors stay transient. restorePrincipal returns the revoked and transient cases separately, so a claim site returns err before the claim and fails the run with revoked after it. The engine logs a warning with the run ID when it does that, and a run under required verification now keeps ErrPrincipalRevoked as its error in place of ErrVerificationUnavailable. Decision.DecidedByPrincipal stays in the stored decision JSON. The REST checkpoint list and the dashboard contract serve checkpoints through Checkpoint.Public, which drops it. Task-run children created through coordination still carry no principal bytes and resume with a nil principal. A test pins that, and the authorization docs say so and say what to key on. New tests cover a second pause on the same run, the revoked reason on run.Error, and a transient decode during ResolveCheckpoint. The two fixtures that apply WithToolAuthorizer straight to an engine now get a codec too.
You can now answer a paused run from an HTTP handler and finish it later. DecideCheckpoint and ProvideResults record the answer, move the run to ready and return without calling the model. ContinueRun claims a ready run from a job on any replica and runs it as the principal that started it. ResolveCheckpoint is DecideCheckpoint followed by ContinueRun, so the decision is recorded before the run moves. WithRejectionsReturnToModel hands a rejection back to the model as a denied tool result. CancelRun takes ready and reclaimable runs, and a run failed for a revoked principal closes its pending checkpoint.
A decision now answers only the pause its checkpoint was opened for. DecideCheckpoint closes a checkpoint the run has moved past as superseded and refuses it, and MarkReady refuses readiness that names another suspension, so a stale checkpoint can't approve the run's next gated call. Checkpoint Resolve writes only on a pending row in every backend. The losing decider gets ErrInvalidState and fires no hook, and a late approval can't overwrite a cancel. Readiness keeps the decider, and ContinueRun records a decision whose checkpoint write was lost. A suspension read that fails before the claim leaves the run ready.
A decision on a run with no current suspension is refused with ErrInvalidState and closes nothing, because another decider may still be recording. Only the ready path closes a checkpoint as superseded, and only when the run waits on a different pause. Every losing decider now gets ErrInvalidState. A decider whose own decision a job already recorded treats that as success, and fires RunReady only while the run is still ready.
api.WithAuthorizer puts all 91 routes behind Authorize, keyed by each route's operation id and whether it writes. A refusal is a 403 unless it wraps a cortex not-found or scope error. With an authorizer, resolveCheckpoint records DecidedBy for the request principal and ignores the body's decided_by. Without one the module is for single-trust deployments only, and the docs now say so. GET /v1/tools lists engine.Tools with kind and retry_safe, and /v1/tools/:name/schema serves the tool's parameters. Checkpoint responses use their own type, so decided_by_principal is gone from the OpenAPI schema too.
The prepared-team recovery scan returned ErrRunConflict when its start of an admitted reply lost to the sweep marking the same run reclaimable. That ended the pass, and the coordinator then failed the whole team. The scan now skips that child like one already started and goes on with the rest; the next pass, or ContinueRun, takes it from there. The new test kills an admitter between its admission and its start in a child process, then has the scan's first start lose once. Also adds the agent_reply counterpart of the lost expiry reclaim test.
StartResumeLeased from another scope now has to be ErrResumeAdmissionNotFound and AdmitResumeLeased ErrInvalidScope, the same refusals plain StartResume and AdmitResume give there.
48 handlers wrote their response with ctx.JSON and also returned it, so forge serialized it a second time. Clients got two concatenated JSON objects and the server logged a superfluous WriteHeader. A 200 is now returned for forge to write; a 201, and the 410 of a replay gap, is written by the handler, which returns nil. A new test drives a success path in every handler file and fails on a second header or a second JSON value; the replay test checks its 200 and 410 bodies the same way.
Authorize gets the operation id and the request context only, so a per-resource rule belongs in the host's own lookups. sendAgentMessage's from stays caller-asserted with an authorizer; it's attribution, and the reply token gates what a reply resumes. The route count line now says every route instead of a number that goes stale.
The skip on ErrRunConflict was silent, so a conflict that kept coming back would loop every interval with nothing in the logs. It now logs a warning with the run id and the error type, nothing else.
… through nexus The nexus adapter now sends every call under a nexus tenant. It takes the tenant from pipeline.WithTenantID on the context, or from a cortex.TenantFunc over the scope (cortex.RootTenant by default), and refuses a call it can't name a tenant for with an error matching cortex.ErrNoScope, before anything reaches the gateway. Each call carries usage metadata: one pair per scope level plus the run and agent the engine now sets on every request, cut to nexus's key, value and count limits. New adapter options: WithTenant, WithMetadata, WithProvider and WithSamplingUnsupported (temperature is dropped for the Claude 5.5 ids by default). NewFunc covers a gateway that only exists once nexus starts, and the Forge extension passes options through WithNexusOptions without changing the option order. Native provider content (signed thinking blocks) comes back on llm.Response and the final stream chunk. The engine keeps it on the assistant message, so it survives a pause, a hand-off and a reclaim and goes back to the model, but never into conversation memory. A guard block from nexus surfaces as llm.ErrContentBlocked and ends the run with safety_block, with no tool from that step dispatched. Task-bound calls keep their metadata beside the bound payload, so request digests don't change. Pins nexus v1.9.1 in the root and extension modules; forge v1.13.2 and grove v1.10.0 are unchanged.
…redaction A turn whose text publication redacted loses its thinking blocks on replay, and native content holds the model's unredacted output at rest. A test pins what the engine sends for a redacted turn.
Shield v1.9.1 evaluates profiles for real, so the adapter now uses them. Every content scan runs under a shield tenant taken from the run's scope (cortex.RootTenant by default, WithTenant to map it) and the app from WithApp, and evaluates the agent's shield_profile. The old refusal of a named profile is gone. A scan that can't run returns safety.ErrUnavailable joined with the shield sentinel it matched and nothing else of shield's error, so the text, the profile name and the tenant stay out of logs. The adapter is also a safety.PublicationGate, and EngineOption installs it as both scanner and gate. Inbound tool results, tool errors and retrieved knowledge go to shield as untrusted, so the model reads shield's quoted text (published as redacted). safety.DecisionFromScan exports the mapping ScannerGate already used. NewToolAuthorizer puts shield's tool-call scan under the host's authorizer. The host answers first and its denial is returned as is. Otherwise the two answers go through scan.Stricter: shield can deny a call or ask for approval, never allow one the host refused, and a host approval keeps its own reason and TTL. The extension takes adapter options through WithShieldOptions. The storeless-shield test is replaced by tests on an engine seeded with presets.InfraAgent.
A run in o1 that names o2's collection by the parsed ID gets weave.ErrCollectionNotFound and no o2 chunks, while o2 still reads it.
With "retrieval" in the default untrusted sources, every run with a knowledge section failed: the engine re-checks a prepared run's knowledge before each pass and expects the bytes it stored, but quoting a quote wrapped it again. Now text that is exactly one shield quote for its source has its inner text scanned as untrusted, so when shield changes nothing the same bytes come back. A forged or partial wrapper is still quoted, with its tags escaped, and an untrusted result that isn't one quote is refused. RunAgent, StreamAgent and an approved resume of a knowledge-backed run now complete under the real adapter with the default sources, with the retrieved text in exactly one wrapper. The fail-closed table also covers shield.ErrProfileInvalid.
MIGRATION.md now says that shield gates publication even next to engine.WithSafety, that its tenant is the scope's first level, and that shield_profile is honoured. The authorization page shows how to block secrets in tool arguments, and its table covers a scan that can't run.
The shield adapter is now a safety.IncrementalGate, so you can set safety.ModeCheckedSegments and stream approved text as the model writes it. Each segment ends after a line break or a sentence end. A run with neither is cut at its last whitespace once it reaches WithSegmentLimit (2048 bytes by default), so a word is never split. A block or a scan that can't run ends the stream, and held text is never published.
A colon no longer ends a segment, and the limit cut skips whitespace next to ':' or '=', so a credential assignment reaches shield with its key. A PEM block is held from its BEGIN line until its matching END line or the final check, so the private key detector sees the whole block. Streamed output now publishes what a whole-message check does for these cases. The docs say exactly what is held whole.
A blank run at the end of pending may sit before an assignment operator that hasn't arrived yet, so the limit cut no longer takes it. Without this, "key = value" and "\"key\" : value" could split when the limit fell right after the key's blank, and the value went out in clear.
WithRunLease now defaults to zero, which leaves leasing off. A default engine drains on Stop the way v1.9.0 did: every admitted run finishes, nothing is handed off, and no run becomes reclaimable, so the lapsed sweep has nothing to do. Turn it on with WithRunLease(ttl) once your host wires RunReady and RunReclaimable to a job that calls ContinueRun, since nothing in cortex does. The tests that exercise leases, hand-offs and reclaims now ask for a lease explicitly. ResolveCheckpoint still records the decision before the run continues. When the continuation fails before its claim, the error now says the decision is recorded and ContinueRun finishes the run.
A run the messaging bus starts for a directive kept its reply route, the
asker's reply token included, in the run's metadata, so run JSON, events
and REST responses carried it. The route now lives in Run.ReplyRoute,
which only the engine reads and which is never serialized. Stores write
it once, at creation, in a new reply_route column (migration
20261101000004 on SQLite and PostgreSQL) and never rewrite it.
The lease owner names the worker's host, so Lease.Owner is no longer
serialized either; the expiry still is. A reclaimed run that stops on an
uncertain call says "another worker" in its error. The owner stays in
the store and the progress snapshot.
A conformance case pins the column on all three stores, and a REST test
checks GET /runs/{id} carries neither the route nor the owner.
tool.New returns a definition and a handler, and neither can carry the retry-safe mark to the engine, so it used to drop tool.RetrySafe() without a word. A host that relied on it got runs that stop as uncertain after a crash. New now panics with a message that points at tool.Register or engine.WithToolRetrySafe, the same way it already panics on an input type it can't build a schema for.
cortex.ErrEngineStopping and cortex.ErrRunInterrupted both came back as 500. A stopping replica now answers 503, so your client can retry on another one, and a run that was handed off answers 409: it is waiting to be continued, not failed. Neither response carries the wrapped error.
A default engine's Stop lets the run finish and hands nothing off, its lease ttl is zero, and the defaults test now lists run leasing among the switches that stay off. With WithRunLease set, Stop hands the run off and another replica's continuer job finishes it with ContinueRun, as the docs show. A reclaimed run is authorized as the alice who started it, never as the job continuing it, for the replayed call and the one after. When the codec reports alice revoked, the reclaim fails the run with ErrPrincipalRevoked before any model call or tool. A ResolveCheckpoint whose continuation is refused before its claim keeps the decision, leaves the run ready, and says so in its error.
MIGRATION.md, the runs and configuration pages, the Forge guide, the README and the CHANGELOG no longer say runs are leased by default. They say leasing is off unless you pass WithRunLease, what turning it on needs (a job that continues RunReady and RunReclaimable runs with ContinueRun, since nothing in cortex does), and show the plugin that queues that job. The retry-safety advice now applies to hosts that turn leasing on. The upgrade guide gains the Weave tenant change, the ResolveCheckpoint error, clock sync between replicas, and keeping v1.9.0's experimental reply recovery off during a leased rollout. The runs page says task-claim runs that reach ready belong to coordination, and the MCP page says to filter tools whose readOnlyHint you don't trust. The CHANGELOG lists the reply token, worker id, tool.New and REST status fixes, the new migration, and the known limits, and the [1.13.0] note now says v1.7.0, which matches git.
On a default engine, an approval that arrived while Stop drained, or under a full run_concurrency cap with a caller deadline, recorded the decision and left the run ready, then failed to continue it. v1.9.0 refused both before writing anything. A decision that continues a run (an approval, or a rejection that goes back to the model) is now admitted first, and the admission is held through the continuation. A refusal says nothing was recorded, and the checkpoint stays pending. The continuation runs on that same ticket rather than a nested one, so with leasing on it still hands off when Stop drains. A rejection that fails the run, and a checkpoint the host wrote itself, only record and need no slot, as before.
The engine refuses every request at admission while it stops, before it writes anything, ResolveCheckpoint included. The 503 body now says so, so your client knows a retry on another replica is safe.
Hooks run on the engine's own goroutines, Stop's hand-off among them, so the continuer the docs show enqueues with a select and a default. The test's copy now does the same.
The continuer example in MIGRATION.md and the runs page enqueues without blocking and calls ContinueRun from a worker goroutine. Hooks run on the sweeper, on Stop's hand-off and in the request that decided, so a blocking send could hold Stop up until its deadline. A dropped notice comes back with the sweeper's next renotify pass. MIGRATION.md, the suspension page and the CHANGELOG say an approval refused while the engine drains, or for want of a run slot, records nothing. The custom store guide says to persist Run.ReplyRoute at creation and leave it alone on update, as with Principal.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
gosec G404 fails CI on math/rand, and the standalone gosec run ignores nolint comments. The jitter only spreads retries, but crypto/rand costs nothing here and keeps the scan clean.
On main the engine and SQLite packages already took about 25 minutes each under race coverage in CI. This branch's tests push the engine package past 30, so the coverage run timed out with no failing test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the v1.10.0 release. After it merges, you cut the tags with:
That tags the root module and api, sentinel, extension, integrations/fabriq, integrations/sandbox, optimization and the new mcpbridge at
<dir>/v1.10.0. Everything is pinned to released tags: forge v1.13.2, grove v1.10.1, nexus v1.9.1, shield v1.9.1 and weave v1.9.2.What you get
Hosts can now drive runs themselves. A paused run is decided with
DecideCheckpointor answered withProvideResults, it moves toready, and your host continues it withContinueRunfrom whichever replica its job runner picks. Turn on leasing withWithRunLeaseand a run whose worker dies becomesreclaimable, andContinueRunpicks it up from its last saved step. A tool call that was already handed to a tool that isn't retry-safe is never sent twice: the run endsincompletewithErrEffectUncertain. Leasing is opt-in, so a host that upgrades and wires nothing keeps the v1.9.0 behaviour, Stop included.Resumed runs carry the principal that started them (
WithPrincipalCodec), every suspension has its own deadline, andRunSuspendedfires on every pause.Tools can be typed now.
tool.Newbuilds the JSON Schema from a Go struct, andRetrySafemarks the ones that are safe to run twice. The newmcpbridgemodule registers an MCP server's tools as cortex tools, with read-only ones marked retry-safe.The adapters do the real work now:
The REST module takes an
Authorizer, which guards every route and decides who approved a checkpoint, and it serves the tool catalog. It also writes each response body once. Before this, most routes wrote two JSON objects into one body, which every normal JSON client rejects.Upgrading
Read
MIGRATION.md, "Upgrading from v1.9.0". The short version: migrations 20261101000001 to 20261101000004 are additive, the nexus adapter refuses calls without a tenant, shield now gates publication even when you pass your ownWithSafety, and weave uses the first scope level as its tenant. If you upgrade straight from v1.8.0, its section covers pauses written at schema 1.The CHANGELOG's
[1.10.0]section lists the known limits, so they don't surprise you later. A turn whose text publication redacted loses its thinking blocks on replay, for one.How it was checked
Every task went through its own review, and the whole branch had a final review before this PR. These ran on the branch:
make fmt,make lint(0 issues in all 11 modules) and goimports-race(about 20 minutes)-v: 2566 passed, none skipped-race, plusmake test-integrationA few timing tests flaked once each while the machine was heavily loaded and passed on every rerun, so if CI trips on
A2ARecipientRecoveryReclaimsReplyin Mongo, rerun it before you dig in.Shield's segment checking was attacked directly before it shipped. Private keys streamed a few bytes at a time (RSA PKCS#1, PKCS#8 and EC, 30 fresh keys each),
password: valueandapi_key=valuelines, and forged quote wrappers in retrieved text all came out the same as a whole-message check.