[shashankram]

Description

Canceled or timed-out messages can leave an actor RUNNING or RESUMING without an active turn, holding worker capacity. The existing Session lifecycle worker now recovers and suspends these actors while keeping the Session READY for the next message.

• Persist recovery in runtime_lifecycle_job, with a runtime, operation kind, generation, phase, availability, and owner. Revocation enqueues work atomically; expiry scans discover abandoned dispatches after controller loss. Successful message delivery adds no job writes.
• Complete interrupted resumes before suspending and recording the durable snapshot, without creating task history. Persisted ownership and suspension intent support restarts and multiple replicas; completion requires the same dispatch and owner.
• Let newer messages supersede cleanup immediately. Workers stop further runtime calls and retries when superseded, and scans prune obsolete jobs without touching newer work.
• Yield workers between retries, with backoff capped at 15 seconds. Previously suspended actors without recorded cleanup intent use a persistent observation schedule: one minute initially, then five minutes. Lifecycle counters and structured logs expose progress.
• Support DB-first upgrades and controller-first rollbacks. Older controllers can clear or replace dispatches without maintaining job metadata; migration rollback preserves Session data.

A resume or suspend already issued to Substrate may finish after a newer message arrives. The message follows normal runtime retry behavior; stale cleanup cannot overwrite its dispatch or snapshot.

Testing

• PostgreSQL coverage for atomic completion, competing owners, restart takeover, persistent schedules, older-controller writes, and populated upgrade/rollback/re-upgrade.
• Workflow coverage for supersession, transient failures, worker fairness, crash recovery, and lifecycle telemetry.
• E2Es for cancellation in RUNNING and RESUMING, verifying the same actor is suspended with a fresh snapshot, released capacity, empty task history, and a successful subsequent message. Updated interrupted-resume deletion coverage.

Change Type

/kind fix

Changelog

Automatically recover and suspend actors left active by canceled or timed-out messages.

[iplay88keys]

🤖 AI-generated review, guided and reviewed by me before posting.

I focused on the migration and on rollbacks within 1.0. I found one rollback issue and have one restart question. Both also apply to #3649. The store and migration tests pass locally at d30cb0b.

[iplay88keys]

LGTM from a migration and rollback perspective. The separate cleanup table is a bit heavy, but the old controller ignores it, so it works here.