Skip to content

fix(runtime): release what the prelude took when State or the logger panics - #142

Merged
ThomasK33 merged 10 commits into
mainfrom
fix/prelude-panic-releases
Oct 30, 2026
Merged

ThomasK33 merged 10 commits into
mainfrom
fix/prelude-panic-releases

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Why

Every open follow-up issue in coder/chat gets one small, reviewed PR, and this one closes #139. When State or the app's slog handler panics during the prelude, net/http recovers the panic, but three things the delivery took stay taken. A redelivery of the event waits on the open acceptance entry until its own request ends. The next delivery for the thread waits for ThreadLockTTL. A burst member whose join log panicked never gets a runner, so later members never run and Shutdown times out.

What changed

  • The prelude defers abandonEventAcceptance and its own releaseOnResolve. The first does nothing once the prelude finished the entry. The second does nothing once the lease is released or handed to the tail. So every release still runs once, and nothing recovers the panic.
  • joinBurstBatch starts the burst runner in a defer. The runner still starts after the join log, and it now also starts when the log panics.
  • takeQueueTurn deletes the superseded waiter before it logs. Without that, the released lease lets the old waiter take the lock behind the newer turn, and the commitPending assert ends the process under DispatchDeferred. ADR 0015 gains the rule as invariant 4.

Scope

  • Covers State and logger panics in the prelude and in the burst join, under both dispatch modes.
  • A logger panic in the supersede log still loses events, because MarkEvent already ran and the platform's retry is deduped. Under debounce the newest event is lost. In takeQueueTurn both the newer event and the waiter it replaced are lost. That is event loss, not a leak, and is tracked in runtime: a panicking slog handler on the supersede log loses queued events #143.
  • The dispatch span of a panicked delivery stays open, as on main.

Tradeoffs

Blast Radius

Only the prelude, the burst join, and the queue supersede path change. On every normal path the new defers find nothing left to release.

Verification

  • Each fix lands after its failing test. Before the fixes, the redelivery and the next delivery for the thread time out after 2 s, and the burst member never runs.
  • Wrong fixes fail the new tests. A silent recover returns 200 or 500 instead of aborting the request. A second release reports lock_release_failed or panics with close of closed channel. A release after the handoff fails the existing lock tests, or admits a second burst member while the first holds the only slot. The lease release without the takeQueueTurn change trips the commitPending assert.
  • The new tests and the Queue, Deferred, Burst, Admission, and Concurrent tests pass 20 of 20 runs under -race. go vet and go test pass in every workspace module, and go test -race passes for the root package.

Closes #139


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

The three tests serve the runtime behind a real httptest.Server, so
net/http recovers each panic. On main each one fails: a redelivery of the
panicked event waits until its client gives up, the next delivery for the
thread never gets the Thread Lock, and a burst member whose join log
panicked never runs.
The prelude defers finishEventAcceptance, which now returns early once the
entry is done. A panic from State or the logger therefore frees the entry:
duplicates that wait on it fail, and a redelivery runs.
A logger panic in the queue supersede log leaves the Thread Lock with the
prelude, like a MarkEvent panic. The lock test also checks that a delivery
which resolves in the prelude releases its lease once.
The prelude defers releaseOnResolve, which now clears the lease it frees.
Routed work takes the lease out first, so the defer frees only a lease a
panic left behind. takeQueueTurn deletes the superseded waiter before it
logs, so a logger panic there cannot leave that waiter pending behind a
newer turn.
joinBurstBatch now starts the runner it promised in a defer. The runner
still starts after the join log, and a logger panic no longer strands the
member, its admission slot, and every later member of the scope.
Comment review: the deferred finish read as if every delivery ended with
the panic error. abandonEventAcceptance owns the open-entry check, so
finishEventAcceptance stays strict, and handLeaseToTail names the handoff.
The ADR rule names locks, slots, and records instead of every resource.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T08:48:54.436397Z 6a0086c Manual request
🔒 Security Review ✅ Completed 2026-10-30T08:53:16.437411Z 6a0086c Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: e804b8c398

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e804b8c398

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

A failure before a gate opened left a request blocked on it, and
srv.Close waited on that request until the go test timeout.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 6a0086c1cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6a0086c1cb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 merged commit f141502 into main Oct 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime: a panic during the prelude still leaks the acceptance entry, the Thread Lock, and a burst slot

1 participant