Skip to content

fix(runtime): supersede an older queue waiter when a newer event takes the free lock - #122

Merged
ThomasK33 merged 3 commits into
mainfrom
fix/queue-supersede-parked-waiter
Oct 29, 2026
Merged

ThomasK33 merged 3 commits into
mainfrom
fix/queue-supersede-parked-waiter

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Why

Under ConcurrencyQueue, the bot must never answer an older message after a newer one (#121). A newer event that found the Thread Lock free took it in its prelude and ran. The older event that waited for the lock still ran afterwards, so the bot answered the old message last.

What changed

  • routedWork: a queue event that took the free lock calls the new supersedeOlderWaiter. Under queueMu, it supersedes and deletes a pending waiter that was admitted earlier, and keeps a newer one. The waiter ends as ignored, and the log records superseded_by.
  • registerPending and supersedeOlderWaiter share displaceLocked, which writes the skip record and closes the waiter's displacement channel.
  • The ConcurrencyQueue GoDoc states the rule.

Scope

Tradeoffs

  • A check for a waiter before AcquireLock was rejected. A TLA+ model of this coordination found a path where an older delivery registers after that check and still runs after the lock taker.
  • The supersede runs after dedupe and routing, not at AcquireLock. So a duplicate or unrouted event that takes the lock never knocks out the waiter.

Blast Radius

Only the queue strategy changes, and only when an event finds the lock free while an older event waits. That older event is now skipped as superseded, which is what the queue contract promises.

Verification

  • TestQueueNewerDeliveryThatTakesTheFreeLockSupersedesParkedWaiter lands first and fails with handled = [A C B] in deferred and sync dispatch. With the fix it passes 5 of 5 runs under -race.
  • Two guard tests fail when the supersede moves to AcquireLock (handled = [A]) or loses its sequence check (handled = [A C]).
  • go test -race ./... and go vet in every workspace module pass. The TLA+ model of the fixed code (not in this repo yet) passes every invariant with up to 4 deliveries.

Closes #121


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

A newer delivery that finds the Thread Lock free takes it in its
prelude. The older delivery that was parked for the lock still runs
afterwards, so the handler sees A, C, B. The test holds B's first lock
poll with a gated State, lets C run, then releases B. It fails under
both dispatch modes.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…s the free lock

Under ConcurrencyQueue, a routed event that takes the free Thread Lock
in its prelude now supersedes the scope's pending waiter when that
waiter was admitted earlier. The waiter stops at its next poll or at
its re-check after AcquireLock, ends as ignored, and the log records
superseded_by. Before, the waiter still ran after the newer event, so
the bot answered an older message last.

The call sits after acceptEvent, so an event that resolves in its
prelude (a duplicate, an ignored or unrouted event) supersedes no one.
A waiter admitted after the lock holder stays and runs next. Two guard
tests pin both rules. registerPending and the new path share
displaceLocked, which writes the skip record.

Closes #121

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T12:47:11.283998Z 8d4b4d9 Manual request
🔒 Security Review ✅ Completed 2026-10-29T12:53:05.732035Z 8d4b4d9 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: f4568acd90

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4568acd90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread queue_edge_test.go Outdated
A check that fails before the test opens a gate left B's lock poll or
A's handler blocked, so the unbounded Shutdown in cleanup waited forever.
The test hit the binary timeout and its own failure message never
printed. A 5s shutdown context lets the failure report.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 8d4b4d9eeb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8d4b4d9eeb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33

Copy link
Copy Markdown
Member Author

Readiness record

Decision: ready to merge, with one tracked follow-up (#123). Not merged: the owner merges.

  • Final commit: 8d4b4d9.
  • CI on that commit: test (pinned) and test (minimum) pass.
  • Local checks on that commit (Go 1.27.1): gofmt clean, go vet ./... passes in all 7 workspace modules, go test -race -count=1 ./... passes, and the three new queue tests pass 5 of 5 runs under -race.
  • Reviews: 4 rounds (I count each completed normal or security review as one).
    1. Normal review on f4568ac: one P2. Cleanup used an unbounded Shutdown, so a check that failed before its gate opened hung the test and hid the failure. A probe confirmed it (45 s timeout, no failure message). 8d4b4d9 bounds the cleanup at 5 s, and the same probe then fails in 8 s with its message. Thread resolved.
    2. Security review on f4568ac: no issues.
    3. Normal review on 8d4b4d9: no major issues.
    4. Security review on 8d4b4d9: no issues.
  • Threads: 1 in total, 0 unresolved (re-checked 6 minutes after the last verdict).
  • Independent advisor (fresh context, given the goal, diff, findings and test results): recommends ready. It reran the suite and the new tests (10 of 10 under -race), and it found no blocker.
  • Follow-up: the advisor found a pre-existing prelude-stall gap that main also has. A conflicted queue delivery that stalls before it registers as the waiter can still run after a newer lock taker. It is tracked in bug(runtime): ConcurrencyQueue runs an older delivery that stalled in its prelude after a newer lock taker #123 (owner: maintainer triage; trigger: before queue is called production-grade, or with the next queue coordination change).
  • Why I stopped: every gate holds on the final commit, and the remaining finding is outside this PR's scope and tracked.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(runtime): ConcurrencyQueue runs an older queued event after a newer one that took the free lock

1 participant