Repository navigation
Queue a webhook's job before remembering its delivery, so a full disk cannot lose it (RA-021) - #90
Merged
Conversation
createJob wrote the job, saved the delivery de-dupe index and only then queued the job. On a full disk the index write threw ENOSPC after the in-memory index already held the mapping: the request answered 500, the job stayed "queued" without running, and the sender's retry was answered as a duplicate of it. A hosted geocode-sentry-alert delivery was lost this way on 2026-11-17 01:35Z. Queue first, then remember; a failed index save is logged as a warning instead of failing a request whose job is already queued. Test: the de-dupe index save throws ENOSPC once; the webhook is accepted (202), the job runs, and the retry is a duplicate of it. Fails on the old order (500), passes with this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (RA-021)
On 2026-11-17 the operator machine's disk filled (0.1 GiB free at 01:27Z). A hosted
geocode-sentry-alertdelivery at 01:35:51Z created job20261117T013551Z-d9mg86, then the request failed withunhandled request error(ENOSPC) and answered 500. The cloud retried at 01:38:35Z and gotduplicate delivery ignored(200) pointing at that job, which is stillqueuedand never ran. Only the cloud reconciliation noticed the loss.Cause:
createJobinsrc/server.tscalledstore.rememberDeliverybeforequeue.enqueue.rememberDeliveryupdates the in-memory index and then writes the file; the write threw, so the job was never queued while the retry still matched it in memory.Change
could not save the delivery de-dupe index; the job is queued) instead of failing a request whose job is already queued. The in-memory entry still makes a retry a duplicate of the queued job.Test
server.test.tscase: the index save throws ENOSPC once; the webhook gets 202, the job succeeds, and the retry is a duplicate of it.expected 500 to be 202.npm test534 tests in 63 files pass;npm run typecheckpasses.Not in this PR
Scheduler.fire(src/scheduler.ts) has the same order (rememberDeliverybeforequeue.enqueue), so a schedule slot can be lost the same way during ENOSPC. It's left for a separate change with its own test.No release is included; shipping needs a version-bump PR.
🤖 Generated with Claude Code