Skip to content

Queue a webhook's job before remembering its delivery, so a full disk cannot lose it (RA-021) - #90

Merged
JOsacky merged 1 commit into
mainfrom
reliability/ra-021-enqueue-before-dedupe
Nov 17, 2026
Merged

JOsacky merged 1 commit into
mainfrom
reliability/ra-021-enqueue-before-dedupe

Conversation

@JOsacky

@JOsacky JOsacky commented Nov 17, 2026

Copy link
Copy Markdown
Member

Problem (RA-021)

On 2026-11-17 the operator machine's disk filled (0.1 GiB free at 01:27Z). A hosted geocode-sentry-alert delivery at 01:35:51Z created job 20261117T013551Z-d9mg86, then the request failed with unhandled request error (ENOSPC) and answered 500. The cloud retried at 01:38:35Z and got duplicate delivery ignored (200) pointing at that job, which is still queued and never ran. Only the cloud reconciliation noticed the loss.

Cause: createJob in src/server.ts called store.rememberDelivery before queue.enqueue. rememberDelivery updates the in-memory index and then writes the file; the write threw, so the job was never queued while the retry still matched it in memory.

Change

  • Queue the job first, then remember its delivery and ingress key.
  • A failure to save the de-dupe index is logged as a warning (could not save the delivery de-dupe index; the job is queued) instead of failing a request whose job is already queued. The in-memory entry still makes a retry a duplicate of the queued job.

Test

  • New server.test.ts case: the index save throws ENOSPC once; the webhook gets 202, the job succeeds, and the retry is a duplicate of it.
  • Control: with the old order the same test fails with expected 500 to be 202.
  • Locally on Node 26.13.0: npm test 534 tests in 63 files pass; npm run typecheck passes.

Not in this PR

Scheduler.fire (src/scheduler.ts) has the same order (rememberDelivery before queue.enqueue), so a schedule slot can be lost the same way during ENOSPC. It's left for a separate change with its own test.

No release is included; shipping needs a version-bump PR.

🤖 Generated with Claude Code

createJob wrote the job, saved the delivery de-dupe index and only then
queued the job. On a full disk the index write threw ENOSPC after the
in-memory index already held the mapping: the request answered 500, the
job stayed "queued" without running, and the sender's retry was answered
as a duplicate of it. A hosted geocode-sentry-alert delivery was lost
this way on 2026-11-17 01:35Z.

Queue first, then remember; a failed index save is logged as a warning
instead of failing a request whose job is already queued.

Test: the de-dupe index save throws ENOSPC once; the webhook is accepted
(202), the job runs, and the retry is a duplicate of it. Fails on the old
order (500), passes with this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JOsacky
JOsacky merged commit d8afb32 into main Nov 17, 2026
7 checks passed
@JOsacky
JOsacky deleted the reliability/ra-021-enqueue-before-dedupe branch November 17, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant