Skip to content

feat(loopback): hold an address with a kernel-released claim, and bound the wait with a holder listing - #44

Merged
Ethan-Arrowood merged 6 commits into
mainfrom
feat/loopback-pool-exhaustion-diagnostic
Nov 16, 2026
Merged

Ethan-Arrowood merged 6 commits into
mainfrom
feat/loopback-pool-exhaustion-diagnostic

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 29, 2026 •

Copy link
Copy Markdown
Member
A girl and a boy pin a board naming which boat holds each dock slip, with a clock reading wait is over, while a man looks on at Lake Powell.

⊙ Problem

When every loopback address was held, allocation looped forever and printed nothing. Worse, the pool leaked addresses for good and gave one address to two runs. It judged liveness by a stored PID, and kept its file under TMPDIR. Both break on machines that run sandboxed or containerized test runners.

Measured on a Linux dev box running sandboxed agents and Docker bench runs, 2026-10-29
  • A stored PID means nothing outside its PID namespace. A runner in a sandbox or container records its own namespace's PID. On the host, one pool held 4 of 32 slots under pid 10, a kernel worker thread up 19 days; another held pid 23, a kernel idle_inject thread. Those slots never came back. The reverse also happens: inside a namespace, a live host runner's PID returns ESRCH, so its address was handed out twice. EPERM (alive, another user) also counted as dead.
  • One pool per TMPDIR. Agents set their own TMPDIR, so the box had 1,083 separate pool files, each handing out 127.0.0.2 first. Concurrent runs collided with EADDRINUSE.
  • Dead holders were freed only when the pool was full, and many runs set the pool count to 1–8.

🤔 Does the root fix belong in this PR?

As specified, yes: the owner asked for it here, on top of the bounded wait. A bigger pool only delays all three failures. No version edit: the feat commit makes the next release 1.1.0, and consumer pins move after it ships.

💡 Solution

⚖️ Alternatives

  • Raise the pool count — delays all three failures; a slot held by a namespace-local PID still never frees.
  • Fix the PID check (ESRCH only, compare start times) — a PID from another namespace names a different process here, and no check on the number can tell.
  • A per-slot OS file lock outside TMPDIR — shared only by processes that see one filesystem; a sandbox with a private /tmp still collides. Node has no flock.

🤔 Should a conflict that a listener on all interfaces explains be ignored?

No. A listener on all interfaces plus a leftover node on the address give the same bind results. Ignoring it could reissue a live node's address. On Linux it throws ForeignListenerError at once instead. A local MQTT broker on 1883 now trips it.

🤔 Is the mixed-version window acceptable without a major bump?

Runs on 1.0.5 or earlier don't see claims, and this version doesn't read their file. Until a machine's runs all upgrade, two can get one address before either node binds. The README says to let old runs finish first. A transitional dual write would bring PIDs back.

🔧 Changes

✅ Verification

  • npm run check is clean. npm test on Linux: 120 tests, 118 pass, 2 skipped, 0 fail. The PID-namespace case skipped locally because AppArmor blocks unshare --user; Linux CI requires it.
  • Real Harper 5.0.10 on Linux: two concurrent startHarper nodes got distinct addresses and each answered as itself. Teardown, then acquire, reused the address. A SIGKILLed runner's surviving node made the next acquire skip its address; killing the node freed it.
  • Acquire plus release, 200 runs: 3.3–4.1 ms before, 0.37 ms after; with 31 of 32 held, 2.5 → 1.5 ms.
Test changes

🤖 Generated by Claude Sonnet 5.5 and Claude Opus 5.5 via Claude Code; posted via @kriszyp.

Related PRs: #33 overlaps (merged; its lock fencing is removed here), #40 overlaps (merged; its foreign-listener refusal now also covers Linux), #42 overlaps (open; it wraps the allocation this PR rewrites), 2 others independent
Complexity: complicated

Origin — the dispatch brief this PR was written from

In @harperfast/integration-testing, make loopback-address-pool exhaustion fail fast with a diagnostic instead of looping silently forever, and prepare a version bump so the already-merged orphan-reaper can ship.

Acceptance

A test covering the exhaustion path: with every loopback address held, acquisition terminates with an error whose message names the pool range and the holding pids, rather than hanging. Existing pool tests still pass. package.json version bumped (minor) with a changelog entry naming PR #30's reaper as the shipped change. No npm publish performed.

Dispatch: task integration-testing-loopback-exhaustion-diagnostic · queued by finding-triage/wave-15 · ran by claude/sonnet/xhigh · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-composer,muse; rounds=6; full=2 @ b5377ed

Review-Attention: study ~9m (decisions: claim-port-source-of-truth, mixed-version-no-interlock, linux-wildcard-refusal-widened, replication-port-unprobed, bounded-wait-default, identity-protocol-on-claim, do-less-alternative) @ b5377ed

kriszyp and others added 3 commits October 29, 2026 07:20
…waiting forever

getNextAvailableLoopbackAddress looped silently when every address was held,
so a wedged run looked like a hang. It now waits up to
HARPER_INTEGRATION_TEST_LOOPBACK_POOL_TIMEOUT_MS (default 10 minutes, past the
unusable-pool quarantine) and throws LoopbackPoolExhaustedError naming the pool
range and the pids holding each address, with a warning at 30 seconds.

Refs #13

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic
…g or rejecting

The pass that freed dead holders' slots slept a second before looking again,
and past the deadline it threw instead of claiming the slot it had just freed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic
Search again after dropping dead holders, under the same lock, rather than
carrying a flag to the next pass. Also tighten the exhaustion tests and trim
comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a timeout mechanism and a new LoopbackPoolExhaustedError for the loopback address pool allocation in getNextAvailableLoopbackAddress. Instead of waiting indefinitely when the pool is full, the allocator will now wait up to a configurable timeout (defaulting to 10 minutes), log a warning after 30 seconds detailing the active PIDs holding the addresses, and eventually throw an error with detailed pool state information if the timeout is reached. The changes also include updated documentation in README.md and CONTRIBUTING.md, as well as a new test suite in test/loopbackAddressPoolExhaustion.test.ts to verify the exhaustion, reclamation, and timeout behaviors. No review comments were provided, and the implementation appears solid with no additional feedback.

@kriszyp
kriszyp marked this pull request as ready for review October 29, 2026 13:57
kriszyp and others added 3 commits October 29, 2026 09:04
… storing a PID

The pool recorded each holder's PID in a JSON file under os.tmpdir(). A runner
in a PID namespace stores a PID that names an unrelated process on the host, so
its slots never came back, and a host runner's PID looks dead from inside a
namespace, so its address was handed out twice. Runs with different TMPDIRs kept
separate pool files and handed out the same addresses.

A process now holds an address by listening on <address>:<claim port> (default
9924, HARPER_INTEGRATION_TEST_LOOPBACK_CLAIM_PORT). The kernel allows one such
listener per network namespace and closes it when its process exits, so a dead
holder's address is free at once, whatever its PID namespace or TMPDIR. The pool
file, its lock, the quarantine and the PID sweep are gone, and with them #44's
deferred lock-wait finding.

Allocation also skips an address whose HTTPS, MQTT or MQTTS port is still bound
on it, besides the operations and HTTP ports. On Linux, a conflict that a bind
of an address outside the pool shares comes from a listener on all interfaces:
that now throws ForeignListenerError at once instead of waiting out the timeout.

The exhaustion message names each holder from its claim's reply: pid, PID
namespace and host name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-44-pool-liveness-root-fix
…tions

- Each pool address gets its own control address (127.255.0.x), which only
  that address's claim holder binds, so two allocations probing different
  addresses cannot mistake each other's probe for a listener on all
  interfaces.
- A claim reports whether its allocation is still checking the address. The
  exhaustion message no longer names such a process as a holder, and lists
  the Harper port that made it give the address up.
- A hostname lookup failure no longer escapes the claim's connection handler,
  and the identity size limit also applies to a reply that ends in a newline.
- Tests release during a wait only after the allocation reports waiting, clean
  up on every exit, and pick their ports while holding them all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-44-pool-liveness-root-fix
…im port

Waiters hold no claim when they diagnose a held pool, so two that diagnose at
once could bind the same control address and one would wrongly report the
claim port as bound on all interfaces. The test helper also marks both of a
holder's early-exit rejections handled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: integration-testing-44-pool-liveness-root-fix
@kriszyp
kriszyp marked this pull request as draft October 29, 2026 15:48
@kriszyp kriszyp changed the title feat(loopback): reject an exhausted pool with its holders instead of waiting forever fix(loopback): hold an address with a kernel-released claim, and bound the wait with a holder listing Oct 29, 2026
@kriszyp kriszyp changed the title fix(loopback): hold an address with a kernel-released claim, and bound the wait with a holder listing feat(loopback): hold an address with a kernel-released claim, and bound the wait with a holder listing Oct 29, 2026
@kriszyp
kriszyp marked this pull request as ready for review October 29, 2026 16:52
@Ethan-Arrowood
Ethan-Arrowood merged commit b707ddc into main Nov 16, 2026
16 checks passed
@Ethan-Arrowood
Ethan-Arrowood deleted the feat/loopback-pool-exhaustion-diagnostic branch November 16, 2026 23:08
github-actions Bot pushed a commit that referenced this pull request Nov 16, 2026
## [1.1.0](v1.0.5...v1.1.0) (2026-11-16)

### Features

* **loopback:** hold an address with a kernel-released claim, and bound the wait with a holder listing ([#44](#44)) ([b707ddc](b707ddc)), closes [#13](#13)

### Documentation

* **readme:** Windows needs no loopback setup ([b7a600e](b7a600e))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 1.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants