Repository navigation
feat(loopback): hold an address with a kernel-released claim, and bound the wait with a holder listing - #44
Conversation
…waiting forever getNextAvailableLoopbackAddress looped silently when every address was held, so a wedged run looked like a hang. It now waits up to HARPER_INTEGRATION_TEST_LOOPBACK_POOL_TIMEOUT_MS (default 10 minutes, past the unusable-pool quarantine) and throws LoopbackPoolExhaustedError naming the pool range and the pids holding each address, with a warning at 30 seconds. Refs #13 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic
…g or rejecting The pass that freed dead holders' slots slept a second before looking again, and past the deadline it threw instead of claiming the slot it had just freed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic
Search again after dropping dead holders, under the same lock, rather than carrying a flag to the next pass. Also tighten the exhaustion tests and trim comments. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-loopback-exhaustion-diagnostic
There was a problem hiding this comment.
Code Review
This pull request introduces a timeout mechanism and a new LoopbackPoolExhaustedError for the loopback address pool allocation in getNextAvailableLoopbackAddress. Instead of waiting indefinitely when the pool is full, the allocator will now wait up to a configurable timeout (defaulting to 10 minutes), log a warning after 30 seconds detailing the active PIDs holding the addresses, and eventually throw an error with detailed pool state information if the timeout is reached. The changes also include updated documentation in README.md and CONTRIBUTING.md, as well as a new test suite in test/loopbackAddressPoolExhaustion.test.ts to verify the exhaustion, reclamation, and timeout behaviors. No review comments were provided, and the implementation appears solid with no additional feedback.
… storing a PID The pool recorded each holder's PID in a JSON file under os.tmpdir(). A runner in a PID namespace stores a PID that names an unrelated process on the host, so its slots never came back, and a host runner's PID looks dead from inside a namespace, so its address was handed out twice. Runs with different TMPDIRs kept separate pool files and handed out the same addresses. A process now holds an address by listening on <address>:<claim port> (default 9924, HARPER_INTEGRATION_TEST_LOOPBACK_CLAIM_PORT). The kernel allows one such listener per network namespace and closes it when its process exits, so a dead holder's address is free at once, whatever its PID namespace or TMPDIR. The pool file, its lock, the quarantine and the PID sweep are gone, and with them #44's deferred lock-wait finding. Allocation also skips an address whose HTTPS, MQTT or MQTTS port is still bound on it, besides the operations and HTTP ports. On Linux, a conflict that a bind of an address outside the pool shares comes from a listener on all interfaces: that now throws ForeignListenerError at once instead of waiting out the timeout. The exhaustion message names each holder from its claim's reply: pid, PID namespace and host name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-44-pool-liveness-root-fix
…tions - Each pool address gets its own control address (127.255.0.x), which only that address's claim holder binds, so two allocations probing different addresses cannot mistake each other's probe for a listener on all interfaces. - A claim reports whether its allocation is still checking the address. The exhaustion message no longer names such a process as a holder, and lists the Harper port that made it give the address up. - A hostname lookup failure no longer escapes the claim's connection handler, and the identity size limit also applies to a reply that ends in a newline. - Tests release during a wait only after the allocation reports waiting, clean up on every exit, and pick their ports while holding them all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-44-pool-liveness-root-fix
…im port Waiters hold no claim when they diagnose a held pool, so two that diagnose at once could bind the same control address and one would wrongly report the claim port as bound on all interfaces. The test helper also marks both of a holder's early-exit rejections handled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: integration-testing-44-pool-liveness-root-fix
|
🎉 This PR is included in version 1.1.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
⊙ Problem
When every loopback address was held, allocation looped forever and printed nothing. Worse, the pool leaked addresses for good and gave one address to two runs. It judged liveness by a stored PID, and kept its file under
TMPDIR. Both break on machines that run sandboxed or containerized test runners.Measured on a Linux dev box running sandboxed agents and Docker bench runs, 2026-10-29
idle_injectthread. Those slots never came back. The reverse also happens: inside a namespace, a live host runner's PID returns ESRCH, so its address was handed out twice.EPERM(alive, another user) also counted as dead.TMPDIR. Agents set their ownTMPDIR, so the box had 1,083 separate pool files, each handing out127.0.0.2first. Concurrent runs collided withEADDRINUSE.🤔 Does the root fix belong in this PR?
As specified, yes: the owner asked for it here, on top of the bounded wait. A bigger pool only delays all three failures. No version edit: the
featcommit makes the next release1.1.0, and consumer pins move after it ships.💡 Solution
9924). The kernel allows one listener per address and closes it when the process exits, whatever its PID namespace orTMPDIR. The pool file and its lock are gone.LoopbackPoolExhaustedError, naming each holder: its pid, plus its PID namespace when that differs from the caller's.⚖️ Alternatives
TMPDIR— shared only by processes that see one filesystem; a sandbox with a private/tmpstill collides. Node has noflock.🤔 Should a conflict that a listener on all interfaces explains be ignored?
No. A listener on all interfaces plus a leftover node on the address give the same bind results. Ignoring it could reissue a live node's address. On Linux it throws
ForeignListenerErrorat once instead. A local MQTT broker on 1883 now trips it.🤔 Is the mixed-version window acceptable without a major bump?
Runs on 1.0.5 or earlier don't see claims, and this version doesn't read their file. Until a machine's runs all upgrade, two can get one address before either node binds. The README says to let old runs finish first. A transitional dual write would bring PIDs back.
🔧 Changes
src/loopbackAddressPool.ts:exclusivebind (so cluster workers cannot share it), identity reply, and a process-wide map shared by every copy of the module.src/index.ts: exportsLoopbackPoolExhaustedError.src/harperLifecycle.ts: teardown's comment describes the claim.README.md: the claim model and the upgrade note, the Linux foreign-listener scope, the error's listing.CONTRIBUTING.md: one source of truth for holders and the test notes..github/workflows/verify-pr.yaml: lifts Ubuntu's user-namespace block so the PID-namespace test runs, and fails it there instead of skipping.✅ Verification
npm run checkis clean.npm teston Linux: 120 tests, 118 pass, 2 skipped, 0 fail. The PID-namespace case skipped locally because AppArmor blocksunshare --user; Linux CI requires it.startHarpernodes got distinct addresses and each answered as itself. Teardown, then acquire, reused the address. A SIGKILLed runner's surviving node made the next acquire skip its address; killing the node freed it.Test changes
test/loopbackAddressPoolExhaustion.test.ts, with holders as child processes (test/loopbackHolder.ts,test/fixtures/loopbackHolder.ts):TMPDIRs, released together, get the one address exactly once;test/loopbackAddressPool.test.ts:test/loopbackAddressPoolProbeFailure.test.ts: checks the claim port instead of the file.test/loopback-lock-fencing.test.ts: deleted with the lock it tested.🤖 Generated by Claude Sonnet 5.5 and Claude Opus 5.5 via Claude Code; posted via @kriszyp.
Related PRs: #33 overlaps (merged; its lock fencing is removed here), #40 overlaps (merged; its foreign-listener refusal now also covers Linux), #42 overlaps (open; it wraps the allocation this PR rewrites), 2 others independent
Complexity: complicated
Origin — the dispatch brief this PR was written from
In @harperfast/integration-testing, make loopback-address-pool exhaustion fail fast with a diagnostic instead of looping silently forever, and prepare a version bump so the already-merged orphan-reaper can ship.
Acceptance
A test covering the exhaustion path: with every loopback address held, acquisition terminates with an error whose message names the pool range and the holding pids, rather than hanging. Existing pool tests still pass. package.json version bumped (minor) with a changelog entry naming PR #30's reaper as the shipped change. No npm publish performed.
Dispatch: task
integration-testing-loopback-exhaustion-diagnostic· queued by finding-triage/wave-15 · ran by claude/sonnet/xhigh · worker kzyp-xps-1Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-composer,muse; rounds=6; full=2 @ b5377ed
Review-Attention: study ~9m (decisions: claim-port-source-of-truth, mixed-version-no-interlock, linux-wildcard-refusal-widened, replication-port-unprobed, bounded-wait-default, identity-protocol-on-claim, do-less-alternative) @ b5377ed