Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/verify-pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,15 @@ jobs:
run: npm run check
- name: Build
run: npm run build
# The loopback pool's PID-namespace test runs its holder under `unshare --user`, which Ubuntu's
# AppArmor default blocks for unprivileged users.
- name: Allow unprivileged user namespaces
if: runner.os == 'Linux'
run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
# Tests run the TypeScript sources directly via Node's native type stripping (Node >= 22.18,
# enforced by devEngines). They spawn short-lived node processes and bind loopback ports, so
# no real Harper instance or loopback-pool setup is required.
- name: Run tests
run: npm test
env:
LOOPBACK_POOL_TEST_REQUIRE_PID_NAMESPACE: ${{ runner.os == 'Linux' && '1' || '' }}
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Contributors are encouraged to communicate with maintainers in issues or other c

Source files are located in `src/`. These are built to the `dist/` directory. The published package includes `dist/`, `scripts/`, and the regular npm metadata and documentation files.

The `src/index.ts` is the source for the main export. This is the public re-export of all the various utilities from `src/harperLifecycle.ts`, `targz.ts`, and more. The `src/run.ts` is the source for the `harper-integration-test-run` bin script. The internal `src/harperInstanceRegistry.ts` publishes each running Harper instance to a shared on-disk registry, and `src/harperMonitor.ts` is the singleton monitor process that reads it and reaps instances whose test runner died without cleaning up (see README's *Orphaned Instance Monitor* section). A registry record covers a whole process group, so the monitor — the only half that can see when that group is finished — is also the only writer that removes one; the lifecycle side registers and never deregisters. The pool behind the loopback utilities, `src/loopbackAddressPool.ts`, never writes an unusable pool file back as an empty pool before its quarantine ends (see README's *Loopback Pool Utilities*): every path that rewrites the file returns early when `readPoolFile` reports it quarantined (`null`), so a new write path must do the same. Harper's `restart` operation replaces the process the harness spawned with a detached one the harness never tracked, so `killHarper` identifies the node by `<dataRootDir>/hdb.pid` once the spawned process has exited (covered by the relaunch tests in `test/harperLifecycle.test.ts`); the runner-exit handlers and the monitor still key on the spawned process and do not follow a restart. And the `scripts/setup-loopback.sh` is the source for the `harper-integration-test-setup-loopback` bin script.
The `src/index.ts` is the source for the main export. This is the public re-export of all the various utilities from `src/harperLifecycle.ts`, `targz.ts`, and more. The `src/run.ts` is the source for the `harper-integration-test-run` bin script. The internal `src/harperInstanceRegistry.ts` publishes each running Harper instance to a shared on-disk registry, and `src/harperMonitor.ts` is the singleton monitor process that reads it and reaps instances whose test runner died without cleaning up (see README's *Orphaned Instance Monitor* section). A registry record covers a whole process group, so the monitor — the only half that can see when that group is finished — is also the only writer that removes one; the lifecycle side registers and never deregisters. The pool behind the loopback utilities, `src/loopbackAddressPool.ts`, has one source of truth for who holds an address: a listening socket on that address's claim port, which the kernel keeps unique per network namespace and closes when its process dies (see README's *Loopback Pool Utilities*). Do not add a second record of holders, such as a file of PIDs: a PID means nothing outside its own PID namespace, and a file is shared only by processes that see the same filesystem, which is not the scope in which addresses collide. A claim joins the process-wide map only once allocation succeeds, and every failure after the bind closes it. Harper's `restart` operation replaces the process the harness spawned with a detached one the harness never tracked, so `killHarper` identifies the node by `<dataRootDir>/hdb.pid` once the spawned process has exited (covered by the relaunch tests in `test/harperLifecycle.test.ts`); the runner-exit handlers and the monitor still key on the spawned process and do not follow a restart. And the `scripts/setup-loopback.sh` is the source for the `harper-integration-test-setup-loopback` bin script.

The package is `"type": "module"` — all source files are ESM by default.

Expand All @@ -29,7 +29,7 @@ Tests live in `test/` (separate from `src/`) and run on the built-in Node.js tes

Internal-only helpers may be exported from their modules for testing (e.g. `runHarperCommand` in `harperLifecycle.ts`) but are deliberately **not** re-exported from `src/index.ts`, keeping them out of the public API.

`loopbackAddressPool.ts` reads its pool file location, address range, and canary ports from the environment once, when it is imported, so `test/loopbackAddressPool.test.ts` sets them before a dynamic import: a private pool file in its own temp directory, and `127.0.0.1`, the one address every OS has without aliases. Its foreign-listener cases need an OS that lets an exclusive bind of that address succeed beside a listener on all interfaces, as macOS and Windows do; where the bind conflicts instead (Linux), the conflict canary refuses the address first and those cases skip themselves. That is why CI includes a macOS job alongside Linux and Windows.
`loopbackAddressPool.ts` reads its address range, claim port, and probe ports from the environment once, when it is imported, so `test/loopbackAddressPool.test.ts` sets them before a dynamic import. It uses `127.0.0.1`, the one address every OS has without aliases, and ephemeral ports, so neither a local Harper nor another test file running at the same time affects it. A listener on all interfaces is found by binding on Linux and by connecting on macOS and Windows, so the foreign-listener cases take a different path on each, which is why CI runs all three. `test/loopbackAddressPoolExhaustion.test.ts` is separate because it needs its own timeout, which is read the same way. Its holders are child processes (`test/fixtures/loopbackHolder.ts`), each with its own `TMPDIR`; one runs in its own PID namespace through `unshare`. That case skips itself where unprivileged user namespaces are blocked (Ubuntu's AppArmor default), so the Linux CI job lifts that restriction and sets `LOOPBACK_POOL_TEST_REQUIRE_PID_NAMESPACE` to turn the skip into a failure.

## Releases

Expand Down
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ npm install --save-dev @harperfast/integration-testing

## Framework Agnosticism

The core lifecycle and loopback pool APIs in this package — `startHarper`, `teardownHarper`, `killHarper`, and the loopback utilities — are **not tied to any specific test runner or framework**. They spawn and manage Harper child processes and allocate loopback addresses safely across concurrent processes using a file-based locking mechanism. This means they can be used with any testing framework that supports async setup and teardown: Node.js `node:test`, Vitest, Playwright, or anything else.
The core lifecycle and loopback pool APIs in this package — `startHarper`, `teardownHarper`, `killHarper`, and the loopback utilities — are **not tied to any specific test runner or framework**. They spawn and manage Harper child processes and allocate loopback addresses safely across concurrent processes, whatever their PID namespace or `TMPDIR`. This means they can be used with any testing framework that supports async setup and teardown: Node.js `node:test`, Vitest, Playwright, or anything else.

The key responsibility these APIs place on you is **thinking critically about Harper instance scope**. Starting a Harper instance has real cost — it spawns a process, allocates a loopback address from the pool, and creates a temporary install directory. How you scope instances depends on your framework and what you're testing. A single instance shared across many assertions in one logical test block may be perfectly appropriate; spinning up a fresh instance for every individual assertion would be wasteful. What matters is that concurrently executing scopes each have their own instance and loopback address — the pool handles cross-process allocation safely, but your setup and teardown hooks determine the boundaries.

Expand Down Expand Up @@ -84,7 +84,7 @@ Each node binds its fixed ports on its own loopback address, so nodes on differe

`getNextAvailableLoopbackAddress`, and therefore `startHarper`, refuses an address when another process already accepts connections on its operations or HTTP port, throwing a `ForeignListenerError` that names the port. Stop the other process or bind it to a specific address (`lsof -nP -iTCP:9926 -sTCP:LISTEN` names it). To run anyway, set `HARPER_INTEGRATION_TEST_ALLOW_FOREIGN_LISTENERS=1`, which turns the error into a warning.

On Linux, binding a pool address beside such a listener already fails, so the pool reports each address as still in use by another node instead.
On Linux, such a listener also stops anything else from binding that port on a single address, Harper included, so there the same error covers Harper's HTTPS, MQTT and MQTTS ports too. With the variable set on Linux, the address is handed out even though a Harper node bound to that address under the other listener cannot be ruled out.

## API

Expand Down Expand Up @@ -287,7 +287,11 @@ try {

### `ForeignListenerError`

Thrown by `getNextAvailableLoopbackAddress`, and so by `startHarper`, when another process already accepts connections on the operations or HTTP port of a newly allocated address (see [Other services listening on Harper's ports](#other-services-listening-on-harpers-ports)). It extends `Error` with `name` set to `'ForeignListenerError'`, plus `loopbackAddress` and `port` properties naming what was reached. The address is returned to the pool before it is thrown. Set `HARPER_INTEGRATION_TEST_ALLOW_FOREIGN_LISTENERS=1` (or `true`) to get a warning instead.
Thrown by `getNextAvailableLoopbackAddress`, and so by `startHarper`, when another process already accepts connections on the operations or HTTP port of a newly allocated address, or on Linux on any of its Harper ports through a listener on all interfaces (see [Other services listening on Harper's ports](#other-services-listening-on-harpers-ports)). It extends `Error` with `name` set to `'ForeignListenerError'`, plus `loopbackAddress` and `port` properties naming what was reached. The address is released before the error is thrown. Set `HARPER_INTEGRATION_TEST_ALLOW_FOREIGN_LISTENERS=1` (or `true`) to get a warning instead.

### `LoopbackPoolExhaustedError`

Thrown by `getNextAvailableLoopbackAddress`, and so by `startHarper`, when no address became free within `HARPER_INTEGRATION_TEST_LOOPBACK_POOL_TIMEOUT_MS` (default `600000`, 10 minutes). It extends `Error` with `name` set to `'LoopbackPoolExhaustedError'`. The message gives the pool range and, for each holding process, the addresses it holds: its pid, the PID namespace when that differs from this process's, and the host name when that differs. It also lists any address held by a process that did not identify itself, and any unclaimed address a Harper node still has bound. The same listing is logged as a `[loopback-pool]` warning once a wait reaches 30 seconds (or a quarter of the timeout, if shorter). See [Loopback Pool Utilities](#loopback-pool-utilities).

### `targz(dirPath)`

Expand All @@ -298,21 +302,17 @@ Packs and compresses a directory into a base64-encoded tar.gz string. Useful for
These are used internally by `startHarper` and `teardownHarper`, but are exported for advanced use cases or custom runner integrations.

- `validateLoopbackAddressPool(): Promise<{ successful: string[]; failed: { loopbackAddress: string; error: Error }[] }>` - Validates all pool addresses can be bound to
- `getNextAvailableLoopbackAddress(): Promise<string>` - Allocates an address from the pool; throws a `ForeignListenerError` if another process already accepts connections on its operations or HTTP port (see [Other services listening on Harper's ports](#other-services-listening-on-harpers-ports))
- `getNextAvailableLoopbackAddress(): Promise<string>` - Allocates an address from the pool, waiting for one to be released if all are held; throws a `ForeignListenerError` if another process already accepts connections on its operations or HTTP port (see [Other services listening on Harper's ports](#other-services-listening-on-harpers-ports)), or a `LoopbackPoolExhaustedError` if the wait outlasts `HARPER_INTEGRATION_TEST_LOOPBACK_POOL_TIMEOUT_MS`
- `releaseLoopbackAddress(address: string): Promise<void>` - Returns an address to the pool
- `releaseAllLoopbackAddressesForCurrentProcess(): Promise<void>` - Releases all addresses held by this process

The pool is one file shared by every process on the machine: `${TMPDIR}/harper-integration-test-loopback-pool.json`, a JSON array with one slot per address holding the PID of the process that reserved it (or `null`), guarded by the `harper-integration-test-loopback-pool.lock` mutex beside it. Updates are published by renaming a complete `*.pending` file over it, so a process killed mid-update leaves the previous pool intact (and possibly an inert `*.pending` file). Releasing an address clears its slot only if this process holds it.

A pool file that is unparseable, not an array, or an empty array can still be produced by a writer that doesn't publish by rename (an older version of this package killed mid-write) or by an outside edit. Such a file is either about to be completed or has lost reservations whose holders may not have bound their address yet. Allocation does refuse an address that a Harper node is already listening on, but that check cannot see a reservation nothing is listening on yet. So the file is quarantined rather than reset:
A process holds an address by listening on its **claim port** (`HARPER_INTEGRATION_TEST_LOOPBACK_CLAIM_PORT`, default `9924`). The kernel allows one listener per address and port, so while one process holds an address, no other process can claim it. The kernel also closes the listener when its process exits, however it exits. This covers every process in one network namespace, which is where two Harper nodes on one address would collide, whatever their PID namespace or `TMPDIR`. Processes in separate network namespaces, such as most containers, have separate loopback interfaces and cannot collide. Every run on a machine must use the same claim port. Nothing is written to disk.

- `getNextAvailableLoopbackAddress()` waits, and both release functions leave the file untouched, until it has gone 6 minutes without modification. That outlasts `startHarper`'s default startup ceiling (2 minutes, 5 under CI), so a `startHarper` that reserved an address before the file broke has bound it or given up by then.
- After that, the pool is reinitialized with every address free and a warning is logged. From then on, an address still in use is protected only by that listening check.
- Deleting the file skips the wait, but only do that once every integration-test run on the machine — including ones using older versions of this package — has stopped.
Allocation takes the first address whose claim port is free, so an address left by a process that died is reused at once. It then skips an address on which Harper's operations, HTTP, HTTPS, MQTT or MQTTS port is still bound, such as by a node that outlived a killed test runner, and tries the next. Replication is not checked: harper-pro's default replication port has no host, so every node binds it on all interfaces. Releasing an address only frees a claim this process holds.

The quarantine bounds, rather than eliminates, the risk of handing out an address twice: a caller that holds an address for longer than that without binding it (for example, with a `startupMaxMs` above 5 minutes) is not covered.
When every address is held, allocation waits up to `HARPER_INTEGRATION_TEST_LOOPBACK_POOL_TIMEOUT_MS` (default `600000`), then throws a [`LoopbackPoolExhaustedError`](#loopbackpoolexhaustederror) naming the holders, rather than waiting forever. Raise the timeout if suites legitimately queue for longer than that. To name its holder, a claim answers a connection with one JSON line: its process's pid, PID namespace and host name. Allocation itself never connects to a claim.

If other processes take over the pool lock as stale (held over 10 seconds) before this process publishes its update, the allocate and release functions retry from the current pool state with a `[loopback-pool]` warning; after four takeovers in a row they reject with an error named `LockOwnershipLostError`.
Versions 1.0.5 and earlier kept the pool as a file of process IDs in `${TMPDIR}/harper-integration-test-loopback-pool.json`. They do not see claims, and this version does not read that file. Until every run on a machine uses this version, the two can hand out one address to two runs before either has started its node. Let runs on the old version finish before switching. The old file can then be deleted.

## Node.js Test Runner

Expand Down
10 changes: 5 additions & 5 deletions src/harperLifecycle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1112,11 +1112,11 @@ export async function teardownHarper(ctx: StartedHarperTestContext): Promise<voi
} else {
// A Harper child escaped the tree kill and still holds this address's ports. Do NOT
// recycle the address — under SO_REUSEPORT a later suite could silently co-bind it
// (the exact failure the loopback pool exists to prevent). Leave the slot parked under
// this process's PID; it is reclaimed when this (per-file) process exits and its PID
// goes dead. The conflict canary in getNextAvailableLoopbackAddress is the backstop if
// the address is somehow handed out before then. The install directory stays too: the
// process holding the ports may still be using it.
// (the exact failure the loopback pool exists to prevent). Keep this process's claim on
// the address; the kernel drops it when this (per-file) process exits. After that, the
// pool's check of the address's Harper ports keeps it from being handed out while the
// escaped child still binds them. The install directory stays too: the process holding
// the ports may still be using it.
console.warn(
`Harper ports on ${ctx.harper.hostname} still in use after teardown (${DEFAULT_PORT_RELEASE_TIMEOUT_MS}ms); NOT recycling the address or removing ${ctx.harper.dataRootDir} (a Harper child outlived the kill). The slot will be reclaimed when this process exits.`
);
Expand Down
1 change: 1 addition & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ export {
releaseLoopbackAddress,
releaseAllLoopbackAddressesForCurrentProcess,
ForeignListenerError,
LoopbackPoolExhaustedError,
} from './loopbackAddressPool.ts';

export { targz } from './targz.ts';
Loading
Loading