diff --git a/CHANGELOG.md b/CHANGELOG.md index 204a3ed..2362af6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) - SemVer. ## [Unreleased] +## [1.1.0] - 2026-10-31 + +### Added +- Multi-key rotation (opt-in). With two or more keys from + `COMMANDCODE_API_KEYS`, `COMMANDCODE_KEYS_FILE` or + `~/.config/commandcode/keys.txt`, the plugin picks the first key that is not + cooling down, so a dead account does not end the turn. `402` and `401`/`403` + block a key for 24h, `429` honours `Retry-After` up to 6h (5m by default), and + `5xx` never blames the key. The soonest recovery wins when every key is + cooling down. State is labels-only (`commandcode-keys-state.json`, 0600) and + every cooldown is logged to `commandcode-rotation.log` without secrets. + Single-key mode keeps the previous behaviour: the plugin never overrides an + `Authorization` header the host already set. + ## [1.0.7] - 2026-10-31 ### Added diff --git a/README.es.md b/README.es.md index ae0a2d4..faeac02 100644 --- a/README.es.md +++ b/README.es.md @@ -72,6 +72,32 @@ substring para credenciales: filtraria la key a hosts parecidos como `/provider/v1/chat/completions` en su URL. Una cabecera `authorization` existente jamas se sobrescribe. +### Rotacion multi-key (opt-in) + +Se pueden agrupar varias keys de Command Code para que una cuenta muerta no corte +el turno. Dos o mas keys activan la rotacion; con una sola se mantiene el camino +de key unica sin cambios. + +```sh +# inline +export COMMANDCODE_API_KEYS="key_one,key_two" + +# o archivo, una por linea (`label=key` o key pelada, comentarios con `#`) +export COMMANDCODE_KEYS_FILE="$HOME/.config/commandcode/keys.txt" +``` + +Los cooldowns se registran por key segun la respuesta que culpa a la credencial: +`402` (creditos insuficientes) y `401`/`403` bloquean 24h, `429` respeta +`Retry-After` hasta 6h (5m por defecto), y `5xx` nunca culpa a la key. Si todas +estan en cooldown gana la que se recupera antes. El estado guarda solo etiquetas +(`/commandcode-keys-state.json`, modo 0600) y cada cooldown se registra en +`/commandcode-rotation.log`. + +En modo rotacion el plugin es dueno de la cabecera `Authorization` para +`api.commandcode.ai`, porque la key que inyecto el host puede ser justo la que +devolvio `402`. En modo key unica se mantiene la regla anterior: inyectar solo si +el request no trae cabecera. + ### Estabilidad de sesion Cuatro hooks independientes, todos fail-open (un error del plugin nunca puede romper @@ -218,6 +244,9 @@ Cada ajuste es una variable de entorno con default seguro. Referencia completa: | --- | --- | --- | | `COMMAND_CODE_API_KEY` | - | Resolucion de la key, maxima prioridad | | `COMMANDCODE_KEY_FILE` | - | Ruta absoluta a un archivo con la key | +| `COMMANDCODE_API_KEYS` | - | Pool de keys; dos o mas activan rotacion | +| `COMMANDCODE_KEYS_FILE` | - | Archivo pool (`label=key` o key pelada por linea) | +| `CMD_ZDR` | - | `1` para zero data retention (`x-cmd-zdr: 1`) | | `COMMANDCODE_ALLOWED_HOSTS` | - | Hosts extra separados por coma autorizados a recibir la key | | `CC_NO_CONNECTION_CLOSE` | `0` | Poner en `1` para dejar de inyectar `Connection: close` | | `COMMANDCODE_STATE_DIR` | `$XDG_STATE_HOME/opencode` | Donde viven los logs y la cache de modelos | @@ -236,6 +265,8 @@ Kill switch en runtime para la higiene de conexion: crea un archivo centinela ll | `/commandcode-sanitizer.log` | Cada reparacion de payload en vuelo, con las correcciones aplicadas | | `/commandcode-watchdog.log` | Streams SSE abortados por inactividad | | `/commandcode-retries.log` | Reintentos de transporte de Command Code: sesion, agente, proveedor, modelo, tipo de error, status HTTP, numero de intento, delay del backoff y mensaje | +| `/commandcode-rotation.log` | Cooldowns de keys: etiqueta, status HTTP, motivo y hora de reset | +| `/commandcode-keys-state.json` | Estado de cooldown por etiqueta (solo etiquetas, 0600) | | `/commandcode-models-cache.json` | Ultimo catalogo de modelos bueno (TTL 6h) | `` por defecto es `~/.local/state/opencode`. diff --git a/README.md b/README.md index f2743c9..762519d 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,31 @@ it would leak the key to lookalike hosts such as `api.commandcode.ai.evil.com` or to any server that merely carries `/provider/v1/chat/completions` in its URL. An existing `authorization` header is never overwritten. +### Multi-key rotation (opt-in) + +Several Command Code keys can be pooled so a dead account does not end the turn. +Two or more keys enable rotation; one key keeps the single-key path untouched. + +```sh +# inline +export COMMANDCODE_API_KEYS="key_one,key_two" + +# or a file, one per line (`label=key` or bare key, `#` comments) +export COMMANDCODE_KEYS_FILE="$HOME/.config/commandcode/keys.txt" +``` + +Cooldowns are recorded per key from the response that blamed the credential: +`402` (insufficient credits) and `401`/`403` block for 24h, `429` honours +`Retry-After` up to 6h (5m by default), and `5xx` never blames the key. When all +keys are cooling down the soonest recovery wins. State is labels-only +(`/commandcode-keys-state.json`, mode 0600) and every cooldown is logged +to `/commandcode-rotation.log`. + +In rotation mode the plugin owns the `Authorization` header for +`api.commandcode.ai`, because the key that the host config injected may be the +one that just returned `402`. Single-key mode keeps the previous rule: inject +only when the request has no header. + ### Session stability Four independent hooks, all fail-open (an error in the plugin can never break a @@ -212,6 +237,9 @@ Every setting is an environment variable with a safe default. Full reference: | --- | --- | --- | | `COMMAND_CODE_API_KEY` | - | Key resolution, highest priority | | `COMMANDCODE_KEY_FILE` | - | Absolute path to a key file | +| `COMMANDCODE_API_KEYS` | - | Pool of keys; two or more enable rotation | +| `COMMANDCODE_KEYS_FILE` | - | Pool file (`label=key` or bare key per line) | +| `CMD_ZDR` | - | Set to `1` for zero data retention (`x-cmd-zdr: 1`) | | `COMMANDCODE_ALLOWED_HOSTS` | - | Comma-separated extra hosts allowed to receive the key | | `CC_NO_CONNECTION_CLOSE` | `0` | Set to `1` to stop injecting `Connection: close` | | `COMMANDCODE_STATE_DIR` | `$XDG_STATE_HOME/opencode` | Where logs and the model cache live | @@ -230,6 +258,8 @@ Runtime kill switch for connection hygiene: create a sentinel file named | `/commandcode-sanitizer.log` | Every in-flight payload repair, with the fixes applied | | `/commandcode-watchdog.log` | Aborted stalled SSE streams | | `/commandcode-retries.log` | Command Code transport retries: session, agent, provider, model, error type, HTTP status, attempt number, backoff delay and message | +| `/commandcode-rotation.log` | Key cooldowns: label, HTTP status, reason and reset time | +| `/commandcode-keys-state.json` | Per-key cooldown state (labels only, 0600) | | `/commandcode-models-cache.json` | Last known good model catalog (6h TTL) | `` defaults to `~/.local/state/opencode`. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 931479f..0f9bf3f 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -12,6 +12,9 @@ hook. | --- | --- | --- | | `COMMAND_CODE_API_KEY` | unset | Highest priority. Used as-is (trimmed, surrounding quotes stripped). | | `COMMANDCODE_KEY_FILE` | unset | Absolute path to a file containing only the key. Ignored when the file is missing or empty, so the cascade keeps going. | +| `COMMANDCODE_API_KEYS` | unset | Comma or whitespace separated pool. Two or more keys enable rotation. | +| `COMMANDCODE_KEYS_FILE` | unset | Pool file, one `label=key` or bare key per line. `#` starts a comment. | +| `CMD_ZDR` | unset | Set to `1` to send `x-cmd-zdr: 1` (zero data retention). | | `XDG_CONFIG_HOME` | `~/.config` | Base for the third resolution step: `$XDG_CONFIG_HOME/commandcode/key.txt`. | ### Resolution order @@ -43,6 +46,32 @@ Code traffic. Every host added here can read the API key. COMMANDCODE_ALLOWED_HOSTS=gateway.internal,proxy.lan ``` +### Key rotation + +A pool of two or more keys activates rotation. With one key (or none) the plugin +keeps the single-key behaviour and never touches an `Authorization` header the +host already set. + +| Source | Format | +| --- | --- | +| `COMMANDCODE_API_KEYS` | `key1,key2` or whitespace separated | +| `COMMANDCODE_KEYS_FILE` | one `label=key` or bare key per line, `#` comments | +| `~/.config/commandcode/keys.txt` | same format, used when neither variable is set | + +Cooldown recorded per key after a response that blames the credential: + +| Status | Cooldown | +| --- | --- | +| `402` insufficient credits | 24h | +| `401` / `403` invalid or not entitled | 24h | +| `429` rate limited | `Retry-After` when present and at most 6h, else 5m | +| `5xx` | none (not the key's fault) | + +When every key is cooling down the one that recovers first is used, so the +request still reaches upstream. State lives in +`/commandcode-keys-state.json` (mode 0600, labels only, never keys) and +each cooldown is appended to `/commandcode-rotation.log`. + ## Transport | Variable | Default | Meaning | @@ -70,6 +99,8 @@ Files written under the state directory: | `commandcode-sanitizer.log` | append | JSON lines, one per repaired payload | | `commandcode-watchdog.log` | append | JSON lines, one per aborted stream | | `commandcode-retries.log` | append | JSON lines, Command Code retries only | +| `commandcode-rotation.log` | append | JSON lines, one per key cooldown (label, status, reason, reset time) | +| `commandcode-keys-state.json` | overwrite | Cooldown state per key label (0600, no secrets) | | `commandcode-models-cache.json` | overwrite | `{ at, models[] }`, 6h TTL | Logs are append-only and never rotated by the plugin. At the observed event rate diff --git a/docs/es/configuration.md b/docs/es/configuration.md index 8d3179f..243bf01 100644 --- a/docs/es/configuration.md +++ b/docs/es/configuration.md @@ -12,6 +12,9 @@ todos los hooks. | --- | --- | --- | | `COMMAND_CODE_API_KEY` | sin definir | Maxima prioridad. Se usa tal cual (recortada y sin comillas envolventes). | | `COMMANDCODE_KEY_FILE` | sin definir | Ruta absoluta a un archivo que contenga solo la key. Se ignora si el archivo falta o esta vacio, para que la cascada continue. | +| `COMMANDCODE_API_KEYS` | sin definir | Pool separado por comas o espacios. Dos o mas keys activan la rotacion. | +| `COMMANDCODE_KEYS_FILE` | sin definir | Archivo pool, una `label=key` o key pelada por linea. `#` inicia comentario. | +| `CMD_ZDR` | sin definir | `1` envia `x-cmd-zdr: 1` (zero data retention). | | `XDG_CONFIG_HOME` | `~/.config` | Base del tercer paso de resolucion: `$XDG_CONFIG_HOME/commandcode/key.txt`. | ### Orden de resolucion @@ -39,6 +42,29 @@ tests. `resolveApiKey()` devuelve solo el string. Usala solo para un gateway propio que realmente termine tu trafico hacia Command Code. Cada host que agregues aqui puede leer la API key. +### Rotacion de keys + +Un pool de dos o mas keys activa la rotacion. Con una sola key (o ninguna) el plugin +mantiene el comportamiento de key unica y nunca toca una cabecera `Authorization` que +el host ya haya puesto. + +| Fuente | Formato | +| --- | --- | +| `COMMANDCODE_API_KEYS` | `key1,key2` o separadas por espacios | +| `COMMANDCODE_KEYS_FILE` | una `label=key` o key pelada por linea, comentarios con `#` | +| `~/.config/commandcode/keys.txt` | mismo formato, usado si no hay variables | + +| Status | Cooldown | +| --- | --- | +| `402` creditos insuficientes | 24h | +| `401` / `403` key invalida o sin permiso | 24h | +| `429` rate limit | `Retry-After` si existe y es <= 6h, si no 5m | +| `5xx` | ninguno (no es culpa de la key) | + +Si todas las keys estan en cooldown se usa la que se recupera primero. El estado vive en +`/commandcode-keys-state.json` (modo 0600, solo etiquetas) y cada cooldown se +registra en `/commandcode-rotation.log`. + ``` COMMANDCODE_ALLOWED_HOSTS=gateway.internal,proxy.lan ``` diff --git a/index.ts b/index.ts index 05cfa77..ee38a4b 100644 --- a/index.ts +++ b/index.ts @@ -2,6 +2,7 @@ import plugin from "./src/index.ts"; export default plugin; export * from "./src/auth/resolver.ts"; +export * from "./src/auth/pool.ts"; export * from "./src/client/api.ts"; export * from "./src/hooks/resilience.ts"; export * from "./src/hooks/zdr.ts"; diff --git a/package.json b/package.json index 70bde93..01b2222 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "opencode-commandcode", - "version": "1.0.7", + "version": "1.1.0", "description": "Production-grade OpenCode plugin for Command Code: dynamic model discovery, stability hooks and quota telemetry", "main": "index.ts", "type": "module", diff --git a/src/auth/pool.ts b/src/auth/pool.ts new file mode 100644 index 0000000..f3ccc8e --- /dev/null +++ b/src/auth/pool.ts @@ -0,0 +1,244 @@ +/** + * Multi-key pool with per-key cooldowns. + * + * A Command Code plan can hold several API keys (one per account). When the + * current key is dead for a while - insufficient credits, a rate-limit window - + * the next request should try the next account instead of failing the turn. + * + * Activation is opt-in and additive: the pool only exists when at least two + * keys are configured, either as + * + * COMMANDCODE_API_KEYS="key1,key2" (comma or whitespace separated) + * COMMANDCODE_KEYS_FILE=/path/to/keys.txt (one `label=key` or bare key per line) + * ~/.config/commandcode/keys.txt (default pool file) + * + * With a single key (or no pool) the plugin keeps its previous behaviour: the + * resolved key from `auth/resolver.ts` is injected only when the host did not + * already send an Authorization header. + * + * Cooldown policy (recorded per key label in the state file): + * + * 402 insufficient_credits -> 24h + * 401 / 403 -> 24h (invalid key, revoked, not entitled) + * 429 rate limited -> Retry-After when present and <= 6h, else 5m + * 5xx / network -> no cooldown (not the key's fault) + */ + +import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { resolveStateDir } from "../hooks/resilience.ts"; + +export interface PoolKey { + /** Stable identifier used in the state file. Never the secret itself. */ + label: string; + /** The API key. Never logged. */ + key: string; +} + +export interface KeyCooldown { + until: number; + reason: string; + status: number; + at: number; +} + +export interface PoolState { + version: 1; + keys: Record; +} + +export const COOLDOWN_DAY_MS = 24 * 60 * 60 * 1000; +export const COOLDOWN_RATE_LIMIT_MS = 5 * 60 * 1000; +export const COOLDOWN_RATE_LIMIT_CAP_MS = 6 * 60 * 60 * 1000; + +const STATE_FILE = "commandcode-keys-state.json"; + +function clean(value: unknown): string { + return typeof value === "string" ? value.trim().replace(/^["']|["']$/g, "") : ""; +} + +function readFileSafe(path: string): string | null { + try { + if (!existsSync(path)) return null; + return readFileSync(path, "utf8"); + } catch { + return null; + } +} + +/** Short non-reversible label so the state file never stores a key. */ +export function keyLabel(key: string, index: number): string { + let hash = 0; + for (let i = 0; i < key.length; i++) { + hash = (hash * 31 + key.charCodeAt(i)) | 0; + } + const suffix = (hash >>> 0).toString(16).padStart(8, "0").slice(0, 8); + return `key${index + 1}-${suffix}`; +} + +/** Parses the `label=key` / bare-key line format. Comments and blanks are skipped. */ +export function parseKeysFile(raw: string | null): PoolKey[] { + if (!raw) return []; + const keys: PoolKey[] = []; + for (const line of raw.split(/\r?\n/)) { + const trimmed = line.trim(); + if (!trimmed || trimmed.startsWith("#")) continue; + const eq = trimmed.indexOf("="); + const label = eq > 0 ? clean(trimmed.slice(0, eq)) : ""; + const key = clean(eq > 0 ? trimmed.slice(eq + 1) : trimmed); + if (!key) continue; + keys.push({ label: label || keyLabel(key, keys.length), key }); + } + return dedupe(keys); +} + +function dedupe(keys: PoolKey[]): PoolKey[] { + const seen = new Set(); + const out: PoolKey[] = []; + for (const entry of keys) { + if (seen.has(entry.key)) continue; + seen.add(entry.key); + out.push(entry); + } + return out; +} + +function poolFileCandidates(): string[] { + const explicit = clean(process.env.COMMANDCODE_KEYS_FILE); + if (explicit) return [explicit]; + const configHome = clean(process.env.XDG_CONFIG_HOME) || join(homedir(), ".config"); + return [join(configHome, "commandcode", "keys.txt")]; +} + +/** + * Loads the pool. Returns an empty array when rotation is not configured, which + * keeps the single-key path untouched. + */ +export function loadKeyPool(env: Record = process.env): PoolKey[] { + const inline = clean(env.COMMANDCODE_API_KEYS); + if (inline) { + const keys = inline + .split(/[,\s]+/) + .map((key, index) => ({ label: keyLabel(key, index), key })) + .filter((entry) => entry.key.length > 0); + const deduped = dedupe(keys); + if (deduped.length > 0) return deduped; + } + + const explicitFile = clean(env.COMMANDCODE_KEYS_FILE); + const candidates = explicitFile + ? [explicitFile] + : [join(clean(env.XDG_CONFIG_HOME) || join(homedir(), ".config"), "commandcode", "keys.txt")]; + for (const candidate of candidates) { + const parsed = parseKeysFile(readFileSafe(candidate)); + if (parsed.length > 0) return parsed; + } + return []; +} + +export function statePath(): string { + return join(resolveStateDir(), STATE_FILE); +} + +export function loadPoolState(path: string = statePath()): PoolState { + const raw = readFileSafe(path); + if (!raw) return { version: 1, keys: {} }; + try { + const data = JSON.parse(raw); + if (data && data.version === 1 && data.keys && typeof data.keys === "object") { + return data as PoolState; + } + } catch { + // corrupt state must never break a request + } + return { version: 1, keys: {} }; +} + +export function savePoolState(state: PoolState, path: string = statePath()): void { + try { + mkdirSync(join(path, ".."), { recursive: true }); + const tmp = `${path}.tmp`; + writeFileSync(tmp, JSON.stringify(state, null, 2) + "\n", { mode: 0o600 }); + renameSync(tmp, path); + } catch { + // best effort: a failed write only costs a retry on the same key + } +} + +function cooldownIndex(state: PoolState) { + return state.keys ?? {}; +} + +/** + * Picks the first key that is not cooling down. When every key is cooling down + * the one that recovers first wins, so the request still has a chance instead + * of failing before it reaches upstream. + */ +export function selectKey(keys: PoolKey[], state: PoolState, now: number = Date.now()): PoolKey | null { + if (keys.length === 0) return null; + const index = cooldownIndex(state); + const available = keys.find((entry) => (index[entry.label]?.until ?? 0) <= now); + if (available) return available; + return [...keys].sort( + (a, b) => (index[a.label]?.until ?? 0) - (index[b.label]?.until ?? 0) + )[0] ?? null; +} + +/** Parses Retry-After (seconds or HTTP date) into milliseconds. */ +export function retryAfterMs(value: string | null, now: number = Date.now()): number | null { + if (!value) return null; + const seconds = Number(value); + if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1000; + const date = Date.parse(value); + if (!Number.isNaN(date)) return Math.max(0, date - now); + return null; +} + +/** + * Maps a response status to a cooldown. Returns null when the key is healthy or + * the failure is not attributable to it. + */ +export function cooldownForStatus( + status: number, + retryAfter: string | null, + now: number = Date.now() +): KeyCooldown | null { + if (status === 402) { + return { until: now + COOLDOWN_DAY_MS, reason: "insufficient credits", status, at: now }; + } + if (status === 401 || status === 403) { + return { until: now + COOLDOWN_DAY_MS, reason: "key rejected", status, at: now }; + } + if (status === 429) { + const fromHeader = retryAfterMs(retryAfter, now); + const wait = + fromHeader !== null && fromHeader <= COOLDOWN_RATE_LIMIT_CAP_MS + ? Math.max(fromHeader, 1000) + : COOLDOWN_RATE_LIMIT_MS; + return { until: now + wait, reason: "rate limited", status, at: now }; + } + return null; +} + +/** Records the cooldown for a key label, if the status deserves one. */ +export function recordResponse( + state: PoolState, + label: string, + status: number, + retryAfter: string | null, + now: number = Date.now(), + path: string = statePath() +): KeyCooldown | null { + const cooldown = cooldownForStatus(status, retryAfter, now); + if (!cooldown) return null; + state.keys = state.keys ?? {}; + state.keys[label] = cooldown; + savePoolState(state, path); + return cooldown; +} + +/** True when the label's cooldown has expired; used to prune on read. */ +export function isCoolingDown(state: PoolState, label: string, now: number = Date.now()): boolean { + return (state.keys?.[label]?.until ?? 0) > now; +} diff --git a/src/index.ts b/src/index.ts index adafe23..4abade4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -14,6 +14,7 @@ import { supportsVision, } from "./hooks/resilience.ts"; import { applyZdrHeader } from "./hooks/zdr.ts"; +import { loadKeyPool, loadPoolState, recordResponse, selectKey } from "./auth/pool.ts"; import { getQuotaReport } from "./tools/quota.ts"; import { describeRetryEvent } from "./telemetry.ts"; import { buildReleaseDate, freeCostEntry } from "./catalog.ts"; @@ -151,12 +152,19 @@ export default { id: "opencode-commandcode", async setup(ctx: any) { + // Multi-key pool (opt-in): rotation only activates with two or more keys. + const keyPool = loadKeyPool(); + const rotationEnabled = keyPool.length >= 2; + const pendingKeyByRequest = new WeakMap(); + const lastKeyBySession = new Map(); + // ---------------------------------------------------------------- 1. Hooks if (ctx?.session && typeof ctx.session.hook === "function") { await ctx.session.hook("http.request", async (event: any) => { const req = event?.request; const url = req?.url || ""; if (!req || !isCommandCodeRequest(url)) return; + let selectedLabel: string | null = null; // A. Connection hygiene (Bun keep-alive vs Cloudflare idle close). if (!isConnectionCloseDisabled()) { @@ -180,13 +188,30 @@ export default { // C. Credential injection, STRICT origin only (no lookalike hosts, // no path-based matching: this is where the API key travels). - if (isCommandCodeHost(url) && !req.headers.has("authorization")) { - const key = resolveApiKey(); - if (key) { - try { - req.headers.set("authorization", `Bearer ${key}`); - } catch { - // ignore + if (isCommandCodeHost(url)) { + if (rotationEnabled) { + // Rotation owns the credential: a request must never keep the key + // that the host config injected, or a cooling-down key would be + // reused after a 402/429 was recorded for it. + const selected = selectKey(keyPool, loadPoolState()); + if (selected) { + try { + req.headers.set("authorization", `Bearer ${selected.key}`); + selectedLabel = selected.label; + pendingKeyByRequest.set(req, selected.label); + if (event.sessionID) lastKeyBySession.set(event.sessionID, selected.label); + } catch { + // immutable headers in this runtime + } + } + } else if (!req.headers.has("authorization")) { + const key = resolveApiKey(); + if (key) { + try { + req.headers.set("authorization", `Bearer ${key}`); + } catch { + // ignore + } } } } @@ -222,6 +247,9 @@ export default { body: newBody, signal: req.signal, }); + if (selectedLabel) { + pendingKeyByRequest.set(event.request, selectedLabel); + } await appendEvent(logPath("commandcode-sanitizer.log"), { at: new Date().toISOString(), @@ -239,6 +267,33 @@ export default { const url = event?.request?.url || ""; if (!isCommandCodeRequest(url)) return; + // Key rotation: record the cooldown implied by this response so the + // next request picks a different account instead of the same dead key. + if (rotationEnabled && isCommandCodeHost(url)) { + const label = + (event?.request ? pendingKeyByRequest.get(event.request) : undefined) ?? + (event?.sessionID ? lastKeyBySession.get(event.sessionID) : undefined); + const status = Number(event?.response?.status ?? 0); + if (label) { + const cooldown = recordResponse( + loadPoolState(), + label, + status, + event?.response?.headers?.get?.("retry-after") ?? null + ); + if (cooldown) { + await appendEvent(logPath("commandcode-rotation.log"), { + at: new Date().toISOString(), + sessionID: event.sessionID, + keyLabel: label, + status, + reason: cooldown.reason, + cooldownUntil: new Date(cooldown.until).toISOString(), + }); + } + } + } + const res = event.response; if (!res || !res.body) return; diff --git a/tests/pool.test.ts b/tests/pool.test.ts new file mode 100644 index 0000000..ee0a354 --- /dev/null +++ b/tests/pool.test.ts @@ -0,0 +1,317 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import plugin from "../src/index.ts"; +import { + COOLDOWN_DAY_MS, + COOLDOWN_RATE_LIMIT_MS, + cooldownForStatus, + isCoolingDown, + keyLabel, + loadKeyPool, + loadPoolState, + parseKeysFile, + retryAfterMs, + selectKey, +} from "../src/auth/pool.ts"; + +const URL_CC = "https://api.commandcode.ai/provider/v1/chat/completions"; + +function makeCtx() { + const hooks: Record Promise)[]> = { + "http.request": [], + "http.response": [], + retry: [], + }; + const ctx: any = { + session: { + hook: async (name: any, cb: any) => { + hooks[name]?.push(cb); + return { dispose: async () => {} }; + }, + synthetic: async () => ({}), + }, + model: { transform: async () => {} }, + tool: { transform: async () => {} }, + command: { transform: async () => {} }, + }; + return { ctx, hooks }; +} + +async function runRequest(hooks: Record Promise)[]>, request: Request) { + const event: any = { request, sessionID: "ses_pool" }; + for (const cb of hooks["http.request"] ?? []) await cb(event); + return event; +} + +async function runResponse( + hooks: Record Promise)[]>, + event: any, + response: Response +) { + const next: any = { ...event, response }; + for (const cb of hooks["http.response"] ?? []) await cb(next); + return next; +} + +describe("keyLabel", () => { + test("is stable and never contains the key", () => { + const a = keyLabel("user_alpha", 0); + const b = keyLabel("user_alpha", 0); + const c = keyLabel("user_beta", 1); + expect(a).toBe(b); + expect(a).not.toBe(c); + expect(a).not.toContain("user_alpha"); + }); +}); + +describe("parseKeysFile", () => { + test("parses bare keys, labelled keys, comments and blanks", () => { + const keys = parseKeysFile("# pool\n\nuser_a\nmain=user_b\n # trailing comment\nuser_c\n"); + expect(keys.map((k) => k.key)).toEqual(["user_a", "user_b", "user_c"]); + expect(keys[1]!.label).toBe("main"); + }); + + test("dedupes repeated keys", () => { + const keys = parseKeysFile("user_a\nuser_a\nuser_b\n"); + expect(keys.map((k) => k.key)).toEqual(["user_a", "user_b"]); + }); + + test("returns an empty pool for empty input", () => { + expect(parseKeysFile(null)).toEqual([]); + expect(parseKeysFile(" \n# only comments\n")).toEqual([]); + }); +}); + +describe("loadKeyPool", () => { + let dir: string; + let file: string; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "cc-pool-")); + file = join(dir, "keys.txt"); + }); + + afterEach(() => { + try { + rmSync(dir, { recursive: true, force: true }); + } catch {} + }); + + test("reads COMMANDCODE_API_KEYS before files", () => { + writeFileSync(file, "file_key\n"); + const keys = loadKeyPool({ + COMMANDCODE_API_KEYS: "k1, k2\nk3", + COMMANDCODE_KEYS_FILE: file, + }); + expect(keys.map((k) => k.key)).toEqual(["k1", "k2", "k3"]); + }); + + test("falls back to COMMANDCODE_KEYS_FILE", () => { + writeFileSync(file, "main=file_key\nsecond=file_key_2\n"); + const keys = loadKeyPool({ COMMANDCODE_KEYS_FILE: file }); + expect(keys.map((k) => k.label)).toEqual(["main", "second"]); + }); + + test("returns an empty pool when nothing is configured", () => { + expect(loadKeyPool({ COMMANDCODE_KEYS_FILE: join(dir, "missing.txt") })).toEqual([]); + }); +}); + +describe("selectKey", () => { + const keys = [ + { label: "a", key: "ka" }, + { label: "b", key: "kb" }, + ]; + + test("prefers the first key that is not cooling down", () => { + const now = 1_000; + const state = { version: 1 as const, keys: { a: { until: now + 1, reason: "x", status: 402, at: now } } }; + expect(selectKey(keys, state, now)?.label).toBe("b"); + }); + + test("fails open with the soonest-recovering key when all are cooling down", () => { + const now = 1_000; + const state = { + version: 1 as const, + keys: { + a: { until: now + 500, reason: "x", status: 402, at: now }, + b: { until: now + 100, reason: "x", status: 429, at: now }, + }, + }; + expect(selectKey(keys, state, now)?.label).toBe("b"); + }); + + test("returns null for an empty pool", () => { + expect(selectKey([], { version: 1, keys: {} }, 0)).toBeNull(); + }); +}); + +describe("cooldownForStatus", () => { + const now = 10_000; + + test("402 blocks the key for a day", () => { + const c = cooldownForStatus(402, null, now)!; + expect(c.until).toBe(now + COOLDOWN_DAY_MS); + expect(c.reason).toBe("insufficient credits"); + }); + + test("401 and 403 block the key for a day", () => { + expect(cooldownForStatus(401, null, now)!.until).toBe(now + COOLDOWN_DAY_MS); + expect(cooldownForStatus(403, null, now)!.until).toBe(now + COOLDOWN_DAY_MS); + }); + + test("429 honours Retry-After within the cap", () => { + const c = cooldownForStatus(429, "60", now)!; + expect(c.until).toBe(now + 60_000); + }); + + test("429 falls back to five minutes without Retry-After", () => { + const c = cooldownForStatus(429, null, now)!; + expect(c.until).toBe(now + COOLDOWN_RATE_LIMIT_MS); + }); + + test("429 ignores an absurd Retry-After", () => { + const c = cooldownForStatus(429, "999999", now)!; + expect(c.until).toBe(now + COOLDOWN_RATE_LIMIT_MS); + }); + + test("5xx is not the key's fault", () => { + expect(cooldownForStatus(500, null, now)).toBeNull(); + expect(cooldownForStatus(503, null, now)).toBeNull(); + }); +}); + +describe("retryAfterMs", () => { + test("parses seconds", () => { + expect(retryAfterMs("30", 0)).toBe(30_000); + }); + + test("parses an HTTP date", () => { + const now = Date.parse("2026-01-01T00:00:00Z"); + expect(retryAfterMs("Thu, 01 Jan 2026 00:01:00 GMT", now)).toBe(60_000); + }); + + test("rejects garbage", () => { + expect(retryAfterMs("soon", 0)).toBeNull(); + expect(retryAfterMs(null, 0)).toBeNull(); + }); +}); + +describe("plugin key rotation", () => { + let stateDir: string; + const previous = { + keys: process.env.COMMANDCODE_API_KEYS, + state: process.env.COMMANDCODE_STATE_DIR, + key: process.env.COMMAND_CODE_API_KEY, + }; + + beforeEach(() => { + stateDir = mkdtempSync(join(tmpdir(), "cc-rot-")); + process.env.COMMANDCODE_STATE_DIR = stateDir; + process.env.COMMANDCODE_API_KEYS = "key_one,key_two"; + delete process.env.COMMAND_CODE_API_KEY; + }); + + afterEach(() => { + for (const [name, value] of [ + ["COMMANDCODE_API_KEYS", previous.keys], + ["COMMANDCODE_STATE_DIR", previous.state], + ["COMMAND_CODE_API_KEY", previous.key], + ] as const) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } + try { + rmSync(stateDir, { recursive: true, force: true }); + } catch {} + }); + + test("first request uses the first key and overrides the host-injected one", async () => { + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + const request = new Request(URL_CC, { + method: "POST", + body: "{}", + headers: { authorization: "Bearer host_config_key" }, + }); + const event = await runRequest(hooks, request); + expect(event.request.headers.get("authorization")).toBe("Bearer key_one"); + }); + + test("a 402 cools the first key down and the next request uses the second", async () => { + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + + const first = await runRequest(hooks, new Request(URL_CC, { method: "POST", body: "{}" })); + expect(first.request.headers.get("authorization")).toBe("Bearer key_one"); + + await runResponse(hooks, first, new Response(null, { status: 402 })); + + const state = loadPoolState(); + const cooled = Object.values(state.keys); + expect(cooled).toHaveLength(1); + expect(cooled[0]!.reason).toBe("insufficient credits"); + + const second = await runRequest(hooks, new Request(URL_CC, { method: "POST", body: "{}" })); + expect(second.request.headers.get("authorization")).toBe("Bearer key_two"); + }); + + test("a 429 cools the key down using Retry-After", async () => { + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + const first = await runRequest(hooks, new Request(URL_CC, { method: "POST", body: "{}" })); + await runResponse(hooks, first, new Response(null, { status: 429, headers: { "retry-after": "90" } })); + const state = loadPoolState(); + const cooled = Object.values(state.keys)[0]!; + expect(cooled.reason).toBe("rate limited"); + const remaining = cooled.until - Date.now(); + expect(remaining).toBeGreaterThan(80_000); + expect(remaining).toBeLessThanOrEqual(91_000); + }); + + test("logs rotation events without the secret", async () => { + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + const first = await runRequest(hooks, new Request(URL_CC, { method: "POST", body: "{}" })); + await runResponse(hooks, first, new Response(null, { status: 402 })); + const log = readFileSync(join(stateDir, "commandcode-rotation.log"), "utf8"); + expect(log).toContain("insufficient credits"); + expect(log).not.toContain("key_one"); + expect(log).not.toContain("key_two"); + }); + + test("a single key keeps the previous behaviour and never overrides the host header", async () => { + process.env.COMMANDCODE_API_KEYS = "only_key"; + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + const request = new Request(URL_CC, { + method: "POST", + body: "{}", + headers: { authorization: "Bearer host_config_key" }, + }); + const event = await runRequest(hooks, request); + expect(event.request.headers.get("authorization")).toBe("Bearer host_config_key"); + }); + + test("never touches a lookalike host", async () => { + const { ctx, hooks } = makeCtx(); + await plugin.setup(ctx); + const request = new Request("https://api.commandcode.ai.evil.com/provider/v1/chat/completions", { + method: "POST", + body: "{}", + }); + const event = await runRequest(hooks, request); + expect(event.request.headers.get("authorization")).toBeNull(); + }); + + test("prunes expired cooldowns on read", () => { + const now = 1_000; + const state = { + version: 1 as const, + keys: { a: { until: now - 1, reason: "x", status: 402, at: now } }, + }; + expect(isCoolingDown(state, "a", now)).toBe(false); + }); +});