CI perf: ci-ok — 2-second check waits up to 2.6 min for a Depot host to boot, serially at the end of every run (~0.5–2.6 min per merge) #1745

Filed by the CI profiler routine.

Measurement

Window: 2026-10-27 13:07 → 20:16 UTC. Sample: every ci-ok job in the window's CI runs that I fetched jobs for (n=21: 9 merge_group, 9 push, 3 pull_request; filter=all).

ci-ok runs one step, Check every needed job, which takes 0–2 s. Its job time is almost all Depot start-up: the gap from the job's started_at to its first step.

Examples:

• merge_group run 40573814234 (Build(deps): Bump zod from 3.25.76 to 4.6.6 in /npm/socket-patch #1702): the last needed job finished at +4.93 min, ci-ok job started at +4.95 on depot-5mz2mw730c, its first step at +7.0, done at +7.08. The merge waited 2.1 min on a host booting to run a 2-second Python check. The 13:07–14:58 merge_group runs had ci-ok at a p50 of 5.13.
• pull_request run 40547522878: ci-ok job took 159 s, with 155 s before the first step.
• merge_group run 40572480618: ci-ok waited 131 s before reporting the failure, so the queue held the eviction for 2.2 min longer.

For comparison, GitHub-hosted ubuntu-latest jobs in the same hour (the Audit GHA Workflows, Merge queue janitor and Merge queue fail-fast runs at 19:48–20:12) were queued 2 s and started their first step 0–1 s after starting (n=3).

The slow starts aren't specific to ci-ok. Across all 172 Depot jobs in the 8 CI runs since 16:17, 64% (110) had a start gap over 15 s: p50 31 s, p90 91 s, max 179 s. That is up from 16% in the 13:07–14:58 burst. That host-level problem is #1512 and needs Depot. ci-ok is the one place where a slow start always lands on the critical path, because it runs alone after every other job, and it's the one place where it can be removed in the repo.

Where the time goes

In the 21 ci-ok jobs: start gap mean 32 s, p90 131 s, max 155 s. The check itself takes 0–2 s. Since 16:17 the mean gap is 75 s (n=6).

Root cause

ci-ok runs on depot-ubuntu-24.04-4 (ci.yml:2884), like the other Linux jobs, so it pays a Depot boot. It needs no cache, toolchain or CPU: one python3 call over toJSON(needs).

Proposed fix

In .github/workflows/ci.yml, job ci-ok, change

runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-4' }}

to runs-on: ubuntu-24.04 (GitHub-hosted, which has python3). Keep name: ci-ok, if: always(), needs and timeout-minutes: 5 as they are.

History: #1372 (10-10) moved ci-ok off ubuntu-latest because the hosted Linux pool was backlogged 9–18 min. That was before #1362 moved the Linux CI jobs to Depot. Today only the light audit/janitor/fail-fast jobs use hosted Linux, and they start in 2 s. Before merging, the PR should still check hosted Linux queue wait over the last few days of those workflows' runs. If it's under 10 s at p90, go ahead.

Expected saving

• Merge-queue critical path: ~0.5 min mean per merge (32 s gap → ~3 s), 2.0–2.5 min on each run during a slow-start episode like the one since 16:17 (3 of 6 runs).
• PR feedback (created → ci-ok): the same ~0.5 min mean, 2+ min in the tail.
• Job-minutes: about 0.5 Depot job-min per run, ~50–100 runs/day ≈ 25–50 Linux job-min/day. Moving them to hosted is free for a public repo.

Coverage and risk

No test moves. ci-ok keeps its name and logic, so the required check stays the same. Risk: a hosted Linux backlog like #1372's would put the wait back. That's why the PR should check queue wait first. If hosted ever backs up, revert the one line.

Effort

S. One line.

ROI

(0.05k job-min → 0.05, + 0.5 critical-path min + 0.5 PR-latency min) × confidence 0.7 ÷ effort 1 ≈ 0.7 (≈ 3.5 during slow-start episodes, when it saves 2.5 + 2.5 min).