Repository navigation
Conversation
A us-release-head-to-head ToolSpec (one score stage, digest-pinned incumbent, candidate and ledger-feed inputs, owned path flags, --workers and --maximum-microsim-batch-size as plan options), a head-to-head resource class (20 CPUs with limit 20, 224 GiB, 4 h 45 min, non-preemptible; $38.95 list-price ceiling per attempt), the d844 plan and a runbook section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as a plan option #1171 merged as 7639ef8, so the plan pins that commit on main. The scorer gained --worker-max-slices after this stage was written; it becomes the third plan option (unset in the d844 plan), and the ownership test and its parser fixture follow. The runbook's launch section no longer carries machine paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t at 16 workers Address the #1190 review: the check-mode import probe now puts the scorer's directory on sys.path and registers the module in sys.modules, so tools that import a sibling (the scorer imports build_us_fiscal_refresh_release) pass the required check; a subprocess test runs the real probe (engine-contract-us). The head-to-head class's memory is a hard limit equal to its request, so an overrun OOM-kills the container instead of billing past the ceiling. The class is 16 CPUs / 192 GiB / 20,400 s (ceiling $38.91 at list); the runbook's ledger path is an operator placeholder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as draft
October 15, 2026 09:21
Keep both sides where main's ACS multispine staging and the head-to-head stage each added a resource class, runner functions, a ToolSpec and runbook sections: six tools are now registered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max cancelled the head-to-head of the September Route A release on 2026-10-15 and asked for it on the d1386 build instead. The candidate is populace-us-2024-b5db384-8b0c1950b65f-20261024T083103Z (H5 sha256 2d9c8586), the scorer pin moves to main 2cfe304 (policyengine-us 2.2.1, the engine the d1386 manifest names), and the run id becomes route-a-h2h-d844-d1386-r1. The incumbent and the c5e5bf8 register stay as d844 fixed them. The class sizing is unchanged here; it waits on the at-scale memory measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, recycle workers The 11 November local probe (6 slices of chunk 5 on the d1386 frame) peaked at 10.98 GB sampled RSS. Its traced heap stepped up once at slice 3 and then grew 22-45 MB per slice, so the plan recycles each worker after 6 slices. The class grows to 224 GiB (16 workers at about 12 GiB plus 32 GiB for the parent). October's slice times put the run at about 7.7 hours on 16 workers, about $59 non-preemptible. The plan therefore runs preemptible, with a 52,200-second budget across attempts. One attempt held to its 53,400-second timeout costs $37.75 at list price, under the $40 cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The r1 review of d1776a0 found the preemptible plan could not finish (λ≈0.67/h against a 7-8 h run with no resume) and that Modal's debian_slim 3.14 image serves Python 3.14.2, which the scorer's recycling guard refuses. - Plan: non-preemptible, no worker_max_slices, 16 workers. - Class head-to-head: 16 CPU and 288 GiB, both limited; the timeout derives from HEAD_TO_HEAD_COST_CAP_USD at the non-preemptible list price (15,705 s at $40). At $40 the run cannot finish; the cap is Max's call. - Check mode reports the image venv's Python and runs a tool's option preflights, so a plan that sets worker_max_slices runs the scorer's _assert_worker_recycling_supported in the image. - tools/us_head_to_head_projection.py projects the end time and worker memory from the streamed scorer log and answers WAIT, GO or NO-GO; tests check its bounds on simulated pools. - Runbook: launch with --run --spawn under launchd, a watcher that stops the app on NO-GO, and the sizing evidence with its gaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… per log stream Review r2 of dc7836e found the time estimate froze after the candidate's first round (PROJ-1), the sizing was at Mac speed (N1), the first verdict came later than stated (N2), the runbook row was stale (N4) and a worker's ru_maxrss can start at the parent's peak on Linux (PROJ-4). - tools/us_head_to_head_projection.py rebuilds each pool from the scorer's lines under ProcessPoolExecutor's on-demand starts (each result received frees a token; a submit with none free starts a worker and blocks the parent), measures each chunk's seconds per slice, the start cost and the first-slice extra, and replays the rest of the run at every poll. Unmeasured chunks scale from the slowest measured result by local chunk weights; the candidate runs at the incumbent's pace times the prior ratio until its own is measured. NO-GO on the time budget or a stall; memory is reported, not judged. Exit 3 on NO-GO. Several log streams merge, each record counted once across them. - tools/us_head_to_head_watch.sh writes each `modal app logs` stream to its own app-N.log, polls the projection, and stops and confirms on NO-GO. - test_support/microcosm_build/h2h_pool_sim.py simulates both pools; the projection tests check the rebuild against it, the estimate's error, a NO-GO for runs that cannot finish and none for runs that fit, and the model against a real three-worker log of the pinned scorer (fixture). - docs/evidence/us-h2h-speed: one-process slice times, the pool run (starts 42/36/44 s; replayed estimates 0.4% to 10% early), simulated 16-worker grids (fits $40 only at Modal speed within 1-2x the Mac's). - Runbook d844 section and table row rewritten; changelog updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Projection (tools/us_head_to_head_projection.py): - Date a token-taking submit by its own `after receiving` line, as the scorer prints it after future.result() and submit_next() (Sol 1). - Resynchronise the worker-start pointer after an unexplained start, so one missed start no longer misdates every later submit (MODEL-4). - A result taken whose slice line is missing is a log gap: the verdict is WAIT with log_gap, and the stall check still applies (WATCHER-5 / ESTIMATE-7). - Candidate start costs count before its first slice (Sol 2, ESTIMATE-3). - A running slice past its chunk's pace is taken to run as far past it again (Sol 5). - With a measured candidate chunk, unmeasured chunks scale the prior by the incumbent's measured ratio (ESTIMATE-1). - The stall allowance runs from --started-at before any record, so a hang before the first slice is a NO-GO, not WAIT forever (WATCHER-3, Sol 4). Watcher (tools/us_head_to_head_watch.sh): - app_state's live and stopped answers are deliberate exit codes; any other exit or an unexpected list shape is unreadable, never stopped (WATCHER-1, -2, Sol 3). - Help is read whole before matching, so pipefail cannot drop --follow or --yes (WATCHER-4). - A dead stream is restarted within WATCH_STEP seconds; a log gap alarms once; a WAIT past the deadline plus WATCH_WAIT_GRACE counts as no verdict (WATCHER-3, -5). - The start check retries, the projection gets a trial run, and a refused start is written to go-no-go.log and ALARM (WATCHER-6). - Each stream claims the next unused app-N.log under noclobber before it starts (WATCHER-7); the header lists every setting (WATCHER-10). - Tests for each, with a fake modal that reports detached apps. Docs, evidence regeneration and the remaining projection tests follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Toy log from review r3 (CPython 3.14, 12 spawned workers, seed 44): all 12 starts explained after a same-second tie, first tasks within one of the toy's own record, chunk paces within 10% (MODEL-4, MODEL-5). - The scorer prints `after receiving` after its next submit, and opens with 2 x workers submits, checked against its source (NUMBERS-3); the real-log test is narrowed and bounds each slice by its chunk's pace. - Logged candidate starts count before its first slice (Sol 2). - An overdue running slice runs as far past its pace again (Sol 5). - A result taken without its slice line is a log gap: WAIT with log_gap, and the stall check still ends it (WATCHER-5 / ESTIMATE-7). - The candidate's fill carries the incumbent's measured departure (ESTIMATE-1). - A prior off on one chunk moves the first estimate by up to that departure's share of the scoring ahead: the false-NO-GO band (ESTIMATE-5). - The rebuild property cuts while a pool is open (NUMBERS-1), and a seeded uniform draw pins how many runs reach an estimate (NUMBERS-2). Each new test fails with its fix reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Module docstring: submits dated by their receive line, the whole-second resync, log gaps, the candidate's carried fill, starts counted before a first slice, overdue slices, the stall from --started-at, and the false-NO-GO band of a first estimate. - projection_grid.py: --chunk5-artifact candidate slows only the candidate's chunk 5 (review r3, ESTIMATE-6); "fits" is defined as the last slice ending 600 s before the budget does (ESTIMATE-4). - Both saved grids reproduce row for row; three candidate-only grids (chunk 5 at 1.2, 1.3 and 1.5 times) are new. - scripts/mutation_check.py: seventeen seeded faults, the r2 ten and the r3 fixes, each run against the test file with a fresh Hypothesis database (NUMBERS-7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runbook and evidence README now state the r3 estimate as built: memory metrics named, the watcher command pinned to python3.14 and the uv-tools modal, stream replay unknown and slice lines a restart misses surfacing as log_gap, WAIT on a log gap with the stall counted from started-at, the within-chunk slowdown (chunk 5 slices 1-12 averaged 227.4 s, 13-29 287.5 s), the candidate-only chunk-5 cases ($19.87-$30.56) and the first estimate's error band, exit codes 4 and 2, the unreadable states, and the WATCH_* settings. projection_grid.py gains --pre-tool (started-at that many seconds before the tool), and the grids carry it in their headers; rows unchanged. The 300 s pre-tool grid shows one false NO-GO at 1,200 s ($3.06) and no other change. The plan tool's NUMBERS-8 comment names the scorer's 20 GiB per-process guard beside the hard limit. The projection test file is ruff-formatted. scripts/mutation_check.py: 17 of 17 seeded faults caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Date a submit that finds the pool full and no token by its receive line, as a token-taking one already was: the scorer prints `after receiving` straight after either. A parent pause otherwise starts the task early and reads its slice as long (Sol r4, 1). - The runner prints its BUDGET line before every tool start. The projection treats scorer lines without one as streams begun late: never GO, `log_gap` "runner" (the watcher alarms), NO-GO against the plan's budget still holds (Sol r4, 2). - The watcher takes "Stopped at" only as null or a timestamp; any other value is an unreadable list, never a confirmed stop. - Docs: start-cost intervals are rebuilt, not measured blocking; restart latency after a second stream end; the pre-tool exception to "no fitting run got NO-GO". - Mutation check: 19 faults (full-pool dating, late-stream GO). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Watcher: refuse to start unless MODAL_PROFILE and MODAL_ENVIRONMENT are set; read an app missing from a valid list as unreadable, never stopped; refuse a run folder whose streams predate started-at; alarm on stream holes as well as log gaps. - Projection: a pool with lines but no opening line is a log gap (WAIT); report stream_holes; wait on records from before started-at; scale the candidate's carried fill by one pooled ratio; date same-second starts to the earlier submit. - Tests: toy pool checked against its per-process truth (committed as a fixture): start costs, misdated slices and chunk-pace bands; the rebuild property cuts at record times; new watcher and projection cases. - Docs: tie bias, unshared departures, the 600 s tail, extrapolated slice totals, model-dated start intervals, attempt folders and workspace pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…faults Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hole raised first used to set the one gap flag, so a lost line found later (log_gap) never alarmed. Each now alarms once on its own; a new test runs a hole, then a lost candidate opening line, and expects both alarms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r5: a worker start lost in a stretch a restarted stream did not replay left the rebuild a worker short, so it dated three slices to end before they start and gave a false NO-GO (14,264 s intact, 15,857 s damaged, against a 14,505 s budget). Such a rebuild is now a log gap: WAIT, the watcher's gap alarm, and the stall check still runs. No intact simulated log has such slices (600 random pools, 2,243 d844-shaped polls, the three logs in the tests). Also from r5: only a BUDGET: line satisfies the late-stream check (an ATTEMPTS: line can carry a longer budget); the watcher confirms a stop only on a timestamped "Stopped at"; the straggler error bound is stated as the slice's real time less its pace; the docs name the all-scored GO as the one GO without a BUDGET line. Two mutation faults added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restarted log stream that does not replay can lose lines before any log gap shows. A lost slice line read as a straggler and a lost start line as one start taking twice as long: both false NO-GOs. After a hole the estimate now waits while a slice that began before the hole ended has no line, and leaves out of the start cost any start whose span (its submit to its line) holds a hole, since starts come one at a time and a lost start line merges two of them. A newest stream with nothing in it yet counts as a hole ending now. Rebuild notes that no intact log gives (a submit with no token and no start, starts it cannot explain) are a log gap, as slices ending before their start already were. Streams are read in numeric order, app-2 before app-10. scripts/lost_lines.py cuts stretches out of simulated d844 and random logs: no GO turned NO-GO in 1,222 d844 cuts or 12,765 random cuts (judged against a budget each run fits and one it overruns); one random cut turned NO-GO into GO, a missed early stop that becomes a log gap at the next poll. A restart that loses nothing changes the verdict at one poll at most across the d844 grid. Six new mutation faults; three old ones re-pointed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…overlaps Review r6 (in-session and GPT-6.1 Sol) found false NO-GOs the hole guards missed. The estimate now: - waits, with the stall check held, while the newest stream is restarted and empty, whether or not a slice is running (SF-2); - waits while a worker start that began before a hole ended has no line (hole_start_suspects), and counts slice suspects from the hole's end; - holds a NO-GO after a hole while the estimate is past the line by less than hole_margin_seconds, the late reading a lost start line can cause (SF-4); - takes the start cost as the lesser of the mean over every start and over those clear of holes, so a restart that lost nothing never reads later than the intact log (Sol S1); - counts a record in the second a restart begins once per stream showing it most, and reports that second in restart_overlaps, so a gap it causes is not read as proven loss (Sol S3; summing turned a replay from inside that second into extra worker starts and false NO-GOs). The watcher restarts an ended stream before it polls, defers a NO-GO from a stream that started under WATCH_SETTLE seconds ago or has not grown since the last poll (restarting it once), and refuses an app created before this attempt's started-at (SF-1, SF-6, SF-7). Tests pin Sol's three cases, the review's scenarios, a ten-stream CLI read and a per-poll intact-log property; the mutation check gains 13 faults and an --only filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r6 found two ways the hole margin failed. A lost start line makes the estimate read late by more than a start when start costs vary (a lost 78 s start among 119 to 328 s ones read 460 s late), and leaving a start whose span holds a hole out of the start cost read a lossless restart late (Sol S1). Both come from guessing what a hole lost. Each worker prints its start line once, so a pool that has logged a start for every worker has lost none. After a hole that could hold a pool's start lines (it ends once the pool opened and begins before the pool's release line), a time NO-GO now waits until that pool logs a start per worker (starts_unconfirmed); a stall still stops the run. A released pool counts only while its starts still set the candidate's start cost. This replaces the hole margin, the lesser-mean start cost, the starts-across-holes exclusion and the start-under-way suspects; the start cost is the plain mean of logged starts again. The four tests that pinned the removed rules now pin the hold, and the watcher's hole alarm says what it does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new test pins that the hold reaches only a pool whose start lines a hole could cut (one ending once the pool opened and beginning before its release line), that a released incumbent is held only until the candidate's first start, and that a stall still stops a held run. The mutation check drops the six faults for the removed margin code, follows the start cost's new call, and adds eight faults for the hold and the plain-mean start cost: 46 in all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
verdict_at now holds a time NO-GO as WAIT while a pool's starts are unconfirmed, the excess is reported where the hold is off and, apart, where it holds, and each line counts the intact log's own NO-GO at the tight budget, which is the estimate's error rather than the damage's. --constant-starts keeps each random pool's first start cost so the two can be told apart; each section prints its elapsed time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eams Answers review r6's runbook items. The app id comes from this attempt's launch log, since every stage is microcosm-us-stage and the watcher only refuses apps created before started-at (SF-6). Every Modal command of an attempt pins the workspace, and the watcher's lsubmit line names the attempt folder literally (N11). The hole paragraph describes the starts-logged hold and restart overlaps; each poll restarts a dead stream first (SF-1); a NO-GO read from a stream that has not grown, or that started under WATCH_SETTLE seconds ago, waits a poll (SF-7). Only an attempt ended from outside the runner writes no receipt (Sol nit 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lost-lines search with one start cost per pool found a false NO-GO the starts hold let through: a lost incumbent start also misplaces that worker's slices in the rebuild, so the incumbent's measured slice time and first-slice extra read wrong (a 62 s chunk read as 127.3 s), and the candidate, paced from them, read 495 s late after the hold had lifted at its first start line. The estimate now reports whether it read any of the incumbent's timing (start cost, pace, first-slice extra, or departures carried into a chunk the candidate has not measured), and a released incumbent short of starts after a hole stays unconfirmed while it does. Two regression tests pin the single-chunk case (pool 94 of the search) and a two-chunk case where only the carried fill reads the incumbent. The hold-scope test now releases when the candidate paces from its own chunk; the seeded faults cover the new rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lost-lines.txt now holds the five sections of the search run on baf2052 (cut-d844, cut-random with varied and with constant start costs, lossless, boundary), byte for byte, with a header giving each command. The README's lost-lines section and the runbook's verdict bullet now describe that search: 0 false NO-GOs in every section; no unheld poll reads later than the intact log in the cut searches; a restart that loses nothing delays an overrunning run's NO-GO until the incumbent's 16th start (up to 8,400 s, $21.39, inferred cause); and a restart that splits one second's identical lines can read 24 s late with the hold off. The runbook bullet also says a time NO-GO can end the run again once the hold on a lost incumbent start lifts, if no log gap shows. lost_lines.py: docstring only (the double-cut sample size); the search ran on the script as it was at baf2052. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stream can hang after its file grew in the same poll interval, so growth no longer clears the hang check (Sol r7 SF1). The first NO-GO after a GO, a WAIT or the watcher's start now starts a new stream and leaves the decision to the next poll, and a NO-GO that comes through that restart stops the app. The old stream runs beside the new one until the next poll has projected both, so a new stream that doesn't replay leaves no hole whose holds would re-arm the restart (r8 check, simulated). The README and runbook now scope the lost-lines search to what it searched: a restart that loses part of one second, with neighbours sharing its stamp, forms no hole (SF2). The runbook adds the uneven-start case and the watcher's own stop rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sol's r8 review of 8ea47a2 asked for changes (0 blocking, 3 should-fix, 1 nit). This commit answers all four in the head-to-head watcher, its tests and the docs. - Should-fix 1 (a restart attempt could authorize a stop on the same stale log): after a restart, the watcher stops the app on a NO-GO only when a stream it started since that restart holds a scorer record with a stamp after both the previous poll and the stream's own start plus WATCH_READY. It reads the records with the projection's own _records parser, refuses to start when its Python cannot read them, and adds no stream for a restart that cannot create its file. Otherwise it raises ALARM once per run of NO-GOs, restarts the stream up to WATCH_RECONNECTS (2) more times and leaves the run to the runner's budget. A stall, or no scorer record an hour after the launch, now raises ALARM instead of stopping the app, so a run whose scorer hangs runs to the runner's 14,505 s budget ($36.94) unless someone acts on the alarm. - Should-fix 2 (a log client that ignores SIGTERM could freeze the watcher in wait): the watcher sends each stream it ends TERM, then KILL if it still runs WATCH_KILL_GRACE (5) seconds later, and reaps it, after each poll's projection, in reconnects and in the exit trap. - Should-fix 3 (a refused or empty replacement could defeat early stopping): a new stream gets WATCH_READY (30) seconds to show a record or keep running. One that ends beside the old stream makes the watcher reconnect one stream at a time, up to WATCH_RECONNECTS tries, then alarm. When the newest stream delivers nothing for two polls, the watcher replaces it once a poll, up to WATCH_RECONNECTS times, then alarms. A WAIT that comes while the newest stream has delivered nothing no longer re-arms the restart. - Nit (the one-poll delay overstated for exit 4): the runbook scopes the 600 s and $1.53 to stops on a NO-GO. A check of the first answer added the previous-poll floor and each stream's own start to the record check, and made the watcher check with ps that a pid is still its own running child before it signals it. The projection's held reason and docstring, a plan-tool comment and three projection-test comments no longer say a stall stops the run; the projection's logic, the scripts and lost-lines.txt do not change. The watcher's test file drops four tests (five cases) that checked the old rules and adds 17 (18 cases), for 68 cases. Through the review folder's harness on this tree, the watcher's test file passed 68 of 68 in two slices and the projection's passed 59 of 59. Against 8ea47a2's watcher, all 18 new cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in main's sixth registered Modal tool (US_PUF_SUPPORT_BASE_2023_2025) beside this branch's US_RELEASE_HEAD_TO_HEAD. The next commit moves the registered-tool count, the runbook intro and its tool list to seven. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merging main (7ff5a26) into this branch put main's new us-puf-support-base-income-2023-2025 registration beside this branch's us-release-head-to-head. Both sides had moved the registered-tool count from five to six, so the merge kept "six" while TOOLS holds seven. CI's engine-free jobs, which test 877977f merged into main, failed test_the_runbook_names_every_registered_tool with assert 7 == 6 (review r9, blocking 3). - The runbook's introduction now counts seven registered tools; its list already named all seven. - The test asserts seven and now also checks that the introduction's hyphenated backticked names form the same set as the registered tool names, so a stale or missing name fails even when the count matches. Run locally on this tree, one test file at a time, test_us_modal_stage_puf_support_base.py passed 172 of 172, the watcher's test file 68 of 68 in two slices and the projection's 59 of 59. test_us_modal_stage_plan_tool.py passed 283 of 284. Its one failure, test_a_release_plan_cannot_pass_the_county_facts, imports Microcosm in a subprocess from the local environment's older checkout, which lacks microcosm.build.program_telemetry; CI's engine-free jobs passed it on 877977f merged into main. The previous count test fails on the merged tree with assert 7 == 6. The full engine-free suites did not run locally. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Brings in #1694, the hand-off guard, which edits the Modal stage runbook and test_us_modal_stage_puf_support_base.py beside this branch's count fix. No conflict; the registered-tool count stays seven. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This registers the US head-to-head scorer as a Modal stage, so the d844 comparison (the d1386 Route A build against
populace-us-2024-spm-20260915) can run on Modal workers instead of the shared Mac. It adds a watcher that stops a run early when its own log shows it will not finish inside its budget.us-release-head-to-head: one stage,score, fortools/score_us_release_head_to_head.py.6496cc43…, the candidate H52d9c8586…(the d1386 buildpopulace-us-2024-b5db384-8b0c1950b65f-20261024T083103Z), and the ledger feedconsumer_facts_us_c5e5bf8.jsonlb8543739…, the register d844 named.populace_us_2024.h5, in separate content-addressed directories, because the scorer's loader needs the.h5suffix.workers,maximum_microsim_batch_sizeandworker_max_slices.head-to-head: 16 CPUs and 288 GiB, each limited at the request, no retries.timeout_for_cap(HEAD_TO_HEAD_COST_CAP_USD, …)sets the timeout at the non-preemptible list price of $0.00254688/s: 15,705 s at $40.docs/us-modal-stage-route-a-h2h-plan.json: pinned tomainat2cfe3043d(carries Score head-to-head household slices in parallel with --workers, bit-identical to sequential #1171). 16 workers, batch size 2,000, one numerical thread per worker, non-preemptible, no worker recycling,max_wall_seconds14,505.tools/us_head_to_head_projection.py(rewritten).ProcessPoolExecutoron 3.9 and later, each result received frees a token. A submit that finds none free starts a worker; the model takes each start to hold up the parent's next submit until the worker's first line.after receivingline, which the scorer prints straight after the submit. It puts a start that whole-second stamps hide back on the submit that made it, with a note, and gives a start in its submit's own second to that submit.WAIT,GOorNO-GO, and givesNO-GO(exit 3) when the estimated end passes the budget or the log stalls. A result taken without its slice line, a pool without its opening line, a rebuild that starts slices after they end, a submit or start the rebuild cannot explain, or scorer lines without the runner'sBUDGET:line count as log gaps:WAITwithlog_gap. Without theBUDGET:line it givesGOonly for a candidate that has scored every slice, and anATTEMPTS:line does not stand in for it. It reports a stretch no stream covers asstream_holes, and the rules under "How the verdict holds through lost log lines" hold the verdict through it. Records from beforestarted-athold the verdict atWAIT. The stall allowance runs fromstarted-atuntil the first record. It reports memory but never judges it.app-2.logbeforeapp-10.log), and counts a record that several streams show as many times as the stream showing it most.tools/us_head_to_head_watch.sh(new, run under launchd).modal app logsstream to its ownapp-N.log, claimed under noclobber, and restarts a stream that ends while the app runs: withinWATCH_STEPseconds the first time between two polls, and otherwise at the next poll, which restarts it before it projects.MODAL_PROFILEorMODAL_ENVIRONMENTunset, when its run folder holds an earlier attempt's streams, when Modal created the app more thanWATCH_CREATED_SLACK(60) seconds beforestarted-at, or when the app list gives no readable creation time.NO-GO, it waits for the next poll when the stream it read started underWATCH_SETTLE(120) seconds ago. A stream can hang after its file grew, so on the firstNO-GOafter aGOorWAIT, or after the watcher starts, it starts a new stream whatever the old one's growth and waits for the next poll. The old stream runs on beside the new one until that poll has projected both, so a restart of a stream that had not hung leaves no hole even when the new one does not replay.NO-GOstops the app only when a stream the watcher started at or after the last restart holds a scorer record with a stamp after both the previous poll (the restart's own, or a later one) and the stream's own start, counted fromWATCH_READY(30) seconds after its launch; the watcher reads records with the projection's own parser. With the previous-poll floor, a stream that delivered after the restart but has since ended or hung cannot vouch for a later poll. Review r9 found that what a new stream only replays can still pass when the stream connects late or a container clock runs ahead (inferred), and that a line the projection never read can pass (reproduced; r9 section). A restart that could not create its file, or a new stream that replays only the stale log, shows no such record, and neither does a scorer that has gone quiet; the log cannot tell those apart. So on a stall, or a run with no scorer record an hour after the launch, the watcher polling every 600 s finds no such record while the scorer stays quiet (inferred): it raisesALARMonce per run ofNO-GOs, restarts the stream up toWATCH_RECONNECTS(2) more times, and otherwise leaves the run to the runner's budget (before the tool starts, the class timeout). When two polls come more than an hour apart, though, one line that a new stream shows after the restart's poll can confirm the next stallNO-GO, and the watcher then stops the app (r9 section, should-fix 4, reproduced). Every stop on aNO-GOcomes at least a poll (600 s, about $1.53) after the firstNO-GO; the stop after three polls with no usable verdict (below) needs noNO-GO. The nextGOorWAITre-arms the restarts, except aWAITthat comes while the newest stream has delivered nothing (newest_stream_empty); a poll with no verdict does not. So aGOorWAITbetweenNO-GOs delays the stop further (r7 section). The watcher confirms each stop inmodal app list. An unreadable or unexpected list, an app missing from the list, or a "Stopped at" other than null or a timestamp in the form the Modal CLI prints never counts as stopped.WATCH_READY(30) seconds to show it works: a scorer record, or still running. One that ends with no record while the old stream runs, as a refused second reader would, makes the watcher end the old stream and reconnect one stream at a time, up toWATCH_RECONNECTStries, then raiseALARM. When the projection finds the newest stream empty at two polls in a row, the watcher replaces it, once a poll, up toWATCH_RECONNECTStimes, then raisesALARM. Each stream it ends gets TERM, then KILL if it still runsWATCH_KILL_GRACE(5) seconds later, and the watcher reaps it, at a poll and when the watcher exits mid-overlap. Before it signals a pid, it checks withpsthat the pid is still its own running child, so a pid that an ended stream has freed gets no signal (whenpscannot answer,kill -0decides).WAITlong past the deadline included) it alarms and stops the app. An unconfirmed stop raisesALARM; a log gap raises it once, and a stream hole once more, on its own.go-no-go.logandALARM.--followtomodal app logsand--yestomodal app stopwhen that command's whole--helptext lists the flag (Modal CLI 1.4 and later need--follow, 1.4.2 and later--yes).docs/evidence/us-h2h-speed/: slice times in one process, a three-worker pool run of the pinned scorer, simulated 16-worker runs (below), the lost-lines search (scripts/lost_lines.py, outputlost-lines.txt) andscripts/mutation_check.py, the seeded-fault check.Measured on the Mac
ru_maxrss, the metric of the scorer's lines and guardru_maxrss)The pool run's parent released the incumbent pool at 20:14:04Z. Its chunk 5 slowed within itself (slices 1–12 averaged 227.4 s and 13–29 averaged 287.5 s, for a reason nobody has found), so the 2,400 s poll read furthest from the end. The candidate's pool never opened: a disk-management session moved the candidate H5's folder to an archive drive at 20:04Z under d3015, while the incumbent's pool ran, and the scorer opens the candidate only after that pool. We did not repeat the run.
Simulated (not measured): when $40 fits
The simulated 16-worker runs use this PR's pool simulator. Each slice takes f times its Mac time, where f is Modal's time over the Mac's, and each worker start costs T. The watcher's polls run against the 14,505 s budget (
projection-grid.txt). A run fits when its last slice ends at least 600 s before the budget does, the estimate's allowance for the comparison and scorecards:NO-GOwithin 600–3,000 s ($1.53–$7.64), and no run that fits gets one.The estimate gives
NO-GOat these times; the watcher stops the app a poll later at the earliest (Summary), about $1.53 more.Each variant has its own file in the evidence folder:
GOuntil the estimate measures the incumbent's chunk 5, thenNO-GOby 5,400 s ($13.75).NO-GOat 7,800–12,000 s ($19.87–$30.56); the rest get it within 600–3,000 s, as in the main grid.started-at300 s before the tool (container start and staging come out of the watcher's budget, not the runner's): the f = 1, T = 300 s run, which fits the runner's budget, gets a falseNO-GOat 1,200 s ($3.06). No other row changes.Only the last variant gives a fitting run a
NO-GO. Every grid scales all chunks by one factor, though. When the prior misses on the chunk the first estimate measures, that estimate moves by the departure's share of the scoring ahead: with chunk 1 at 1.1× the prior, late by up to a tenth of it. The watcher can stop a run that fits with less spare than that, and nothing undoes a stop.test_a_prior_off_on_one_chunk_moves_the_first_estimate_by_its_sharepins this band.Every grid also gives each run one start cost. With uneven start costs the estimate can give a fitting run a
NO-GOwith no line lost. In the lost-lines search below, with 265 s to spare, the undamaged log givesNO-GOat 7,102 of 35,336 polls in a d844 shape whose first worker start costs 320 s and the rest 200 s. In random pools with varied start costs it givesNO-GOat 54,346 of 1,668,873 polls. With each random pool's first start cost for every start, it gives none (lost-lines.txt). For example, in that d844 shape against an 11,574 s budget and polled every 5 s, the whole log givesNO-GOfrom +670 to +1,780 s (11,839 s at +1,200 s) and again from +3,975 to +4,770 s (the README's lost-lines section). Polling every 600 s, the watcher stops the run between +1,270 and +1,780 s at 103 of 120 poll phases (inferred from the watcher's rule, assuming the scorer prints a line after both the poll before and the new stream's start;runs/r8-fix/fix-evidence/uneven_phases.pyin the review folder). It stops the run when it polls at +900 and +1,500 s, say, and not when it polls at +1,200 and +1,800 s.The simulator and the estimate implement one model, so the grids only show what the watcher does when Modal behaves like that model. The runbook's measured Modal slowdown for single-core materialize work, 3.3–7.6×, would not fit.
How the verdict holds through lost log lines
The watcher restarts a
modal app logsstream that ends, and before it acts on aNO-GOit starts a new stream beside one that may have hung. Nobody has tried whether a fresh stream replays what the app printed. One that does not, when the watcher starts it after the last stream ended or hung, leaves a stretch no stream covers (a hole), and a line lost in it can make the estimate read late, toward a falseNO-GO. At this head, five rules and the watcher's checks hold the verdict through a hole:hole_suspects). A slice still running clears it when its line arrives; a lost one becomes a log gap once the parent takes its result, and the gap holdsWAITfor the rest of the run.NO-GOwaits asWAITuntil that pool has logged a start for every worker (starts_unconfirmed, the starts hold). A stall still givesNO-GO. At polls 600 s apart, while the scorer stays quiet, the watcher raisesALARMon it (inferred) and leaves the run to the runner's budget (r8 section), but when two polls come more than an hour apart it can stop the app (r9 section). The estimate takes the plain mean of the logged starts as the start cost, so a restart that loses nothing gives the intact log's estimate.reads_incumbentin the code, that no report shows). For a released incumbent short of starts, the hold lasts until the candidate has its own start cost, first-slice extra and pace, and no chunk the candidate has not measured carries the incumbent's departures from the prior.newest_stream_empty).restart_overlaps, and the gap reason and the watcher's alarm say a gap may come from it and not from lost lines. One that repeats none, having lost lines from that second, raises no flag (below).NO-GOthat the projection took from a stale log (r9 section).A restart that loses only part of one second's lines leaves no hole, so none of the five rules applies to it; the watcher's restart before a stop still does (below).
What baf2052 changed
The starts hold as of aa2addb let go of a released incumbent at the candidate's first start line. The lost-lines search with one start cost per random pool found false
NO-GOs that it let through: 202 polls at the fitting budget, all in seed 3's pool 94, where the unheld estimate read up to 495.2 s later than the intact log's (review-folder runr7-fix/search-aa2addb, not committed). In that pool a restart at +634 s that does not replay loses the incumbent's first worker-start line. The rebuild places that worker's slices on another and reads the incumbent's 62 s chunk as 127.3 s, and the candidate, paced from it, reads 495 s late at +1,385 s, after the hold had let go at +1,184 s. baf2052 adds the rule above, which keeps the hold while the candidate reads the incumbent's timing. At baf2052 the search finds 0 falseNO-GOs and 0 s of unheld excess in the same pools, and in the pools with varied start costs, which had no falseNO-GObefore, the unheld excess drops from 21.6 s (pool 69) to 0 s. The d844, lossless and boundary sections print the same results at both commits.test_a_lost_incumbent_start_holds_while_the_candidate_paces_from_itpins pool 94;test_the_hold_lasts_while_the_candidate_fill_carries_the_incumbentpins a two-chunk pool in which only the candidate's fill for its second chunk reads the incumbent and no log gap shows.The lost-lines search at baf2052 (simulated, not measurements)
scripts/lost_lines.pydamages simulated logs in some of the ways a restarted stream can: it removes stretches of whole seconds, or splits the log there with nothing lost, and it restarts a stream inside one second without losing a line. It reads the pieces as the watcher's streams and compares each poll's verdict and estimate with the intact log's. The cut and boundary sections judge each poll at two budgets per shape: the true end plus 265 s (d844's spare at f = 1, T = 300 s), where the run fits, and the true end less 600 s, where it overruns. The lossless section polls against the plan's 14,505 s and splits shapes by whether they fit it. The search takes the damaged estimate less the intact one as the excess, where both logs have an estimate, and reports it apart for held polls (where the starts hold applies) and unheld ones. The table counts polls;lost-lines.txtholds the output, with each section's command and totals.NO-GOs; largest unheld excess 0 s; held polls up to 3,300 s lateGOwhere the intact log givesNO-GO(missed stops) at 3,004 of 45,161 polls at f = 3, T = 100 s and 816 of 30,199 at f = 1, T = 40 sNO-GOs and 0 s largest unheld excess in both; held polls up to 5,039 and 5,664 s lateNO-GOin the 10 shapes that fit; 11 to 66 polls per shape turnGOintoWAITNO-GO; per shape, the latest firstNO-GOcomes 600 s (f = 3, T = 40 or 100 s) to 8,400 s (T = 600 s) after the intact log's ($1.53 to $21.39)NO-GOs but reads up to 24 s late in the random pools (0 s in both d844 shapes), and 245 of 6,685 polls turnGOintoWAITwith no estimateNO-GOuntil the incumbent's 16th start. The poll counts match that in all 18 overrunning shapes.after receivinglines when a split falls between them (inferred; the search does not name the pool). A run whose intact estimate had less spare than that at such a poll would get a falseNO-GO.validate).NO-GOcomes late anyway (only the candidate's chunk 5 slower), or Modal's own streams.Losing part of one second's lines (simulated; the search does not cover it)
The projection marks a hole only where a stream's first record comes after the second the earlier streams reached (
_stream_holescomparesfirst > reach). So a restart that loses some of one second's lines, while lines on both sides of the loss share that second, leaves no hole and raises no hold, and a lost slice line reads as a straggler until the parent's receive line shows the gap. The cut search removes whole seconds and the boundary arms lose no lines, so neither covers this.For example, take seed 3's pool 7 (
random_pools(3)inscripts/lost_lines.py: 11 workers, three chunks, start costs of 221, 325 and 369 s), with its own slice times as the prior. A restart in the 08:34:57Z second (+2,097 s) that loses the incumbent's chunk 2 slice 3/7 keeps its neighbours, slices 2/7 and 4/7, which share that stamp. Polled at +2,400 s, the intact log estimates 11,535.2 s and the damaged one 11,718.2 s, with no flag; the last slice plus the 600 s allowance ends at 11,678 s. Against an 11,700 s budget the damaged log givesNO-GOthere and the intact oneGO, though the intact log givesNO-GOitself from +2,895 to +3,110 s (up to 11,704.0 s, likely from its uneven start costs; inferred). Against 11,710 s, polled every 5 s through the candidate's last slice, only the damaged log givesNO-GO, from +2,255 to +2,460 s; from +2,495 s its receive line shows the gap and the log gap holdsWAIT. The README gives the pool's whole draw, and the review folder'sruns/r8-fix/repro/holds the scripts.The projection gives that false
NO-GOfor 205 s, less than a poll, and the watcher stops the app only on aNO-GOthat comes through a restart a poll later, with a line the scorer printed after both the poll before and the new stream's start. We applied that rule to the projection's verdicts (runs/r8-fix/fix-evidence/pool7_next_poll.pyin the review folder; inferred, not a run of the watcher, and assuming the new stream shows such a line). At 11,710 s the watcher does not stop the run: a restart that replays puts the lost line back (GOat all 42 restart times on the 5 s grid), and one that does not leaves the log gap (WAIT). At 11,700 s a restart that replays meets the intact log's ownNO-GOat 34 of those 42 times, and the watcher stops the run. The watcher could stop a run that fits on a falseNO-GOthat lasts longer than a poll; we haven't searched for one, and #1680 tracks it.Review r9 (877977f), open
Review r9 (3 blocking, 5 should-fix, 1 nit) returned REQUEST_CHANGES. Its sandbox could not run the harness or reach GitHub, so the reviewer ran the projection on logs in memory and inferred each watcher action from the shell code. We reproduced blocking 1 and should-fix 4 with the watcher, through tests on a scratch copy of its test file that drive it against the fake
modalcommand (runs/r10-triage/scratch/in the review folder, not committed). Each of the three new tests asserts the stop the review predicts, so its pass means we reproduced that stop; all three passed, with the committed control below (runs/r10-triage/logs/repro_b1_sf4_run2.txt).80532c73bbfixes blocking 3 and patches nothing else: d3034 decides whether the watcher keeps the stop that most of the other eight concern.NO-GO. The watcher reads records for its freshness check after the projection has read the logs, and only the projection leaves out stamps after the time it judges atNO-GOat 16,125 s. The second stream then caught up to +12,600 s before the freshness read, and the watcher stopped the app (exit 3); the caught-up log givesGOat 14,240 s. Lines that carry stamps after the poll's own time, in the log before the projection ran, gave the same stop.test_a_restart_that_shows_only_the_stale_log_does_not_stop_the_app, which gives the watcher the same streams up to the +12,600 s poll without the late lines, passed: no stopWATCH_READY(30) seconds to connect or a container clock runs ahead of the Mac's, and a reader that delivered and ended within the interval still countsmainand this branch each registered a sixth Modal tool, so the merge registers seven where the runbook and a test said six. CI's engine-free jobs failed that test on 877977f80532c73bb(Tests)NO-GOcan stop the app. With polls more than an hour apart, one scorer line that a new stream shows after the restart's poll confirms the next stallNO-GOBUDGET:line, so the +4,000 s poll gave a stallNO-GOand the watcher started a new stream. That stream showed one line at +4,100 s and then nothing; the +8,000 s poll gave a stallNO-GOagain, and the watcher stopped the app (exit 3) with noALARMWAIT(hole_suspects) that re-arms the restart, and a reader that exits just before every poll keeps deferring the stop underWATCH_SETTLEwith no alarmpsgives no answer,kill -0lets the watcher signal an unrelated process that took a reaped reader's pidWATCH_READYout of the timing rule, and one "review r9" tag beside the 16,125 s example should cite r8The review's table of earlier findings rates the answers to r8's should-fix 1 and 3 partial, through blocking 1 and 2 and should-fix 5 and 6. It also notes that a projection test's name,
test_the_hold_reaches_pools_a_hole_could_cut_and_a_stall_still_stops, still speaks of a stall stopping the run.Review r8 (8ea47a2), answered
Sol's review (0 blocking, 3 should-fix, 1 nit) returned REQUEST_CHANGES.
app-2.log, or a new stream that replays only the frozen prefix and hangs, let the +12,600 sNO-GO(16,125 s, where the whole log givesGOat 14,240 s) stop a run that fitsNO-GOstops the app only when one of those streams holds a scorer record with a stamp after both the previous poll and that stream's own start (counted fromWATCH_READYseconds after its launch, on the projection's clock). The watcher reads the records with the projection's own parser (_records), and a restart that cannot create its file adds no stream. Otherwise it raisesALARMonce per run ofNO-GOs, restarts again up toWATCH_RECONNECTStimes, then leaves the run to the runner's budget. A stalled scorer reads the same as a stream stuck on a stale prefix, so a stall, or a run with no scorer record an hour after launch, now raisesALARMinstead of stopping the app (cost below), though it can still stop the app when two polls come more than an hour apart (r9 section, should-fix 4). A new stream that delivers one fresh line and then hangs, or a delay every stream shares, can still give a staleNO-GO(inferred), and review r9 found more ways (r9 section). Tests:test_a_restart_that_cannot_create_its_file_does_not_stop_the_app,test_a_restart_that_shows_only_the_stale_log_does_not_stop_the_app(replacestest_a_no_go_that_comes_through_the_restart_stops_the_app),test_a_hung_stream_is_restarted_and_a_stall_it_still_shows_stops_nothingandtest_a_hang_before_any_scorer_record_alarms_and_stops_nothing. The tests of the stop itself now use a d844 run that overruns, whose restarted stream carries records past the restartwaitbefore it reaches the stopWATCH_KILL_GRACE(5) seconds, sends KILL if the stream still runs, and reaps it; a zombie counts as ended. The same path runs after each poll's projection, in reconnects and in the exit trap.test_a_stream_that_ignores_term_is_killed_and_reaped_before_the_stopchecks that stream 1 has exited beforeapp stopruns and that its pid has gone;test_a_watcher_that_ends_during_the_overlap_ends_the_replaced_streamchecks that the watcher sent KILL and that the pid has gone after it exits.test_a_go_after_the_restart_rearms_it_for_the_next_no_gonow checks which streams run at each projectionWAIT, which re-armed the restartWATCH_READY(30) seconds to show it works: a scorer record, or still running. One that ends with no record while the old one runs makes the watcher end the old one and reconnect one stream at a time, up toWATCH_RECONNECTStries, then raiseALARM. When the projection finds the newest stream empty at two polls in a row, the watcher replaces it, once a poll, up toWATCH_RECONNECTStimes, then raisesALARM. AWAITthat comes while the newest stream has delivered nothing no longer re-arms the restart, so when Modal serves one reader at a time, the watcher falls back to one new stream, and the nextNO-GOstops the app (inferred). If Modal both refuses a second reader and does not replay, each reconnect leaves a hole whoseWAITre-arms the restart, so only the runner's budget may end a run that overruns (inferred). Tests:test_a_refused_second_stream_falls_back_to_one_stream_and_stops_a_poll_later,test_a_refused_restart_beside_the_replaced_stream_falls_back_to_one_stream,test_streams_that_are_always_refused_raise_an_alarm_and_stop_nothing,test_a_new_stream_that_runs_but_prints_nothing_is_replaced_then_alarmed,test_a_silent_second_reader_is_replaced_alone_and_the_next_no_go_stopsandtest_a_wait_on_an_empty_new_stream_does_not_rearm_the_restart(replacestest_a_wait_after_the_restart_rearms_it_for_the_next_no_go). Nobody has tried Modal's own behaviour with two readersNO-GONO-GO, and say the stop after three polls with no usable verdict needs noneAt polls 600 s apart, the watcher no longer stops a run whose scorer hangs (inferred); when two polls come more than an hour apart it still can (r9 section, should-fix 4). It raises
ALARM, and the run goes on to the runner's 14,505 s budget ($36.94 at the class's list price) unless someone acts on the alarm; a hang before the tool starts goes on to the class timeout (15,705 s, $40.00). At 8ea47a2 the watcher stopped such a run a poll after the projection's stallNO-GO, which comes once no scorer record has arrived for an hour (or four times the slowest slice, if longer). On a timeNO-GOthat a fresh record confirms, the watcher still stops the app.A check of that answer before commit found that the watcher judged streams it started after the restart (the empty-stream replacement, a mid-interval restart, a reconnect) against the restart's poll. Records such a stream had only replayed, or a stream that had since ended had delivered, could then confirm a
NO-GOthat the projection read from a log a poll or more old. 877977f judges each stream against both the previous poll and its own start.test_a_replacement_that_only_replays_does_not_stop_the_app,test_a_stream_that_delivered_then_ended_cannot_vouch_for_a_later_pollandtest_records_from_before_a_new_stream_could_connect_do_not_confirmpin this. Without the previous-poll floor, only the second of the three fails, and with each stream's start set to 0, only the third (seeded faults;runs/r9-fix/logs/patch_run7_mutant_nolastpoll.txtandpatch_run7_mutant_nosinceat.txtin the review folder). The check also found that the watcher could send KILL to a pid that an ended stream had freed; it now checks withpsthat a pid is still its own running child before it signals it (test_ending_a_stream_signals_only_the_watchers_own_child). In their first run, two of this round's tests failed on their own expectations, not on the watcher: one let the fake's app list confirm a stop it meant to fail, and one pinned a poll count that depends on timing. We corrected both tests. The projection's held reason and docstring, a plan-tool comment and three projection-test comments no longer say a stall stops the run, though a projection test's name still does, and a stall can still stop the run when two polls come more than an hour apart (r9 section).The watcher also refuses to start when its interpreter cannot run the record check (
test_a_python_that_cannot_read_records_is_refused_at_the_start), since every stop on aNO-GOnow rests on it. On the tree committed as 877977f, the watcher's test file passed 68 of 68 and the projection's 59 of 59 (Tests). Against 8ea47a2's watcher, all 18 new test cases fail (runs/r9-fix/logs/patch_run8_r9tests_on_8ea_group1.txtandpatch_run9_r9tests_on_8ea_group2.txt);test_a_go_after_the_restart_rearms_it_for_the_next_no_go, which ran with them, passes there, since its new assertions check for a fault 8ea47a2 does not have. Against this round's watcher before the check's fixes, the four tests the check added all fail (patch_run5_new_tests_on_r9pre.txtfor the pid test andpatch_run6_new_tests_on_r9pre.txtfor the other three).Review r7 (00b0ee6), answered
Sol's review (0 blocking, 2 should-fix, 1 nit) returned REQUEST_CHANGES. We rate should-fix 1 P1, should-fix 2 P1 only while the docs claimed to cover it, and the nit P3.
NO-GOat the +12,000 s poll, estimating 14,925 s against the 14,505 s budget. The whole log givesGOat 14,240 s there, and the watcher stopped the runNO-GOafter aGOorWAIT, or after the watcher starts: it starts a new stream whatever the old one's growth and leaves the verdict to the next poll. The old stream runs on beside the new one until that poll has projected both, then ends, so a restart of a stream that had not hung leaves no hole even when the new one does not replay. ANO-GOthat comes through the restart stops the app. One latch allows one restart per run ofNO-GOs; aGOorWAITre-arms it, and a poll with no verdict does not. Every stop on aNO-GOnow comes at least a poll (600 s, about $1.53) after the firstNO-GO, and later when aGOorWAITfalls betweenNO-GOs (checks below); the stop after three polls with no usable verdict needs noNO-GO. Since 877977f, a stop on such aNO-GOalso needs a line the scorer printed after both the poll before and the new stream's start (r8 section). Teststest_a_stream_that_hangs_after_growing_is_restarted_before_a_no_go(the review's case:GO,NO-GO, restart,GO, no stop),test_a_no_go_that_comes_through_the_restart_stops_the_app,test_a_go_after_the_restart_rearms_it_for_the_next_no_go,test_a_restart_that_does_not_replay_leaves_no_hole_and_stops_a_poll_later,test_a_wait_after_the_restart_rearms_it_for_the_next_no_goandtest_a_failed_poll_after_the_restart_does_not_rearm_it; each fails against 00b0ee6's watcher (Tests), and 877977f replaces the second and the fifth (r8 section)NO-GOwith no damage flaglost_lines.py's ("the ways a restart can"). 8ea47a2 left both byte for byte, so the projection and the scripts stayed as 00b0ee6 had them, and #1680 carries them. 877977f rewords another sentence of that docstring and one held reason (r8 section), and leaves both quoted passages and the scripts as they wereNO-GOfrom +670 to +1,780 s (11,839 s at +1,200 s) and again from +3,975 to +4,770 s, with no damage flag. A watcher that polls at +900 and +1,500 s stops the run (inferred from its rule). We haven't measured how much worker start-up varies on Modal. The runbook's stop sentence now reads "aNO-GOthat comes through a restart". Its hang paragraph says that a new stream that does not replay holds the stall check until it delivers, so when the scorer has stopped printing, only the runner's budget ends the runChecks of the answers
Three checks read a first answer to should-fix 1, which ended the old stream at the restart: a model of the watcher's poll loop over the projection (simulated, not a run of the watcher;
runs/r8-fix/verify-sf1-evidence/emu.pyin the review folder), seeded faults in the watcher, and a read of the docs.WAIT, theWAITre-armed the restart, and each laterNO-GOrestarted again. In the model, 4 of 44 d844 runs that overrun never stopped before the runner's budget, up to 13,305 s ($33.89) after a watcher that acts on the firstNO-GO, and neither did 14 of 78 runs in seed 3's random pools. The d844 runs overran the plan's 14,505 s (16 runs) or budgets 600 or 1,800 s short of the true end (28), and the random pools budgets 600 s short. 8ea47a2 keeps the old stream until the next poll has projected both. In the same model it stops every overrunning d844 run at most 600 s after a watcher that acts on the firstNO-GOdoes, and the random pools up to 2,400 s ($6.11) after. In two of those 78, the runner's budget ends the run less than a poll after the firstNO-GO. With 265 s to spare, the modelled watcher stops 3 of 92 runs that fit, where one that acts on the firstNO-GOstops 11.test_a_restart_that_does_not_replay_leaves_no_hole_and_stops_a_poll_laterfails when the restart ends the old stream. The model takes everyNO-GOthat comes through a restart as a stop; since 877977f the watcher also needs a fresh scorer line, which a run still scoring prints within a poll (inferred), so the figures assume one.modal app logsstreams of one app at once, which the overlap needs. At 8ea47a2, a refused second stream left the newest file empty, whoseWAIT(newest_stream_empty) re-armed the restart, so only the runner's budget ended a run that overran (inferred from the code). Since 877977f, a refused new stream falls back to one stream at a time, and aWAITon an empty newest stream re-arms nothing; if Modal also does not replay, the hole a reconnect leaves still gives aWAITthat re-arms it (r8 section, Not established).GO, and one that also cleared it on a failed poll, each passed all 52 tests of the first answer.test_a_wait_after_the_restart_rearms_it_for_the_next_no_goandtest_a_failed_poll_after_the_restart_does_not_rearm_itnow catch them. At 8ea47a2 two faults in the new code still passed every test: dropping the end of the replaced stream, and dropping the exit trap that ends it when the watcher ends first. Either left an extramodal app logsprocess running until the app ended and changed no verdict (inferred from the code). Since 877977f,test_a_go_after_the_restart_rearms_it_for_the_next_no_gochecks which streams run at each projection, andtest_a_watcher_that_ends_during_the_overlap_ends_the_replaced_streamchecks that the replaced stream's process has gone after the watcher exits, so each fault should fail one of them (inferred from the code; nobody has run either fault against 877977f).test_a_hung_stream_is_restarted_once_before_a_no_go_is_acted_onnow waits for the restarted stream to print before its second poll: read earlier, the empty new stream givesWAIT, which re-arms the restart, and when that stream started printing 1 s late, the test ran out its 120 s.test_a_stream_that_grows_again_resets_the_hang_restartbecametest_a_restart_that_shows_a_fresh_slice_keeps_the_run_going, the behaviour it checks. 877977f replaces the first withtest_a_hung_stream_is_restarted_and_a_stall_it_still_shows_stops_nothing, since at polls 600 s apart a stall no longer stops the run (r8 section; r9 section for polls further apart).NO-GOtimes as stop times. They now give the restart's second, the watcher's outcome and the poll of delay.Review r6 (f6655af), answered
Sol's review (0 blocking, 3 should-fix, 1 nit) and the in-session four-lens review (0 blocking, 7 should-fix, 12 nits) both returned REQUEST_CHANGES, and the in-session review's skeptics upheld every should-fix and refuted no finding. The starts hold replaces two guesses about what a hole lost. f6655af left any start whose span covered a hole out of the start cost (
starts_across_holes). The first fix commit, fd9ee94, added a margin of one start per hole, which a lost 78 s start among 119 to 328 s ones overran: it read 460 s late. 5985499 removed both and added the starts hold.NO-GOat 14,683.7 s for a run that ends at 14,440 sGO, and the incumbent showsstarts_unconfirmeduntil its 16th start (test_a_lossless_restart_reads_as_the_intact_log_and_holds_only_a_no_go). Sol's second case, a candidate whose starts take 600 s against the incumbent's 100 s, split losslessly as its pool opens, should now read as the intact log. The hold keeps itsNO-GOasWAITuntil the candidate's 16th start, though: a late stop, which the runner's budget bounds (inferred from the rule, not run)NO-GOat 10,665 s for a run that ends at 10,210 snewest_stream_empty). Once it delivers, the hole lies inside the candidate's pool, so the starts hold applies until the candidate logs all 13 starts (the rule; the test stops at +7,200 s). Teststest_a_worker_start_lost_before_a_new_stream_delivers_waitsandtest_an_empty_new_stream_holds_between_pools_and_during_a_startNO-GOat +1,200 s for the d844 shape at 265 s spare (test_a_replay_from_inside_the_last_second_counts_nothing_twice). So the count rule stays.restart_overlaps(rules, above) now names such a restart in the gap reason and the watcher's gap alarm (test_a_restart_inside_a_second_of_repeated_lines_is_reported_not_guessed). The boundary search above gives the cost: up to 24 s late, 245 polls ofWAIT, one missed stopmodal app stopfrom the watcher or by hand, the class timeout or a container crash. A scorer failure or the runner's own budget stop pushes its state and writes a receipt, whichverify-receiptrejects as not completedtest_a_second_stream_end_in_one_interval_is_restarted_before_the_polltest_an_empty_new_stream_holds_between_pools_and_during_a_start). Scenario C, a lost incumbent start with no slice running, falls under the starts hold; the d844 cut search, which cuts every worker-start line at f = 1.01 and 1.02 among others, gives 0 falseNO-GOstest_a_start_lost_in_a_hole_holds_a_time_no_go_until_every_worker_startspins the damaged estimate (4,112 s against the intact 3,672 s) and the holdtest_slice_suspects_after_a_real_hole_hold_on_their_own(M2),test_suspects_count_from_the_end_of_the_hole(M3) and the log-gap assertion intest_a_log_the_model_cannot_explain_is_reported(M15), each with a seeded fault. The fault check now seeds 50 faults, the hold's included (result under Tests)WATCH_CREATED_SLACK(60) seconds beforestarted-at, or one whose "Created at" it cannot read. The runbook takes the id from this attempt's launch log, never frommodal app list, since every stage runs as the appmicrocosm-us-stage. The check misses an app another session launches afterstarted-at; the launch log guards against that. Teststest_an_app_not_created_by_this_attempt_is_refused_at_the_startandtest_an_app_created_just_before_started_at_is_watchedNO-GO, the watcher checks that the newest stream's file grew since the last poll. If it did not, it restarts the stream (once until it grows again) and leaves the call to the next poll; aNO-GOread from a stream underWATCH_SETTLE(120) seconds old waits too. When the restarted stream replays, as the tests' fake does, the watcher stops the silent run at the next poll, about $1.53 more. If it does not replay and the app prints nothing more, the empty new stream also holds the stall check, and the runner's budget ends the run (inferred from the code). Teststest_a_hung_stream_is_restarted_once_before_a_no_go_is_acted_on,test_a_stream_that_grows_again_resets_the_hang_restart(nowtest_a_restart_that_shows_a_fresh_slice_keeps_the_run_going) andtest_a_no_go_from_a_stream_that_just_started_waits_for_the_next_poll. 8ea47a2 replaced the growth check (r7 section), and 877977f's watcher alarms on a stall instead of stopping the run, though it can still stop the run when two polls come more than an hour apart (r8 and r9 sections)NO-GOcomes: up to 8,400 s ($21.39)GOandNO-GOWAITones includedNO-GOwhile its pool has logged fewer starts than workers: the candidate's for the rest of its pool, the incumbent's until its release and then while the candidate's estimate reads its timingtest_an_intact_log_raises_no_damage_flag_at_any_poll(evenly spaced record times of each drawn pool, one stream and a whole replay) andtest_the_fixtures_raise_no_damage_flag_at_any_record_timemain()test_the_cli_reads_ten_streams_in_the_watchers_orderlsubmitline names the attempt folder literallystarted-at, which errs toward less spendEarlier reviews
This description's edit history keeps each round's full table.
BUDGET:exceptions.GOfrom streams that began late.started-at, missed starts put back on their submit, and help captured whole.Invariants (tested)
NO-GOat its first estimate before the deadline.NO-GO. With a prior off on one chunk (the band above) or uneven start costs (above) this does not hold.log_gap, hole flag or hold at evenly spaced record times of each drawn pool, on one stream and on a whole replay, or at any record time of the three fixtures.timeout_for_capreturns the longest whole second inside any cap at the class price, and refuses a zero cap or price.scripts/mutation_check.pyseeds 50 faults in the projection and runs the test file against each with a fresh Hypothesis database (result under Tests;uv run python docs/evidence/us-h2h-speed/scripts/mutation_check.pyreruns it). Neither 00b0ee6 nor 8ea47a2 changes the projection or its tests; 877977f changes only text there (Tests).Not established
GO, in that window.NO-GOto a run with 265 s to spare (above); we haven't timed how much start costs vary on Modal.started-atto the tool's start on Modal, which comes out of the watcher's budget.modal app logsstream replays the backlog, whether Modal serves two streams of one app at once, how a client behaves on TERM, and how often streams end or hang. Before acting on the firstNO-GOafter aGOorWAIT, the watcher starts a new stream beside the old one in case the old one hung, even after it grew. It stops the app on a laterNO-GOonly when a stream it started since that restart shows a line the scorer printed after both the previous poll and the stream's own start, counted fromWATCH_READYseconds after its launch on the assumption, not measured, that a client connects within that time. That shows a new connection delivered a fresh line since the previous poll, not that it still delivers at the poll that reads it: a new stream that does so and then hangs, or a delay every stream shares, can still give a staleNO-GO(inferred). At polls 600 s apart a stall gives no fresh line, so the watcher alarms on it and leaves it to the runner's budget; a timeNO-GOwith no scorer line since the previous poll (600 s earlier) alarms too, until a line arrives (inferred). When two polls come more than an hour apart, a stall can stop the app (review r9's should-fix 4, which we reproduced against the fake). A line that reaches the log after the projection has read it, or that carries a stamp after the poll's own time, can confirm aNO-GOthat the projection took from a stale log (r9's blocking 1, reproduced against the fake), and a stream that connects later thanWATCH_READYallows, or a container clock that runs ahead, can let a replay pass (r9's blocking 2, inferred). When the old stream had hung and the new one does not replay, the new one leaves a hole, so the holds can delay the stop. If Modal refuses a second stream, the watcher reconnects one stream at a time, which gives up the overlap, so a reconnect that does not replay leaves a hole too. If Modal both refuses a second stream and does not replay, the hole's holds give aWAITthat re-arms the restart, so each run ofNO-GOs can end in another suchWAIT, and only the runner's budget may end a run that overruns (inferred). Only the tests' fakemodalhas exercised the readiness check, the serial reconnect, the empty-stream replacement and TERM/KILL. The watcher's other Modal calls (app list,app stop) and the projection have no timeouts, so a hung one would freeze its loop and leave only the runner's budget.BUDGET:line shows as a log gap, and so does a lost worker start once the rebuild needs the missing worker; in a pool with fewer slices than two per worker it may never need it, and only the starts hold covers the loss. A lost line that leaves the rebuild consistent (a receive line, say) shows at most as the hole it leaves, though it can still move the estimate: the boundary split's 24 s likely comes from receive lines the parse drops (inferred). A restart that loses part of one second's lines leaves no hole, and a slice line lost there reads as a straggler until its receive line shows the gap. Until then the estimate can give a falseNO-GOto a run that fits (seed 3's pool 7, above). A stream that drops lines without ending leaves no hole.Tests
edb1dc5ef5mergesmainat 2f45706 on top of80532c73bb. That merge brings in Refuse an occupied base path in both Modal hand-off blocks #1694, which edits the runbook andtest_us_modal_stage_puf_support_base.py. Git reports no conflict, and the count stays seven. No local run covers the merged tree; CI runs it.80532c73bbcarries a clean merge ofmainat 7ff5a26 and moves the registered-tool count from six to seven in the runbook's introduction and intest_the_runbook_names_every_registered_tool. The test now also checks that the introduction's hyphenated backticked names form the same set as the registered tools' names, so a stale or missing name fails even when the count matches. The merge leaves the watcher, the projection, the scorer, the plan, the evidence folder and the watcher's and projection's test files as they were at 877977f. On the tree committed as80532c73bb, through the review folder's harness on 2026-11-20 (UTC),test_us_modal_stage_puf_support_base.pypassed 172 of 172, the watcher's test file 68 of 68 in two slices (-k stream, 24 of 24;-k "not stream", 44 of 44) and the projection's 59 of 59. The engine-freetest_us_modal_stage_plan_tool.pypassed 283 of 284. Its one failure,test_a_release_plan_cannot_pass_the_county_facts, imports Microcosm in a subprocess, which the harness leaves on the shared environment's older checkout, one withoutmicrocosm.build.program_telemetry. That test depends on neither the runbook's text nor the tool count, and CI's engine-free jobs on 877977f failed only the count test. On a scratch copy, the previous count test fails on the merged tree withassert 7 == 6, and the new one fails on the merged runbook, on a stale extra tool name that the previous one passes and on a dropped name (runs/r10-triage/logs/count_check_demo.txtin the review folder). The full engine-free suites and the engine-workflow plan-tool tests did not run locally, and CI has not yet run onedb1dc5ef5.tools/modal_us_stage_plan.pyonly comments; it leaveslost-lines.txtand the scripts as they were. On the tree committed as 877977f, through the review folder's harness on 2026-11-19 and 2026-11-20 (UTC), the watcher's test file passed 68 of 68, run in two slices (-k stream, 24 of 24;-k "not stream", 44 of 44), and the projection's test file passed 59 of 59. The plan tool's tests did not run for its comment-only change, and the full engine-free suites did not run locally.WAITat the next poll. In the last two its growth check re-arms the restart, so it stops at the secondNO-GOin theWAITtest and restarts again in the failed-poll test.lost-lines.txtor the scripts. So the seeded-fault and lost-lines results below still describe the projection at 8ea47a2.runs/r7-fix/mutation-baf2052/full.txtin the review folder).lost-lines.txtholds the five sections' output byte for byte under a four-line header.lost-lines.txt, the runbook's verdict bullet and a docstring inlost_lines.py. The plan, the plan tool and the stage registration have not changed since 5b32556. At that commit the stage registration and plan tool tests and the ACS local release tool, ACS multispine staging and CI test plan tests passed (this description's earlier revisions record it), andtools/modal_us_stage_plan.py validateaccepted the plan ($39.23 at the maximum wall, $40.00 at the timeout).main), CI passed lint, wheels, installed-emitter, select-countries, integration-uk, engine-uk and engine-us (3.13 and 3.14). Both engine-free jobs failed one test,test_the_runbook_names_every_registered_tool(9,108 passed, 57 skipped):mainnow registers a sixth Modal tool of its own (US_PUF_SUPPORT_BASE_2023_2025intools/modal_us_stage_plan.py), so the merge registers seven, where the runbook and the test say six. On 877977f (the run of 2026-11-20 00:31Z, which also tests this PR merged intomain), CI passed the same jobs, and both engine-free jobs failed that test again withassert 7 == 6(1 failed, 9,173 passed, 57 skipped in each).80532c73bbcarries the merge and moves the count to seven (above). The engine-contract parser-probe test, outside the local suites above, runs in CI.This PR made no Modal upload, check or paid run.
🤖 Generated with Claude Code