ci: the race job lays its guards out by what each weighs

.github/race-weights

@@ -0,0 +1,63 @@
+# What each guard weighs in the race job of ci.yml, in seconds under the race
+# detector. .github/scripts/race_parts.py lays these out over the parts, the
+# heaviest first, and sends every test not named here by the checksum of its
+# name, counted at the "unlisted" weight.
+#
+# Taken on 2026-11-03 from eight parts of two runs, each test at the longest
+# time any of them took. The race job runs on whatever processor the runner
+# is given, and the same part took 689 s on one and 1202 s on another, every
+# test of it slower by the same factor. The longest time is the slowest
+# processor's, so the parts are even where they are closest to the timeout.
+# Listed: every test that took 10 s or more. "unlisted" is the mean of the rest.
+#
+# A test that is not here and takes 60 s or more fails its part with the line
+# to add. A test that is here and no longer exists fails every part.
+unlisted 0.58
+348.3 TestAFormatRefusesASizeItCannotDescribe
+343.3 TestAFilmStoppedHalfWayLeavesNoHelperRunning
+329.9 TestEveryFormatHitsTheOrderedSizeExactly
+162.6 TestEveryScreenStillDrawsItsStoredPicture
+155.5 TestACompressedZipIsEitherWrittenAtItsSizeOrRefusedInWords
+142.2 TestEveryTileOfAFilmIsGav1dsCodingOfItsPixelsAndDecodesAsItDoesAlone
+130.0 TestTheMinimumThisToolPrintsStandsWhateverTheSeedIs
+117.9 TestFreeBoxesCarryPaddingOfAnyLength
+99.9 TestEveryBase64TextSizeIsExact
+93.8 TestAFilmPictureLargerThanAnyAV1LevelIsRefusedNotBroken
+67.9 TestADeclaredDefaultIsTheOneTheFormatUses
+64.3 TestNoTrackedFileCarriesPrivateContent
+63.8 TestNoGeneratorHoldsTheWholeFileInMemory
+57.3 TestAFilmSaysItCarriesItsLabelOnlyWhenAPictureShowsItWhole
+55.3 TestAFilmOfANamedSizeKeepsEveryPictureInsideItsReserve
+53.4 TestTheArchiveScanLooksIntoTheArchivesInside
+38.5 TestAContainerStopsBetweenTheFilesItPlans
+37.5 TestEveryPictureNamedAfterARefusalShowsOne
+36.7 TestTheArchiveScanReadsTheListArchiveZipReads
+35.9 TestTheArchiveScanFindsEveryTrapItNames
+35.5 TestJxlStillFitsWhenTheQualityIsNotTheOneTheCeilingsWereMeasuredAt
+30.8 TestOurOwnGeneratorsHaveNotDrifted
+29.9 TestTheSameSeedGivesTheSameBytes
+27.7 TestTheCeilingCountsTheWholeRunNotOneTarget
+26.8 TestEveryJxlRungStillFitsInsideItsDeclaredCeiling
+26.6 TestTheMinimumStandsWhateverTheSeedIsUnderEveryDeclaredChoice
+24.9 TestEveryEMLSettingSurvivesItsReferenceTool
+24.2 TestEveryLinkOnTheSiteLeadsSomewhere
+24.2 TestAPictureSaysItCarriesItsLabelOnlyWhenItsTextIsWhole
+20.9 TestAFilmsProgressMovesWithItsPicturesNotItsPadding
+20.7 TestAFilmHoldsTheFramesTheManifestDeclares
+20.7 TestTheWindowBinaryStartsWithoutAConsole
+19.9 TestAFilmWhoseSquareCoversItsLabelSaysSoInItsPlan
+19.9 TestTheSiteSaysWhatTheToolSays
+19.8 TestChoosingAToolLaysTheScreenOutAsIfItOpenedWithIt
+19.8 TestTheArchiveScanHoldsNoMoreOfANameThanTheName
+16.6 TestEveryLanguageDescribesEveryDamage
+16.3 TestTheSameWorkDoneAgainLeavesTheWindowAsItWas
+16.2 TestABuildWithNoWindowInItSaysSoAndKeepsStandardOutputEmpty
+16.0 TestEveryLanguageDescribesEverythingTheProgramCanProduce
+15.4 TestAvifStillFitsWhenTheQualityIsNotTheOneTheCeilingsWereMeasuredAt
+15.2 TestAFinishedRunInTheWindowSaysItsManifestIsTooBigToReadBack
+15.1 TestABuildWithoutTheBuildTagsRefusesAndSaysWhy
+14.9 TestAShortcutCannotStartASecondRunDuringTheFirst
+14.6 TestTheLicenceCommandListsWhatTheBinaryLinks
+14.4 TestEveryFormatSurvivesItsReferenceTool
+12.2 TestTheMemoryFlagReachesThePlanOnEveryCommandThatPlans
+11.2 TestTheCommandLineHoldsTheWholeRunToItsMemorySetting

.github/scripts/race_parts.py

@@ -0,0 +1,151 @@
+"""Which guards one part of the race job runs, and whether the weights still hold.
+
+    race_parts.py split NAMES WEIGHTS PART PARTS > mine.txt
+    race_parts.py unlisted WEIGHTS LOG
+
+split prints the names of NAMES (one per line, as go test -list gives them)
+that part PART of PARTS runs. The tests WEIGHTS names are laid out by their
+weight, the heaviest first, each into the part that is lightest so far. Every
+other test goes where the checksum of its name sends it and counts as the
+"unlisted" weight, so adding or removing a light test moves no other test, and
+the heavy ones are laid over the light ones rather than beside them.
+
+unlisted reads the log of a part (go test -v) and fails when a test WEIGHTS
+does not name took UNLISTED_CEILING seconds or more. Such a test is laid out as
+if it were light, and the parts stop being even without anything saying so.
+
+Both fail closed, with a sentence that names what to change: a weight for a
+test the binary does not list (renamed or removed), a line that is not a
+weight, a name given twice, no "unlisted" line, or a part outside the parts.
+"""
+
+import math
+import re
+import sys
+import zlib
+
+# The weights list every test that took 10 s or more on the slowest processor
+# measured, so an unlisted test is under 10 s there, and no test measured took
+# more than 2.7 times as long on one processor as on another. Six times the
+# listing threshold is a test that is new or has grown, not one the machine
+# made slow.
+UNLISTED_CEILING = 60.0
+
+# A duration as go test writes it. A looser one let "1.2.3s" through to float(),
+# which ended the script with a traceback instead of passing the line by.
+TOP = re.compile(r"^--- (?:PASS|FAIL|SKIP): (\S+) \(([0-9]+(?:\.[0-9]+)?)s\)$")
+
+
+def refuse(message):
+    print(message, file=sys.stderr)
+    sys.exit(1)
+
+
+def read_lines(path, what):
+    try:
+        with open(path, encoding="utf-8") as f:
+            return f.read().splitlines()
+    except UnicodeDecodeError:
+        refuse("the %s %s is not UTF-8 text, so it is not one this job wrote." % (what, path))
+    except OSError as err:
+        refuse("cannot read the %s %s: %s." % (what, path, err.strerror or err))
+
+
+def read_weights(path):
+    unlisted = None
+    weights = {}
+    for number, line in enumerate(read_lines(path, "weights"), 1):
+        text = line.strip()
+        if not text or text.startswith("#"):
+            continue
+        fields = text.split()
+        where = "%s line %d (%s)" % (path, number, text)
+        if len(fields) != 2:
+            refuse("%s is not a weight. Each line is seconds and a test name, or \"unlisted\" and seconds." % where)
+        if fields[0] == "unlisted":
+            if unlisted is not None:
+                refuse("%s gives the unlisted weight a second time." % where)
+            unlisted = seconds(fields[1], where)
+            continue
+        took, name = seconds(fields[0], where), fields[1]
+        if name in weights:
+            refuse("%s names %s a second time. Keep one line for it." % (where, name))
+        weights[name] = took
+    if unlisted is None:
+        refuse("%s has no \"unlisted\" line, so the tests it does not name weigh nothing. "
+               "Add one with the mean time of those tests." % path)
+    return unlisted, weights
+
+
+def seconds(text, where):
+    try:
+        value = float(text)
+    except ValueError:
+        value = math.nan
+    if not math.isfinite(value) or value < 0:
+        refuse("%s gives %s, and a weight is a number of seconds, nought or more." % (where, text))
+    return value
+
+
+def lay_out(names, unlisted, weights, parts):
+    """Return the part of every name and the weight each part carries."""
+    gone = [name for name in weights if name not in names]
+    if gone:
+        refuse("the weights name %s, which the test binary does not list - a test renamed or removed. "
+               "Take the line out of the weights, or give it the test's new name." % ", ".join(sorted(gone)))
+    part_of = {}
+    load = [0.0] * parts
+    for name in names:
+        if name not in weights:
+            part = zlib.crc32(name.encode("utf-8")) % parts
+            part_of[name] = part
+            load[part] += unlisted
+    for name in sorted(weights, key=lambda n: (-weights[n], n)):
+        part = min(range(parts), key=lambda p: (load[p], p))
+        part_of[name] = part
+        load[part] += weights[name]
+    return part_of, load
+
+
+def split(names_path, weights_path, part_text, parts_text):
+    try:
+        part, parts = int(part_text), int(parts_text)
+    except ValueError:
+        refuse("the part and the number of parts are whole numbers, and these are %s and %s." % (part_text, parts_text))
+    if parts < 1 or not 0 <= part < parts:
+        refuse("part %d of %d does not exist. Parts count from nought and stay below their number." % (part, parts))
+    names = list(dict.fromkeys(n.strip() for n in read_lines(names_path, "test list") if n.strip()))
+    unlisted, weights = read_weights(weights_path)
+    part_of, load = lay_out(set(names), unlisted, weights, parts)
+    mine = [name for name in names if part_of[name] == part]
+    for name in mine:
+        print(name)
+    print("part %d of %d runs %d of %d tests, about %.0f s by the weights (the parts: %s s)" % (
+          part, parts, len(mine), len(names), load[part], ", ".join("%.0f" % l for l in load)), file=sys.stderr)
+
+
+def check_unlisted(weights_path, log_path):
+    _, weights = read_weights(weights_path)
+    long = []
+    for line in read_lines(log_path, "log"):
+        found = TOP.match(line)
+        if found and found.group(1) not in weights and float(found.group(2)) >= UNLISTED_CEILING:
+            long.append((found.group(1), float(found.group(2))))
+    if long:
+        lines = "\n".join("  %.1f %s" % (took, name) for name, took in long)
+        refuse("these tests took %.0f s or more and the weights do not name them, so the parts are laid out "
+               "as if they were light:\n%s\nAdd each line above to %s." % (UNLISTED_CEILING, lines, weights_path))
+
+
+def main():
+    args = sys.argv[1:]
+    if len(args) == 5 and args[0] == "split":
+        split(*args[1:])
+    elif len(args) == 3 and args[0] == "unlisted":
+        check_unlisted(*args[1:])
+    else:
+        refuse("usage: race_parts.py split NAMES WEIGHTS PART PARTS | race_parts.py unlisted WEIGHTS LOG")
+
+
+if __name__ == "__main__":
+    main()

.github/workflows/ci.yml

@@ -1049,7 +1049,9 @@ jobs:
         # a new definition was the weekly sweep, days after it merged. And so
         # is .github/build-tags, which every command of the job reads and which
         # decides the files it compiles - the same reason as go.mod, named by
-        # an outside review of the pull request that split the job.
+        # an outside review of the pull request that split the job. And so are
+        # the weights and the script that lay its parts out (O298), since a
+        # change to either decides which guards each part runs.
         #
         # Anything unclear counts as touched. A first push, a branch with no
         # comparable parent, a pull request event with no before - all answer
@@ -1058,7 +1060,7 @@ jobs:
         # in somebody else's file.
         run: |
           set -euo pipefail
-          watched='internal/format/registry.go internal/damage/damage.go cmd/tfg/main.go internal/gui/window/run.go internal/gui/run_cgo.go internal/gui/window/tidy.go internal/audit/parallel.go internal/engine/parallel.go internal/tool/tool.go internal/gui/window/tools.go internal/format/video/ahead.go go.mod .github/workflows/ci.yml .github/build-tags'
+          watched='internal/format/registry.go internal/damage/damage.go cmd/tfg/main.go internal/gui/window/run.go internal/gui/run_cgo.go internal/gui/window/tidy.go internal/audit/parallel.go internal/engine/parallel.go internal/tool/tool.go internal/gui/window/tools.go internal/format/video/ahead.go go.mod .github/workflows/ci.yml .github/build-tags .github/race-weights .github/scripts/race_parts.py'
           # On a pull request there is no "before" - the field belongs to a push
           # - so this asked for something empty and every pull request answered
           # "touched". That quietly undid the decision of 2026-08-20, because
@@ -1130,10 +1132,10 @@ jobs:
     # The ceiling is the sum of the limits inside it, with room left over, so
     # that each of them fires first and names what was slow: about four minutes
     # to set up and compile (the compile alone took 3m03s on the first run),
-    # twenty for the guards, five for the other packages. Measured on the first
-    # run, split by a sorted list: 677, 635, 164 and 154 s of guards in the
-    # four parts, 14m55s for the longest part. The two slowest guards take 329
-    # and 222 s, and wherever they land together, that part is the slow one.
+    # twenty for the guards, five for the other packages. The guards are laid
+    # out by what each weighs (.github/race-weights), because a split by name
+    # gave parts of 1250, 1158, 795 and 526 s on the slowest processor
+    # measured, two of them at the twenty minutes (O298).
     strategy:
       fail-fast: false
       matrix:
@@ -1172,14 +1174,13 @@ jobs:
 
       - name: the machine this part runs on
         # O298, 2026-11-03. Part 2 ran past its twenty minutes in one run of
-        # three since 2026-10-26, main included, while parts 0, 1 and 3 of the
-        # same runs took the time they always take. In a slow run every test of
-        # the part from the fourth on is about 1.75 times slower, the single
-        # threaded ones that read files among them. The log names the runner's
-        # image and region and nothing about the machine, so a slower machine
-        # and something inside the part could not be told apart. This says what
-        # the machine is, and the step after the tests says what its processor
-        # did meanwhile - the next slow run answers which of the two it was.
+        # three since 2026-10-26, main included, and the log named the runner's
+        # image and region and nothing about the machine. With this step the
+        # answer was the processor: the same part took 1202 s on an AMD EPYC
+        # 7763 and 689 s on an EPYC 9V45, every test of it slower by the same
+        # factor, while part 3 took 525 and 530 s on two 7763s. The weights
+        # the parts are laid out by come from these numbers, so this stays to
+        # say which processor a slow part had when the weights stop fitting.
         run: |
           set -euo pipefail
           echo "processors: $(nproc)"
@@ -1203,13 +1204,16 @@ jobs:
         # because a file in the package builds only without it, and it keeps
         # Fuzz and Example because go test runs those as tests too.
         #
-        # A test's part is the checksum of its name (cksum) modulo the number
-        # of parts, so adding or removing a test moves no other test. It was
-        # every fourth name of a sorted list at first, and one guard added on
-        # this job's own pull request moved every name after it - the part
-        # holding the slowest guards changed from one commit to the next. The
-        # parts are even only on average, and which one is slow now changes only
-        # with what is in it.
+        # The tests .github/race-weights names are laid out by weight, the
+        # heaviest first, each into the part that is lightest so far. Every
+        # other test goes by the checksum of its name, so adding or removing a
+        # light test moves no other test. It was every fourth name of a sorted
+        # list at first, and one guard added on this job's own pull request
+        # moved every name after it. Then the checksum alone, which is even only
+        # on average, and with three guards of 330 to 350 s it was not. A test
+        # missing from the weights that takes a minute or more fails the part
+        # with the line to add, and a weight for a test that is gone fails
+        # every part, so the list cannot fall behind the tests quietly.
         #
         # Three things fail the part rather than let it pass on nothing -
         # go test with a -run pattern that matches no test exits 0. An index the
@@ -1223,7 +1227,8 @@ jobs:
         # and stays below the job's ceiling, so a slow part fails as a test with
         # a name rather than as a killed job with no output. Go's ten minutes
         # killed this job on 2026-08-25, before a timeout was stated here.
-        # Twenty minutes is 1.8 times the slowest part measured.
+        # By the weights the heaviest part is 961 s on the slowest processor
+        # measured, and twenty minutes is 1.25 times that.
         #
         # The processor's counters and the clock are read just before the
         # tests, for the step after this one (O298).
@@ -1240,12 +1245,7 @@ jobs:
           fi
           go test -tags "$(cat .github/build-tags)" ./internal/guard/ -race -list '.*' > listed.txt || { cat listed.txt; exit 1; }
           grep -E '^(Test|Fuzz|Example)' listed.txt > names.txt || { echo "the test binary listed no Test, Fuzz or Example function. What it printed:"; cat listed.txt; exit 1; }
-          while IFS= read -r name; do
-            read -r sum _ < <(printf '%s' "$name" | cksum)
-            if [ $((sum % PARTS)) -eq "$PART" ]; then
-              echo "$name"
-            fi
-          done < names.txt > mine.txt
+          python3 .github/scripts/race_parts.py split names.txt .github/race-weights "$PART" "$PARTS" > mine.txt
           planned=$(wc -l < mine.txt)
           echo "part $PART of $PARTS runs $planned of $(wc -l < names.txt) tests"
           if [ "$planned" -eq 0 ]; then
@@ -1263,6 +1263,7 @@ jobs:
             echo "Each of these is a name the -run pattern did not reach, or a test the binary skipped before it started."
             exit 1
           fi
+          python3 .github/scripts/race_parts.py unlisted .github/race-weights part.log
         shell: bash
 
       - name: what the processor did while this part ran