From 7bafbd9413914f41492e369c55b5aa6445288c7c Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 3 Nov 2026 15:17:49 +0100 Subject: [PATCH 1/5] ci: the race job says what machine each part ran on Part 2 of the race detector job has run past its twenty minutes in about one run of three, while the other parts of the same runs took their usual time. The log does not say what the machine was, so a slower machine and something inside the part could not be told apart. Each part now prints its processors, their model and the memory before the tests, and after them how the machine's processor time was spent while the tests ran (busy, idle, waiting on the disk, taken by the host) and which processes still running have used the most of it. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 77ce1321..b039b5ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1170,6 +1170,24 @@ jobs: sudo apt-get install -y --no-install-recommends libgl1-mesa-dev libwayland-dev libx11-dev libxkbcommon-dev xorg-dev shell: bash + - name: the machine this part runs on + # O298, 2026-11-03. Part 2 ran past its twenty minutes in one run of + # three since 2026-10-26, main included, while parts 0, 1 and 3 of the + # same runs took the time they always take. In a slow run every test of + # the part from the fourth on is about 1.75 times slower, the single + # threaded ones that read files among them. The log names the runner's + # image and region and nothing about the machine, so a slower machine + # and something inside the part could not be told apart. This says what + # the machine is, and the step after the tests says what its processor + # did meanwhile - the next slow run answers which of the two it was. + run: | + set -euo pipefail + echo "processors: $(nproc)" + lscpu | grep -E '^(Model name|Thread\(s\) per core|Core\(s\) per socket|Socket\(s\)|CPU max MHz|L3 cache|Hypervisor vendor|Virtualization type):' || true + free -m + echo "load average: $(cat /proc/loadavg)" + shell: bash + - name: test this part under the race detector # A data race is the one defect class here that nothing else notices. It # does not change a size, and on the run that happens to interleave the @@ -1206,6 +1224,9 @@ jobs: # a name rather than as a killed job with no output. Go's ten minutes # killed this job on 2026-08-25, before a timeout was stated here. # Twenty minutes is 1.8 times the slowest part measured. + # + # The processor's counters and the clock are read just before the + # tests, for the step after this one (O298). env: PART: ${{ strategy.job-index }} PARTS: ${{ strategy.job-total }} @@ -1232,6 +1253,8 @@ jobs: echo "With this few tests, use fewer parts in strategy.matrix.part." exit 1 fi + head -n 1 /proc/stat > stat-before.txt + date +%s > time-before.txt go test -tags "$(cat .github/build-tags)" ./internal/guard/ -count=1 -race -timeout 20m -v -run "^($(paste -sd'|' mine.txt))\$" 2>&1 | tee part.log missing=$(comm -23 <(sort mine.txt) <(sed -n 's/^=== RUN \([^/]*\)$/\1/p' part.log | sort -u)) if [ -n "$missing" ]; then @@ -1242,6 +1265,37 @@ jobs: fi shell: bash + - name: what the processor did while this part ran + # The other half of the step that says what the machine is (O298). The + # counters are the whole machine's, read just before the tests and + # again now, so time a host took away from this machine shows as taken + # by the host, and a process the part left running shows below with + # the time it has used. Runs after red tests as well, because a slow + # part is a red one. + if: ${{ !cancelled() }} + run: | + set -euo pipefail + if [ ! -s stat-before.txt ]; then + echo "the tests of this part never started, so there is nothing to compare" + exit 0 + fi + read -r _ user0 nice0 system0 idle0 iowait0 irq0 softirq0 steal0 _ < stat-before.txt + read -r _ user1 nice1 system1 idle1 iowait1 irq1 softirq1 steal1 _ < <(head -n 1 /proc/stat) + busy=$(( (user1 + nice1 + system1 + irq1 + softirq1) - (user0 + nice0 + system0 + irq0 + softirq0) )) + idle=$(( idle1 - idle0 )) + iowait=$(( iowait1 - iowait0 )) + steal=$(( steal1 - steal0 )) + total=$(( busy + idle + iowait + steal )) + echo "over $(( $(date +%s) - $(cat time-before.txt) )) s on $(nproc) processors:" + if [ "$total" -gt 0 ]; then + echo " busy $(( 100 * busy / total ))%, idle $(( 100 * idle / total ))%, waiting on the disk $(( 100 * iowait / total ))%, taken by the host $(( 100 * steal / total ))%" + fi + echo "load average: $(cat /proc/loadavg)" + echo "the processes that have used the most processor time, still running now:" + ps -eo pid,ppid,etime,time,pcpu,comm --sort=-time > processes.txt + head -n 12 processes.txt + shell: bash + - name: the other packages under the race detector # Every package outside internal/guard, in part 0 alone. None holds a # test today - the first run took one second for all of them - and one From cd80910277eef13496abbc94c2a91e51c9f86664 Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 3 Nov 2026 15:28:30 +0100 Subject: [PATCH 2/5] deps: golang.org/x/net 0.62.0 for the window Two published advisories name golang.org/x/net 0.60.0 (GO-2026-6860 and GO-2026-6866), both in its html package. The window links that package only through the toolkit's SVG reader, govulncheck finds neither in code it reaches, and the command line tool does not link golang.org/x/net at all, so no generated file changes. The licence file is byte for byte the same. Among the packages the window links, only html/parse.go and html/render.go changed. No module entered or left the graph, and golang.org/x/text and golang.org/x/sys stay where they were. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 +++++ THIRD-PARTY-NOTICES.md | 6 +++++- go.mod | 2 +- go.sum | 4 ++-- internal/guard/mutation/entries.py | 2 +- 5 files changed, 14 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c2f0367..b3c4bcb8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -125,6 +125,11 @@ because it turns other people's test suites red. - **Built with Go 1.27.3.** It fixes a vulnerability in `crypto/x509` of the Go standard library, which only the desktop window links. The command line tool was not affected. Building from source now needs Go 1.27.3. +- **The window is built with `golang.org/x/net` 0.62.0.** The version before + had two published vulnerabilities in the package that reads and writes HTML. + The window uses the package only to read the SVG pictures it draws, neither + of the two is in code it reaches, and the command line tool does not contain + the package at all. No generated file changes. ## [0.5.0] - 2026-10-20 diff --git a/THIRD-PARTY-NOTICES.md b/THIRD-PARTY-NOTICES.md index d07cb68a..1ff36db7 100644 --- a/THIRD-PARTY-NOTICES.md +++ b/THIRD-PARTY-NOTICES.md @@ -233,6 +233,10 @@ module, was compared with the one at the version before and is byte for byte the same. No module entered or left the graph, so the counts above did not move. +`golang.org/x/net` went to v0.62.0 on 2026-11-03 on its own, because two +published advisories name v0.60.0. Its licence file is byte for byte the one +at v0.60.0, and no module entered or left the graph. + Two arrived later than the rest. `github.com/FyshOS/fancyfs` came in on the same day with the folder picker, because the toolkit's dialog package imports it to decorate folder icons. Its licence was read before it was @@ -285,7 +289,7 @@ is the module's own MIT, carried unchanged. | `github.com/srwiley/rasterx` | v0.0.0-20220730225603-2ab79fcdd4ef | BSD-3-Clause | (c) 2018, Steven R Wiley | | `github.com/yuin/goldmark` | v1.8.2 | MIT | (c) 2019 Yusuke Inuzuka | | `golang.org/x/image` | v0.46.0 | BSD-3-Clause | 2009 The Go Authors. | -| `golang.org/x/net` | v0.60.0 | BSD-3-Clause | 2009 The Go Authors. | +| `golang.org/x/net` | v0.62.0 | BSD-3-Clause | 2009 The Go Authors. | | `golang.org/x/sys` | v0.49.0 | BSD-3-Clause | 2009 The Go Authors. | The five licence texts follow, one copy each. They differ only in the copyright diff --git a/go.mod b/go.mod index 77e84f92..c7cc8e89 100644 --- a/go.mod +++ b/go.mod @@ -122,7 +122,7 @@ require ( github.com/stretchr/testify v1.11.1 // indirect github.com/yuin/goldmark v1.8.2 // indirect golang.org/x/image v0.46.0 // indirect - golang.org/x/net v0.60.0 // indirect + golang.org/x/net v0.62.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 87161d03..7fec9ea9 100644 --- a/go.sum +++ b/go.sum @@ -72,8 +72,8 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/image v0.46.0 h1:b1+oYj0Jbp6K5MDT4i4/eZpYlk3V8SJhhDKh6LBHAyQ= golang.org/x/image v0.46.0/go.mod h1:3B3W05VGVQyuXucLINLjXKrqISASfi4Xj+iCVkLMwew= -golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU= -golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= +golang.org/x/net v0.62.0 h1:18ELalfM5+XNRwzM2nYqXsZQ9ByS1uhj2O/U7+5jqRY= +golang.org/x/net v0.62.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= diff --git a/internal/guard/mutation/entries.py b/internal/guard/mutation/entries.py index 9f90f5fa..895596bb 100644 --- a/internal/guard/mutation/entries.py +++ b/internal/guard/mutation/entries.py @@ -4305,7 +4305,7 @@ # and would prove nothing about whether drift is caught. "the notices file goes back to naming a version that is no longer built", "THIRD-PARTY-NOTICES.md", - "| `golang.org/x/net` | v0.60.0 |", + "| `golang.org/x/net` | v0.62.0 |", "| `golang.org/x/net` | v0.35.0 |", "TestTheNoticesFileNamesTheVersionsThatAreActuallyBuilt", ), From aa377b03e75f55cbdbab18221778b99754c071fd Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 3 Nov 2026 15:53:30 +0100 Subject: [PATCH 3/5] security: three more mcp advisories under semgrep wait for the same date Three advisories published against mcp 1.29.0 are fixed in 1.30.0, and every semgrep release, 1.183.0 included, asks for mcp==1.29.0. Nor can the package be left out: semgrep's command line imports it when it starts. Two are about the MCP client and one about an MCP server checking bearer tokens. The CI job runs semgrep scan and nothing else, which starts neither. The entries end on 2026-11-15 with the two already there, so all of them come back on one reminder. Co-Authored-By: Claude Opus 5.5 --- .github/osv-scanner.toml | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/.github/osv-scanner.toml b/.github/osv-scanner.toml index a3384e70..7555ed41 100644 --- a/.github/osv-scanner.toml +++ b/.github/osv-scanner.toml @@ -44,3 +44,36 @@ reason = "semgrep pins mcp==1.29.0 in every release, and the advisory is about a id = "GHSA-xcq7-p3pf-p88h" ignoreUntil = 2026-11-15 reason = "semgrep pins opentelemetry-api~=1.37.0 in every release, and the advisory is about parsing an inbound tracestate header in a server, which semgrep scan never runs" + +# Three more advisories against mcp 1.29.0, published 2026-11-02, all fixed in +# the same 1.30.0 as the first entry above. +# +# semgrep 1.183.0 of 2026-10-29, the newest, still asks for mcp==1.29.0, so no +# set pip can install holds the fix. Nor can mcp be left out of the install: +# semgrep's command line imports mcp.server.auth.settings when it starts, and +# without the package `semgrep --version` itself fails (measured 2026-11-03 in +# python:3.14-slim with this directory's requirements). +# +# Two are about the MCP client - its HTTP transports following a redirect to +# another host with the request's headers and body, and its fetching of $ref +# addresses a server names in an output schema. The third is about an MCP +# server accepting a bearer token issued for a different resource server. The +# job runs `semgrep scan` and nothing else, which starts neither a client nor +# a server, on a runner that lives for one scan. +# +# The same date as the entries above: all of them wait on a semgrep that moves +# its pin, and all of them come back on one reminder. +[[IgnoredVulns]] +id = "GHSA-5h93-6whr-6q8j" +ignoreUntil = 2026-11-15 +reason = "semgrep pins mcp==1.29.0 in every release, and the advisory is about the MCP client following cross-origin redirects, which semgrep scan never starts" + +[[IgnoredVulns]] +id = "GHSA-rwrf-2pqf-9j8j" +ignoreUntil = 2026-11-15 +reason = "semgrep pins mcp==1.29.0 in every release, and the advisory is about the MCP client fetching server-chosen $ref addresses, which semgrep scan never starts" + +[[IgnoredVulns]] +id = "GHSA-w4fh-qvv9-3v23" +ignoreUntil = 2026-11-15 +reason = "semgrep pins mcp==1.29.0 in every release, and the advisory is about an MCP server checking bearer tokens, which semgrep scan never starts" From 972e8eab27c855a9c88ccdb878cb0a3d57ec12b7 Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 3 Nov 2026 16:25:29 +0100 Subject: [PATCH 4/5] engine: a run that will hold more than its memory setting says so first A run holds its whole plan to the end, and every file adds its answer and its manifest entry as it finishes, about 520 B in each of the five formats measured. The plan is weighed while it is built and what the files add was not, so 600 000 text files in a container of 1 GB passed every check, wrote for fifteen minutes and were ended by the system with nothing said and no manifest. With Docker's default swap the same run finished. engine.PredictHold works out the heap as the run starts plus what its files will add, against what the memory setting gives the run. The command line prints a note before the first byte, dry runs included, because a process the system ends prints nothing afterwards. The window says the same after a preview and after a run. A note and not a refusal: whether such a run is ended depends on swap the program cannot see. Three guards: the command line under --memory 1mb and 64gb, the window on a machine of 2 MiB and of 16 GB, and the forecast's 520 B against what a run of 2000 files adds to the heap, measured in a process of its own. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 ++ internal/cli/generate.go | 1 + internal/cli/record.go | 19 +++ internal/engine/engine.go | 18 +-- internal/engine/holdforecast.go | 85 +++++++++++ internal/guard/holdforecast_test.go | 212 ++++++++++++++++++++++++++++ internal/guard/mutation/entries.py | 86 ++++++++++- internal/gui/text/locale/en.json | 5 + internal/gui/text/locale/pl.json | 1 + internal/gui/text/text.go | 9 ++ internal/gui/window/run.go | 4 +- internal/gui/window/runreport.go | 15 ++ 12 files changed, 443 insertions(+), 20 deletions(-) create mode 100644 internal/engine/holdforecast.go create mode 100644 internal/guard/holdforecast_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index b3c4bcb8..d6b26971 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,14 @@ because it turns other people's test suites red. 1 GB, 500 000 text files are written where 400 000 were killed. Files come out byte for byte the same. +- **A run that will hold more memory than its setting gives it says so before + it starts.** The note names both sizes and comes before the first file is + written on the command line, and after Preview or the run in the window, so + a run the system ends still leaves the reason in its log. In a container of + 1 GB, 600 000 text files were ended after fifteen minutes with nothing said. + The same run finished where the container had swap, so it is warned rather + than refused. + - **A Docker image.** `ghcr.io/donislawdev/tfg` holds the Linux command line of a release and nothing else, for x64 and Arm, and runs with no network. It is made from the published archive once its checksum and attestation check diff --git a/internal/cli/generate.go b/internal/cli/generate.go index d33d5729..6d05f946 100644 --- a/internal/cli/generate.go +++ b/internal/cli/generate.go @@ -430,6 +430,7 @@ func produce(ctx context.Context, targets []engine.Target, opt engine.Options, g echoBoundaries(targets, planned, errOut) forecast, warned := echoManifestReach(ctx, planned, opt, errOut) opt.Record = forecast + echoHold(planned, opt, errOut) if g.dryRun { fmt.Fprintln(errOut, "dry run - nothing was written.") diff --git a/internal/cli/record.go b/internal/cli/record.go index 69c0982a..d387067e 100644 --- a/internal/cli/record.go +++ b/internal/cli/record.go @@ -106,6 +106,25 @@ func echoManifestReach(ctx context.Context, planned []engine.PlannedFile, opt en return &record, true } +// echoHold says so when a run will hold more than its memory setting gives it +// (O296, engine.PredictHold says why and what was measured). +// +// Before the first byte, and that is the point of it: a run the system ends +// prints nothing afterwards, so the sentence has to be in the log already. A +// note rather than a refusal, because the same run finishes where there is swap +// to carry it. And on a dry run, for the reason echoManifestReach gives. +func echoHold(planned []engine.PlannedFile, opt engine.Options, errOut io.Writer) { + hold := engine.PredictHold(len(planned), opt) + if !hold.Over { + return + } + fmt.Fprintf(errOut, + "note: this run will hold about %s in memory by its end, more than the %s the memory setting gives it, "+ + "so it will run slower and the system may end it before the manifest is written. "+ + "Split the run, or give it more with --memory if this machine has the room.\n", + core.HumanBytes(hold.Holds), core.HumanBytes(hold.Room)) +} + // echoManifestOutcome says what the manifest came to, when that is not what // the run was told before its first byte. // diff --git a/internal/engine/engine.go b/internal/engine/engine.go index 9da9f2d0..4a5d49de 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -196,17 +196,6 @@ func (o Options) availableBytes(path string) (int64, error) { return core.AvailableBytes(path) } -// memoryLimits is the memory setting worked out on this machine. Asked once -// by planning and once by the run, rather than carried between them, because -// the two are separate calls and a caller may plan without running. -func (o Options) memoryLimits() core.MemoryLimits { - machine := o.MachineMemory - if machine == nil { - machine = core.MachineMemory - } - return o.Memory.On(machine) -} - // Result is what a run produced. type Result struct { Manifest *manifest.Manifest @@ -242,6 +231,12 @@ type Result struct { // saved knows the real length afterwards, in Record of SaveRecord. Record RecordSize + // Hold is what this run was worked out to hold at its end, against what + // its memory setting gives it (PredictHold), before its first byte. A + // window says it after a preview and after a run, and the command line + // says it before the run from its own forecast. + Hold HoldForecast + // reservation is the run's hold on its manifest name, taken before the // first file and handed to SaveRecord, which saves through it. Nil for a // dry run and for a run refused before it reserved anything. @@ -533,6 +528,7 @@ func Run(ctx context.Context, files []PlannedFile, opt Options) (*Result, error) } limits := opt.memoryLimits() res.noteMemory(limits) + res.Hold = predictHold(len(files), limits) if opt.DryRun { // Nothing is written. Every entry is what would have been produced, diff --git a/internal/engine/holdforecast.go b/internal/engine/holdforecast.go new file mode 100644 index 00000000..664be3e4 --- /dev/null +++ b/internal/engine/holdforecast.go @@ -0,0 +1,85 @@ +// Part of package engine. See engine.go. +package engine + +import "github.com/donislawdev/TestingFilesGenerator/internal/core" + +// Whether a run will hold more than its memory setting gives it, said before +// its first byte (O296, docs/FILE-CEILING-2026-11-02.md section 13). +// +// A run holds its whole plan to the end, and every file adds its answer and its +// manifest entry as it finishes. The plan is already weighed while it is built +// (planmemory.go), and what the files add is not, so a run of 600 000 text files +// in a container of 1 GB passed every check, wrote for fifteen minutes and was +// ended by the system with no sentence and no manifest. With Docker's default +// swap the same run finished, so this says and does not refuse: whether a run +// past its setting is ended depends on swap the program cannot see. +// +// Measured against the setting rather than against the machine. Under auto the +// setting is half the machine, and that is also the limit the runtime is handed +// (processmemory.go): past it the collector works harder and the run is slower, +// which is certain, and the system may end it, which is not. In 1 GB, 300 000 +// text files held about 420 MB and finished at the usual speed, 400 000 and +// 500 000 held more than the 512 MiB and finished more slowly, and 600 000 held +// about 840 MB and was ended. + +// entryHolds is what one file adds to the memory of a run from the moment it is +// written to the end of the run: its answer and its manifest entry. +// +// Measured 2026-11-03 with the runsize probe, 20 000 files each, as the live +// heap after writing less the live heap after planning: 520 B a file for txt, +// png and zip, 519 for docx and 518 for pdf. The entry does not grow with the +// format, because the properties it carries are the target's own map, shared by +// reference. TestAFileAddsWhatTheForecastSaysToTheRunsMemory keeps the number +// honest. +const entryHolds = 520 + +// memoryLimits is the memory setting worked out on this machine. Asked once +// by planning and once by the run, rather than carried between them, because +// the two are separate calls and a caller may plan without running. +// +// Here rather than in engine.go since 2026-11-03, when the forecast below took +// that file past its ceiling: this file is about what a run holds against its +// setting, and this is where the setting meets the machine. +func (o Options) memoryLimits() core.MemoryLimits { + machine := o.MachineMemory + if machine == nil { + machine = core.MachineMemory + } + return o.Memory.On(machine) +} + +// HoldForecast is how much a run will hold at its end, against what its memory +// setting gives it. +type HoldForecast struct { + // Holds is the heap as the run starts, its plan among it, and what every + // file will add as it is written. + Holds int64 + // Room is what the memory setting gives the run. Nought when it gives no + // limit to measure against - off, or a machine that could not be asked. + Room int64 + // Over says Holds is past Room. + Over bool +} + +// PredictHold works out what a run of this many files will hold, for the +// sentence both surfaces say before the first byte. +// +// The heap is read the cheap way first, and that reading is never less than +// what is live, so a run it says has room has room. Only a run that may not is +// read again after a collection. That costs what heapInUse says it does, and it +// is paid by a run that is about to be warned. +func PredictHold(files int, opt Options) HoldForecast { + return predictHold(files, opt.memoryLimits()) +} + +func predictHold(files int, limits core.MemoryLimits) HoldForecast { + if limits.Run <= 0 || limits.Unknown != nil { + return HoldForecast{} + } + added := int64(files) * entryHolds + if held := int64(heapHeldAtMost()) + added; held <= limits.Run { + return HoldForecast{Holds: held, Room: limits.Run} + } + holds := int64(heapInUse()) + added + return HoldForecast{Holds: holds, Room: limits.Run, Over: holds > limits.Run} +} diff --git a/internal/guard/holdforecast_test.go b/internal/guard/holdforecast_test.go new file mode 100644 index 00000000..34a6d8a9 --- /dev/null +++ b/internal/guard/holdforecast_test.go @@ -0,0 +1,212 @@ +package guard + +// A run that will hold more than its memory setting gives it says so before its +// first byte (O296, docs/FILE-CEILING-2026-11-02.md section 13). +// +// Measured before this existed: 600 000 text files in a container of 1 GB passed +// every check, wrote for fifteen minutes and were ended by the system, with no +// sentence and no manifest. The same run with Docker's default swap finished, +// which is why this is a note and not a refusal. A run the system ends prints +// nothing afterwards, so the command line has to say it before it writes. + +import ( + "context" + "os" + "os/exec" + "runtime" + "strings" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/engine" + "github.com/donislawdev/TestingFilesGenerator/internal/gui/parts" + "github.com/donislawdev/TestingFilesGenerator/internal/gui/text" +) + +// holdsMore is the fixed start of the command line's sentence. +const holdsMore = "note: this run will hold about " + +// The command line says it before the dry run's last line, and says nothing +// when the setting has room. +// +// One megabyte is less than this test process holds before the run starts, so +// the forecast is past it whatever else ran before, and twenty text files plan +// far under it, so the plan's own ceiling lets the run through to be warned. +// Sixty four gigabytes is more than any runner this suite meets. +func TestARunThatWillHoldMoreThanItsMemorySaysSoBeforeItsFirstByte(t *testing.T) { + dry := func(memory string) string { + return runCLI(t, "generate", "--format", "txt", "--size", "200b", "--count", "20", + "--memory", memory, "--dry-run", "--out", t.TempDir()) + } + const last = "dry run - nothing was written." + + said := dry("1mb") + end := strings.Index(said, last) + if end < 0 { + t.Fatalf("the dry run under --memory 1mb was not accepted, so nothing here was asked about what it holds.\nIt said:\n%s", said) + } + at := strings.Index(said, holdsMore) + if at < 0 { + t.Fatalf("a run that will hold more than its 1 MiB said nothing about it.\n"+ + "A run past its memory setting is slower and may be ended by the system with no manifest, "+ + "and a run that is ended prints nothing afterwards.\nIt said:\n%s", said) + } + if at > end { + t.Errorf("the note came after the run's last line rather than before its first byte:\n%s", said) + } + if !strings.Contains(said[at:], "1.0 MB the memory setting gives it") { + t.Errorf("the note does not name the 1 MiB the setting gives the run:\n%s", said) + } + + quiet := dry("64gb") + if !strings.Contains(quiet, last) { + t.Fatalf("the dry run under --memory 64gb was not accepted, so its silence proves nothing.\nIt said:\n%s", quiet) + } + if strings.Contains(quiet, holdsMore) { + t.Errorf("a run with room under its setting was warned anyway, which teaches somebody to stop reading the line:\n%s", quiet) + } +} + +// The window says it after a preview and after a run, from the same forecast. +// +// The machine is the stand in host's, so auto is half of what the guard says +// it has: two megabytes, a megabyte for the run, which this process already +// holds more than. Left alone the host is a machine of sixteen gigabytes, and +// the same preview is quiet. The run is asked as well as the preview, because a +// preview is a button somebody may not press. +func TestTheWindowSaysWhenARunWillHoldMoreThanItsMemory(t *testing.T) { + marker := text.RunHoldsMoreThanItsMemory("HOLDS", "ROOM") + fixed := marker[:strings.Index(marker, "HOLDS")] + written := text.Written(0) + written = written[strings.LastIndex(written, " ")+1:] + + ask := func(machine int64, button, accepted string) string { + content, w, host := screenInAWindowWithHost(t, text.TabOneTarget()) + host.machine = machine + picker, ok := controlUnder(content, text.FieldFormat()).(*parts.Chooser) + if !ok { + t.Fatal("the format field is not a list to choose from, so this guard read the wrong tree") + } + picker.SetSelected("txt") + fill(t, content, text.FieldSize(), "200b") + fill(t, content, text.FieldCount(), "20") + fill(t, content, text.FieldOutputDir(), t.TempDir()) + press(t, content, button) + join(host) + settle(content, w) + _, status := runMessages(content) + if status == nil { + t.Fatal("the screen has no status line, so this guard read the wrong tree") + } + if !strings.Contains(status.Text, accepted) { + t.Fatalf("%s was not accepted, so nothing here was asked about what the run holds.\nIt said:\n%s", button, status.Text) + } + return status.Text + } + + if said := ask(2<<20, text.ButtonPreview(), text.AndNothingWrittenYet()); !strings.Contains(said, fixed) { + t.Errorf("a preview on a machine of 2 MiB said nothing about the run holding more than its setting.\n"+ + "The command line says this before the first byte.\nIt said:\n%s", said) + } + if said := ask(2<<20, text.ButtonGenerate(), written); !strings.Contains(said, fixed) { + t.Errorf("a run on a machine of 2 MiB, never previewed, said nothing about holding more than its setting:\n%s", said) + } + if said := ask(0, text.ButtonPreview(), text.AndNothingWrittenYet()); strings.Contains(said, fixed) { + t.Errorf("a preview on a machine of sixteen gigabytes was warned anyway:\n%s", said) + } +} + +// What the forecast counts for a file is what a file adds to the run's memory. +// +// The forecast carries one number for every file, measured with the runsize +// probe. A number typed beside the code and compared with nothing goes stale in +// silence, so this measures it again the way the probe did - the live heap after +// the run against the live heap before it, the plan and the result both held - +// and asks the forecast what it adds for the same files. +// +// In a process of its own. The heap of this package's process is shared by +// every guard that ran before (O162), and in a batch of eighty eight of them the +// same 2000 files measured 199 B a file against 534 alone - something freed +// between the two readings. A small run first, so whatever a first run builds +// once is not counted against the files. +func TestAFileAddsWhatTheForecastSaysToTheRunsMemory(t *testing.T) { + const name = "TestAFileAddsWhatTheForecastSaysToTheRunsMemory" + if os.Getenv(holdChild) != "1" { + run := exec.Command(os.Args[0], "-test.run=^"+name+"$", "-test.v") + run.Env = append(os.Environ(), holdChild+"=1") + out, err := run.CombinedOutput() + if err != nil || !strings.Contains(string(out), "--- PASS: "+name) { + t.Fatalf("%s did not pass in its own process (%v):\n%s", name, err, out) + } + return + } + + const files = 2000 + opt := engine.Options{OutDir: t.TempDir(), Seed: 1, Command: "test", Memory: core.MemoryOff()} + warm := opt + warm.OutDir = t.TempDir() + if _, err := engine.Run(context.Background(), planOf(t, 10, warm), warm); err != nil { + t.Fatalf("the run before the measured one: %v", err) + } + planned := planOf(t, files, opt) + + before := liveHeap() + res, err := engine.Run(context.Background(), planned, opt) + if err != nil { + t.Fatalf("the run: %v", err) + } + after := liveHeap() + runtime.KeepAlive(planned) + if len(res.Manifest.Files) != files { + t.Fatalf("the run recorded %d files of %d, so this did not measure what a run of them holds", len(res.Manifest.Files), files) + } + measured := float64(int64(after)-int64(before)) / files + + // A setting far over this process, so the forecast is worked out rather + // than left at nought, and a million files so that what it adds outweighs + // anything the heap does between the two readings. + room := engine.Options{Memory: core.MemoryOf(1 << 50)} + runtime.GC() + none := engine.PredictHold(0, room).Holds + runtime.GC() + million := engine.PredictHold(1_000_000, room).Holds + forecast := float64(million-none) / 1_000_000 + + if ratio := forecast / measured; ratio < 0.7 || ratio > 1.4 { + t.Errorf("the forecast counts %.0f B for a file and a run of %d text files added %.0f B a file.\n"+ + "Measure it again with tools/probes/runsize and write the number in internal/engine/holdforecast.go.", + forecast, files, measured) + } + + // And past the cheap reading: a setting 256 MiB over this heap has room + // for no files and none for a million, so the second reading, after a + // collection, is the one that answers - and it has to count the files too. + tight := engine.Options{Memory: core.MemoryOf(none + 256<<20)} + if engine.PredictHold(0, tight).Over { + t.Fatalf("a run of no files under a setting 256 MiB over this heap was judged over it, so the half below proves nothing") + } + if !engine.PredictHold(1_000_000, tight).Over { + t.Errorf("a million files, about 500 MB of entries, under a setting 256 MiB over this heap were not judged over it") + } +} + +// holdChild puts the measurement above in a process of its own. +const holdChild = "TFG_HOLD_FORECAST_CHILD" + +// planOf plans count text files of 200 B. +func planOf(t *testing.T, count int, opt engine.Options) []engine.PlannedFile { + t.Helper() + planned, err := engine.Plan([]engine.Target{txtTarget("t", count, 200)}, opt) + if err != nil { + t.Fatalf("planning %d files: %v", count, err) + } + return planned +} + +// liveHeap is the heap after a collection. +func liveHeap() uint64 { + runtime.GC() + var m runtime.MemStats + runtime.ReadMemStats(&m) + return m.HeapAlloc +} diff --git a/internal/guard/mutation/entries.py b/internal/guard/mutation/entries.py index 895596bb..35ac93a2 100644 --- a/internal/guard/mutation/entries.py +++ b/internal/guard/mutation/entries.py @@ -53,15 +53,15 @@ ( "the preview stops warning that the record will be too big to read back", "internal/gui/window/run.go", - "r.say(append(append([]string{said}, manifestReachNote(res.Record)...), runNotesOf(res)...)...)", - "r.say(append([]string{said}, runNotesOf(res)...)...)", + "r.say(append(append(append([]string{said}, manifestReachNote(res.Record)...), holdNote(res)...), runNotesOf(res)...)...)", + "r.say(append(append([]string{said}, holdNote(res)...), runNotesOf(res)...)...)", "TestTheWindowSaysWhenItsManifestWillBeTooBigToReadBack", ), ( "a finished run stops warning, so somebody who never pressed Preview is told nothing", "internal/gui/window/run.go", - "said := append([]string{outcome}, manifestReachNote(saved.Size)...)", - "said := []string{outcome}", + "said := append(append([]string{outcome}, manifestReachNote(saved.Size)...), holdNote(res)...)", + "said := append([]string{outcome}, holdNote(res)...)", "TestAFinishedRunInTheWindowSaysItsManifestIsTooBigToReadBack", ), ( @@ -6218,7 +6218,7 @@ ), ( "the ceiling a caller does not name stops being the real one, so ordinary short runs are refused", - "internal/engine/engine.go", + "internal/engine/holdforecast.go", "\treturn o.Memory.On(machine)", "\treturn core.MemoryOf(1).On(machine)", "TestAShortOrdinaryRunIsNotRefusedByThePlanCeiling", @@ -16241,8 +16241,8 @@ ( "a preview in the window does not say what it said about its memory", "internal/gui/window/run.go", - "manifestReachNote(res.Record)...), runNotesOf(res)...)...)", - "manifestReachNote(res.Record)...), notesOf(nil)...)...)", + "holdNote(res)...), runNotesOf(res)...)...)", + "holdNote(res)...), notesOf(nil)...)...)", "TestTheWindowSaysWhatARunSaidAboutItsMemory", ), ( @@ -16405,4 +16405,76 @@ "return c.readable()", "TestAFilmWhoseSquareCoversItsLabelSaysSoInItsPlan", ), + ( + # O296: the forecast counts a tenth of what a file adds. The guard + # measures a run of 2000 text files the way the runsize probe did and + # compares it with what the forecast adds for a million. + "the forecast of what a run holds counts a tenth of what a file adds", + "internal/engine/holdforecast.go", + "const entryHolds = 520", + "const entryHolds = 52", + "TestAFileAddsWhatTheForecastSaysToTheRunsMemory", + ), + ( + # The answer itself turned off, asked through the command line: a run + # under --memory 1mb must say it before the dry run's last line. + "a run that will hold more than its memory setting is never told on the command line", + "internal/engine/holdforecast.go", + "return HoldForecast{Holds: holds, Room: limits.Run, Over: holds > limits.Run}", + "return HoldForecast{Holds: holds, Room: limits.Run, Over: holds > limits.Run && false}", + "TestARunThatWillHoldMoreThanItsMemorySaysSoBeforeItsFirstByte", + ), + ( + # The same, asked through the window: a preview and a run on a machine + # of 2 MiB. + "a run that will hold more than its memory setting is never told in the window", + "internal/engine/holdforecast.go", + "return HoldForecast{Holds: holds, Room: limits.Run, Over: holds > limits.Run}", + "return HoldForecast{Holds: holds, Room: limits.Run, Over: holds > limits.Run && false}", + "TestTheWindowSaysWhenARunWillHoldMoreThanItsMemory", + ), + ( + # The second reading, after a collection, forgets the files. Only a run + # near its setting gets there, and the guard asks one whose heap fits + # and whose million files do not. + "the forecast after a collection forgets what the files add", + "internal/engine/holdforecast.go", + "\tholds := int64(heapInUse()) + added\n", + "\tholds := int64(heapInUse()) + added*0\n", + "TestAFileAddsWhatTheForecastSaysToTheRunsMemory", + ), + ( + # The cheap reading forgets the files, so a run of a million is + # forecast to hold what a run of none does. + "the cheap forecast of what a run holds forgets what the files add", + "internal/engine/holdforecast.go", + "if held := int64(heapHeldAtMost()) + added; held <= limits.Run {", + "if held := int64(heapHeldAtMost()) + added*0; held <= limits.Run {", + "TestAFileAddsWhatTheForecastSaysToTheRunsMemory", + ), + ( + # The command line stops asking. echoHold stays declared, so the tree + # still builds. + "the command line no longer says before its first byte that a run will hold too much", + "internal/cli/generate.go", + "\techoHold(planned, opt, errOut)\n", + "", + "TestARunThatWillHoldMoreThanItsMemorySaysSoBeforeItsFirstByte", + ), + ( + "a preview in the window no longer says that the run will hold too much", + "internal/gui/window/run.go", + "manifestReachNote(res.Record)...), holdNote(res)...), runNotesOf", + "manifestReachNote(res.Record)...), []string(nil)...), runNotesOf", + "TestTheWindowSaysWhenARunWillHoldMoreThanItsMemory", + ), + ( + # A run pressed straight away, never previewed, is the person the + # sentence after the run is for. + "a finished run in the window no longer says that it held too much", + "internal/gui/window/run.go", + "manifestReachNote(saved.Size)...), holdNote(res)...)", + "manifestReachNote(saved.Size)...), []string(nil)...)", + "TestTheWindowSaysWhenARunWillHoldMoreThanItsMemory", + ), ] diff --git a/internal/gui/text/locale/en.json b/internal/gui/text/locale/en.json index 18692570..2d569a2b 100644 --- a/internal/gui/text/locale/en.json +++ b/internal/gui/text/locale/en.json @@ -841,6 +841,11 @@ "hash": "sha256-86c4b674d84e", "other": "The first attempt to open a window did not succeed - usually a graphics driver without OpenGL 2.1. Starting again with the software renderer shipped beside the program." }, + "RunHoldsMoreThanItsMemory": { + "description": "Shown in the window. Carries these values, each of which has to stay spelled exactly that way: {{.Holds}}, {{.Room}}.", + "hash": "sha256-e40fb5c32602", + "other": "this run will hold about {{.Holds}} in memory by its end, more than the {{.Room}} the memory setting gives it, so it will run slower and the system may end it before its record is written. Split the run, or give it more Memory in Preferences if this machine has the room." + }, "SectionBase": { "description": "The heading over a group of fields.", "hash": "sha256-cbef193cc958", diff --git a/internal/gui/text/locale/pl.json b/internal/gui/text/locale/pl.json index 2986d42b..8609d52a 100644 --- a/internal/gui/text/locale/pl.json +++ b/internal/gui/text/locale/pl.json @@ -167,6 +167,7 @@ "RendererNotShipped": { "hash": "sha256-bfadaecbdd36", "other": "Dla tego systemu nie jest dostarczany programowy renderer, więc ta flaga niczego tu nie zmienia." }, "RendererStartFailed": { "hash": "sha256-3540e225c773", "other": "Ponowne uruchomienie programu z programowym rendererem nie powiodło się: {{.Error}}." }, "RendererStartingAgain": { "hash": "sha256-86c4b674d84e", "other": "Pierwsza próba otwarcia okna nie powiodła się - zwykle to sterownik grafiki bez OpenGL 2.1. Uruchamiam ponownie z programowym rendererem dostarczonym obok programu." }, + "RunHoldsMoreThanItsMemory": { "hash": "sha256-e40fb5c32602", "other": "ten przebieg będzie do końca trzymał w pamięci około {{.Holds}}, więcej niż {{.Room}}, które daje mu ustawienie pamięci, więc będzie wolniejszy, a system może go zakończyć, zanim powstanie jego zapis. Podziel przebieg albo daj mu więcej w sekcji Pamięć w Preferencjach, jeśli ta maszyna ma na to miejsce." }, "SectionBase": { "hash": "sha256-cbef193cc958", "other": "Budowanie na presecie" }, "SectionCarriedBeside": { "hash": "sha256-4cc79104fe33", "other": "Dostarczane obok programu w Windows" }, "SectionCarriedCode": { "hash": "sha256-b0f1aba6039b", "other": "Wkompilowany kod innych autorów" }, diff --git a/internal/gui/text/text.go b/internal/gui/text/text.go index be4ff714..d41fb553 100644 --- a/internal/gui/text/text.go +++ b/internal/gui/text/text.go @@ -238,6 +238,15 @@ func ManifestTooLargeToRead(size, limit string) string { map[string]any{"Size": size, "Limit": limit}) } +// RunHoldsMoreThanItsMemory is said when a run will hold more than its memory +// setting gives it - the window's half of the note the command line prints +// before the first byte (O296, engine.PredictHold). +func RunHoldsMoreThanItsMemory(holds, room string) string { + return sayf("RunHoldsMoreThanItsMemory", + "this run will hold about {{.Holds}} in memory by its end, more than the {{.Room}} the memory setting gives it, so it will run slower and the system may end it before its record is written. Split the run, or give it more Memory in Preferences if this machine has the room.", + map[string]any{"Holds": holds, "Room": room}) +} + // WorkingOutTheCost is what a preview says while it is going. // // A preview does disk work - it asks how much room there is and whether any of diff --git a/internal/gui/window/run.go b/internal/gui/window/run.go index f91b1fee..05d040ce 100644 --- a/internal/gui/window/run.go +++ b/internal/gui/window/run.go @@ -540,7 +540,7 @@ func (r *runner) previewFinished(res *engine.Result, planned []engine.PlannedFil r.line.measured(opt.OutDir, room.free) } said := r.line.said(exactly(planned), opt.OutDir) + text.AndNothingWrittenYet() - r.say(append(append([]string{said}, manifestReachNote(res.Record)...), runNotesOf(res)...)...) + r.say(append(append(append([]string{said}, manifestReachNote(res.Record)...), holdNote(res)...), runNotesOf(res)...)...) } // formatsOf is what kinds of file the run would produce, each named once. @@ -726,7 +726,7 @@ func (r *runner) runFinished(res *engine.Result, runErr, saveErr error, room dis // said only how many files - so the one thing this tool makes that others // do not was, from a window, something you found in the folder afterwards. outcome := text.SaidWithManifest(outcomeText(res, runErr), manifestNameOf(saved.Manifest)) - said := append([]string{outcome}, manifestReachNote(saved.Size)...) + said := append(append([]string{outcome}, manifestReachNote(saved.Size)...), holdNote(res)...) // Instructions that were due and are not there are said, not failed: the // manifest was saved and holds the same facts. Silence would leave a button // missing with no reason given. diff --git a/internal/gui/window/runreport.go b/internal/gui/window/runreport.go index 15a73123..a356d888 100644 --- a/internal/gui/window/runreport.go +++ b/internal/gui/window/runreport.go @@ -43,6 +43,21 @@ func manifestReachNote(record engine.RecordSize) []string { text.HumanBytes(record.Manifest), text.HumanBytes(manifest.MaxBytes))} } +// holdNote is the window's half of the note the command line prints before the +// first byte when a run will hold more than its memory setting gives it. +// +// Read off the engine (engine.Result.Hold), worked out by the function the +// command line asks, so the two surfaces cannot disagree about one run. A +// preview says it before anything is written, and a run says it afterwards, +// when it explains why the run was slow. A run that never started hands in +// nothing, and nothing is said. +func holdNote(res *engine.Result) []string { + if res == nil || !res.Hold.Over { + return nil + } + return []string{text.RunHoldsMoreThanItsMemory(text.HumanBytes(res.Hold.Holds), text.HumanBytes(res.Hold.Room))} +} + // progressText is the line under the bar. Bytes rather than files, because one // large file is a run where the file count says nothing for minutes. // From 2dd72e58914a8566256ea2749f8d6ffd197753d0 Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 3 Nov 2026 16:27:10 +0100 Subject: [PATCH 5/5] ci: the race job's processor summary says what its counters measure iowait is time the processors sat idle while disk I/O was pending, not time a test waited for the disk. steal depends on the platform and Microsoft documents it as not applicable to Azure, so nought there proves nothing about the host. The process list ranks by processor time since each process started, which ELAPSED dates. The labels now say so. The changelog entry for golang.org/x/net no longer joins two sentences with a comma. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 16 ++++++++++------ CHANGELOG.md | 6 +++--- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b039b5ca..882d4147 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1268,10 +1268,14 @@ jobs: - name: what the processor did while this part ran # The other half of the step that says what the machine is (O298). The # counters are the whole machine's, read just before the tests and - # again now, so time a host took away from this machine shows as taken - # by the host, and a process the part left running shows below with - # the time it has used. Runs after red tests as well, because a slow - # part is a red one. + # again now. iowait is time the processors sat idle while disk I/O was + # pending, not time a test spent waiting for it, and the kernel calls + # the counter unreliable. steal depends on the platform: Microsoft + # documents it as not applicable to Azure, so nought there proves + # nothing about the host. The processes are ranked by the processor + # time each has used since it started, which ELAPSED dates, so one the + # part left running stands out from the runner's own. Runs after red + # tests as well, because a slow part is a red one. if: ${{ !cancelled() }} run: | set -euo pipefail @@ -1288,10 +1292,10 @@ jobs: total=$(( busy + idle + iowait + steal )) echo "over $(( $(date +%s) - $(cat time-before.txt) )) s on $(nproc) processors:" if [ "$total" -gt 0 ]; then - echo " busy $(( 100 * busy / total ))%, idle $(( 100 * idle / total ))%, waiting on the disk $(( 100 * iowait / total ))%, taken by the host $(( 100 * steal / total ))%" + echo " busy $(( 100 * busy / total ))%, idle $(( 100 * idle / total ))%, idle with disk I/O pending (iowait) $(( 100 * iowait / total ))%, steal (platform dependent) $(( 100 * steal / total ))%" fi echo "load average: $(cat /proc/loadavg)" - echo "the processes that have used the most processor time, still running now:" + echo "the processes still running now, by processor time used since each started (ELAPSED):" ps -eo pid,ppid,etime,time,pcpu,comm --sort=-time > processes.txt head -n 12 processes.txt shell: bash diff --git a/CHANGELOG.md b/CHANGELOG.md index d6b26971..ca7131e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -135,9 +135,9 @@ because it turns other people's test suites red. tool was not affected. Building from source now needs Go 1.27.3. - **The window is built with `golang.org/x/net` 0.62.0.** The version before had two published vulnerabilities in the package that reads and writes HTML. - The window uses the package only to read the SVG pictures it draws, neither - of the two is in code it reaches, and the command line tool does not contain - the package at all. No generated file changes. + The window uses the package only to read the SVG pictures it draws. Neither + vulnerability is in code the window reaches, and the command line tool does + not contain the package at all. No generated file changes. ## [0.5.0] - 2026-10-20