Skip to content

Fix slow Go rescan from per-dep go.sum parse (#1424) - #1746

Open
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
agent/fix-go-sum-zip-versions-scan
Open

Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
agent/fix-go-sum-zip-versions-scan

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Fixes #1424

Summary

Hosted Go scans (golang/hosted, golang/rescan) got 23–34% slower after #1403. This PR gets most of that time back without changing what the rewriter decides.

Root cause

#1403 made the hosted Go rewriter (patch/redirect/mod.rs, the stale-pin cross-check) ask go.sum which versions of each patched module the build loads: GoSumEditor::zip_versions(module), once per patched dep. That call split every go.sum line on whitespace (3–4 SplitWhitespace::next calls per line), so a rescan cost (patched deps × go.sum lines) tokenizations. Callgrind on the bench fixture (1200 modules, 30 patched) put about 30% of all instructions there.

Fix

  • zip_versions drops a line whose first token can't be module with one byte-prefix compare (trim_start + strip_prefix(module) + a whitespace check) before tokenizing. Only lines that pass are parsed, through the new single-line parse_go_sum_line, which go_sum_lines now uses too. trim_start and split_whitespace share the Unicode whitespace rule, so any line whose first token equals module still passes, and the result is identical.
  • I didn't build a module → versions map once before the loop. The loop edits go.sum between deps (it prunes the replaced original's lines and removes stale hosted lines), and two deps can share a module path at different versions, so a map built once would go stale.

Test evidence

New unit tests in vendor/go_sum_edit.rs:

  • zip_versions_reads_only_the_modules_own_zip_lines: a shared path prefix (example.com/a vs example.com/ab / example.com/a/v2), /go.mod lines, leading tabs and spaces, two-token and one-token lines, extra tokens, case and CRLF, in both the Text and the split-Lines editor states.
  • zip_versions_matches_the_full_line_reader: compares against tokenizing every line (go_sum_lines) over NBSP and tab spacing and the empty module, in both editor states.

Both tests pass before and after the fix by design: they pin behavior, so the prefilter can't change an answer. The regression itself is a speed one, and the red→green evidence is the bench. Same 4-vCPU machine, interleaved, socket-patch-bench compare --base <main 4654b7c> --head <1994b4d> -f '^golang/' --runs 20:

scenario base wall head wall Δ wall [95% CI] Δ CPU requests
golang/hosted 116.1 ms 96.3 ms -16.6% [-19.5, -11.9] -15.1% 64
golang/rescan 119.6 ms 90.5 ms -20.8% [-22.7, -13.7] -20.1% 64

The regression added +23% (hosted) and +34% (rescan), which would need -19% and -25% to undo completely, so this gets most of it back. (cb8bd65 also removes a per-line str::lines() iterator, which can only help. I didn't bench again after it.)

Local runs:

  • cargo fmt --all -- --check, cargo clippy --workspace --all-features -- -D warnings: clean.
  • cargo test --workspace --all-features --no-fail-fast (two passes because the container restarted): about 7,600 passed, 7 failed. None of the 7 involve Go, and every one is a sandbox limit. Five force a write or remove failure with chmod, which can't fail as root (in_process_redirect::redirect_json_mode_write_failures_emit_error_envelope, partial_lockfile_write_failure_exits_1_and_writes_no_ledger, vlt::scan_redirect_vlt_heal_invalidation_failure_warns, repair_invariants::repair_cleanup_failure_is_reported_in_json_and_silent_modes, repair_render::repair_exits_zero_and_stays_quiet_when_lock_file_unremovable). Two in e2e_bun_lockb need patches-api.socket.dev, which this sandbox's proxy blocks ("tunnel error").
  • Go suites as go-compatibility.yml runs them (go 1.24.7, SOCKET_PATCH_GO_E2E_REQUIRED=1): e2e_golang_hosted_build 22/22, e2e_golang_hosted_state 29/29, e2e_golang_build 27/27, e2e_golang_workspace_build 16/16, e2e_vendor_golang_build 38/38, e2e_vex_lockfile -- golang:: 22/22.
  • Wrappers (npm/, pypi/, gem/) only dispatch to the binary, so they need no change.

Per-issue checklist

/code-review high

No correctness findings. Fixed: lines that pass the prefilter are now parsed directly instead of going back through str::lines(), and the equivalence test now covers the split-Lines state too.

Not fixed:

  • Use split_whitespace().next() == Some(module) as the prefilter. It's equivalent, but it walks the first token through the Unicode whitespace iterator. The byte prefix compare stops at the first differing byte, and that is where the speedup comes from.
  • The loop still makes other full passes over go.sum per dep (upsert_module_lines, remove_exact_module_version_lines, the stale-prefix removal), and scan_hosted_replace walks go.mod per dep. Those predate Fix Go replace for unselected versions (#391, #392, #509) #1403 and aren't part of this regression. Making the whole rewrite linear means an index that stays correct across the loop's edits and keeping the editor in Lines form throughout. That's a larger change, worth doing in a follow-up if huge monorepo go.sum files turn out to matter.

https://claude.ai/code/session_011f9aeNcKctDtmV7cb82g7W


Note

Low Risk
Logic change is localized to go.sum scanning with equivalence tests; output should match prior behavior while improving performance.

Overview
Speeds up hosted Go rescans by avoiding full whitespace tokenization of every go.sum line when collecting zip versions for a module.

GoSumEditor::zip_versions now prefilters lines with a cheap prefix check (trim_start + strip_prefix(module) + whitespace boundary) and only runs parsing on candidates. Line parsing is centralized in a new parse_go_sum_line, which go_sum_lines delegates to so behavior stays the same.

Tests lock module-path matching (prefix collisions, /go.mod lines, odd spacing, editor text vs split-lines state) and assert zip_versions still matches the old “tokenize every line” path.

Reviewed by Cursor Bugbot for commit cb8bd65. Configure here.


Generated by Claude Code

Assisted-by: Claude Code:claude-opus-5-5
Hosted Go scans got about a third slower in #1403. For each patched
dependency the rewriter asks go.sum which versions the build loads,
and that tokenized every go.sum line every time. On a project with
1200 modules and 30 patches this was about 30% of the scan's work.

Lines that can't belong to the module are now skipped with one
prefix check before tokenizing, so the answer stays the same. Tests
pin the edge cases: shared path prefixes, /go.mod lines, odd
whitespace and short lines.

Fixes #1424

Assisted-by: Claude Code:claude-opus-5-5
Lines that pass the module prefilter are now parsed directly instead
of going back through the multi-line reader. The equivalence test now
also checks the editor after an edit, when it holds split lines.

Refs #1424

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 27, 2026 21:40
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit cb8bd65. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 27, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review at head cb8bd65.

  • CI: 82/82 checks on cb8bd65 passed or skipped (32 success, 50 skipped by path filters). Mergeable with main; no main-wide failures.
  • Bugbot: reviewed cb8bd65, no new issues. No open review threads.
  • Reviewer: one file, crates/socket-patch-core/src/vendor/go_sum_edit.rs. It parses go.sum once per rescan instead of once per patched dep.
  • Slack: not announced, because this run had no Slack send tool.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perf regression: golang/rescan wall +34% (1d523d30, #1403)

2 participants