Repository navigation
Fix slow Go rescan from per-dep go.sum parse (#1424) - #1746
Open
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
Open
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
Conversation
Assisted-by: Claude Code:claude-opus-5-5
Hosted Go scans got about a third slower in #1403. For each patched dependency the rewriter asks go.sum which versions the build loads, and that tokenized every go.sum line every time. On a project with 1200 modules and 30 patches this was about 30% of the scan's work. Lines that can't belong to the module are now skipped with one prefix check before tokenizing, so the answer stays the same. Tests pin the edge cases: shared path prefixes, /go.mod lines, odd whitespace and short lines. Fixes #1424 Assisted-by: Claude Code:claude-opus-5-5
Lines that pass the module prefilter are now parsed directly instead of going back through the multi-line reader. The equivalence test now also checks the editor after an edit, when it holds split lines. Refs #1424 Assisted-by: Claude Code:claude-opus-5-5
Mikola Lysenko (mikolalysenko)
marked this pull request as ready for review
October 27, 2026 21:40
Collaborator
Author
|
BugBot review Generated by Claude Code |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit cb8bd65. Configure here.
Collaborator
Author
|
Burn-down agent: labeled Ready for review at head
Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM Description written by Claude Code:claude-opus-5-5
Fixes #1424
Summary
Hosted Go scans (
golang/hosted,golang/rescan) got 23–34% slower after #1403. This PR gets most of that time back without changing what the rewriter decides.Root cause
#1403 made the hosted Go rewriter (
patch/redirect/mod.rs, the stale-pin cross-check) ask go.sum which versions of each patched module the build loads:GoSumEditor::zip_versions(module), once per patched dep. That call split every go.sum line on whitespace (3–4SplitWhitespace::nextcalls per line), so a rescan cost (patched deps × go.sum lines) tokenizations. Callgrind on the bench fixture (1200 modules, 30 patched) put about 30% of all instructions there.Fix
zip_versionsdrops a line whose first token can't bemodulewith one byte-prefix compare (trim_start+strip_prefix(module)+ a whitespace check) before tokenizing. Only lines that pass are parsed, through the new single-lineparse_go_sum_line, whichgo_sum_linesnow uses too.trim_startandsplit_whitespaceshare the Unicode whitespace rule, so any line whose first token equalsmodulestill passes, and the result is identical.Test evidence
New unit tests in
vendor/go_sum_edit.rs:zip_versions_reads_only_the_modules_own_zip_lines: a shared path prefix (example.com/avsexample.com/ab/example.com/a/v2),/go.modlines, leading tabs and spaces, two-token and one-token lines, extra tokens, case and CRLF, in both the Text and the split-Lines editor states.zip_versions_matches_the_full_line_reader: compares against tokenizing every line (go_sum_lines) over NBSP and tab spacing and the empty module, in both editor states.Both tests pass before and after the fix by design: they pin behavior, so the prefilter can't change an answer. The regression itself is a speed one, and the red→green evidence is the bench. Same 4-vCPU machine, interleaved,
socket-patch-bench compare --base <main 4654b7c> --head <1994b4d> -f '^golang/' --runs 20:golang/hostedgolang/rescanThe regression added +23% (hosted) and +34% (rescan), which would need -19% and -25% to undo completely, so this gets most of it back. (cb8bd65 also removes a per-line
str::lines()iterator, which can only help. I didn't bench again after it.)Local runs:
cargo fmt --all -- --check,cargo clippy --workspace --all-features -- -D warnings: clean.cargo test --workspace --all-features --no-fail-fast(two passes because the container restarted): about 7,600 passed, 7 failed. None of the 7 involve Go, and every one is a sandbox limit. Five force a write or remove failure withchmod, which can't fail as root (in_process_redirect::redirect_json_mode_write_failures_emit_error_envelope,partial_lockfile_write_failure_exits_1_and_writes_no_ledger,vlt::scan_redirect_vlt_heal_invalidation_failure_warns,repair_invariants::repair_cleanup_failure_is_reported_in_json_and_silent_modes,repair_render::repair_exits_zero_and_stays_quiet_when_lock_file_unremovable). Two ine2e_bun_lockbneedpatches-api.socket.dev, which this sandbox's proxy blocks ("tunnel error").go-compatibility.ymlruns them (go 1.24.7,SOCKET_PATCH_GO_E2E_REQUIRED=1):e2e_golang_hosted_build22/22,e2e_golang_hosted_state29/29,e2e_golang_build27/27,e2e_golang_workspace_build16/16,e2e_vendor_golang_build38/38,e2e_vex_lockfile -- golang::22/22.npm/,pypi/,gem/) only dispatch to the binary, so they need no change.Per-issue checklist
zip_versionsno longer tokenizes the whole go.sum per dep. Behavior is pinned by the two unit tests above, and the speedup shows in the bench table./code-review high
No correctness findings. Fixed: lines that pass the prefilter are now parsed directly instead of going back through
str::lines(), and the equivalence test now covers the split-Lines state too.Not fixed:
split_whitespace().next() == Some(module)as the prefilter. It's equivalent, but it walks the first token through the Unicode whitespace iterator. The byte prefix compare stops at the first differing byte, and that is where the speedup comes from.upsert_module_lines,remove_exact_module_version_lines, the stale-prefix removal), andscan_hosted_replacewalks go.mod per dep. Those predate Fix Go replace for unselected versions (#391, #392, #509) #1403 and aren't part of this regression. Making the whole rewrite linear means an index that stays correct across the loop's edits and keeping the editor in Lines form throughout. That's a larger change, worth doing in a follow-up if huge monorepo go.sum files turn out to matter.https://claude.ai/code/session_011f9aeNcKctDtmV7cb82g7W
Note
Low Risk
Logic change is localized to go.sum scanning with equivalence tests; output should match prior behavior while improving performance.
Overview
Speeds up hosted Go rescans by avoiding full whitespace tokenization of every
go.sumline when collecting zip versions for a module.GoSumEditor::zip_versionsnow prefilters lines with a cheap prefix check (trim_start+strip_prefix(module)+ whitespace boundary) and only runs parsing on candidates. Line parsing is centralized in a newparse_go_sum_line, whichgo_sum_linesdelegates to so behavior stays the same.Tests lock module-path matching (prefix collisions,
/go.modlines, odd spacing, editor text vs split-lines state) and assertzip_versionsstill matches the old “tokenize every line” path.Reviewed by Cursor Bugbot for commit cb8bd65. Configure here.
Generated by Claude Code