Skip to content

perf(lock): resolve lockfile-missing packages in parallel - #3054

Open
mikeland73 wants to merge 4 commits into
mainfrom
claude/perf-parallel-resolve-6796d5
Open

mikeland73 wants to merge 4 commits into
mainfrom
claude/perf-parallel-resolve-6796d5

Conversation

@mikeland73

Copy link
Copy Markdown
Collaborator

Implements Tier 3 item 13 of specs/perf-improvements.md.

What

  • lock.File.Resolve is concurrency-safe. A sync.Mutex guards Packages and its entries, but is only held to read and write the entry. The search API / nix flake prefetch request runs outside the lock. Concurrent resolutions of the same key share one request via singleflight, so e.g. many legacy packages calling Stdenv() at once only resolve nixpkgs once. Get, Remove, Save, Tidy, UpdateStdenv, SetOutputsForPackage, HasAllowInsecurePackages and isDirty take the lock too.
  • FillNarInfoCache resolves in parallel. Binary cache eligibility (which calls resolve() for packages missing from the lockfile) now runs in an errgroup with a limit of 8, preserving input order. This is on the ensureStateIsUpToDate path, so devbox add a b c resolves new packages concurrently.
  • Race fix in featureflag.Enabled: it wrote to a plain map to log env var overrides once; now a sync.Map. Found by -race in the new tests.
  • multi sync passes *lock.File to cuecfg.WriteFile (vet copylocks, since File now has a mutex).

Code that indexes lockfile.Packages directly (e.g. update.go, plugin.go) is unchanged; it runs serially and is documented as not safe to mix with concurrent File method calls.

Tests

  • TestResolveConcurrent: 5 packages × 4 callers. A fake search server blocks until all 5 requests are in flight at once (fails if Resolve holds the lock during the request), checks one request per package, and that every caller gets the stored entry.
  • TestResolveConcurrentError: failures reach all callers, aren't stored, and are retried.
  • TestEligibleForBinaryCacheResolvesConcurrently: same barrier approach for FillNarInfoCache's eligibility checks; verified it fails with SetLimit(1).
  • go test -race ./internal/devpkg/... ./internal/lock/... ./internal/boxcli/... passes.

🤖 Generated with Claude Code

mikeland73 and others added 2 commits November 13, 2026 09:28
lock.File.Resolve is now safe to call concurrently. A mutex guards the
Packages map, but the search or flake request runs outside of it, and
concurrent resolutions of the same package share one request via
singleflight. Other File methods that touch Packages take the lock too.

FillNarInfoCache uses this to check binary cache eligibility (which
resolves packages that aren't locked yet) with up to 8 resolves at once
instead of one at a time. This also speeds up `devbox add a b c`.

Also fixes a data race in featureflag.Enabled, which wrote to a plain map
to log env var overrides once, and passes *lock.File to cuecfg.WriteFile
in multi sync since File now holds a mutex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 13, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

The PR resolves missing lockfile entries in parallel, shares requests for the same package, and makes feature-flag logging safe across goroutines.

  • Devbox checks package eligibility for the binary cache in parallel.
  • Lockfile lookups can run together without duplicating the same request.
  • Feature-flag overrides are logged once even when callers run together.
  • The performance plan and vendor hash reflect this change.

Reviews (2) · Last reviewed commit: "fix: update vendor-hash; release blocked..." · Reviewed by Greptile

Comment thread internal/lock/lockfile_test.go Resolved
The singleflight import adds a vendored package, which changes the
flake's Go modules hash.

The concurrency tests now release blocked handlers in a cleanup, so a
failure fails after the timeout instead of hanging in httptest.Server.Close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikeland73

Copy link
Copy Markdown
Collaborator Author

@greptileai please re-review. The hang you flagged in TestResolveConcurrent is fixed in 1f9acb2 (blocked handlers are released in cleanup before the server closes), and the vendor-hash is updated. If there are no remaining issues, please approve.


Written by Claude on behalf of / prompted by @mikeland73.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant