Repository navigation
perf(lock): resolve lockfile-missing packages in parallel - #3054
Open
mikeland73 wants to merge 4 commits into
Open
mikeland73 wants to merge 4 commits into
mikeland73 wants to merge 4 commits into
Conversation
lock.File.Resolve is now safe to call concurrently. A mutex guards the Packages map, but the search or flake request runs outside of it, and concurrent resolutions of the same package share one request via singleflight. Other File methods that touch Packages take the lock too. FillNarInfoCache uses this to check binary cache eligibility (which resolves packages that aren't locked yet) with up to 8 resolves at once instead of one at a time. This also speeds up `devbox add a b c`. Also fixes a data race in featureflag.Enabled, which wrote to a plain map to log env var overrides once, and passes *lock.File to cuecfg.WriteFile in multi sync since File now holds a mutex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Medium impact] The PR appears safe to merge.
|
The singleflight import adds a vendored package, which changes the flake's Go modules hash. The concurrency tests now release blocked handlers in a cleanup, so a failure fails after the timeout instead of hanging in httptest.Server.Close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
@greptileai please re-review. The hang you flagged in Written by Claude on behalf of / prompted by @mikeland73. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Tier 3 item 13 of
specs/perf-improvements.md.What
lock.File.Resolveis concurrency-safe. Async.MutexguardsPackagesand its entries, but is only held to read and write the entry. The search API /nix flake prefetchrequest runs outside the lock. Concurrent resolutions of the same key share one request viasingleflight, so e.g. many legacy packages callingStdenv()at once only resolve nixpkgs once.Get,Remove,Save,Tidy,UpdateStdenv,SetOutputsForPackage,HasAllowInsecurePackagesandisDirtytake the lock too.FillNarInfoCacheresolves in parallel. Binary cache eligibility (which callsresolve()for packages missing from the lockfile) now runs in anerrgroupwith a limit of 8, preserving input order. This is on theensureStateIsUpToDatepath, sodevbox add a b cresolves new packages concurrently.featureflag.Enabled: it wrote to a plain map to log env var overrides once; now async.Map. Found by-racein the new tests.multi syncpasses*lock.Filetocuecfg.WriteFile(vet copylocks, sinceFilenow has a mutex).Code that indexes
lockfile.Packagesdirectly (e.g.update.go,plugin.go) is unchanged; it runs serially and is documented as not safe to mix with concurrentFilemethod calls.Tests
TestResolveConcurrent: 5 packages × 4 callers. A fake search server blocks until all 5 requests are in flight at once (fails ifResolveholds the lock during the request), checks one request per package, and that every caller gets the stored entry.TestResolveConcurrentError: failures reach all callers, aren't stored, and are retried.TestEligibleForBinaryCacheResolvesConcurrently: same barrier approach forFillNarInfoCache's eligibility checks; verified it fails withSetLimit(1).go test -race ./internal/devpkg/... ./internal/lock/... ./internal/boxcli/...passes.🤖 Generated with Claude Code