Repository navigation
Imports: a name the source refuses becomes a failed item, and the scan goes on - #892
Merged
Merged
Conversation
…n goes on The scan expands each listed name with requests of its own (a NuGet feed's versions, an npm packument, a registry's tag list). Any failure other than a 404 failed the whole page, and with it the import; a resume started again at the page before that name and failed the same way. Each name's expansion is now retried on its own while its failure may pass, and a failure that remains (401, 403, 500, ...) is recorded as an item that fails with the source's answer, without counting towards the failures in a row that stop an import. Only a failure of the listing itself, or a source that cannot be reached at all, fails the import. Fixes #752 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJbek1nXahgwuyMAZaQ
schmitch
force-pushed
the
claude/752-import-refused-name-fails-item
branch
from
November 4, 2026 00:15
65b90a7 to
e008e97
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #752
What changed
The scan expands each listed name with requests of its own: a NuGet feed's versions (
feed.versions), an npm packument (registry.package), a registry's tag list (source_tags). Before, any failure there other than a 404 failed the whole page throughtry_collectinexpand, and so the import. A resume started again at the page before that name and failed the same way.In
crates/server/src/import/mod.rs:expandreads each name throughexpand_read/read_name. Each name's expansion gets its own retries with the same backoff as a listing page (PAGE_ATTEMPTS). A failure that is still there after the retries (401, 403, 500, an invalid answer) becomes aFound::faileditem that carries the reason inItemSource::error, and the scan goes on with the next name.copy_itemfinishes such an item (empty version,errorset) asfailedwith that detail. It goes throughOk(copied("failed", …)), so it does not count towardsMAX_CONSECUTIVE_FAILURES. Raw's refused paths are handled the same way./-/all) still fails the import, as before. So doesSourceError::Unreachable(the source cannot be reached at all), so that a source that goes away is not recorded as a page of failed names.retryingnow wraps a newretried, which returns theSourceErrorrather than anEnd. The behaviour of listing reads is unchanged.Tests
New tests in
crates/server/tests/import.rs. The mock source gainsrefused, which answers chosen paths with a chosen status:a_package_the_feed_refuses_is_a_failed_item_and_the_search_goes_on: the feed's search lists Acme.A/B/C, andflat/acme.b/index.jsonanswers 403. Acme.A and Acme.C are imported, and Acme.B is a failed item whose detail names the 403.a_package_the_npm_registry_fails_on_is_a_failed_item_and_the_names_go_on: namedalpha,beta,gamma, andbeta's packument answers 500 every time, so it goes through the retries.alphaandgammaare imported, andbetafails with the 500.an_image_whose_tags_are_refused_is_a_failed_item_and_the_catalog_goes_on: the catalog (paged two at a time) liststeam-a/app,team-b/app,tools/cli, andteam-b/app/tags/listanswers 401. The other two are imported, andteam-b/appis a failed item.Without the change in
mod.rs, all three tests fail: the import endsfailed. With it, the wholeimporttest binary passes (23 tests), as do theimportunit tests. I also rancargo fmt --all --check,cargo clippy -p forge-server --all-targets -D warningsandcargo xtask changelog --check.Changelog
docs/changes/2.2.0/752.fixed.mdNot fixed here
Failure::message, which is shared with proxy repositories, so in an import it names the wrong thing.Generated by Claude Code