Skip to content

Imports: a name the source refuses becomes a failed item, and the scan goes on - #892

Merged
schmitch merged 1 commit into
mainfrom
claude/752-import-refused-name-fails-item
Nov 4, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/752-import-refused-name-fails-item

Conversation

@schmitch

@schmitch schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes #752

What changed

The scan expands each listed name with requests of its own: a NuGet feed's versions (feed.versions), an npm packument (registry.package), a registry's tag list (source_tags). Before, any failure there other than a 404 failed the whole page through try_collect in expand, and so the import. A resume started again at the page before that name and failed the same way.

In crates/server/src/import/mod.rs:

  • expand reads each name through expand_read / read_name. Each name's expansion gets its own retries with the same backoff as a listing page (PAGE_ATTEMPTS). A failure that is still there after the retries (401, 403, 500, an invalid answer) becomes a Found::failed item that carries the reason in ItemSource::error, and the scan goes on with the next name.
  • npm scopes: each package of a named scope is handled the same way, so one refused package does not fail the scope's other packages.
  • copy_item finishes such an item (empty version, error set) as failed with that detail. It goes through Ok(copied("failed", …)), so it does not count towards MAX_CONSECUTIVE_FAILURES. Raw's refused paths are handled the same way.
  • A failure of the listing itself (catalog, search, /-/all) still fails the import, as before. So does SourceError::Unreachable (the source cannot be reached at all), so that a source that goes away is not recorded as a page of failed names.
  • retrying now wraps a new retried, which returns the SourceError rather than an End. The behaviour of listing reads is unchanged.

Tests

New tests in crates/server/tests/import.rs. The mock source gains refused, which answers chosen paths with a chosen status:

  • a_package_the_feed_refuses_is_a_failed_item_and_the_search_goes_on: the feed's search lists Acme.A/B/C, and flat/acme.b/index.json answers 403. Acme.A and Acme.C are imported, and Acme.B is a failed item whose detail names the 403.
  • a_package_the_npm_registry_fails_on_is_a_failed_item_and_the_names_go_on: named alpha, beta, gamma, and beta's packument answers 500 every time, so it goes through the retries. alpha and gamma are imported, and beta fails with the 500.
  • an_image_whose_tags_are_refused_is_a_failed_item_and_the_catalog_goes_on: the catalog (paged two at a time) lists team-a/app, team-b/app, tools/cli, and team-b/app/tags/list answers 401. The other two are imported, and team-b/app is a failed item.

Without the change in mod.rs, all three tests fail: the import ends failed. With it, the whole import test binary passes (23 tests), as do the import unit tests. I also ran cargo fmt --all --check, cargo clippy -p forge-server --all-targets -D warnings and cargo xtask changelog --check.

Changelog

docs/changes/2.2.0/752.fixed.md

Not fixed here

  • A refused image's failure detail reads "the upstream registry refused the proxy's credentials". That wording comes from the registry client's Failure::message, which is shared with proxy repositories, so in an import it names the wrong thing.
  • Retrying such a failed item on its own does not read the name at the source again: it fails again with the stored reason. A later run of the import scans the name again.

Generated by Claude Code

…n goes on

The scan expands each listed name with requests of its own (a NuGet
feed's versions, an npm packument, a registry's tag list). Any failure
other than a 404 failed the whole page, and with it the import; a resume
started again at the page before that name and failed the same way.

Each name's expansion is now retried on its own while its failure may
pass, and a failure that remains (401, 403, 500, ...) is recorded as an
item that fails with the source's answer, without counting towards the
failures in a row that stop an import. Only a failure of the listing
itself, or a source that cannot be reached at all, fails the import.

Fixes #752

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbek1nXahgwuyMAZaQ
@schmitch
schmitch force-pushed the claude/752-import-refused-name-fails-item branch from 65b90a7 to e008e97 Compare November 4, 2026 00:15
@schmitch
schmitch added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit f75439e Nov 4, 2026
19 checks passed
@schmitch
schmitch deleted the claude/752-import-refused-name-fails-item branch November 4, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Imports: one package or image the source refuses fails the whole scan, and every resume fails on it again

2 participants