diff --git a/README.md b/README.md index 0e7b55a4..9d4ad264 100644 --- a/README.md +++ b/README.md @@ -559,6 +559,12 @@ docker pull localhost:8080/library/nginx:latest The registry is pull-only. BuildKit cache exports (`--cache-to type=registry`) and other pushes need a separate registry. +An image's page in the web UI (`/ui/package/oci/{name}`) lists its tags, the +most recently seen first, with the digest each one pointed at when the +upstream last confirmed it, through a pull or a mirror run. Its versions are +the cached blobs, by digest, and a blob's page names the tags whose cached +image contains it. + SBOMs, signatures and attestations attached to an image are found through the OCI 1.1 referrers API (`GET /v2/{name}/referrers/{digest}`). `oras discover` and `notation verify` use it, and so does cosign for its newer signature diff --git a/docs/architecture.md b/docs/architecture.md index c498cc7f..d8b8d2e2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -214,11 +214,21 @@ image_manifests ( recorded_at INTEGER NOT NULL, -- Unix ms; BIGINT on Postgres PRIMARY KEY (package_purl, digest) ) + +image_tags ( + package_purl TEXT NOT NULL, -- the repository, as its blobs are cached + tag TEXT NOT NULL, + digest TEXT NOT NULL, -- the manifest the tag pointed at + seen_at INTEGER NOT NULL, -- Unix ms the upstream last confirmed it; BIGINT on Postgres + PRIMARY KEY (package_purl, tag) +) ``` `metadata_dates` holds the `Last-Modified` served with rewritten metadata, such as npm packuments, Composer documents, PyPI simple pages and Helm indexes. A package's date is the second it started serving its current representation, and a new representation is only dated in a later second. Keeping it in the database means replicas sharing one serve the same date, and never give one representation's date to another, so `If-Modified-Since` cannot return 304 for a body the client does not have. -`image_manifests` is filled only with `storage.retention.keep_latest` set and a retention rule for `oci`. It holds which manifests and blobs each served container manifest references, so the retention sweep can keep the blobs of the newest images of a repository. The sweep also drops the rows of images that are no longer among the newest. +`image_manifests` holds which manifests and blobs each served container manifest references. The retention sweep reads it to keep the blobs of the newest images of a repository, and the web UI to name the tags whose image contains a blob. An hourly prune, which runs with or without a retention rule, drops the rows of images that are no longer among the newest and that no tag points at. + +`image_tags` holds the tags of a container image repository, each with the digest it pointed at when the upstream last confirmed it, for the web UI. The same prune drops tags the upstream has not confirmed for 90 days. A tag served as one platform's manifest of the index it points at, to a client that accepts no index, keeps the index's digest. On PostgreSQL, `INTEGER PRIMARY KEY` becomes `SERIAL`, `DATETIME` becomes `TIMESTAMP`, `INTEGER DEFAULT 0` booleans become `BOOLEAN DEFAULT FALSE`, and size/count columns use `BIGINT`. diff --git a/docs/configuration.md b/docs/configuration.md index b75650fc..4f0d61fc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -131,9 +131,9 @@ With this, the files of the two highest cached versions of a package are never e For `oci` there are no versions to order, so `keep_latest` counts images instead: it keeps the blobs of the images of a repository that a tag most recently pointed at. With `keep_latest: 2` and one tag such as `beta` in use, that is the image `beta` points at now and the one it pointed at before. The count is per repository, not per tag, so with several tags in use set it at least that high. A blob that a kept image shares with an older one stays. -To know which blobs an image needs, the proxy records what each manifest it serves references, in the `image_manifests` table, while `keep_latest` is set and a retention rule applies to `oci`. This has a few consequences: +To know which blobs an image needs, the proxy records what each manifest it serves references, in the `image_manifests` table. This has a few consequences: -- An image only counts once its tag was pulled or mirrored with `keep_latest` set, and its blobs are known once its platform manifest was pulled too. Until then they expire by their rule as before. +- An image only counts once its tag was pulled or mirrored, and its blobs are known once its platform manifest was pulled too. Until then they expire by their rule as before. The records are pruned hourly, also without a retention rule: while `keep_latest` is off, those of images that no tag points at any more go, so they don't count when it is turned on, and so do tags not pulled for 90 days. - An image pulled only by digest is not counted and not kept. - A multi-platform tag counts as one image, also when a client that accepts no image index was served one platform's manifest for it. - Homebrew bottles and Helm charts pulled over OCI are images here: the newest versions of a bottle or chart are those whose tags were last asked for. diff --git a/internal/database/database_test.go b/internal/database/database_test.go index 8a2e0574..bc07bdba 100644 --- a/internal/database/database_test.go +++ b/internal/database/database_test.go @@ -613,7 +613,7 @@ func createTestPostgresDB(t *testing.T) *DB { // Drop and recreate every table CreateSchema creates for clean test state; // leftover migration records make the next CreateSchema fail on the // migrations primary key. - tables := []string{"artifacts", "pending_deletes", "versions", "packages", "vulnerabilities", "metadata_cache", "metadata_dates", "image_manifests", "job_runs", "fetch_leases", "upstream_holds", "metadata_snapshots", "projects", "project_components", "migrations", "schema_info"} + tables := []string{"artifacts", "pending_deletes", "versions", "packages", "vulnerabilities", "metadata_cache", "metadata_dates", "image_manifests", "image_tags", "job_runs", "fetch_leases", "upstream_holds", "metadata_snapshots", "projects", "project_components", "migrations", "schema_info"} for _, table := range tables { _, _ = db.Exec("DROP TABLE IF EXISTS " + table + " CASCADE") } diff --git a/internal/database/image_manifests.go b/internal/database/image_manifests.go index c88af1b8..0db9e8e0 100644 --- a/internal/database/image_manifests.go +++ b/internal/database/image_manifests.go @@ -21,6 +21,10 @@ const ( imageWalkBatch = 500 ) +// imagePruneBatch is how many manifests one prune query deletes, well below +// the bind limits of SQLite and PostgreSQL. A variable so tests can lower it. +var imagePruneBatch = 500 + // ImageManifest is what one container image manifest of a repository // references. type ImageManifest struct { @@ -99,45 +103,100 @@ func (db *DB) ImageManifestRepositories() ([]string, error) { return repositories, err } -// PruneImageManifests removes the manifests of a repository that its n -// newest images do not reference and that were recorded before the given -// time. The time leaves a manifest recorded shortly before the one that -// references it in place. +// PruneImageManifests removes the manifests of a repository that neither +// its n newest images nor the images its tags point at reference, and that +// were recorded before the given time. The time leaves a manifest recorded +// shortly before the one that references it in place. What the tags point +// at stays for ImageTagsContaining. func (db *DB) PruneImageManifests(packagePURL string, n int, before time.Time) error { - _, manifests, err := db.walkNewestImages(packagePURL, n) + roots, err := db.newestImages(packagePURL, n) + if err != nil { + return err + } + var tagged []imageManifestRow + err = db.Select(&tagged, db.Rebind(` + SELECT digest, refs FROM image_manifests WHERE package_purl = ? + AND digest IN (SELECT digest FROM image_tags WHERE package_purl = ?) + `), packagePURL, packagePURL) + if err != nil { + return err + } + _, manifests, err := db.walkImages(packagePURL, append(roots, tagged...)) if err != nil { return err } - query := `DELETE FROM image_manifests WHERE package_purl = ? AND recorded_at < ?` - args := []any{packagePURL, before.UnixMilli()} - if len(manifests) > 0 { - query, args, err = sqlx.In(query+` AND digest NOT IN (?)`, packagePURL, before.UnixMilli(), manifests) + keep := make(map[string]struct{}, len(manifests)) + for _, digest := range manifests { + keep[digest] = struct{}{} + } + + var old []string + err = db.Select(&old, db.Rebind(` + SELECT digest FROM image_manifests WHERE package_purl = ? AND recorded_at < ? + `), packagePURL, before.UnixMilli()) + if err != nil { + return err + } + drop := old[:0] + for _, digest := range old { + if _, ok := keep[digest]; !ok { + drop = append(drop, digest) + } + } + // Deleted by name in batches: a list of the kept ones can outgrow what + // one query may bind. + for start := 0; start < len(drop); start += imagePruneBatch { + batch := drop[start:min(start+imagePruneBatch, len(drop))] + query, args, err := sqlx.In(`DELETE FROM image_manifests WHERE package_purl = ? AND digest IN (?)`, packagePURL, batch) if err != nil { return fmt.Errorf("building prune query: %w", err) } + if _, err := db.Exec(db.Rebind(query), args...); err != nil { + return err + } } - _, err = db.Exec(db.Rebind(query), args...) - return err + return nil +} + +// ImageManifestRefs returns the digests a recorded manifest of a repository +// references, or none for one that is not recorded. +func (db *DB) ImageManifestRefs(packagePURL, digest string) ([]string, error) { + var row imageManifestRow + err := db.Get(&row, db.Rebind(` + SELECT digest, refs FROM image_manifests WHERE package_purl = ? AND digest = ? + `), packagePURL, digest) + if errors.Is(err, sql.ErrNoRows) { + return nil, nil + } + if err != nil { + return nil, err + } + return row.refs(), nil } // walkNewestImages returns every digest the n newest images of a repository // reference, themselves included, and those among them that are recorded // manifests. func (db *DB) walkNewestImages(packagePURL string, n int) (map[string]struct{}, []string, error) { - digests := map[string]struct{}{} - if n <= 0 { - return digests, nil, nil - } rows, err := db.newestImages(packagePURL, n) if err != nil { return nil, nil, err } + return db.walkImages(packagePURL, rows) +} - var manifests []string - for _, row := range rows { +// walkImages returns every digest the given manifests of a repository +// reference, themselves included, and those among them that are recorded +// manifests. +func (db *DB) walkImages(packagePURL string, roots []imageManifestRow) (map[string]struct{}, []string, error) { + digests := map[string]struct{}{} + for _, row := range roots { digests[row.Digest] = struct{}{} } - for depth := 0; len(rows) > 0; depth++ { + + var manifests []string + var err error + for depth, rows := 0, roots; len(rows) > 0; depth++ { var next []string for _, row := range rows { manifests = append(manifests, row.Digest) @@ -165,6 +224,9 @@ func (db *DB) walkNewestImages(packagePURL string, n int) (map[string]struct{}, // manifest that another tagged one references is that image seen another // way: it is no image of its own and lends the other its tag time. func (db *DB) newestImages(packagePURL string, n int) ([]imageManifestRow, error) { + if n <= 0 { + return nil, nil + } var rows []imageManifestRow err := db.Select(&rows, db.Rebind(` SELECT digest, refs, tagged_at FROM image_manifests diff --git a/internal/database/image_tags.go b/internal/database/image_tags.go new file mode 100644 index 00000000..e0d55d3e --- /dev/null +++ b/internal/database/image_tags.go @@ -0,0 +1,133 @@ +package database + +import ( + "database/sql" + "errors" + "time" +) + +// ImageTag is a tag of a container image repository as the proxy last saw +// it upstream. +type ImageTag struct { + Tag string `db:"tag"` + Digest string `db:"digest"` + // SeenAt is when the upstream last said the tag points at Digest. + SeenAt time.Time `db:"-"` +} + +type imageTagRow struct { + Tag string `db:"tag"` + Digest string `db:"digest"` + SeenAt int64 `db:"seen_at"` +} + +// RecordImageTag stores that a tag of a repository pointed at digest at the +// given time. An older sighting than the stored one changes nothing. +func (db *DB) RecordImageTag(packagePURL, tag, digest string, seenAt time.Time) error { + _, err := db.Exec(db.Rebind(` + INSERT INTO image_tags (package_purl, tag, digest, seen_at) VALUES (?, ?, ?, ?) + ON CONFLICT (package_purl, tag) DO UPDATE SET digest = excluded.digest, seen_at = excluded.seen_at + WHERE image_tags.seen_at < excluded.seen_at + `), packagePURL, tag, digest, seenAt.UnixMilli()) + return err +} + +// ImageTag returns a tag of a repository as stored, and whether there is one. +func (db *DB) ImageTag(packagePURL, tag string) (ImageTag, bool, error) { + var row imageTagRow + err := db.Get(&row, db.Rebind(` + SELECT tag, digest, seen_at FROM image_tags WHERE package_purl = ? AND tag = ? + `), packagePURL, tag) + if errors.Is(err, sql.ErrNoRows) { + return ImageTag{}, false, nil + } + if err != nil { + return ImageTag{}, false, err + } + return ImageTag{Tag: row.Tag, Digest: row.Digest, SeenAt: time.UnixMilli(row.SeenAt)}, true, nil +} + +// TouchImageTag moves the time a tag was last seen forward and leaves its +// digest. +func (db *DB) TouchImageTag(packagePURL, tag string, seenAt time.Time) error { + _, err := db.Exec(db.Rebind(` + UPDATE image_tags SET seen_at = ? WHERE package_purl = ? AND tag = ? AND seen_at < ? + `), seenAt.UnixMilli(), packagePURL, tag, seenAt.UnixMilli()) + return err +} + +// PruneImageTags removes the tags the upstream has not confirmed since the +// given time, in every repository. +func (db *DB) PruneImageTags(before time.Time) error { + _, err := db.Exec(db.Rebind(`DELETE FROM image_tags WHERE seen_at < ?`), before.UnixMilli()) + return err +} + +// ImageTags returns up to limit tags of a repository, the most recently seen +// first. +func (db *DB) ImageTags(packagePURL string, limit int) ([]ImageTag, error) { + var rows []imageTagRow + err := db.Select(&rows, db.Rebind(` + SELECT tag, digest, seen_at FROM image_tags WHERE package_purl = ? + ORDER BY seen_at DESC, tag LIMIT ? + `), packagePURL, limit) + if err != nil { + return nil, err + } + tags := make([]ImageTag, len(rows)) + for i, row := range rows { + tags[i] = ImageTag{Tag: row.Tag, Digest: row.Digest, SeenAt: time.UnixMilli(row.SeenAt)} + } + return tags, nil +} + +// ImageTagsContaining returns up to limit tags of a repository whose image +// contains digest, the most recently seen first: tags pointing at digest, +// at a manifest that references it, or at an index listing such a manifest. +// It reads the references recorded in image_manifests. +func (db *DB) ImageTagsContaining(packagePURL, digest string, limit int) ([]string, error) { + var manifests []imageManifestRow + err := db.Select(&manifests, db.Rebind(` + SELECT digest, refs FROM image_manifests WHERE package_purl = ? + `), packagePURL) + if err != nil { + return nil, err + } + referencedBy := make(map[string][]string) + for _, manifest := range manifests { + for _, ref := range manifest.refs() { + referencedBy[ref] = append(referencedBy[ref], manifest.Digest) + } + } + containing := map[string]struct{}{digest: {}} + for next := []string{digest}; len(next) > 0; { + var parents []string + for _, child := range next { + for _, parent := range referencedBy[child] { + if _, seen := containing[parent]; !seen { + containing[parent] = struct{}{} + parents = append(parents, parent) + } + } + } + next = parents + } + + var rows []imageTagRow + err = db.Select(&rows, db.Rebind(` + SELECT tag, digest, seen_at FROM image_tags WHERE package_purl = ? ORDER BY seen_at DESC, tag + `), packagePURL) + if err != nil { + return nil, err + } + var tags []string + for _, row := range rows { + if len(tags) == limit { + break + } + if _, ok := containing[row.Digest]; ok { + tags = append(tags, row.Tag) + } + } + return tags, nil +} diff --git a/internal/database/image_tags_test.go b/internal/database/image_tags_test.go new file mode 100644 index 00000000..aa2948b6 --- /dev/null +++ b/internal/database/image_tags_test.go @@ -0,0 +1,289 @@ +package database + +import ( + "fmt" + "slices" + "testing" + "time" +) + +const testTagRepo = "pkg:oci/pihole/pihole" + +func recordTag(t *testing.T, db *DB, repo, tag, digest string, seen time.Time) { + t.Helper() + if err := db.RecordImageTag(repo, tag, digest, seen); err != nil { + t.Fatalf("RecordImageTag(%s): %v", tag, err) + } +} + +// The tags of a repository are listed most recently seen first, each with +// the digest it pointed at when last seen. +func TestImageTagsListsTagsMostRecentlySeenFirst(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now().Truncate(time.Millisecond) + recordTag(t, db, testTagRepo, "2026.10.0", "sha256:old", now.Add(-48*time.Hour)) + recordTag(t, db, testTagRepo, "latest", "sha256:old", now.Add(-time.Hour)) + recordTag(t, db, testTagRepo, "2026.11.0", "sha256:new", now.Add(-time.Minute)) + recordTag(t, db, "pkg:oci/library/alpine", "latest", "sha256:alpine", now) + + // latest moves to the new image. + recordTag(t, db, testTagRepo, "latest", "sha256:new", now) + // A sighting older than the stored one changes nothing. + recordTag(t, db, testTagRepo, "2026.11.0", "sha256:other", now.Add(-time.Hour)) + + tags, err := db.ImageTags(testTagRepo, 10) + if err != nil { + t.Fatalf("ImageTags: %v", err) + } + want := []ImageTag{ + {Tag: "latest", Digest: "sha256:new", SeenAt: now}, + {Tag: "2026.11.0", Digest: "sha256:new", SeenAt: now.Add(-time.Minute)}, + {Tag: "2026.10.0", Digest: "sha256:old", SeenAt: now.Add(-48 * time.Hour)}, + } + if len(tags) != len(want) { + t.Fatalf("ImageTags = %+v, want %+v", tags, want) + } + for i := range want { + if tags[i].Tag != want[i].Tag || tags[i].Digest != want[i].Digest || !tags[i].SeenAt.Equal(want[i].SeenAt) { + t.Errorf("tag %d = %+v, want %+v", i, tags[i], want[i]) + } + } + + if tags, err := db.ImageTags(testTagRepo, 1); err != nil || len(tags) != 1 || tags[0].Tag != "latest" { + t.Errorf("ImageTags(limit 1) = %+v, %v; want latest only", tags, err) + } + }) +} + +func TestMigrateAddsImageTags(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + if _, err := db.Exec("DROP TABLE image_tags"); err != nil { + t.Fatal(err) + } + if _, err := db.Exec("DELETE FROM migrations WHERE name = '025_add_image_tags'"); err != nil { + t.Fatal(err) + } + if err := db.MigrateSchema(); err != nil { + t.Fatalf("MigrateSchema: %v", err) + } + recordTag(t, db, testTagRepo, "latest", "sha256:a", time.Now()) + if tags, err := db.ImageTags(testTagRepo, 10); err != nil || len(tags) != 1 { + t.Errorf("ImageTags after migration = %+v, %v; want one tag", tags, err) + } + }) +} + +func recordReferences(t *testing.T, db *DB, repo, manifest string, references ...string) { + t.Helper() + if err := db.RecordImageManifest(ImageManifest{PackagePURL: repo, Digest: manifest, Refs: references}); err != nil { + t.Fatalf("RecordImageManifest(%s): %v", manifest, err) + } +} + +// The tags whose image contains a digest: the manifest a tag points at, the +// platform manifests of an index, and their configs and layers. +func TestImageTagsContaining(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now() + recordReferences(t, db, testTagRepo, "sha256:index", "sha256:amd64", "sha256:arm64") + recordReferences(t, db, testTagRepo, "sha256:amd64", "sha256:amd64-config", "sha256:shared", "sha256:amd64-layer") + recordReferences(t, db, testTagRepo, "sha256:single", "sha256:single-config", "sha256:shared") + // The same digests in another repository say nothing about this one. + recordReferences(t, db, "pkg:oci/library/alpine", "sha256:alpine", "sha256:single-config") + recordReferences(t, db, "pkg:oci/library/alpine", "sha256:index", "sha256:alpine-layer") + recordReferences(t, db, "pkg:oci/library/alpine", "sha256:single", "sha256:amd64") + recordTag(t, db, "pkg:oci/library/alpine", "latest", "sha256:alpine", now) + + recordTag(t, db, testTagRepo, "multi", "sha256:index", now) + recordTag(t, db, testTagRepo, "single", "sha256:single", now.Add(-time.Minute)) + recordTag(t, db, testTagRepo, "old", "sha256:index", now.Add(-time.Hour)) + + for _, tt := range []struct { + digest string + want []string + }{ + {"sha256:index", []string{"multi", "old"}}, + {"sha256:amd64", []string{"multi", "old"}}, + {"sha256:arm64", []string{"multi", "old"}}, + {"sha256:amd64-layer", []string{"multi", "old"}}, + {"sha256:shared", []string{"multi", "single", "old"}}, + {"sha256:single-config", []string{"single"}}, + {"sha256:alpine-layer", nil}, + {"sha256:unknown", nil}, + } { + got, err := db.ImageTagsContaining(testTagRepo, tt.digest, 10) + if err != nil { + t.Fatalf("ImageTagsContaining(%s): %v", tt.digest, err) + } + if !slices.Equal(got, tt.want) { + t.Errorf("ImageTagsContaining(%s) = %v, want %v", tt.digest, got, tt.want) + } + } + + if got, _ := db.ImageTagsContaining(testTagRepo, "sha256:shared", 2); !slices.Equal(got, []string{"multi", "single"}) { + t.Errorf("ImageTagsContaining(limit 2) = %v, want multi and single", got) + } + }) +} + +// The limit applies to the tags that match, so a match among older tags is +// found however many newer tags there are. +func TestImageTagsContainingLimitsTheMatches(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now() + recordReferences(t, db, testTagRepo, "sha256:old", "sha256:layer") + recordTag(t, db, testTagRepo, "oldest", "sha256:old", now.Add(-time.Hour)) + for i := range 5 { + recordTag(t, db, testTagRepo, fmt.Sprintf("v%d", i), "sha256:new", now) + } + if got, _ := db.ImageTagsContaining(testTagRepo, "sha256:layer", 2); !slices.Equal(got, []string{"oldest"}) { + t.Errorf("ImageTagsContaining = %v, want oldest", got) + } + }) +} + +// An index can list another index, so the walk from a blob up to the tags +// goes as far as the references do, and a manifest that references itself +// does not stop it. +func TestImageTagsContainingFollowsNestedIndexes(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + recordReferences(t, db, testTagRepo, "sha256:outer", "sha256:inner") + recordReferences(t, db, testTagRepo, "sha256:inner", "sha256:platform", "sha256:inner") + recordReferences(t, db, testTagRepo, "sha256:platform", "sha256:layer") + recordTag(t, db, testTagRepo, "nested", "sha256:outer", time.Now()) + got, err := db.ImageTagsContaining(testTagRepo, "sha256:layer", 10) + if err != nil || !slices.Equal(got, []string{"nested"}) { + t.Errorf("ImageTagsContaining = %v, %v; want nested", got, err) + } + }) +} + +// Pruning keeps what the tags of a repository point at, however many images +// retention keeps, so a blob's page still finds its tags. +func TestPruneImageManifestsKeepsWhatTagsPointAt(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now() + recordReferences(t, db, testTagRepo, "sha256:index", "sha256:platform") + recordReferences(t, db, testTagRepo, "sha256:platform", "sha256:layer") + recordReferences(t, db, testTagRepo, "sha256:moved", "sha256:moved-layer") + recordReferences(t, db, testTagRepo, "sha256:pinned", "sha256:pinned-layer") + recordTag(t, db, testTagRepo, "stable", "sha256:moved", now.Add(-time.Hour)) + recordTag(t, db, testTagRepo, "stable", "sha256:index", now) + // A tag of another repository holds nothing here. + recordTag(t, db, "pkg:oci/library/alpine", "latest", "sha256:pinned", now) + + // No image is among the newest none, whatever the count says. + if err := db.PruneImageManifests(testTagRepo, -1, now.Add(time.Minute)); err != nil { + t.Fatalf("PruneImageManifests: %v", err) + } + for digest, want := range map[string]bool{ + "sha256:index": true, "sha256:platform": true, "sha256:moved": false, "sha256:pinned": false, + } { + if _, ok, _, err := db.ImageManifestState(testTagRepo, digest); err != nil || ok != want { + t.Errorf("%s recorded = %v, %v; want %v", digest, ok, err, want) + } + } + if got, _ := db.ImageTagsContaining(testTagRepo, "sha256:layer", 10); !slices.Equal(got, []string{"stable"}) { + t.Errorf("ImageTagsContaining after prune = %v, want stable", got) + } + }) +} + +// Tags the upstream has not confirmed since the given time are dropped, in +// every repository. +func TestPruneImageTags(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now() + recordTag(t, db, testTagRepo, "old", "sha256:a", now.Add(-100*24*time.Hour)) + recordTag(t, db, testTagRepo, "latest", "sha256:b", now) + recordTag(t, db, "pkg:oci/library/alpine", "3.19", "sha256:c", now.Add(-100*24*time.Hour)) + + if err := db.PruneImageTags(now.Add(-90 * 24 * time.Hour)); err != nil { + t.Fatalf("PruneImageTags: %v", err) + } + if tags, _ := db.ImageTags(testTagRepo, 10); len(tags) != 1 || tags[0].Tag != "latest" { + t.Errorf("tags = %+v, want latest only", tags) + } + if tags, _ := db.ImageTags("pkg:oci/library/alpine", 10); len(tags) != 0 { + t.Errorf("alpine tags = %+v, want none", tags) + } + }) +} + +// A tag's stored digest and time are read back, and a missing tag says so. +func TestImageTag(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now().Truncate(time.Millisecond) + recordTag(t, db, testTagRepo, "latest", "sha256:a", now) + tag, ok, err := db.ImageTag(testTagRepo, "latest") + if err != nil || !ok || tag.Digest != "sha256:a" || !tag.SeenAt.Equal(now) { + t.Errorf("ImageTag(latest) = %+v, %v, %v; want sha256:a at %v", tag, ok, err, now) + } + if _, ok, err := db.ImageTag(testTagRepo, "missing"); err != nil || ok { + t.Errorf("ImageTag(missing) ok = %v, %v; want false", ok, err) + } + if _, ok, err := db.ImageTag("pkg:oci/library/alpine", "latest"); err != nil || ok { + t.Errorf("ImageTag in another repository ok = %v, %v; want false", ok, err) + } + }) +} + +// TouchImageTag moves a tag's time forward and leaves its digest. +func TestTouchImageTag(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now().Truncate(time.Millisecond) + recordTag(t, db, testTagRepo, "latest", "sha256:a", now.Add(-time.Hour)) + if err := db.TouchImageTag(testTagRepo, "latest", now); err != nil { + t.Fatal(err) + } + if err := db.TouchImageTag(testTagRepo, "latest", now.Add(-2*time.Hour)); err != nil { + t.Fatal(err) + } + tag, _, err := db.ImageTag(testTagRepo, "latest") + if err != nil || tag.Digest != "sha256:a" || !tag.SeenAt.Equal(now) { + t.Errorf("tag = %+v, %v; want sha256:a at %v", tag, err, now) + } + }) +} + +// The references a manifest records are read back; an unknown one has none. +func TestImageManifestRefs(t *testing.T) { + runWithBothDatabases(t, func(t *testing.T, db *DB) { + recordReferences(t, db, testTagRepo, "sha256:index", "sha256:amd64", "sha256:arm64") + if refs, err := db.ImageManifestRefs(testTagRepo, "sha256:index"); err != nil || !slices.Equal(refs, []string{"sha256:amd64", "sha256:arm64"}) { + t.Errorf("refs = %v, %v", refs, err) + } + if refs, err := db.ImageManifestRefs(testTagRepo, "sha256:unknown"); err != nil || refs != nil { + t.Errorf("unknown refs = %v, %v; want none", refs, err) + } + if refs, err := db.ImageManifestRefs("pkg:oci/library/alpine", "sha256:index"); err != nil || refs != nil { + t.Errorf("refs in another repository = %v, %v; want none", refs, err) + } + }) +} + +// The delete runs in batches, so a repository with more manifests to keep +// than a query can bind is pruned all the same. +func TestPruneImageManifestsInBatches(t *testing.T) { + defer func(n int) { imagePruneBatch = n }(imagePruneBatch) + imagePruneBatch = 2 + runWithBothDatabases(t, func(t *testing.T, db *DB) { + now := time.Now() + for i := range 5 { + kept := fmt.Sprintf("sha256:kept-%d", i) + recordReferences(t, db, testTagRepo, kept, "sha256:layer") + recordTag(t, db, testTagRepo, fmt.Sprintf("v%d", i), kept, now) + recordReferences(t, db, testTagRepo, fmt.Sprintf("sha256:gone-%d", i), "sha256:layer") + } + if err := db.PruneImageManifests(testTagRepo, 0, now.Add(time.Minute)); err != nil { + t.Fatalf("PruneImageManifests: %v", err) + } + for i := range 5 { + for digest, want := range map[string]bool{fmt.Sprintf("sha256:kept-%d", i): true, fmt.Sprintf("sha256:gone-%d", i): false} { + if _, ok, _, err := db.ImageManifestState(testTagRepo, digest); err != nil || ok != want { + t.Errorf("%s recorded = %v, %v; want %v", digest, ok, err, want) + } + } + } + }) +} diff --git a/internal/database/schema.go b/internal/database/schema.go index 1c298fab..4fec0500 100644 --- a/internal/database/schema.go +++ b/internal/database/schema.go @@ -160,6 +160,14 @@ CREATE TABLE IF NOT EXISTS image_manifests ( PRIMARY KEY (package_purl, digest) ); +CREATE TABLE IF NOT EXISTS image_tags ( + package_purl TEXT NOT NULL, + tag TEXT NOT NULL, + digest TEXT NOT NULL, + seen_at INTEGER NOT NULL, + PRIMARY KEY (package_purl, tag) +); + CREATE TABLE IF NOT EXISTS upstream_holds ( host TEXT NOT NULL PRIMARY KEY, until_ms INTEGER NOT NULL @@ -348,6 +356,14 @@ CREATE TABLE IF NOT EXISTS image_manifests ( PRIMARY KEY (package_purl, digest) ); +CREATE TABLE IF NOT EXISTS image_tags ( + package_purl TEXT NOT NULL, + tag TEXT NOT NULL, + digest TEXT NOT NULL, + seen_at BIGINT NOT NULL, + PRIMARY KEY (package_purl, tag) +); + CREATE TABLE IF NOT EXISTS upstream_holds ( host TEXT NOT NULL PRIMARY KEY, until_ms BIGINT NOT NULL @@ -565,6 +581,7 @@ var migrations = []migration{ {"022_add_artifact_mirrored_at", migrateAddArtifactMirroredAt}, {"023_add_metadata_upstream_url", migrateAddMetadataUpstreamURL}, {"024_add_image_manifests", migrateAddImageManifests}, + {"025_add_image_tags", migrateAddImageTags}, } // isTableNotFound returns true if the error indicates a missing table. @@ -1070,6 +1087,22 @@ func migrateAddImageManifests(db *DB) error { return nil } +// migrateAddImageTags creates the table of the tags of container image +// repositories that the web UI lists. +func migrateAddImageTags(db *DB) error { + // BIGINT is an integer column on SQLite too. + if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS image_tags ( + package_purl TEXT NOT NULL, + tag TEXT NOT NULL, + digest TEXT NOT NULL, + seen_at BIGINT NOT NULL, + PRIMARY KEY (package_purl, tag) + )`); err != nil { + return fmt.Errorf("creating image_tags table: %w", err) + } + return nil +} + // EnsureMetadataCacheTable creates the metadata_cache table if it doesn't exist. func (db *DB) EnsureMetadataCacheTable() error { has, err := db.HasTable("metadata_cache") diff --git a/internal/handler/container_image_refs.go b/internal/handler/container_image_refs.go index 7cdc5212..c99237dc 100644 --- a/internal/handler/container_image_refs.go +++ b/internal/handler/container_image_refs.go @@ -19,13 +19,13 @@ const maxImageManifestRefs = 4096 var imageSubjectTagPattern = regexp.MustCompile(`^sha256-[a-f0-9]{64}(\.[a-z0-9]+)?$`) // recordImageManifest notes which manifests and blobs a served manifest -// references, and for a tag that it pointed at the manifest, so retention -// can keep the blobs of the newest images of a repository. It does nothing -// unless storage.retention.keep_latest covers container images. The -// retention sweep drops the records of images that are no longer among the -// newest. +// references, and for a tag that it pointed at the manifest. Retention reads +// the records to keep the blobs of the newest images of a repository, and +// the web UI to name the tags whose image contains a blob. The retention +// sweep drops the records of images that are no longer among the newest and +// that no tag points at. func (h *ContainerHandler) recordImageManifest(cacheName, reference string, manifest *cachedContainerManifest, stale bool) { - if h.proxy.KeepLatestImages <= 0 || h.proxy.DB == nil { + if h.proxy.DB == nil { return } // A manifest asked for by digest is recorded under that digest, the one diff --git a/internal/handler/container_image_refs_test.go b/internal/handler/container_image_refs_test.go index 92837ba9..1460fff1 100644 --- a/internal/handler/container_image_refs_test.go +++ b/internal/handler/container_image_refs_test.go @@ -10,11 +10,10 @@ import ( const imageRefsRepo = "team/app" -func imageRefsHandler(t *testing.T, keepLatest int) (*ContainerHandler, *fakeImageRegistry, fakeImage) { +func imageRefsHandler(t *testing.T) (*ContainerHandler, *fakeImageRegistry, fakeImage) { t.Helper() registry, image := newFakeImageRegistry(t) proxy := newMirrorTestProxy(t, registry) - proxy.KeepLatestImages = keepLatest return NewContainerHandlerWithRegistry(proxy, "http://proxy.example", registry.server.URL), registry, image } @@ -54,7 +53,7 @@ func (r *fakeImageRegistry) moveTag(tag, digest string) { // A pull by tag makes the manifest an image of the repository, and the // manifests pulled by digest after it say which blobs it needs. func TestContainerPullRecordsWhatAnImageReferences(t *testing.T) { - h, _, image := imageRefsHandler(t, 1) + h, _, image := imageRefsHandler(t) pullImageManifest(t, h, "multi") if at, ok := imageTaggedAt(t, h, image.indexDigest); !ok || at.IsZero() { @@ -80,23 +79,24 @@ func TestContainerPullRecordsWhatAnImageReferences(t *testing.T) { } } -// Without keep_latest nothing is recorded. -func TestContainerPullRecordsNothingWithoutKeepLatest(t *testing.T) { - h, _, image := imageRefsHandler(t, 0) +// Manifests are recorded without keep_latest too: the web UI reads them to +// name the tags whose image contains a blob. +func TestContainerPullRecordsWithoutKeepLatest(t *testing.T) { + h, _, image := imageRefsHandler(t) pullImageManifest(t, h, "multi") pullImageManifest(t, h, image.arm64Digest) - if _, ok := imageTaggedAt(t, h, image.indexDigest); ok { - t.Errorf("index recorded though keep_latest is off") + if _, ok := imageTaggedAt(t, h, image.indexDigest); !ok { + t.Errorf("index not recorded") } - if _, ok := imageTaggedAt(t, h, image.arm64Digest); ok { - t.Errorf("platform manifest recorded though keep_latest is off") + if _, ok := imageTaggedAt(t, h, image.arm64Digest); !ok { + t.Errorf("platform manifest not recorded") } } // Referrers and cosign tags hold what refers to an image. They are recorded // like a pull by digest, so they don't count as images of their own. func TestContainerPullOfReferrersTagIsNotAnImage(t *testing.T) { - h, registry, image := imageRefsHandler(t, 1) + h, registry, image := imageRefsHandler(t) subject := "sha256-" + image.indexDigest[len("sha256:"):] for _, tag := range []string{subject, subject + ".sig", subject + ".att"} { registry.moveTag(tag, image.singleDigest) @@ -112,7 +112,7 @@ func TestContainerPullOfReferrersTagIsNotAnImage(t *testing.T) { func TestContainerPullAgainMovesTagTimeForward(t *testing.T) { for _, method := range []string{http.MethodGet, http.MethodHead} { t.Run(method, func(t *testing.T) { - h, _, image := imageRefsHandler(t, 1) + h, _, image := imageRefsHandler(t) pullImageManifest(t, h, "single") db := h.proxy.DB if _, err := db.Exec(db.Rebind("UPDATE image_manifests SET tagged_at = ?"), 1000); err != nil { @@ -136,7 +136,7 @@ func TestContainerPullAgainMovesTagTimeForward(t *testing.T) { // When a tag moves, the image it points at now is the newest, and the one // it pointed at before no longer is. func TestContainerMovedTagMakesTheNewImageNewest(t *testing.T) { - h, registry, image := imageRefsHandler(t, 1) + h, registry, image := imageRefsHandler(t) registry.moveTag("stable", image.singleDigest) pullImageManifest(t, h, "stable") if _, ok := newestImageDigests(t, h)[image.singleLayer]; !ok { @@ -162,7 +162,7 @@ func TestContainerMovedTagMakesTheNewImageNewest(t *testing.T) { // A scheduled mirror run resolves tags itself, and records them like a pull. func TestMirrorImageRecordsWhatAnImageReferences(t *testing.T) { - h, _, image := imageRefsHandler(t, 1) + h, _, image := imageRefsHandler(t) platform, err := ParsePlatform("linux/arm64") if err != nil { t.Fatal(err) diff --git a/internal/handler/container_image_tags.go b/internal/handler/container_image_tags.go new file mode 100644 index 00000000..88277825 --- /dev/null +++ b/internal/handler/container_image_tags.go @@ -0,0 +1,47 @@ +package handler + +import ( + "slices" + "time" +) + +// recordImageTag notes which digest a tag of the repository pointed at when +// the upstream last confirmed it, for the web UI to list. A copy served stale +// during an outage confirms nothing. +func (h *ContainerHandler) recordImageTag(cacheName, reference string, manifest *cachedContainerManifest, stale bool) { + if h.proxy.DB == nil || stale || manifestDigestReferencePattern.MatchString(reference) || + imageSubjectTagPattern.MatchString(reference) || !mirrorDigestPattern.MatchString(manifest.contentDigest) { + return + } + seen := manifest.fetchedAt + if seen.IsZero() { + seen = time.Now() + } + packagePURL := canonicalPackagePURL("oci", cacheName) + stored, known, err := h.proxy.DB.ImageTag(packagePURL, reference) + if err != nil { + h.proxy.Logger.Warn("failed to read image tag", "package_purl", packagePURL, "tag", reference, "error", err) + return + } + // Times are stored in milliseconds. A fresh cache hit repeats what is + // stored, and the one writer SQLite has should not hold the pull up. + if known && seen.UnixMilli() <= stored.SeenAt.UnixMilli() { + return + } + if known && stored.Digest != manifest.contentDigest && h.referencedBy(packagePURL, stored.Digest, manifest.contentDigest) { + // One platform's manifest of the index the tag points at, served to + // a client that accepts no index: the same image seen another way. + err = h.proxy.DB.TouchImageTag(packagePURL, reference, seen) + } else { + err = h.proxy.DB.RecordImageTag(packagePURL, reference, manifest.contentDigest, seen) + } + if err != nil { + h.proxy.Logger.Warn("failed to record image tag", "package_purl", packagePURL, "tag", reference, "error", err) + } +} + +// referencedBy reports whether the recorded manifest parent references digest. +func (h *ContainerHandler) referencedBy(packagePURL, parent, digest string) bool { + refs, err := h.proxy.DB.ImageManifestRefs(packagePURL, parent) + return err == nil && slices.Contains(refs, digest) +} diff --git a/internal/handler/container_image_tags_test.go b/internal/handler/container_image_tags_test.go new file mode 100644 index 00000000..1f381cea --- /dev/null +++ b/internal/handler/container_image_tags_test.go @@ -0,0 +1,203 @@ +package handler + +import ( + "context" + "net/http" + "net/http/httptest" + "slices" + "strings" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/database" +) + +const imageTagsRepo = "team/app" + +func imageTagsHandler(t *testing.T) (*ContainerHandler, *fakeImageRegistry, fakeImage) { + t.Helper() + registry, image := newFakeImageRegistry(t) + proxy := newMirrorTestProxy(t, registry) + return NewContainerHandlerWithRegistry(proxy, "http://proxy.example", registry.server.URL), registry, image +} + +func requestTagManifest(t *testing.T, h *ContainerHandler, method, reference string) { + t.Helper() + request := httptest.NewRequest(method, "/"+imageTagsRepo+"/manifests/"+reference, nil) + request.Header.Set("Accept", containerdManifestAccept) + response := httptest.NewRecorder() + h.Routes().ServeHTTP(response, request) + if response.Code != http.StatusOK { + t.Fatalf("%s %s: status %d: %s", method, reference, response.Code, response.Body.String()) + } +} + +func imageTags(t *testing.T, h *ContainerHandler) map[string]database.ImageTag { + t.Helper() + tags, err := h.proxy.DB.ImageTags(canonicalPackagePURL("oci", imageTagsRepo), 100) + if err != nil { + t.Fatalf("ImageTags: %v", err) + } + byTag := make(map[string]database.ImageTag, len(tags)) + for _, tag := range tags { + byTag[tag.Tag] = tag + } + return byTag +} + +// A pull by tag, with a GET or only a HEAD as Docker and containerd send, +// records the digest the tag points at. Pulls by digest name no tag. +func TestContainerPullRecordsImageTags(t *testing.T) { + h, _, image := imageTagsHandler(t) + requestTagManifest(t, h, http.MethodGet, "multi") + requestTagManifest(t, h, http.MethodHead, "single") + requestTagManifest(t, h, http.MethodGet, image.arm64Digest) + + tags := imageTags(t, h) + if len(tags) != 2 { + t.Fatalf("tags = %+v, want multi and single", tags) + } + for tag, digest := range map[string]string{"multi": image.indexDigest, "single": image.singleDigest} { + got := tags[tag] + if got.Digest != digest || time.Since(got.SeenAt) > time.Minute { + t.Errorf("tag %s = %+v, want %s seen just now", tag, got, digest) + } + } +} + +// Referrers and cosign tags hold what refers to an image, not an image, so +// they are not listed. +func TestContainerPullOfReferrersTagRecordsNoTag(t *testing.T) { + h, registry, image := imageTagsHandler(t) + subject := "sha256-" + image.indexDigest[len("sha256:"):] + for _, tag := range []string{subject, subject + ".sig", subject + ".att"} { + registry.mu.Lock() + registry.manifests["/v2/"+imageTagsRepo+"/manifests/"+tag] = registry.manifests["/v2/"+imageTagsRepo+"/manifests/"+image.singleDigest] + registry.mu.Unlock() + requestTagManifest(t, h, http.MethodGet, tag) + } + if tags := imageTags(t, h); len(tags) != 0 { + t.Errorf("tags = %+v, want none", tags) + } +} + +// A copy served stale during an outage says nothing new about the tag. +func TestContainerStaleManifestKeepsTagTime(t *testing.T) { + h, registry, _ := imageTagsHandler(t) + requestTagManifest(t, h, http.MethodGet, "multi") + db := h.proxy.DB + if _, err := db.Exec(db.Rebind("UPDATE image_tags SET seen_at = ?"), 1000); err != nil { + t.Fatal(err) + } + registry.offline.Store(true) + requestTagManifest(t, h, http.MethodGet, "multi") + if got := imageTags(t, h)["multi"].SeenAt; !got.Equal(time.UnixMilli(1000)) { + t.Errorf("seen at = %v after a stale serve, want unchanged", got) + } +} + +// A scheduled mirror run resolves tags itself and records them like a pull. +func TestMirrorImageRecordsImageTags(t *testing.T) { + h, _, image := imageTagsHandler(t) + ref := ImageRef{Name: imageTagsRepo, References: []string{"multi"}} + if _, err := h.MirrorImage(context.Background(), ref, nil); err != nil { + t.Fatalf("MirrorImage: %v", err) + } + if got := imageTags(t, h)["multi"]; got.Digest != image.indexDigest { + t.Errorf("tag multi after a mirror run = %+v, want %s", got, image.indexDigest) + } +} + +// Pulled manifests record what they reference, so a blob's page finds the +// tags whose image contains it: the manifest the tag points at, the platform +// manifests it lists, and their configs and layers, as far as clients pulled +// them. +func TestContainerPullRecordsWhatImagesContain(t *testing.T) { + h, _, image := imageTagsHandler(t) + // containerd: a GET of the tag, then the platform manifest by digest. + requestTagManifest(t, h, http.MethodGet, "multi") + requestTagManifest(t, h, http.MethodGet, image.arm64Digest) + // Docker: a HEAD of the tag, then the manifest by digest. + requestTagManifest(t, h, http.MethodHead, "single") + requestTagManifest(t, h, http.MethodGet, image.singleDigest) + + for name, tt := range map[string]struct { + digest string + want []string + }{ + "index": {image.indexDigest, []string{"multi"}}, + "platform": {image.arm64Digest, []string{"multi"}}, + "layer": {image.arm64Layer, []string{"multi"}}, + "single layer": {image.singleLayer, []string{"single"}}, + "unpulled layer": {image.amd64Layer, nil}, + "unknown": {"sha256:" + strings.Repeat("0", 64), nil}, + } { + got, err := h.proxy.DB.ImageTagsContaining(canonicalPackagePURL("oci", imageTagsRepo), tt.digest, 10) + if err != nil { + t.Fatal(err) + } + if !slices.Equal(got, tt.want) { + t.Errorf("%s: tags = %v, want %v", name, got, tt.want) + } + } +} + +// A mirror run reads the manifests through the handler, so it records what +// they reference like a pull. +func TestMirrorImageRecordsWhatImagesContain(t *testing.T) { + h, _, image := imageTagsHandler(t) + ref := ImageRef{Name: imageTagsRepo, References: []string{"multi"}} + if _, err := h.MirrorImage(context.Background(), ref, nil); err != nil { + t.Fatalf("MirrorImage: %v", err) + } + got, err := h.proxy.DB.ImageTagsContaining(canonicalPackagePURL("oci", imageTagsRepo), image.amd64Layer, 10) + if err != nil || !slices.Equal(got, []string{"multi"}) { + t.Errorf("tags containing the amd64 layer after a mirror run = %v, %v; want multi", got, err) + } +} + +// A fresh cache hit repeats what is stored, so it writes nothing: SQLite has +// one writer, and every cached pull would queue behind it. +func TestContainerCachedPullWritesNoTag(t *testing.T) { + h, _, _ := imageTagsHandler(t) + h.proxy.MetadataTTL = time.Hour + db := h.proxy.DB + if _, err := db.Exec(`CREATE TABLE tag_writes (n INTEGER)`); err != nil { + t.Fatal(err) + } + if _, err := db.Exec(`CREATE TRIGGER count_tag_writes BEFORE INSERT ON image_tags BEGIN INSERT INTO tag_writes VALUES (1); END`); err != nil { + t.Fatal(err) + } + for range 3 { + requestTagManifest(t, h, http.MethodGet, "multi") + requestTagManifest(t, h, http.MethodHead, "multi") + } + var writes int + if err := db.Get(&writes, `SELECT COUNT(*) FROM tag_writes`); err != nil { + t.Fatal(err) + } + if writes != 1 { + t.Errorf("%d tag writes, want 1", writes) + } +} + +// A client that accepts no index is served one platform's manifest for a +// tag. That is the image the tag points at seen another way, so the tag keeps +// the index's digest and only its time moves. A digest the stored one does +// not reference is a new image. +func TestContainerPlatformManifestKeepsTheTagsIndex(t *testing.T) { + h, _, image := imageTagsHandler(t) + requestTagManifest(t, h, http.MethodGet, "multi") + later := imageTags(t, h)["multi"].SeenAt.Add(time.Minute) + + h.recordImageTag(imageTagsRepo, "multi", &cachedContainerManifest{contentDigest: image.amd64Digest, fetchedAt: later}, false) + if got := imageTags(t, h)["multi"]; got.Digest != image.indexDigest || !got.SeenAt.Equal(later) { + t.Errorf("tag after a platform manifest = %+v, want %s at %v", got, image.indexDigest, later) + } + + moved := later.Add(time.Minute) + h.recordImageTag(imageTagsRepo, "multi", &cachedContainerManifest{contentDigest: image.singleDigest, fetchedAt: moved}, false) + if got := imageTags(t, h)["multi"]; got.Digest != image.singleDigest || !got.SeenAt.Equal(moved) { + t.Errorf("tag after another image = %+v, want %s at %v", got, image.singleDigest, moved) + } +} diff --git a/internal/handler/container_manifest.go b/internal/handler/container_manifest.go index fa857f37..26869faa 100644 --- a/internal/handler/container_manifest.go +++ b/internal/handler/container_manifest.go @@ -116,6 +116,7 @@ func (h *ContainerHandler) writeManifest(w http.ResponseWriter, r *http.Request, return } h.recordImageManifest(cacheName, reference, manifest, stale) + h.recordImageTag(cacheName, reference, manifest, stale) writeContainerManifest(w, r, manifest, stale) } diff --git a/internal/handler/container_mirror.go b/internal/handler/container_mirror.go index 19a52fc7..422f0129 100644 --- a/internal/handler/container_mirror.go +++ b/internal/handler/container_mirror.go @@ -257,6 +257,7 @@ func (m *imageMirror) mirrorTag(ctx context.Context, tag string) (string, error) m.h.storeContainerManifestForAccept(ctx, m.registryURL, m.upstreamName, tag, accept, cacheAccept, manifest) // The walk recorded the manifest as one asked for by digest. m.h.recordImageManifest(m.cacheName, tag, manifest, false) + m.h.recordImageTag(m.cacheName, tag, manifest, false) return digest, nil } diff --git a/internal/handler/handler.go b/internal/handler/handler.go index f245aaff..9786eeac 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -244,12 +244,6 @@ type Proxy struct { DirectServeTTL time.Duration HTTPClient *http.Client - // KeepLatestImages is storage.retention.keep_latest where a retention - // rule applies to container images. Above zero, the container handler - // records what each served manifest references, so retention can keep - // the blobs of that many images per repository. - KeepLatestImages int - // RateLimits is the gate under the upstream clients. Artifact fetches // check it before calling Fetcher, so held hosts never reach its breaker. RateLimits UpstreamHolds diff --git a/internal/handler/profile.go b/internal/handler/profile.go index e9cde1af..8e6ad611 100644 --- a/internal/handler/profile.go +++ b/internal/handler/profile.go @@ -14,7 +14,7 @@ func (p *Proxy) WithCooldown(policy CooldownPolicy, shortest time.Duration) *Pro GradleReadOnly: p.GradleReadOnly, HostedEnabled: p.HostedEnabled, GradleMaxUploadSize: p.GradleMaxUploadSize, NPMFullMetadata: p.NPMFullMetadata, DirectServe: p.DirectServe, DirectServeTTL: p.DirectServeTTL, - HTTPClient: p.HTTPClient, KeepLatestImages: p.KeepLatestImages, + HTTPClient: p.HTTPClient, RateLimits: p.RateLimits, HostedNPMScopes: p.HostedNPMScopes, HostedComposerVendors: p.HostedComposerVendors, StreamArtifacts: p.StreamArtifacts, Scanners: p.Scanners, ScanSigningKey: p.ScanSigningKey, ScanFetchBaseURL: p.ScanFetchBaseURL, diff --git a/internal/handler/profile_test.go b/internal/handler/profile_test.go index c2a768f5..a85d6ca3 100644 --- a/internal/handler/profile_test.go +++ b/internal/handler/profile_test.go @@ -34,7 +34,6 @@ func TestWithCooldownCopiesProxyFields(t *testing.T) { p.NPMFullMetadata = true p.DirectServe = true p.DirectServeTTL = time.Minute - p.KeepLatestImages = 2 p.RateLimits = registryclient.NewRateLimitGate(http.DefaultTransport) p.HostedNPMScopes = []string{"@example"} p.HostedComposerVendors = []string{"example"} diff --git a/internal/server/dashboard.go b/internal/server/dashboard.go index 604a4bf0..6c897e17 100644 --- a/internal/server/dashboard.go +++ b/internal/server/dashboard.go @@ -74,6 +74,8 @@ type PackageShowData struct { UsedBy []VersionUsage // Cooldown is nil with cooldown off. Cooldown *PackageCooldown + // Tags lists a container image's tags, the most recently seen first. + Tags []database.ImageTag } // VersionShowData contains data for rendering the version show page. @@ -89,6 +91,9 @@ type VersionShowData struct { // HasDeletableArtifact is true when a stored artifact was fetched from // the upstream rather than pushed. HasDeletableArtifact bool + // ImageTags names the tags whose cached image contains a container + // image's blob. + ImageTags []string } // SearchPageData contains data for rendering the search results page. diff --git a/internal/server/image_records_prune.go b/internal/server/image_records_prune.go new file mode 100644 index 00000000..16be3bc0 --- /dev/null +++ b/internal/server/image_records_prune.go @@ -0,0 +1,45 @@ +package server + +import ( + "context" + "time" + + "github.com/git-pkgs/proxy/internal/retention" +) + +const ( + // imageRecordPruneInterval is how often container tag and manifest + // records are pruned. + imageRecordPruneInterval = time.Hour + // imageTagMaxAge is how long a tag the upstream has not confirmed is + // kept, and with it what its image references. + imageTagMaxAge = 90 * 24 * time.Hour +) + +// startImageRecordPruning prunes the container tag and manifest records once +// at startup and then every imageRecordPruneInterval. They are written on +// every install, so this runs whether or not a retention rule is set. +func (s *Server) startImageRecordPruning(ctx context.Context) { + keep := 0 + if rules, err := s.cfg.RetentionRules(retention.Default); err == nil { + keep = rules.KeepLatest + } + prune := func() { + s.runExclusive(ctx, "image-record-pruning", func() { + pruneImageRecords(s.db, s.logger, keep, time.Now()) + }) + } + s.background.Go(func() { + prune() + ticker := time.NewTicker(imageRecordPruneInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + prune() + } + } + }) +} diff --git a/internal/server/image_records_prune_test.go b/internal/server/image_records_prune_test.go new file mode 100644 index 00000000..24703c7a --- /dev/null +++ b/internal/server/image_records_prune_test.go @@ -0,0 +1,66 @@ +package server + +import ( + "context" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/database" + "github.com/git-pkgs/proxy/internal/retention" +) + +// Container records are pruned without any retention rule: they are written +// on every install, for the web UI. +func TestImageRecordPruningRunsWithoutRetention(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + if rules, err := ts.server.cfg.RetentionRules(retention.Default); err == nil && rules.MinPositive() != 0 { + t.Fatalf("test config has retention rules: %+v", rules) + } + const repo = "pkg:oci/team/app" + db := ts.db + now := time.Now() + if err := db.RecordImageManifest(database.ImageManifest{PackagePURL: repo, Digest: "sha256:tagged", Refs: []string{"sha256:layer"}}); err != nil { + t.Fatal(err) + } + if err := db.RecordImageManifest(database.ImageManifest{PackagePURL: repo, Digest: "sha256:untagged", Refs: []string{"sha256:layer"}}); err != nil { + t.Fatal(err) + } + if _, err := db.Exec(db.Rebind(`UPDATE image_manifests SET recorded_at = ?`), now.Add(-2*imageRecordGrace).UnixMilli()); err != nil { + t.Fatal(err) + } + if err := db.RecordImageTag(repo, "latest", "sha256:tagged", now); err != nil { + t.Fatal(err) + } + if err := db.RecordImageTag(repo, "stale", "sha256:untagged", now.Add(-imageTagMaxAge-time.Hour)); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + ts.server.startImageRecordPruning(ctx) + defer func() { + cancel() + ts.server.background.Wait() + }() + + deadline := time.Now().Add(5 * time.Second) + for { + _, recorded, _, err := db.ImageManifestState(repo, "sha256:untagged") + if err != nil { + t.Fatal(err) + } + if !recorded { + break + } + if time.Now().After(deadline) { + t.Fatal("the manifest of a tag not confirmed for too long was not pruned") + } + time.Sleep(10 * time.Millisecond) + } + if _, recorded, _, err := db.ImageManifestState(repo, "sha256:tagged"); err != nil || !recorded { + t.Errorf("the manifest latest points at was pruned: %v, %v", recorded, err) + } + if tags, err := db.ImageTags(repo, 10); err != nil || len(tags) != 1 || tags[0].Tag != "latest" { + t.Errorf("tags = %+v, %v; want latest only", tags, err) + } +} diff --git a/internal/server/image_tags_ui_test.go b/internal/server/image_tags_ui_test.go new file mode 100644 index 00000000..9fdde3c7 --- /dev/null +++ b/internal/server/image_tags_ui_test.go @@ -0,0 +1,147 @@ +package server + +import ( + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/database" +) + +// The page of a container image lists its tags, most recently seen first, +// with the digest each one points at. +func TestPackageShowPageListsImageTags(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + + pkg := &database.Package{PURL: "pkg:oci/pihole/pihole", Ecosystem: "oci", Name: "pihole/pihole"} + if err := ts.db.UpsertPackage(pkg); err != nil { + t.Fatalf("UpsertPackage: %v", err) + } + const layer = "sha256:7373c5bccb12b43ec422ccd65b3cc9b55ee6e86ad3cee28189322c6419a72b02" + if err := ts.db.UpsertVersion(&database.Version{PURL: pkg.PURL + "@" + layer, PackagePURL: pkg.PURL}); err != nil { + t.Fatalf("UpsertVersion: %v", err) + } + const newDigest = "sha256:66c798d79a95c2691f280329c3611dba1746117c61c9f331907ca35a06a52778" + const oldDigest = "sha256:1111111111111111111111111111111111111111111111111111111111111111" + now := time.Now() + for tag, at := range map[string]struct { + digest string + seen time.Time + }{ + "2026.10.0": {oldDigest, now.Add(-48 * time.Hour)}, + "2026.11.0": {newDigest, now.Add(-time.Minute)}, + "latest": {newDigest, now}, + } { + if err := ts.db.RecordImageTag(pkg.PURL, tag, at.digest, at.seen); err != nil { + t.Fatalf("RecordImageTag: %v", err) + } + } + + w := httptest.NewRecorder() + ts.handler.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/ui/package/oci/pihole/pihole", nil)) + if w.Code != http.StatusOK { + t.Fatalf("status %d", w.Code) + } + body := w.Body.String() + if !strings.Contains(body, ">Tags<") { + t.Error("page has no Tags section") + } + latest, current, older := strings.Index(body, ">latest<"), strings.Index(body, ">2026.11.0<"), strings.Index(body, ">2026.10.0<") + if latest < 0 || current < 0 || older < 0 || latest > current || current > older { + t.Errorf("tags at %d, %d, %d; want latest, 2026.11.0, 2026.10.0 in that order", latest, current, older) + } + if !strings.Contains(body, newDigest) || !strings.Contains(body, oldDigest) { + t.Error("page does not show the digests the tags point at") + } +} + +// Packages of other ecosystems have no tags section. +func TestPackageShowPageHasNoTagsOutsideOCI(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + pkg := &database.Package{PURL: "pkg:npm/no-tags", Ecosystem: "npm", Name: "no-tags"} + if err := ts.db.UpsertPackage(pkg); err != nil { + t.Fatalf("UpsertPackage: %v", err) + } + w := httptest.NewRecorder() + ts.handler.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/ui/package/npm/no-tags", nil)) + if strings.Contains(w.Body.String(), "Tags (") { + t.Error("npm package page has a Tags section") + } +} + +// The page of an image's blob names the tags whose image contains it. +func TestVersionShowPageListsImageTagsContainingTheBlob(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + + pkg := &database.Package{PURL: "pkg:oci/pihole/pihole", Ecosystem: "oci", Name: "pihole/pihole"} + if err := ts.db.UpsertPackage(pkg); err != nil { + t.Fatalf("UpsertPackage: %v", err) + } + const layer = "sha256:7373c5bccb12b43ec422ccd65b3cc9b55ee6e86ad3cee28189322c6419a72b02" + if err := ts.db.UpsertVersion(&database.Version{PURL: pkg.PURL + "@" + layer, PackagePURL: pkg.PURL}); err != nil { + t.Fatalf("UpsertVersion: %v", err) + } + const image = "sha256:66c798d79a95c2691f280329c3611dba1746117c61c9f331907ca35a06a52778" + if err := ts.db.RecordImageManifest(database.ImageManifest{PackagePURL: pkg.PURL, Digest: image, Refs: []string{layer}}); err != nil { + t.Fatal(err) + } + now := time.Now() + for tag, digest := range map[string]string{"latest": image, "2026.11.0": image, "2026.10.0": "sha256:other"} { + if err := ts.db.RecordImageTag(pkg.PURL, tag, digest, now); err != nil { + t.Fatal(err) + } + } + + w := httptest.NewRecorder() + ts.handler.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/ui/package/oci/pihole/pihole/"+layer, nil)) + if w.Code != http.StatusOK { + t.Fatalf("status %d: %s", w.Code, w.Body.String()) + } + body := w.Body.String() + if !strings.Contains(body, "Tags:") || !strings.Contains(body, ">latest<") || !strings.Contains(body, ">2026.11.0<") { + t.Error("version page does not name the tags containing the blob") + } + if strings.Contains(body, ">2026.10.0<") { + t.Error("version page names a tag whose image lacks the blob") + } +} + +// The tags searched for a blob are not cut to the ones the image's page +// lists, so a blob of an old tag still names it. +func TestVersionShowPageSearchesPastTheListedTags(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + + pkg := &database.Package{PURL: "pkg:oci/team/app", Ecosystem: "oci", Name: "team/app"} + if err := ts.db.UpsertPackage(pkg); err != nil { + t.Fatal(err) + } + const layer = "sha256:1111111111111111111111111111111111111111111111111111111111111111" + if err := ts.db.UpsertVersion(&database.Version{PURL: pkg.PURL + "@" + layer, PackagePURL: pkg.PURL}); err != nil { + t.Fatal(err) + } + if err := ts.db.RecordImageManifest(database.ImageManifest{PackagePURL: pkg.PURL, Digest: "sha256:old", Refs: []string{layer}}); err != nil { + t.Fatal(err) + } + now := time.Now() + if err := ts.db.RecordImageTag(pkg.PURL, "oldest", "sha256:old", now.Add(-time.Hour)); err != nil { + t.Fatal(err) + } + for i := range maxImageTagsShown + 1 { + if err := ts.db.RecordImageTag(pkg.PURL, fmt.Sprintf("v%d", i), "sha256:new", now); err != nil { + t.Fatal(err) + } + } + + w := httptest.NewRecorder() + ts.handler.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/ui/package/oci/team/app/"+layer, nil)) + if !strings.Contains(w.Body.String(), ">oldest<") { + t.Errorf("status %d: version page does not name the oldest tag", w.Code) + } +} diff --git a/internal/server/retention.go b/internal/server/retention.go index 5003dfd7..179a05a5 100644 --- a/internal/server/retention.go +++ b/internal/server/retention.go @@ -4,10 +4,8 @@ import ( "context" "log/slog" "slices" - "strings" "time" - "github.com/git-pkgs/proxy/internal/config" "github.com/git-pkgs/proxy/internal/database" "github.com/git-pkgs/proxy/internal/metrics" "github.com/git-pkgs/proxy/internal/retention" @@ -103,8 +101,6 @@ func sweepRetention(ctx context.Context, db *database.DB, logger *slog.Logger, return } - pruneImageRecords(db, logger, rules.KeepLatest, now) - // Without the buffered hits written, an artifact being served right now // can look expired, so skip this sweep rather than clear it. if err := db.FlushHits(); err != nil { @@ -161,29 +157,13 @@ func sweepRetention(ctx context.Context, db *database.DB, logger *slog.Logger, } } -// keepLatestImages returns storage.retention.keep_latest if a retention rule -// can evict container image blobs, and zero otherwise: without such a rule -// nothing reads what the container handler would record. -func keepLatestImages(cfg *config.Config) int { - rules, err := cfg.RetentionRules(retention.Default) - if err != nil || rules.KeepLatest <= 0 { - return 0 - } - if rules.For("oci", "") > 0 { - return rules.KeepLatest - } - for packagePURL, d := range rules.Packages { - if d > 0 && strings.HasPrefix(packagePURL, "pkg:oci/") { - return rules.KeepLatest - } - } - return 0 -} - -// pruneImageRecords drops the container manifest records that the keep -// newest images of their repository do not reference, which is all of them -// once keep_latest is off. +// pruneImageRecords drops the container tags the upstream has not confirmed +// for imageTagMaxAge, then the manifest records that neither the keep newest +// images of their repository nor the images its tags point at reference. func pruneImageRecords(db *database.DB, logger *slog.Logger, keep int, now time.Time) { + if err := db.PruneImageTags(now.Add(-imageTagMaxAge)); err != nil { + logger.Warn("retention: failed to prune image tags", "error", err) + } repositories, err := db.ImageManifestRepositories() if err != nil { logger.Warn("retention: failed to list image manifest records", "error", err) diff --git a/internal/server/retention_keep_latest_oci_test.go b/internal/server/retention_keep_latest_oci_test.go index 5531087b..10e2ea5b 100644 --- a/internal/server/retention_keep_latest_oci_test.go +++ b/internal/server/retention_keep_latest_oci_test.go @@ -4,14 +4,12 @@ import ( "context" "encoding/json" "io" - "log/slog" "net/http" "net/http/httptest" "sync" "testing" "time" - "github.com/git-pkgs/proxy/internal/config" "github.com/git-pkgs/proxy/internal/database" "github.com/git-pkgs/proxy/internal/handler" "github.com/git-pkgs/proxy/internal/retention" @@ -121,16 +119,14 @@ func (r *keepImagesRegistry) tag(tag string, image keepImage) { r.tags[tag] = image.index } -// newKeepImagesProxy serves the registry through a proxy that keeps the -// blobs of keepLatest images per repository. -func newKeepImagesProxy(t *testing.T, registry *keepImagesRegistry, keepLatest int) (*httptest.Server, *database.DB) { +// newKeepImagesProxy serves the registry through a proxy. +func newKeepImagesProxy(t *testing.T, registry *keepImagesRegistry) (*httptest.Server, *database.DB) { t.Helper() db, store := setupEvictionTest(t) fetcher := fetch.NewFetcher(fetch.WithHTTPClient(registry.srv.Client()), fetch.WithMaxRetries(0)) t.Cleanup(func() { _ = fetcher.Close() }) proxy := handler.NewProxy(db, store, fetcher, fetch.NewResolver(), discardLogger()) proxy.HTTPClient = registry.srv.Client() - proxy.KeepLatestImages = keepLatest srv := httptest.NewServer(handler.NewContainerHandlerWithRegistry(proxy, "http://proxy.test", registry.srv.URL).Routes()) t.Cleanup(srv.Close) return srv, db @@ -206,7 +202,7 @@ func TestSweepRetentionKeepsBlobsOfNewestImages(t *testing.T) { {keepLatest: 2, wantCached: [3]bool{false, true, true}}, } { registry := newKeepImagesRegistry(t) - srv, db := newKeepImagesProxy(t, registry, tt.keepLatest) + srv, db := newKeepImagesProxy(t, registry) images := []keepImage{registry.addImage(t, "first"), registry.addImage(t, "second"), registry.addImage(t, "third")} for _, image := range images { registry.tag("beta", image) @@ -226,12 +222,11 @@ func TestSweepRetentionKeepsBlobsOfNewestImages(t *testing.T) { } } -// A sweep drops the records of images that are no longer among the newest, -// so the table holds a few images per repository, and all records once -// keep_latest is turned off. -func TestSweepRetentionPrunesImageRecords(t *testing.T) { +// The prune drops the records of images that are no longer among the newest +// and that no tag points at, so the table holds a few images per repository. +func TestPruneImageRecords(t *testing.T) { registry := newKeepImagesRegistry(t) - srv, db := newKeepImagesProxy(t, registry, 1) + srv, db := newKeepImagesProxy(t, registry) images := []keepImage{registry.addImage(t, "first"), registry.addImage(t, "second")} for _, image := range images { registry.tag("beta", image) @@ -248,8 +243,7 @@ func TestSweepRetentionPrunesImageRecords(t *testing.T) { return ok } - rules := retention.Rules{Ecosystems: map[string]time.Duration{"oci": retentionDay}, KeepLatest: 1} - sweepRetention(context.Background(), db, discardLogger(), retention.Default, rules, now) + pruneImageRecords(db, discardLogger(), 1, now) for _, digest := range []string{images[0].index, images[0].manifest} { if recorded(digest) { t.Errorf("manifest %s of the older image is still recorded", digest) @@ -261,55 +255,11 @@ func TestSweepRetentionPrunesImageRecords(t *testing.T) { } } - rules.KeepLatest = 0 - sweepRetention(context.Background(), db, discardLogger(), retention.Default, rules, now) - if recorded(images[1].index) { - t.Error("records are left after keep_latest was turned off") - } -} - -// Manifests are recorded only when a retention rule can evict container -// blobs, since nothing reads the records otherwise. -func TestKeepLatestImagesNeedsAnOCIRule(t *testing.T) { - for _, tt := range []struct { - name string - def string - ecosystems map[string]string - packages map[string]string - keepLatest int - want int - }{ - {name: "no keep_latest", ecosystems: map[string]string{"oci": "1d"}, want: 0}, - {name: "no rule", keepLatest: 2, want: 0}, - {name: "rule for another ecosystem", ecosystems: map[string]string{"npm": "7d"}, keepLatest: 2, want: 0}, - {name: "oci rule", ecosystems: map[string]string{"oci": "1d"}, keepLatest: 2, want: 2}, - {name: "default", def: "30d", keepLatest: 2, want: 2}, - {name: "default with oci exempt", def: "30d", ecosystems: map[string]string{"oci": "0"}, keepLatest: 2, want: 0}, - {name: "rule for one repository", packages: map[string]string{"pkg:oci/library/nginx": "3d"}, keepLatest: 2, want: 2}, - {name: "rule for another ecosystem's package", packages: map[string]string{"pkg:npm/lodash": "3d"}, keepLatest: 2, want: 0}, - } { - t.Run(tt.name, func(t *testing.T) { - cfg := config.Default() - cfg.Storage.Retention.Default = tt.def - cfg.Storage.Retention.Ecosystems = tt.ecosystems - cfg.Storage.Retention.Packages = tt.packages - cfg.Storage.Retention.KeepLatest = tt.keepLatest - if got := keepLatestImages(cfg); got != tt.want { - t.Errorf("keepLatestImages() = %d, want %d", got, tt.want) - } - }) - } -} - -// proxy mirror runs in a process of its own and has to record the images it -// mirrors as serve does, or the sweep would not know their blobs. -func TestMirrorProxyRecordsImagesForKeepLatest(t *testing.T) { - cfg := config.Default() - cfg.Storage.Retention.Ecosystems = map[string]string{"oci": "1d"} - cfg.Storage.Retention.KeepLatest = 2 - proxy, _ := NewMirrorProxy(cfg, nil, nil, slog.New(slog.NewTextHandler(io.Discard, nil))) - if proxy.KeepLatestImages != 2 { - t.Errorf("KeepLatestImages = %d, want 2", proxy.KeepLatestImages) + // Without keep_latest the records of the image the tag points at stay, + // for the web UI. + pruneImageRecords(db, discardLogger(), 0, now) + if !recorded(images[1].index) || !recorded(images[1].manifest) { + t.Error("the records of the image the tag points at are gone after keep_latest was turned off") } } @@ -317,7 +267,7 @@ func TestMirrorProxyRecordsImagesForKeepLatest(t *testing.T) { // blobs are evicted by age as without keep_latest. func TestSweepRetentionKeepLatestIgnoresImagesPulledByDigest(t *testing.T) { registry := newKeepImagesRegistry(t) - srv, db := newKeepImagesProxy(t, registry, 1) + srv, db := newKeepImagesProxy(t, registry) image := registry.addImage(t, "pinned") pullImage(t, srv.URL, image.index, image) now := time.Now() @@ -335,7 +285,7 @@ func TestSweepRetentionKeepLatestIgnoresImagesPulledByDigest(t *testing.T) { // manifests by digest. The HEAD alone makes the image one of the newest. func TestSweepRetentionKeepsBlobsOfImagesPulledByHead(t *testing.T) { registry := newKeepImagesRegistry(t) - srv, db := newKeepImagesProxy(t, registry, 1) + srv, db := newKeepImagesProxy(t, registry) images := []keepImage{registry.addImage(t, "first"), registry.addImage(t, "second")} for _, image := range images { registry.tag("beta", image) diff --git a/internal/server/server.go b/internal/server/server.go index d9cf5051..ffcb65ad 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -308,7 +308,6 @@ func (s *Server) serve(listener net.Listener) error { return fmt.Errorf("configuring scanners: %w", err) } configureMetadata(proxy, s.cfg) - proxy.KeepLatestImages = keepLatestImages(s.cfg) proxy.GradleReadOnly = s.cfg.Gradle.BuildCache.ReadOnly proxy.HostedEnabled = s.cfg.Hosted.Enabled proxy.HostedNPMScopes = s.cfg.Hosted.NPMScopes @@ -465,6 +464,7 @@ func (s *Server) serve(listener net.Listener) error { s.background.Go(func() { s.startEvictionLoop(bgCtx) }) retentionResolvers := s.retentionResolvers(proxy) s.background.Go(func() { s.startRetentionLoop(bgCtx, retentionResolvers) }) + s.startImageRecordPruning(bgCtx) s.background.Go(func() { s.startAPKSnapshotPruning(bgCtx) }) s.background.Go(func() { s.startReclaimLoop(bgCtx) }) @@ -642,7 +642,6 @@ func NewMirrorProxy(cfg *config.Config, db *database.DB, store storage.Storage, configureMirrorResolvers(proxy, cfg.Upstream) proxy.HTTPClient = metadataClient proxy.RateLimits = s.rateLimits - proxy.KeepLatestImages = keepLatestImages(cfg) return proxy, s.newContainerHandler(proxy) } @@ -1225,6 +1224,12 @@ func (s *Server) handlePackagePath(w http.ResponseWriter, r *http.Request) { } } +const ( + // maxImageTagsShown is how many tags a container image's page lists, + // and how many a blob's page names. + maxImageTagsShown = 100 +) + func (s *Server) showPackage(w http.ResponseWriter, r *http.Request, ecosystem, name string) { pkg, err := s.db.GetPackageByEcosystemName(ecosystem, name) if err != nil { @@ -1267,6 +1272,11 @@ func (s *Server) showPackage(w http.ResponseWriter, r *http.Request, ecosystem, Cooldown: s.cooldown.forPackage(pkg.Ecosystem, pkg.Name), } data.UsedBy = s.packageUsedBy(pkg, versions) + if pkg.Ecosystem == "oci" { + if data.Tags, err = s.db.ImageTags(pkg.PURL, maxImageTagsShown); err != nil { + s.logger.Error("failed to get image tags", "error", err) + } + } if err := s.templates.Render(w, "package_show", data); err != nil { s.logger.Error("failed to render package show", "error", err) @@ -1307,6 +1317,13 @@ func (s *Server) showVersion(w http.ResponseWriter, r *http.Request, ecosystem, // browsed, so it must not be advertised as cached. hasCached := firstBrowsableArtifact(artifacts) != nil + var imageTags []string + if pkg.Ecosystem == "oci" { + if imageTags, err = s.db.ImageTagsContaining(pkg.PURL, version, maxImageTagsShown); err != nil { + s.logger.Error("failed to get image tags", "error", err) + } + } + data := VersionShowData{ Layout: s.layoutFor(r), Package: pkg, @@ -1317,6 +1334,7 @@ func (s *Server) showVersion(w http.ResponseWriter, r *http.Request, ecosystem, LicenseCategory: categorizeLicense(ver.License), HasCachedArtifact: hasCached, HasDeletableArtifact: hasDeletableArtifact(artifacts), + ImageTags: imageTags, } if err := s.templates.Render(w, "version_show", data); err != nil { diff --git a/internal/server/templates/pages/package_show.html b/internal/server/templates/pages/package_show.html index afbdfa33..815016df 100644 --- a/internal/server/templates/pages/package_show.html +++ b/internal/server/templates/pages/package_show.html @@ -80,6 +80,26 @@

Used by

{{end}} +{{if .Tags}} +
+
+

Tags

+

The most recently seen tags, up to 100, with the digest each pointed at when the upstream last confirmed it

+
+
+ {{range .Tags}} +
+
+ {{.Tag}} + {{.Digest}} +
+ {{.SeenAt.UTC.Format "2006-01-02 15:04"}} UTC +
+ {{end}} +
+
+{{end}} + {{if .Versions}}
diff --git a/internal/server/templates/pages/version_show.html b/internal/server/templates/pages/version_show.html index 468f24cd..96499d84 100644 --- a/internal/server/templates/pages/version_show.html +++ b/internal/server/templates/pages/version_show.html @@ -43,6 +43,9 @@

Version Info

{{if .Version.PublishedAt.Valid}}
Published:
{{.Version.PublishedAt.Time.Format "2006-01-02 15:04:05"}}
{{end}} + {{if .ImageTags}} +
Tags:
{{range $i, $tag := .ImageTags}}{{if $i}}, {{end}}{{$tag}}{{end}}
+ {{end}} {{if .Version.Integrity.Valid}}
Integrity:
{{.Version.Integrity.String}}
{{end}}