Skip to content

feat(encryption): [9/N] Read encrypted manifest lists - #4159

Open
xanderbailey wants to merge 1 commit into
apache:mainfrom
xanderbailey:xb/encryption-read-manifest-list
Open

xanderbailey wants to merge 1 commit into
apache:mainfrom
xanderbailey:xb/encryption-read-manifest-list

Conversation

@xanderbailey

@xanderbailey xanderbailey commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Working towards #3222

Rationale for this change

We need to be able to read encrypted manifest lists which means we need the Table to be able to construct an EncryptionManager.

  • Table takes an optional kms_client, and Table.encryption_manager builds the manager from the table's current metadata.
  • Snapshot.manifests takes an optional encryption_manager. When the snapshot has a key-id the manifest list is decrypted, and it raises if no manager is given.
  • Scans and inspect pass the table's encryption manager through.
  • The manager is passed around separately rather than wrapped into FileIO, matching iceberg-rust ([7/N] feat(encryption) [7/N] Read encrypted manifest list iceberg-rust#2453). <- this is the main design decision to make for this PR and is worth considering carefully. I've chosen to take the approach we used in rust because it actually simplifies things down the line a lot but very happy to hear arguments for the alternative.

Catalogs don't pass a KMS client yet, that comes in a follow-up.

Are these changes tested?

Yes. A manifest list encrypted by iceberg-rust is checked in under tests/table/encryption, and scanning and inspecting that table reads it. There's also a round trip of an encrypted manifest list with entries.

Are there any user-facing changes?

Table and the scans take a new optional argument. Nothing changes for unencrypted tables.

@xanderbailey

Copy link
Copy Markdown
Contributor Author

@kevinjqliu would be interested on your take here for how this looks vs how Java does it. I'm pretty happy with following the same pattern we have in rust but curious for your thoughts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant