Repository navigation
feat(encryption): [9/N] Read encrypted manifest lists - #4159
Open
xanderbailey wants to merge 1 commit into
Open
xanderbailey wants to merge 1 commit into
xanderbailey wants to merge 1 commit into
Conversation
Contributor
Author
|
@kevinjqliu would be interested on your take here for how this looks vs how Java does it. I'm pretty happy with following the same pattern we have in rust but curious for your thoughts. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Working towards #3222
Rationale for this change
We need to be able to read encrypted manifest lists which means we need the Table to be able to construct an
EncryptionManager.Tabletakes an optionalkms_client, andTable.encryption_managerbuilds the manager from the table's current metadata.Snapshot.manifeststakes an optionalencryption_manager. When the snapshot has akey-idthe manifest list is decrypted, and it raises if no manager is given.inspectpass the table's encryption manager through.FileIO, matching iceberg-rust ([7/N] feat(encryption) [7/N] Read encrypted manifest list iceberg-rust#2453). <- this is the main design decision to make for this PR and is worth considering carefully. I've chosen to take the approach we used in rust because it actually simplifies things down the line a lot but very happy to hear arguments for the alternative.Catalogs don't pass a KMS client yet, that comes in a follow-up.
Are these changes tested?
Yes. A manifest list encrypted by iceberg-rust is checked in under
tests/table/encryption, and scanning and inspecting that table reads it. There's also a round trip of an encrypted manifest list with entries.Are there any user-facing changes?
Tableand the scans take a new optional argument. Nothing changes for unencrypted tables.