Repository navigation
Reject v3-only features on pre-v3 tables #4156
Description
Feature Request / Improvement
Several features are only valid in format v3 (spec, Appendix E), e.g. the new types (variant, geometry, geography, unknown, timestamp_ns, timestamptz_ns), non-null initial-default, encryption (encryption.key-id, encryption-keys, snapshot key-id), deletion vectors, and row lineage.
PyIceberg handles these inconsistently on pre-v3 tables today. Some checks only run at create_table, some features are silently dropped, and others are accepted. We should follow Java here, but Java doesn't cover every case either, so we need to go further.
Rules
- Reject v3 features on pre-v3 tables. Creating a pre-v3 table, or changing its metadata (schema, properties, partition specs, sort orders), must raise if the change introduces a v3-only feature. Never silently accept, drop, or ignore it.
- Allow reads and repairs. Loading and scanning a pre-v3 table that already contains v3-only features must work. Metadata changes that only remove v3-only features (e.g.
remove_properties("encryption.key-id"), dropping a column) must succeed, so the table can be repaired from invalid to valid. - Check the version on the write side. Writers must check the format version before writing v3-only structures (e.g. deletion vectors, encrypted manifest lists, row lineage fields), and raise instead of falling back to a pre-v3 equivalent.
- Reject everything else on an invalid table. If a pre-v3 table contains v3-only features, reject all other writes until it is repaired: data writes (append, overwrite, delete), metadata updates other than repairs, and format version upgrades. Upgrading must not turn an invalid table into a valid one.
Java coverage
| Rule | Java |
|---|---|
| 1. Reject v3 features | Types and initial-default are checked on every schema change (Schema.checkCompatibility). Encryption properties are checked only at table creation (EncryptionUtil.checkCompatibility), not on property updates |
| 2. Reads and repairs | Allowed, since metadata isn't validated on load |
| 3. Write-side checks | Deletion vectors are checked on commit (validateDeleteFileForVersion). Manifest list encryption is not gated by version |
| 4. Reject on invalid tables | Not covered: upgrades and other writes are not blocked |
Note: Java doesn't fully handle this yet either. Where PyIceberg goes further than Java (rules 1, 3 and 4), we should raise the gaps on the dev list so implementations stay consistent.
Activity
Ankitraj-builds commentedon Oct 26, 2026
I opened #4160 for the first piece of rule 1: schema changes on v1/v2 tables now raise for v3-only types (timestamp_ns, unknown, etc.) and for a non-null initial-default. Before, the format version check only ran in create_table, so update_schema().add_column(...) accepted these.
Dropping a v3-only column still works, so a table can be repaired. Happy to take the next pieces (properties like encryption.key-id, then the write-side checks) as separate PRs if that works for you.