Repository navigation
Conversation
Implement issue #1058 for the Java reference implementation. - Writer: TsFileWriter.addColumn/dropColumn/renameColumn mutate the registered TableSchema (FIELD columns only; TAG/ATTRIBUTE/TIME rejected). - On-disk: rename history (aliases) and drop history (tombstones) are stored in a tsFileProperties entry ("tableSchemaEvolution") written only when evolution occurred, so no-evolution files stay byte-compatible. The tableSchemaMap keeps the union/current schema. - Reader: renamed columns resolve via aliases (the index lookup expands historical names and merges their series); aligned reads now align value chunks to time chunks by file offset so columns added or renamed across chunk groups read back with nulls for the groups they were absent from. - Type changes are rejected, never coerced. Tests: TableSchemaEvolutionTest covers cross-chunk-group rename, rename-then-write, tag-column rename rejection, name/tombstone collision, add duplicate, drop-then-re-add with a different type, drop hiding, add-nulls-in-old-groups, and no-evolution byte compatibility.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #1086 +/- ##
===========================================
+ Coverage 66.49% 66.62% +0.12%
===========================================
Files 778 779 +1
Lines 56559 56818 +259
Branches 9098 9161 +63
===========================================
+ Hits 37607 37853 +246
- Misses 17301 17314 +13
Partials 1651 1651 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #1058 (milestone 2.5.1) for the Java reference implementation.
What
Table-model columns can now be evolved between writes via three new
TsFileWritermethods:addColumn(table, name, type[, encoding, compression])dropColumn(table, name)renameColumn(table, oldName, newName)Only FIELD columns are evolvable; TAG / ATTRIBUTE / TIME are rejected with a clear error (TAG participates in device-id encoding). Type changes are rejected, never silently coerced to STRING.
On-disk format (backward compatible)
tableSchemaMapkeeps the union/current column list (renamed names, added columns; dropped columns removed).tsFilePropertiesentry, key"tableSchemaEvolution", written only when a table has evolved. Files written without any evolution are byte-identical to before this change, and old readers ignore the unknown property (defined "ignore" behavior — no version bump).Read semantics
The aligned read path now aligns value chunks to time chunks by file offset instead of position, so a column added or renamed across chunk groups (fewer chunks than the time column) reads back correctly instead of throwing
IndexOutOfBoundsException.Scope / follow-ups
This is the Java reference implementation. C++/Python/C parity and append/recovery of the evolution history are follow-ups (detailed in the sibling design notes). The full Java unit test suite passes;
TableSchemaEvolutionTestadds 8 tests covering cross-chunk-group rename, rename-then-write, tag rename rejection, name/tombstone collision, add duplicate, drop-then-re-add with a different type, drop hiding, add-nulls-in-old-groups, and no-evolution byte compatibility.