Skip to content

feat: table-model schema evolution (add/drop/rename columns) - #1086

Open
ColinLeeo wants to merge 1 commit into
developfrom
colin/table-schema-evolution-da9fdc
Open

ColinLeeo wants to merge 1 commit into
developfrom
colin/table-schema-evolution-da9fdc

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Implements #1058 (milestone 2.5.1) for the Java reference implementation.

What

Table-model columns can now be evolved between writes via three new TsFileWriter methods:

  • addColumn(table, name, type[, encoding, compression])
  • dropColumn(table, name)
  • renameColumn(table, oldName, newName)

Only FIELD columns are evolvable; TAG / ATTRIBUTE / TIME are rejected with a clear error (TAG participates in device-id encoding). Type changes are rejected, never silently coerced to STRING.

On-disk format (backward compatible)

  • tableSchemaMap keeps the union/current column list (renamed names, added columns; dropped columns removed).
  • Rename history (aliases, historical name -> current name) and drop history (tombstones) are stored in a single tsFileProperties entry, key "tableSchemaEvolution", written only when a table has evolved. Files written without any evolution are byte-identical to before this change, and old readers ignore the unknown property (defined "ignore" behavior — no version bump).

Read semantics

  • Rename: data written under the old name is returned under the new name. The reader expands the query with historical aliases and merges the old+new series (time-sorted union).
  • Add: old chunk groups return null for the new column.
  • Drop: the column is hidden from results; its data stays on disk until compaction.

The aligned read path now aligns value chunks to time chunks by file offset instead of position, so a column added or renamed across chunk groups (fewer chunks than the time column) reads back correctly instead of throwing IndexOutOfBoundsException.

Scope / follow-ups

This is the Java reference implementation. C++/Python/C parity and append/recovery of the evolution history are follow-ups (detailed in the sibling design notes). The full Java unit test suite passes; TableSchemaEvolutionTest adds 8 tests covering cross-chunk-group rename, rename-then-write, tag rename rejection, name/tombstone collision, add duplicate, drop-then-re-add with a different type, drop hiding, add-nulls-in-old-groups, and no-evolution byte compatibility.

Implement issue #1058 for the Java reference implementation.

- Writer: TsFileWriter.addColumn/dropColumn/renameColumn mutate the registered
  TableSchema (FIELD columns only; TAG/ATTRIBUTE/TIME rejected).
- On-disk: rename history (aliases) and drop history (tombstones) are stored in a
  tsFileProperties entry ("tableSchemaEvolution") written only when evolution
  occurred, so no-evolution files stay byte-compatible. The tableSchemaMap keeps
  the union/current schema.
- Reader: renamed columns resolve via aliases (the index lookup expands historical
  names and merges their series); aligned reads now align value chunks to time
  chunks by file offset so columns added or renamed across chunk groups read back
  with nulls for the groups they were absent from.
- Type changes are rejected, never coerced.

Tests: TableSchemaEvolutionTest covers cross-chunk-group rename, rename-then-write,
tag-column rename rejection, name/tombstone collision, add duplicate,
drop-then-re-add with a different type, drop hiding, add-nulls-in-old-groups, and
no-evolution byte compatibility.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.95804% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.62%. Comparing base (545e8f0) to head (8f4563e).

Files with missing lines Patch % Lines
...a/org/apache/tsfile/file/metadata/TableSchema.java 90.51% 11 Missing ⚠️
...a/org/apache/tsfile/read/TsFileSequenceReader.java 93.15% 5 Missing ⚠️
...le/metadata/AbstractAlignedTimeSeriesMetadata.java 86.20% 4 Missing ⚠️
...che/tsfile/file/metadata/TableSchemaEvolution.java 92.30% 2 Missing ⚠️
...ain/java/org/apache/tsfile/write/TsFileWriter.java 91.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1086      +/-   ##
===========================================
+ Coverage    66.49%   66.62%   +0.12%     
===========================================
  Files          778      779       +1     
  Lines        56559    56818     +259     
  Branches      9098     9161      +63     
===========================================
+ Hits         37607    37853     +246     
- Misses       17301    17314      +13     
  Partials      1651     1651              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants