Repository navigation
[C++][Feature] Add ALP float/double codec with SIMD - #966
Merged
Merged
Conversation
ColinLeeo
force-pushed
the
colin/alp-codec
branch
from
September 17, 2026 10:37
8869fe8 to
4e74d03
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #966 +/- ##
===========================================
+ Coverage 66.55% 66.82% +0.26%
===========================================
Files 782 788 +6
Lines 56709 57363 +654
Branches 9134 9255 +121
===========================================
+ Hits 37743 38330 +587
- Misses 17304 17370 +66
- Partials 1662 1663 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Validate the block factor/exponent and the page value/block counts when decoding. A damaged header could index past the power-of-ten tables or drive a multi-gigabyte reservation. - Decode packed fields wider than 57 bits with the scalar tail. The 64-bit SIMD unpack rebuilds each value from a single 64-bit window and silently truncated fields of 59, 61, 62 and 63 bits. - Scale values in the SIMD encoder in the same order as the scalar reference, so both builds emit identical bytes. - Always re-score the fewest-exception candidate and the unscaled (0, 0) candidate in the exponent/factor search. The 8-sample screen ranks a candidate that rejects every sample as the cheapest, so a wide-range block could be shortlisted only with candidates that encode nothing and silently degrade to PLAIN. - Accept ALP in AlignedChunkReader and in set_datatype_encoding's range assertion, so aligned/table columns written with ALP can be read back.
An aligned device's schema is fixed at the first registration on develop; registering the two ALP columns separately makes the second call fail with E_INVALID_ARG.
Record that the codec is an independent implementation of the SIGMOD 2024 ALP algorithm, shares no code with the MIT-licensed reference implementation, and needs no LICENSE/NOTICE change; note that scheme 2 is reserved for ALP-RD.
ColinLeeo
force-pushed
the
colin/alp-codec
branch
from
November 12, 2026 07:25
4ba8d63 to
9009910
Compare
- Add a table-model round-trip with NULL rows, multiple pages and SNAPPY compression, exercising AlignedChunkReader's multi-value path. - Assert the chunk header stores ALP for FLOAT/DOUBLE after a write, so a silent codec downgrade cannot pass the round-trip tests. - Decode an encoded page from an unaligned byte buffer. - Document that ALP is C++/Python-only for now: db_common.h, the C wrapper header, the Python enum, the design doc, the Java enum comment and the compatibility matrix all warn that Java cannot read ALP columns, and the matrix completeness bound records why ALP is not swept yet.
# Conflicts: # cpp/CMakeLists.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in ALP encoding for
FLOATandDOUBLEto the C++ implementation,and expose the new encoding through the Python API.
The existing Gorilla/Chimp codecs are not a good fit for fixed-precision
decimal-like training data. ALP converts such values to integers, applies
frame-of-reference and bit packing, and stores exceptions for values that
cannot be represented exactly. This PR adds:
TSEncoding::ALP = 15, supported only forFLOATandDOUBLE.TSEncoding.ALP = 15value in Python, mapped through the C wrapperand Cython layer.
buffers.
EncoderFactory/DecoderFactory.The default FLOAT/DOUBLE encoding remains Gorilla.
Java is intentionally untouched. The wire format is language-neutral so a
Java decoder can be added later without changing encoded bytes.
Design
cpp/docs/alp-codec-design.mddescribes thepage/block format, exception handling, and the relationship to TS_2DIFF.
cpp/docs/alp-vs-gorilla-benchmark.mdrecords the ALP/Gorilla/TS_2DIFF comparison.
cpp/docs/alp-profiling-analysis.mdrecords the flame-graph analysis and the optimizations that followed it.
The codec is bit-exact by construction: the encoder decodes each candidate
integer back to the original type and stores an exception when the round-trip
is not bitwise equal. Non-decimal or high-entropy columns fall back to PLAIN;
ALP remains opt-in and should be selected per column.
Benchmark
Codec-level benchmark on Apple Silicon, best of 5, 1M values:
The benchmark also includes TS_2DIFF as a decimal-data reference. TS_2DIFF has
a much higher ratio on this workload, but the current C++ TS_2DIFF path is not
bit-exact for all FLOAT/DOUBLE values; ALP is the lossless option.
Testing
./mvnw -P with-cpp ... clean verifypassed locally with all bundleddependencies enabled:
974/974tests passed, with only the existingdisabled/skipped tests not run.
-Denable.simd=OFFalso passed.factory integration, special values, truncated payloads, and end-to-end
TsFile write/read with ALP.
TSEncoding.ALPand an ALP FLOAT/DOUBLEround-trip through the Cython layer.
supported Python versions; Spotless, cppcheck, RAT, CodeQL, and the
cross-language fixture validation jobs also pass.
Compatibility
ALP writes a new encoding byte. Older readers that do not know encoding
15will reject ALP chunks, so the codec is opt-in and the default encodings are
unchanged. Existing files are unaffected.
Licensing and provenance
The codec is an independent implementation of the ALP algorithm
(Xinyu Zeng et al., "ALP: Adaptive Lossless Floating-Point Compression",
SIGMOD 2024, DOI 10.1145/3626717). It shares no source code with the authors'
MIT-licensed reference implementation (
cwida/ALP); the wire format, theEncoder/Decoderintegration and the SIMD kernels are original and carrythe Apache License 2.0 like the rest of the repository, so no
LICENSE/NOTICEchange is required. Scheme value 2 remains reserved for theALP-RD variant. See
cpp/docs/alp-codec-design.md#references-and-provenance.