Skip to content

[C++][Feature] Add ALP float/double codec with SIMD - #966

Merged
ColinLeeo merged 19 commits into
developfrom
colin/alp-codec
Nov 13, 2026
Merged

ColinLeeo merged 19 commits into
developfrom
colin/alp-codec

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add an opt-in ALP encoding for FLOAT and DOUBLE to the C++ implementation,
and expose the new encoding through the Python API.

The existing Gorilla/Chimp codecs are not a good fit for fixed-precision
decimal-like training data. ALP converts such values to integers, applies
frame-of-reference and bit packing, and stores exceptions for values that
cannot be represented exactly. This PR adds:

  • TSEncoding::ALP = 15, supported only for FLOAT and DOUBLE.
  • The same TSEncoding.ALP = 15 value in Python, mapped through the C wrapper
    and Cython layer.
  • A self-contained ALP page/block wire format with ALP and PLAIN fallback.
  • Scalar reference encoder/decoder plus SIMDe AVX2-style kernels.
  • SIMD value conversion, SIMD bit unpacking, and word-based packing.
  • A two-stage exponent/factor search and direct decoding from wrapped page
    buffers.
  • Opt-in integration with the existing EncoderFactory/DecoderFactory.
    The default FLOAT/DOUBLE encoding remains Gorilla.

Java is intentionally untouched. The wire format is language-neutral so a
Java decoder can be added later without changing encoded bytes.

Design

The codec is bit-exact by construction: the encoder decodes each candidate
integer back to the original type and stores an exception when the round-trip
is not bitwise equal. Non-decimal or high-entropy columns fall back to PLAIN;
ALP remains opt-in and should be selected per column.

Benchmark

Codec-level benchmark on Apple Silicon, best of 5, 1M values:

Type Codec Ratio Encode MB/s Decode MB/s
FLOAT decimal ALP 1.773 539.3 2646.9
FLOAT decimal Gorilla 1.457 342.0 1389.3
DOUBLE decimal ALP 5.882 1055.3 6470.7
DOUBLE decimal Gorilla 0.970 276.7 2546.8

The benchmark also includes TS_2DIFF as a decimal-data reference. TS_2DIFF has
a much higher ratio on this workload, but the current C++ TS_2DIFF path is not
bit-exact for all FLOAT/DOUBLE values; ALP is the lossless option.

Testing

  • Full ./mvnw -P with-cpp ... clean verify passed locally with all bundled
    dependencies enabled: 974/974 tests passed, with only the existing
    disabled/skipped tests not run.
  • The CI compatibility build with -Denable.simd=OFF also passed.
  • New C++ tests cover scalar round-trip, SIMD/scalar parity, encoder/decoder
    factory integration, special values, truncated payloads, and end-to-end
    TsFile write/read with ALP.
  • New Python test verifies TSEncoding.ALP and an ALP FLOAT/DOUBLE
    round-trip through the Cython layer.
  • CI passes on Ubuntu, macOS, Windows/MinGW, and Windows/MSVC across the
    supported Python versions; Spotless, cppcheck, RAT, CodeQL, and the
    cross-language fixture validation jobs also pass.

Compatibility

ALP writes a new encoding byte. Older readers that do not know encoding 15
will reject ALP chunks, so the codec is opt-in and the default encodings are
unchanged. Existing files are unaffected.

Licensing and provenance

The codec is an independent implementation of the ALP algorithm
(Xinyu Zeng et al., "ALP: Adaptive Lossless Floating-Point Compression",
SIGMOD 2024, DOI 10.1145/3626717). It shares no source code with the authors'
MIT-licensed reference implementation (cwida/ALP); the wire format, the
Encoder/Decoder integration and the SIMD kernels are original and carry
the Apache License 2.0 like the rest of the repository, so no
LICENSE/NOTICE change is required. Scheme value 2 remains reserved for the
ALP-RD variant. See cpp/docs/alp-codec-design.md#references-and-provenance.

@codecov-commenter

codecov-commenter commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.57055% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.82%. Comparing base (10626d0) to head (57f2829).

Files with missing lines Patch % Lines
cpp/src/encoding/alp/alp_decoder.h 45.12% 44 Missing and 1 partial ⚠️
cpp/src/encoding/alp/alp_encoder.h 75.67% 9 Missing ⚠️
cpp/src/encoding/alp/alp_simde.h 95.65% 5 Missing ⚠️
cpp/src/encoding/alp/alp_scalar.h 98.63% 4 Missing ⚠️
cpp/src/encoding/decoder.h 0.00% 2 Missing ⚠️
cpp/src/common/global.h 50.00% 1 Missing ⚠️
cpp/src/encoding/alp/alp_format.h 98.57% 1 Missing ⚠️
cpp/src/reader/chunk_reader.cc 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #966      +/-   ##
===========================================
+ Coverage    66.55%   66.82%   +0.26%     
===========================================
  Files          782      788       +6     
  Lines        56709    57363     +654     
  Branches      9134     9255     +121     
===========================================
+ Hits         37743    38330     +587     
- Misses       17304    17370      +66     
- Partials      1662     1663       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ColinLeeo ColinLeeo changed the title Colin/alp codec [C++][Feature] Add ALP float/double codec with SIMD Sep 18, 2026
- Validate the block factor/exponent and the page value/block counts when
  decoding. A damaged header could index past the power-of-ten tables or
  drive a multi-gigabyte reservation.
- Decode packed fields wider than 57 bits with the scalar tail. The 64-bit
  SIMD unpack rebuilds each value from a single 64-bit window and silently
  truncated fields of 59, 61, 62 and 63 bits.
- Scale values in the SIMD encoder in the same order as the scalar
  reference, so both builds emit identical bytes.
- Always re-score the fewest-exception candidate and the unscaled (0, 0)
  candidate in the exponent/factor search. The 8-sample screen ranks a
  candidate that rejects every sample as the cheapest, so a wide-range
  block could be shortlisted only with candidates that encode nothing and
  silently degrade to PLAIN.
- Accept ALP in AlignedChunkReader and in set_datatype_encoding's range
  assertion, so aligned/table columns written with ALP can be read back.
An aligned device's schema is fixed at the first registration on develop;
registering the two ALP columns separately makes the second call fail with
E_INVALID_ARG.
Record that the codec is an independent implementation of the SIGMOD 2024
ALP algorithm, shares no code with the MIT-licensed reference
implementation, and needs no LICENSE/NOTICE change; note that scheme 2 is
reserved for ALP-RD.
- Add a table-model round-trip with NULL rows, multiple pages and SNAPPY
  compression, exercising AlignedChunkReader's multi-value path.
- Assert the chunk header stores ALP for FLOAT/DOUBLE after a write, so a
  silent codec downgrade cannot pass the round-trip tests.
- Decode an encoded page from an unaligned byte buffer.
- Document that ALP is C++/Python-only for now: db_common.h, the C wrapper
  header, the Python enum, the design doc, the Java enum comment and the
  compatibility matrix all warn that Java cannot read ALP columns, and the
  matrix completeness bound records why ALP is not swept yet.
@ColinLeeo
ColinLeeo merged commit 4c2cc4b into develop Nov 13, 2026
61 of 62 checks passed
@HTHou
HTHou deleted the colin/alp-codec branch November 13, 2026 03:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants