[tae898]

What does this PR do?

A column of record links or of lists of numbers inside a binary column frame was one tagged JSON array of the column (the json kind), parsed per cell in Python. This is rank 10 of #388. Both writers (ColumnFrames of the native image and ColumnBatcher of the bridge jar) now write two more column kinds, and the one shared decoder arcadedb/_columnar.py reads them:

• rid: two int64 per row, the bucket and the position of a record link. to_arrow gives a string column of #bucket:position, as the column of RID objects degraded to before. to_columns and to_dataframe give the wheel's own RID object: the native wheel builds its RID from the pair, and the JVM bridge puts the engine's RID objects into the side array in row order (the header names where they start, side), so the JVM wheel still hands out the engine's own RID, which a bucket and a position cannot be turned back into on the Python side without the database. to_arrow never touches the side array.
• li8 and lf8: a list of int64 or of float64 per row. The buffer is int32 offsets (rows + 1), a bitmap with one bit per element of the flat array for a null element, then the flat int64 or float64 array of every list's elements. A null row is the frame's null bit, an empty list is two equal offsets, None inside a list is a bit of the bitmap. NaN, the infinities and -0.0 are float64 values. to_columns gives Python lists of exact int or float, to_arrow gives list<int64> or list<double> over the frame's own offsets and flat array.

What keeps the json kind: maps, nested lists, lists of strings, booleans or RIDs, a list that mixes ints and floats (each element keeps its own type there), a list of nothing but nulls or empty lists (Arrow types it list<null>, an element type cannot be read from it), embedded records, a RID that is not valid (negative bucket or position), a column that mixes kinds (a column of int lists in one row and float lists in another is json), Set, primitive arrays, and a column with more than 2**27 list elements in one frame (int32 offsets). The decoder keeps reading json for RID and list columns, so a frame of an older writer still decodes. The first 256 rows of a native result are written as a frame in Python from the JSON reply (the values themselves, json kind); that code is unchanged, and the decoder gives the same values and the same Arrow types for both forms.

Related issues

Refs #388 and #315.

Performance changes

Not measured: no timed A/B has run, so this PR states no speed or memory figure.

Figure definition for the A/B (pre-registered before it runs): the operation is a columnar read of one query, to_columns(), to_arrow(), to_dataframe() and to_polars(); the unit of one operation is one such read of the whole result; sizes are 1, 100 and 100,000 rows, plus the batch_size that splits the 100,000 rows into several frames; shapes are a SELECT @rid column, a link property, an int list column, a float list column, and the control shapes that keep json (a map column, a nested list column) which must not change. Both wheels, each against the released 26.11.1.post3 wheel of the same distribution, ABBA with an A/A noise run, latency and peak RSS with anon growth as information figures. The parity digest is tests/test_frame_rid_list_kinds.py::test_readers_match_the_tag_fixture: all four readers on a 700-row table of these shapes, per column the type and a digest of the exact values (type tags, repr of floats, NaN and -0.0 apart from 0.0), compared with the output of the released 26.11.1.post3 wheels stored in tests/fixtures/frame_kinds_post3.json. The no-change control is the map and the nested list columns, and a result with no RID or list column.

What the change allocates per frame, per column: a rid column is 16 bytes per row of buffer on each side (the JVM also holds one reference per row in the side array); an li8 or lf8 column is 4 bytes per row of offsets, one bit per element, 8 bytes per element, one copy of the flat array on each side of the crossing. to_columns then builds one Python list per row and one Python number per element (the JSON route built the same objects, plus the text and the parse tree); to_arrow builds no per-element Python object. The frame still holds one batch, never the whole result.

Docs

No page under docs/ or README.md describes the frame kinds; the module docstrings of arcadedb/_columnar.py, ColumnFrames.java and ColumnBatcher.java and the one of arcadedb/_native/colframes.py are updated.

How I checked it

• tests/test_column_frames.py (hand-built frames, no database): rid, li8 and lf8 with nulls, empty lists, None inside, NaN, the infinities, -0.0 and integers beyond 2**53; the decoder against the JSON form of the same column (values, Python types, Arrow type); the hook that makes the RID objects; and a json frame of RIDs, lists and maps still reads. Six of the seven fail on the decoder of the tag.
• tests/test_frame_rid_list_kinds.py (both wheels, no branch on the backend except the stored output): which kind each shape gets, read from the frame headers (this fails while a writer still writes json), the four readers against the row route (to_list) cell by cell including the type and text of every RID, and against the stored output of the released wheels.
• The Java was not compiled on my machine (no local build, per the bench rules): the two CI runs below are the first compile.

Checklist

• A regression test fails on the old code and passes on the new (decoder tests: checked against the tag's decoder; writer tests: by their construction, they read the frame kinds the tag's writers do not write)
• The test suite that the change can affect passes (CI: native and JVM runs dispatched on this branch)
• uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code