Repository navigation
Glue: JSON text as an intermediate on bulk and per-row paths, ten ranked candidates (none timed yet) #388
Description
Several paths of both wheels still put data into JSON text on the Java side and parse it again in Python. For bulk, typed or wide data that costs a text format, a parse and one dict per row, and the columnar routes of #331, #335 and #383 show what the binary route buys. I went through every place of the glue that uses JSON as an intermediate (a code read of main at 65e7c8f: 42 uses classified from 338 grep hits in 29 files) to see which ones are worth removing.
None of this is timed. Every cost named here is a number of crossings or per-row work read from the code, so nothing below is a speed claim. Each item gets its own measured table and a pre-registered A/B (ABBA, A/A control, against the released wheel) before its pull request, as for #382 to #385.
17 of the 42 uses sit on a bulk or per-row path. Ten are worth the work, ranked by how much data goes through them and how small the fix is:
| rank | wheel | site on main |
what the JSON carries | when | replacement | rebuild | check first |
|---|---|---|---|---|---|---|---|
| 1 | JVM | src/arcadedb/core.py:948 |
insert_many rows as one JSON text per chunk (10,000 rows), tagged values for date, datetime, Decimal, bytes, numpy and pandas scalars; float vectors... |
per chunk of 10,000 rows | route homogeneous chunks through rows_to_columns and DocumentBatcher.insertColumns (exists); keep JSON for shapes columns cannot carry |
none (PY) | yes, plus a stored-type differential test |
| 2 | native | src/arcadedb/_native/database.py:648 |
row count (count() walks the whole result) |
per count() call; O(rows) |
arcadedb_n_count returning a long; PY partial: count len(batch) without building NativeResult |
native image for full | no, the loop is the evidence |
| 3 | native | native/src/main/java/org/humemai/nativelib/ArcadeNative.java:205 |
transaction state {ok, active, depth, commits} | per begin, commit, rollback, is_transaction_active, stack mark |
return one long (active bit, depth, commits packed) from arcadedb_n_tx |
native image | no, counts are the evidence; measure only to size the win |
| 4 | native | src/arcadedb/_native/colframes.py:160 |
head of a columnar read: up to 256 JSON-decoded rows re-packed into a frame with a Python-written JSON header | once per to_arrow/to_columns/to_dataframe/to_polars call |
ask Java for frame 1 directly (arcadedb_n_next_col from the first call), retire _head_frame |
native image + PY | yes |
| 5 | native | src/arcadedb/_native/schema_objects.py:331 |
approximate vector search: the query vector as a JSON list of d numbers, hits as a JSON list of [rid, score] | per search | arcadedb_n_vindex_bin taking a float32 pointer and count; hits as one packed buffer |
native image | yes: how often approximate is called in the benchmark lanes |
| 6 | native | native/src/main/java/org/humemai/nativelib/NativeCodec.java:83 |
query and command result rows (property dict per row, plus a meta array per row: rid, type, v/e/d) | first reply up to 256 rows (_FIRST_BATCH), then 2048 rows per crossing (_NEXT_BATCH); per... |
binary row batches (frame per batch decoded to column lists, rows built by zip) for typed or wide results; keep JSON for narrow plain rows | native image | yes: json.loads is C, zip-of-columns is Python per row |
| 7 | native | src/arcadedb/_native/extras.py:193 |
GraphBatch create_vertices: rows as JSON, RIDs back as a JSON list of text |
per chunk (commit_every rows) |
frame (insert_cols layout) in, packed long[] of RIDs back |
native image | yes |
| 8 | native | src/arcadedb/_native/extras.py:575 |
async create_record: one JSON op {type, kind, props} per record, plus callbacks |
per record | a create_records batch entry taking a frame; or buffer in Python and use insert_parallel |
native image | no, crossings are the evidence |
| 9 | JVM | src/arcadedb/graph_batch.py:262 |
GraphBatch create_vertices: rows as JSON text, RIDs back as one ';'-joined text |
per chunk (_BULK_CHUNK rows) |
a ColumnBatcher-style typed insert for vertices (DocumentBatcher.insertColumns already builds vertex types) |
none (PY) or bridge jar | yes |
| 10 | both | native/src/main/java/org/humemai/nativelib/ColumnFrames.java:322 |
the json column kind inside binary column frames (nested lists, maps, RIDs, embedded documents, mixed-type columns): one tagged JSON array per column | per frame, per cell of such a column | add column kinds: rid (two int64), list of int64/float64 (ragged offsets), keep json for the rest | native image and bridge jar | yes |
Also found, not in the first audit: the query vector of an approximate vector search crosses as a JSON list (rank 5), and the json cell kind inside binary frames (rank 10) makes RID, list and map columns go through text. arcadedb_command in ArcadeLib.java has no Python caller and can go.
What stays: error replies, control and configuration messages, frame headers and column specs (once per call or per failure), statement parameters (small, and float32 vectors already go binary), the JVM to_dict and to_list reads where the code comments say JSON is the cheaper container on JPype (only the object hook is removable, that is #382), and the shapes that a column cannot carry (widen the columnar path first). The JSON that a caller asks for (to_json_list, iter_json_batches, to_json) never goes.
Rank 3 is #384, and rank 4 is the second part of #381. The rest has no pull request yet. I take them in the order of the table, the first ones needing no native rebuild, and I post the A/B of each here.
Activity
JVM insert_many: a homogeneous chunk crosses as typed columns, not as…
GraphBatch (JVM): create_vertices and new_edges send rows of exact sc…
Tests: the async create_record tests import numpy at the top, so a mi…
Tests: the type dump of the create_record parity test uses literal st…
Tests: the frame kinds of RID and number-list columns, and the stored…
Tests: format the stored fixture with prettier (formatting only, refs #…
Tests: fix the stand-in and three expectations of the native GraphBat…
Tests: the returned RIDs are checked by the engine's lookup of each t…
11 remaining items
Tests: take tests/test_import_database.py from main (the post3 tag co…
Tests: take tests/test_import_database.py from main (the post3 tag co…
JVM insert_many typed columns: a row the JSON text cannot carry keeps…
JVM: get_rid, get_vertex, get_edge, get_element and Document.get_rid …
Tests: the crossing test counts only the call (proxies off around beg…
JVM: Document.get_rid stays on the plain path, no bridge call (refs #388
Native create_vertices: a chunk of fewer than 64 rows keeps the JSON …
Tests: the signature table of lib._load uses only the aliases it defi…
Native async create_record: the first 32 records of an executor go on…
Native create_vertices: the comment of the 64 row floor names the mic…
Tests: a def in the floor test, not an assigned lambda (ruff E731 of …
Native async create_record: a hand-over of fewer than 64 collected re…
Tests: nosec B404 and B603 on the subprocess of the floor-from-enviro…
Native async create_record: the first 96 records of an executor go on…
Native async create_record: the first 128 records of an executor go o…