[tae898]

What does this PR do?

On the arcadedb (native) wheel, find_nearest_approximate sent the query vector to the library as a JSON list of d numbers (text that Python printed and Java parsed into a list of boxed doubles, then narrowed to a float[]) and read the hits back as a JSON list of [rid, score] that Python parsed again. Now a query that float32 carries crosses as a pointer and a count, a filter crosses as int64 (bucket, position) pairs, and the library writes the hits into a buffer that Python provides. It is the same shape as the other address entries (arcadedb_n_insert_colp, arcadedb_n_exec_bin, arcadedb_n_gb_bin). This is rank 5 of the scan in #388, the one that was not in the first audit.

The new entry, in NativeExtras.java:

@CEntryPoint(name = "arcadedb_n_vindex_bin")
public static long vindexBin(IsolateThread thread, long dbh, CCharPointer type, CCharPointer prop, int k,
    CFloatPointer query, long dims, CCharPointer allowed, long allowedPairs, CCharPointer out, long capHits)

It returns the number of hits n. When n <= capHits the hits are in out, three int64 words each: the RID's bucket, its position, and the score as the bits of a double. When n is larger nothing is written and Python asks again with room for n. -1 is an error, and its text is read with arcadedb_n_last_error, as for arcadedb_n_tx. arcadedb_n_vindex stays for the metadata and build ops and for the queries the new entry cannot carry. Both entries share one lookup of the bucket indexes (vectorParts) and one search loop (approximateHits).

On the Python side (schema_objects.py, lib.py), NativeVectorIndex._run_approximate tries _approximate_packed and falls back to _approximate_json, which is the old code. The merge of the bucket answers (_sort_results, cut to k) and the record loads by RID are unchanged.

What the answer is, and what stays on the JSON route

The call returns the same list as before: the same records in the same order, the same RID texts, and scores of the same type with the same bits. A score on the JSON route is the double that the number text of the engine's float parses to (Float.toString, then Python's float()), not the float widened. The new entry computes that same double in Java with Double.parseDouble(Float.toString(f)), so the two routes agree by construction and not by a rounding argument. That is why the buffer holds a double and not a float32. The query is narrowed to float32 in Python with numpy's cast, which rounds to nearest even as Java's floatValue() does, and the tests compare both routes bit for bit, with exact ties and a subnormal in the query.

A query crosses by address when it is a one-dimensional float16, float32 or float64 array of the native byte order, or a list or tuple of float and int (an int beyond 2**53 is not one). Everything else keeps the JSON route, which returns the same hits: a generator, a NumPy scalar, None or a bool in a list, an integer array, a 2-d array, an empty query, and a NaN, an infinity or a value that float32 overflows to. A filter crosses as int64 pairs when every entry is plain #bucket:position text (a record is its get_rid() text, as before); anything else goes to the engine's own RID parser through the JSON route. A k beyond a C int, an answer buffer of more than a gibibyte, a missing numpy, ARCADEDB_NATIVE_VINDEX_BIN=0 and a library built before this entry also keep the JSON route. The switch and the older-library fallback are the ones the other address entries have.

What it allocates

Per call, in Python: the float32 query array (d x 4 bytes; none when the caller passes a contiguous float32 array, which is used as it is), the answer buffer (24 bytes per hit; room for 8 bucket indexes of k hits at first, so 192 x k bytes, and larger once for a type with more bucket indexes, remembered per index), and for a filter of m RIDs the int64 pair array (16 x m bytes) plus a transient list of 2m short strings from the digit scan. After the call: one tuple per hit and one RID text per hit, as the JSON route built. In Java: the float[] of d floats and the HashSet of the filter, as before, and no StringBuilder for the hits and no parsed list of boxed doubles for the query. Nothing holds more than one query or one answer.

Other searches

Only the approximate search used this JSON shape. find_nearest (also with include_vector=False, and find_nearest_by_key) runs vectorNeighbors as a statement: its query is already a float32 array in the side buffer of arcadedb_n_exec_bin, and its hits are result rows (the rank 6 path of #388), a different Java op, so this entry does not cover it and I changed nothing there. get_metadata, get_stats, get_size and build_graph_now use the metadata and build ops of arcadedb_n_vindex or statements, once per call and with no vector, and stay as they are.

Related issues

Refs #388 and #315.

Performance changes

Not measured yet, and no number is stated here. The change removes the number text of the query on both sides and the hit text of the answer; the tests pin what crosses, not how long it takes. The figure for the timed A/B is pre-registered as follows.

• Operation: one find_nearest_approximate(query, k) call on the arcadedb (native) wheel, against the released 26.11.1.post3 wheel as the baseline, ABBA with an A/A control, one fresh process per run.
• Data: a PRODUCT-quantized index with the default bucket count and enough vectors per bucket to be trained (at least pq_clusters), so the search is approximate. The sizes that matter: d of 128, 768 and 1536; k of 1, 10 and 100; a query as a float32 array (the form that is not copied) and as a float64 array (one narrowing copy); one filtered case with a few thousand RIDs.
• Unit: one call, including the k record loads by RID that the call makes, so the figure is what a caller sees. The loads are unchanged by this PR; k of 1 is where they weigh least.
• Parity digest: a hash over the list of (RID text, record id, score bits) of every search of the query set, from both wheels, which must be equal. The tests in this PR pin the same fact for a small database.
• No-change control: find_nearest(query, k) on the same index (the statement route, untouched by this change), measured in the same runs.
• Memory: peak RSS and anon growth as information figures, not a gate.

Docs

docs/modules/ROOT/pages/how-to/vectors.adoc: one paragraph after the quantization rules on how find_nearest_approximate crosses on arcadedb, which queries keep the text route, and the switch. No other page or the README describes it.

How I checked it

The tests are in the first commit and the change in the second, so the first commit fails on the old code: every test uses lib.VINDEX_BIN or lib.vindex_bin, which the parent does not have.

• tests/test_native_vindex_glue.py needs no library. A stand-in for arcadedb_n_vindex_bin reads the query and the filter by their addresses, as the library does, and writes hits into the buffer. It pins the floats the library is given for every numeric form (bit for bit against a C cast through struct, so numpy is not its own witness), the filter pairs and the plain-text rule, the buffer size and the second call, the order, RID texts and score bits against the JSON route, and each shape that keeps the JSON route.
• tests/test_native_vindex_bin.py needs the native library (not collected on the JVM wheel): a four-bucket PRODUCT index of 1200 vectors, k of 3 and 500, three query forms, the scores of both routes compared as packed doubles, queries that float32 rounds, a filter by text and by record, the second call for an answer larger than the first buffer, and the same outcome as the JSON route for each shape that keeps it and for a wrong dimension and an unindexed property.
• Run on this branch, light checks only: uvx pre-commit run --files on the changed files, Bandit as CI runs it, compile() and pyflakes. I did not run the tests: the laptop's queue guard blocks the test runner, and there is no JDK 25 here for the Java, so CI has to prove the Java compiles and every test passes. A draft pull request skips the platform legs, so I dispatched the native test workflow on the branch myself.

CI: the native workflow that I dispatched on the branch (run 43568546290) passed on all four platforms. The Java compiles, and the parity suite passed with no failure, error or skip, including test_native_vindex_bin.py (19 tests) and test_native_vindex_glue.py (51 tests) on linux-x86_64 with Python 3.12. That run was on the commit before I rebased the branch onto main. The pre-commit check of this pull request then failed with ruff F821 on the ctypes type letter I in lib.py, because main renamed it to N. I renamed it in my entry, rebased, and every hook of main's configuration passes locally. The native workflow has not run on the rebased commit: the only code difference from the tested one is that one-letter rename.

Checklist

• A regression test fails on the old code and passes on the new (the first needs the CI run to be seen)
• The test suite that the change can affect passes (the native run above; the rebased commit is not re-run)
• uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code