Repository navigation
Ingest integrity: count and content checks after every ingest path (parallel and async included) and a stress run for rare losses #336
Description
A goal for both wheels: no ingest path may lose, duplicate or silently alter a row, and a test must be able to catch it. Today the suite checks values and error messages path by path, but there is no one place that checks the same three things after every ingest call: the record count, an order-independent digest of every stored column, and the same two after closing and reopening the database. The new native read frames (#331) lost rows for a small batch_size and only CI found it, which is the kind of loss I want a test to find before a review does.
Scope: insert_many, insert_columns, insert_arrow, insert_dataframe, graph_batch (create_vertices, new_edges), the async executor, and time-series append, each sequential and with parallel=True or async where it exists, on both wheels. Cases: sizes at every chunk boundary (n = k * chunk and one either side of it, for the 100,000 JSON chunk, commit_every and batch_size), several writer threads on one type and on different types, a bad row in the middle of a batch (duplicate key, wrong type, refused value) with the documented outcome checked exactly (earlier chunks stored, the error raised, nothing silently dropped), a failure inside an async callback that must reach the caller, and close() straight after the last call.
Method: a CI test file for correctness on every platform (with a mutation check, so each test is shown to fail on code with a rows-dropped fault), and a stress script that repeats the cycles in fresh processes on the laptop. A run that finds no loss is reported as 0 of N, not as absence: with N cycles the rate is below 3 / N at 95 percent.
Done when: the test file is on the release branch and green on both wheels, the stress run has a published N, and every loss it finds has its own issue.
Activity
tests: ingest integrity, count and digest checks after every ingest p…
tae898 commentedon Nov 12, 2026
The stress run for this issue is done, and it found no loss: 0 of 300 cycles lost, duplicated or changed a row on each of the 14 ingest paths, on each of 4 builds, 16,800 cycles in all.
A cycle is a fresh database, one randomized ingest, a check,
close(), a reopen from disk, and the same check again. The check reads the record count three ways and an order-independent digest of every column two ways, and it wants every key exactly once, all against the generated data and not against what the path reports. The cycles vary the row count (up to 59,310, with the chunk and commit boundaries in the mix), the chunk and commit sizes, and the number of writer threads, and 1,372 of them inject a failure (1,240 duplicate keys and 132 values of the wrong type) intoinsert_many,insert_columns, and the parallelinsert_arrowandinsert_dataframe. Every one of those raised the error and ended as documented: exactly the committed batches stored on the sequential paths and, on the parallel paths, no key stored twice and the same count after the reopen as after the call. No call raised where nothing was injected.The paths are
insert_many(list, generator, andparallel=True),insert_columnsandinsert_arrowandinsert_dataframe(each sequential and parallel),create_vertices,new_edges, the asynccreate_recordat parallel level 1 and 4, andappend_samples. The builds are the releasedarcadedbandarcadedb-jvm26.11.1 wheels and the post1 candidate wheels of both, 12 fresh processes of 25 cycles per path and build, with a fixed seed so any failing cycle can be replayed. It started on 2026-11-10 on my laptop (Intel Core Ultra X9 388H, Linux 7.0.0, cores 8 to 11, Python 3.12, JPype1 1.7.1).With none found, the loss rate per path is below 3 / 300, which is 1 percent at 95 percent: this rules out a common loss and says nothing about a rarer one. The test file of PR #338 (
tests/test_ingest_integrity.py, with a mutation proof for every family) is on main, the CI jobs run the whole suite with no skips accepted, and theTest native (arcadedb)andTest JVM (arcadedb-jvm)runs on main atc90b3f99are green: https://github.com/ArcadeData/arcadedb-python/actions/runs/42946619491 and https://github.com/ArcadeData/arcadedb-python/actions/runs/42946619223.The published
26.11.1.post2wheels were not stress-run, and the candidate wheels predate some of its later changes, the JVM Arrow vector path of #348 among them. The "done when" here is a green test file on both wheels, a published N, and an issue for every loss, and no loss was found, so I am closing it; I will open a new issue with the replay line if a later run or CI finds one.