Skip to content

paperless.3: NEON on Linux arm64, leak and crash fixes - #3

Merged
stumpylog merged 13 commits into
paperlessfrom
integration/paperless-3
Nov 11, 2026
Merged

stumpylog merged 13 commits into
paperlessfrom
integration/paperless-3

Conversation

@stumpylog

Copy link
Copy Markdown
Owner

Brings NEON on Linux arm64 and a set of leak and crash fixes into the integration line for v0.1.10-paperless.3. See CHANGELOG-paperless.md.

NEON on Linux arm64 (ported from photostructure/sqlite-vec). The Makefile enabled NEON only on Darwin arm64, so Linux aarch64 wheels ran the scalar loop for every distance, including the cosine float kernel paperless-ngx uses.

  • Make l1_int8_neon compile with GCC (from the GCC part of photostructure a5aec74).
  • Enable NEON on Linux arm64, selected from $(CC) -dumpmachine rather than uname (photostructure 5fb0b3d, 350c67a).
  • Not built on aarch64 before this PR. The aarch64 build job here is the first real build. Float32 distances can differ from the scalar results in the last digits.

Crash and leak fixes (photostructure 9a01d16, fdc3282, 7c03b26, hand-adapted):

  • Keep a vec0 table whose DROP TABLE fails; don't crash when the chunk lookup cannot be prepared.
  • Free the SQL string of every aux-column UPDATE; close the blob handle when a long-text metadata write fails; free a rowids buffer on a failed blob read.

Fork-only leak fixes, found with an ASan/LeakSanitizer harness:

  • Free the _metadatachunksNN shadow table names when a table is freed.
  • Free the aux CREATE TABLE and INSERT SQL strings. The INSERT one leaked on every insert into a table with an aux column.

Tests:

  • tests/test-error-paths.py: the DROP TABLE and missing-_rowids crashes. Both crashed before.
  • tests/test-leaks.c and make test-leaks, run by the asan-unit job. Not covered: the failed blob write and failed blob read behind two of the fixes, which have no known way to trigger.
  • On an x86 VM, the full suite passes (286 passed, 204 snapshots) and the harness is clean under ASan/LSan and valgrind. On the pre-fix tree it fails.

Left out: cosine zero vectors, OPTIMIZE, partition-key UPDATE, MMR, and the other fork candidates paperless-ngx does not use.

🤖 Generated with Claude Code

stumpylog and others added 13 commits November 11, 2026 09:25
SQLite keeps a virtual table whose xDestroy fails, but vec0Destroy freed
its resources anyway, so the failed DROP TABLE's rollback, or the next
query, DROP, or close, used freed memory. Free the table only when the
destroy succeeds. The two error messages xDestroy set go too: SQLite never
reports them, and on a table that survived they could leak.

Ported from photostructure/sqlite-vec@9a01d16 (Matthew McEachen),
adapted to this tree's vec0Destroy.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_get_chunk_position() reset and cleared the bindings of its cached
statement even when preparing it had failed, and sqlite3_clear_bindings()
dereferences a NULL statement, so a point query or column read crashed when
the _rowids table was missing or an interrupt stopped the prepare.

Ported from photostructure/sqlite-vec@fdc3282 (Matthew McEachen).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0Update_UpdateAuxColumn() built its UPDATE statement with
sqlite3_mprintf() and never freed it, so every UPDATE of an auxiliary
column leaked the string.

Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which
found it through vlasky/sqlite-vec@5ae4fed. This is the auxiliary-column
part of that commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_write_metadata_value() ignored the result of the view write and, on
every early exit, skipped sqlite3_blob_close(), so a failed write left a
blob handle open and sqlite3_close() returned SQLITE_BUSY. Close the
handle on every path and report its error when nothing else failed. Keep
the write error of the integer, float and boolean kinds, which the final
close used to overwrite.

Also free the SQL string of the long-text INSERT, UPDATE and DELETE
statements, which leaked, and return SQLITE_OK rather than SQLITE_DONE
from them.

Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which
found it through vlasky/sqlite-vec@5ae4fed. This is the metadata-write
part of that commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_metadata_filter_text() allocated a buffer for the chunk's rowids and
returned without freeing it when reading the blob failed.

Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which
found it through vlasky/sqlite-vec@5ae4fed. This is the last part of that
commit that applies here: its vec0_init hunk is not needed because this
tree's parser already sets the pNew->num*Columns counters as it goes, and
its top-k ownership hunk belongs to MMR reranking, which this tree does not
have.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_init() allocates a _metadatachunksNN name per metadata column, and
only xRename ever freed them, so every connection to a table with a
metadata column leaked one string per column when the table was freed.

Found with an ASan/LeakSanitizer harness; not from another fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_init() leaked the CREATE TABLE for the auxiliary shadow table on every
create or connect, and vec0Update_Insert() leaked the INSERT for it on every
insert into a table with an auxiliary column.

Found with an ASan/LeakSanitizer harness; not from another fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A point query after the _rowids shadow table is gone must report an error,
and a DROP TABLE whose shadow table drop fails must leave the table usable
for the next query, DROP and close. Both crashed before the preceding
commits; the second runs in a child process for that reason.

Ported from photostructure/sqlite-vec@9a01d16 and @fdc3282 (Matthew McEachen).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tests/test-leaks.c runs the paths the preceding commits fix in one process:
auxiliary column insert and update, long-text metadata writes, rejected
CREATE VIRTUAL TABLEs, a failed DROP TABLE, and a point query with the
_rowids table gone. Under AddressSanitizer a leak or use after free fails
the process. `make test-leaks` builds it, and the asan-unit job in
analyze.yaml runs it with the same flags as the unit tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vabdq_s8 returns signed lanes, but vpaddlq_u8 takes unsigned ones, and GCC
rejects the implicit signed/unsigned vector conversions that Clang accepts.
Reinterpret the absolute differences as unsigned before the widening
pairwise adds, and the sums back to signed afterwards. Same results.

Needed before NEON is enabled on Linux arm64, where the release build uses
GCC. Checked with clang --target=aarch64 -flax-vector-conversions=none:
five errors in l1_int8_neon before, none after.

Ported from the GCC part of photostructure/sqlite-vec@a5aec74 (Matthew
McEachen). Its int8 L2 overflow fix and its MSVC include are not part of
this commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Makefile enabled NEON only on Darwin arm64, so Linux aarch64 builds ran
the scalar loop for every distance, including the cosine float kernel that
paperless-ngx's vec0 table uses. NEON is part of the arm64 baseline and GCC
and Clang define __ARM_NEON for aarch64 with no -m flag, so Linux needs only
the define. OMIT_SIMD=1 still builds without it.

Ask the compiler for its target rather than uname: 32-bit Raspberry Pi OS
runs a 64-bit kernel, so uname reports aarch64 while gcc targets 32-bit ARM,
which lacks the AArch64-only intrinsics the NEON kernels use.

Float32 distances can differ from the scalar loops' in the last digits,
since the NEON kernels sum in a different order.

Not built or run on aarch64 here: the Makefile selection was checked with
stand-in compilers (aarch64 gets NEON; x86, 32-bit ARM and OMIT_SIMD=1 do
not), and the NEON source was syntax-checked cross-compiled. The aarch64
CI job is the first real build.

Ported from photostructure/sqlite-vec@5fb0b3d and @350c67a (Matthew
McEachen), adapted to this Makefile.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@stumpylog
stumpylog merged commit 29ee3ec into paperless Nov 11, 2026
7 of 8 checks passed
@stumpylog
stumpylog deleted the integration/paperless-3 branch November 18, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant