Repository navigation
paperless.3: NEON on Linux arm64, leak and crash fixes - #3
Merged
Merged
Conversation
SQLite keeps a virtual table whose xDestroy fails, but vec0Destroy freed its resources anyway, so the failed DROP TABLE's rollback, or the next query, DROP, or close, used freed memory. Free the table only when the destroy succeeds. The two error messages xDestroy set go too: SQLite never reports them, and on a table that survived they could leak. Ported from photostructure/sqlite-vec@9a01d16 (Matthew McEachen), adapted to this tree's vec0Destroy. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_get_chunk_position() reset and cleared the bindings of its cached statement even when preparing it had failed, and sqlite3_clear_bindings() dereferences a NULL statement, so a point query or column read crashed when the _rowids table was missing or an interrupt stopped the prepare. Ported from photostructure/sqlite-vec@fdc3282 (Matthew McEachen). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0Update_UpdateAuxColumn() built its UPDATE statement with sqlite3_mprintf() and never freed it, so every UPDATE of an auxiliary column leaked the string. Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which found it through vlasky/sqlite-vec@5ae4fed. This is the auxiliary-column part of that commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_write_metadata_value() ignored the result of the view write and, on every early exit, skipped sqlite3_blob_close(), so a failed write left a blob handle open and sqlite3_close() returned SQLITE_BUSY. Close the handle on every path and report its error when nothing else failed. Keep the write error of the integer, float and boolean kinds, which the final close used to overwrite. Also free the SQL string of the long-text INSERT, UPDATE and DELETE statements, which leaked, and return SQLITE_OK rather than SQLITE_DONE from them. Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which found it through vlasky/sqlite-vec@5ae4fed. This is the metadata-write part of that commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_metadata_filter_text() allocated a buffer for the chunk's rowids and returned without freeing it when reading the blob failed. Ported from photostructure/sqlite-vec@7c03b26 (Matthew McEachen), which found it through vlasky/sqlite-vec@5ae4fed. This is the last part of that commit that applies here: its vec0_init hunk is not needed because this tree's parser already sets the pNew->num*Columns counters as it goes, and its top-k ownership hunk belongs to MMR reranking, which this tree does not have. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_init() allocates a _metadatachunksNN name per metadata column, and only xRename ever freed them, so every connection to a table with a metadata column leaked one string per column when the table was freed. Found with an ASan/LeakSanitizer harness; not from another fork. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vec0_init() leaked the CREATE TABLE for the auxiliary shadow table on every create or connect, and vec0Update_Insert() leaked the INSERT for it on every insert into a table with an auxiliary column. Found with an ASan/LeakSanitizer harness; not from another fork. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A point query after the _rowids shadow table is gone must report an error, and a DROP TABLE whose shadow table drop fails must leave the table usable for the next query, DROP and close. Both crashed before the preceding commits; the second runs in a child process for that reason. Ported from photostructure/sqlite-vec@9a01d16 and @fdc3282 (Matthew McEachen). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tests/test-leaks.c runs the paths the preceding commits fix in one process: auxiliary column insert and update, long-text metadata writes, rejected CREATE VIRTUAL TABLEs, a failed DROP TABLE, and a point query with the _rowids table gone. Under AddressSanitizer a leak or use after free fails the process. `make test-leaks` builds it, and the asan-unit job in analyze.yaml runs it with the same flags as the unit tests. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
vabdq_s8 returns signed lanes, but vpaddlq_u8 takes unsigned ones, and GCC rejects the implicit signed/unsigned vector conversions that Clang accepts. Reinterpret the absolute differences as unsigned before the widening pairwise adds, and the sums back to signed afterwards. Same results. Needed before NEON is enabled on Linux arm64, where the release build uses GCC. Checked with clang --target=aarch64 -flax-vector-conversions=none: five errors in l1_int8_neon before, none after. Ported from the GCC part of photostructure/sqlite-vec@a5aec74 (Matthew McEachen). Its int8 L2 overflow fix and its MSVC include are not part of this commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Makefile enabled NEON only on Darwin arm64, so Linux aarch64 builds ran the scalar loop for every distance, including the cosine float kernel that paperless-ngx's vec0 table uses. NEON is part of the arm64 baseline and GCC and Clang define __ARM_NEON for aarch64 with no -m flag, so Linux needs only the define. OMIT_SIMD=1 still builds without it. Ask the compiler for its target rather than uname: 32-bit Raspberry Pi OS runs a 64-bit kernel, so uname reports aarch64 while gcc targets 32-bit ARM, which lacks the AArch64-only intrinsics the NEON kernels use. Float32 distances can differ from the scalar loops' in the last digits, since the NEON kernels sum in a different order. Not built or run on aarch64 here: the Makefile selection was checked with stand-in compilers (aarch64 gets NEON; x86, 32-bit ARM and OMIT_SIMD=1 do not), and the NEON source was syntax-checked cross-compiled. The aarch64 CI job is the first real build. Ported from photostructure/sqlite-vec@5fb0b3d and @350c67a (Matthew McEachen), adapted to this Makefile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings NEON on Linux arm64 and a set of leak and crash fixes into the integration line for
v0.1.10-paperless.3. SeeCHANGELOG-paperless.md.NEON on Linux arm64 (ported from photostructure/sqlite-vec). The Makefile enabled NEON only on Darwin arm64, so Linux aarch64 wheels ran the scalar loop for every distance, including the cosine float kernel paperless-ngx uses.
l1_int8_neoncompile with GCC (from the GCC part of photostructurea5aec74).$(CC) -dumpmachinerather thanuname(photostructure5fb0b3d,350c67a).Crash and leak fixes (photostructure
9a01d16,fdc3282,7c03b26, hand-adapted):DROP TABLEfails; don't crash when the chunk lookup cannot be prepared.UPDATE; close the blob handle when a long-text metadata write fails; free a rowids buffer on a failed blob read.Fork-only leak fixes, found with an ASan/LeakSanitizer harness:
_metadatachunksNNshadow table names when a table is freed.CREATE TABLEandINSERTSQL strings. TheINSERTone leaked on every insert into a table with an aux column.Tests:
tests/test-error-paths.py: theDROP TABLEand missing-_rowidscrashes. Both crashed before.tests/test-leaks.candmake test-leaks, run by theasan-unitjob. Not covered: the failed blob write and failed blob read behind two of the fixes, which have no known way to trigger.Left out: cosine zero vectors,
OPTIMIZE, partition-keyUPDATE, MMR, and the other fork candidates paperless-ngx does not use.🤖 Generated with Claude Code