Skip to content

Cheaper float32 cosine KNN on x86 (v0.1.10-paperless.4) - #4

Merged
stumpylog merged 3 commits into
paperlessfrom
perf/avx-caps-cosine-qnorm
Nov 12, 2026
Merged

stumpylog merged 3 commits into
paperlessfrom
perf/avx-caps-cosine-qnorm

Conversation

@stumpylog

Copy link
Copy Markdown
Owner

Fork-only change for the unreleased v0.1.10-paperless.4. No upstream PR.

Changes

  • 0a241f9 Centralize AVX CPU capability detection into a vec_avx_caps struct (avx, avx2, fma) filled once in sqlite3_vec_init. The l2, cosine and hamming dispatch sites read it instead of each caching __builtin_cpu_supports("avx2") in a function-local static. Idea and struct from feature/simd-additional (277aed8); the rest of that branch is not carried.
  • 43e3c80 Compute the cosine query norm once per KNN query, not once per row, with an AVX2+FMA kernel that accumulates only dot and |a|^2 (gated on vec_avx_caps.avx2 and .fma). Rows that cannot beat the k-th distance are dropped before min_idx().
  • acf062d Changelog entry.

Measurements (i7-9750H VM, -O3, d=768 float32 cosine)

Tested

  • make test-unit passes; tests/test-loadable.py gives 46 passed, 4 skipped (on the VM, x86_64).
  • Not yet run: ASan/LeakSanitizer job, aarch64 build. That is what this PR is for.

Not carried

The rest of feature/simd-additional. paperless-ngx only uses float32 cosine, and that branch's cosine_float_avx2 (2c64ea3) measured about 14% slower than the asg017#364 kernel with the data in L1.

🤖 Generated with Claude Code

stumpylog and others added 3 commits November 12, 2026 11:25
The three AVX dispatch sites each cached __builtin_cpu_supports("avx2") in a
function-local static. Fill one vec_avx_caps struct (avx, avx2, fma) in
sqlite3_vec_init and read it at the sites, so a new kernel can gate on fma
without another lazy cache.

Idea and struct from feature/simd-additional (277aed8).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k-th

A float32 cosine KNN recomputed the query's squared magnitude for every
stored row. Compute it once per query and use an AVX2+FMA kernel that
accumulates only dot and |a|^2, gated on vec_avx_caps.avx2 and .fma.

Once the global top k is full, drop rows that are not strictly nearer than
the k-th before min_idx(): merge_sorted_lists keeps the earlier row on a
tie and ranks NaN last, so such a row can never enter the result.

A full scan of 60k 768-dim vectors runs 181.8M instead of 109.9M
instructions. Wall time moves little there (about 3%): the scan is bound
by memory bandwidth. float32 distances can differ in the last digits.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@stumpylog
stumpylog merged commit 155c9f7 into paperless Nov 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant