Skip to content

Port asg017's experimental IVF index - #12

Open
MayCXC wants to merge 2 commits into
photostructure:mainfrom
MayCXC:ivf-index-photostructure
Open

MayCXC wants to merge 2 commits into
photostructure:mainfrom
MayCXC:ivf-index-photostructure

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

Ports asg017's experimental IVF index, the third of its approximate-nearest-neighbour indexes, as asg017's main has it at 04d28bd. #8 (rescore) and #11 (DiskANN) have landed, so this is two commits on main at 5458604, and it carries the fixes their reviews made to rescore and DiskANN wherever IVF has the same code or behavior.

What it adds. A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a float vector column indexed by ivf(), which clusters its rows around nlist centroids (128 by default) with k-means and keeps them in cells of up to 64 rows, each cell holding rows of one centroid. A KNN query reads the cells of the nprobe centroids nearest its vector (10 by default) and the cells of the rows no clustering has placed yet, so until compute-centroids runs it reads every row. With quantizer=int8 or quantizer=binary, the cells and centroids hold the vectors quantized, the float vectors are kept by rowid besides, and a query ranks the k * oversample rows nearest by quantized distance by their exact distances.

create virtual table v using vec0(emb float[768] indexed by ivf(nlist=16, nprobe=4));
-- after the inserts
insert into v(v) values ('compute-centroids');
select rowid, distance from v where emb match ? and k = 10;
insert into v(v) values ('nprobe=8');  -- this connection's queries, until SQLite reconnects the table

The commands compute-centroids, compute-centroids:{"nlist": N, "max_iterations": N, "seed": N}, set-centroid:N, assign-vectors and clear-centroids build the index, and nprobe=N sets the nprobe of a connection's queries until SQLite reconnects the table, as after VACUUM, ALTER TABLE ... RENAME, or another connection's schema change. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream names them. The prebuilt binaries are built without it.

What it ports. asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main has them at 04d28bd (merged in e2c38f3), with the refusal of a binary quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6), the rename of its shadow tables (8b81f40), the finalizing of every cached statement on commit (fe94171), and the fixes of the unmerged asg017#313: an insert takes the first free slot of a cell, where upstream takes the slot after its count and overwrites a live row once a delete leaves a hole; assign-vectors reads the centroids at the right stride and by their ids, which set-centroid can leave apart; clear-centroids and set-centroid keep a quantized column's cells and centroids quantized; and the int8 quantizer ranks by the column's metric, where upstream ranks by L2.

Where it differs from upstream.

  • ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter ranks the rows it names by their exact distances, distance constraints filter the rows the query read, and MMR reranks them. Upstream reads only the query vector and k, so it returns rows outside a rowid filter or a distance constraint.
  • A quantized column's point query and full scan read its float vectors from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point query reads a float vector's length past them.
  • Whether a column has centroids is read from _info at each use. Upstream caches it per connection, so a connection that queried before another one ran compute-centroids reads only the cells of no centroid, which are gone, and finds no rows. Two connections to one file, 100 rows of float[8] in ivf(nlist=4, nprobe=4): a KNN on the first returns rows 7, 97 and 40; the second runs compute-centroids; the same KNN on the first then returns nothing on asg017's main, and 7, 97 and 40 here.
  • The cells are found by centroid through the index of a UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY, which ALTER TABLE RENAME renames along with its table. Upstream creates an index named after the table, which keeps its name through a rename, so a table created later under the old name fails to create its own; nothing inside xRename can drop an index, since OP_Destroy refuses while the ALTER TABLE statement reads. Every statement on the cells names its columns and addresses a cell by rowid, so either implementation opens the other's tables.
  • Only float columns can be indexed by ivf(). Upstream accepts int8 and bit columns and sizes their cells for float vectors, so it reads past an int8 vector and ranks a row's own vector at a distance such as 6.6e10.
  • Errors reach the statement, so an interrupt stays an interrupt. Upstream ignores the results of its writes to the cells and of its blob reads and writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate, which SQLite refuses while the statement writes. An interrupted command fails with interrupted and the statement's rollback undoes it; upstream reports SQL logic error, and the connection's KNN queries then find no rows. A statement that fails, as on a full database or a dropped shadow table, names the cell, row, centroid or vector it was reading or writing with SQLite's reason, as main's DiskANN statements do, and a cell, centroid or vector blob whose size the column's options do not give fails with a message naming it.
  • A cached statement is reset as soon as it has run, so VACUUM works after a KNN query.
  • The commands apply to every IVF column of the table; upstream applies them to the first. compute-centroids:{...} is read with SQLite's json_each(), and its keys nlist, max_iterations and seed take integers in range; upstream finds them with strstr() and reads them with atoi(). The commands take digits only, so nprobe=5x fails with nprobe must be a whole number from 1 to 2147483647. A CREATE spells ivf and its options in full, as main's CREATE does for DiskANN's; a table upstream created with a prefix, as in quant=int8, still opens.
  • vec0's xShadowName reports the four IVF tables, as it does the rescore and DiskANN tables; upstream's reports none of them.
  • k-means indexes its arrays with size_t; upstream multiplies int indexes, which overflow past 2^31 floats.
  • optimize, which upstream lacks, leaves an IVF column's rows, which are keyed by rowid, as they are.

The IVF code shares the DiskANN code's helpers, which take names for both: diskann_stmt() becomes vec0_cached_stmt(), and vec0_parse_diskann_int() becomes vec0_parse_option_int(). The commit adds an ### Added entry under [Unreleased] in CHANGELOG.md and names the IVF tables in its xShadowName entry, and ARCHITECTURE.md and the README describe the index.

The fixes main's reviews made, where IVF has the same code or behavior.

  • 4667770, 809fc3d: on connect, indexed by ivf(...) and its option names and quantizer values accept a prefix, as upstream's parser does, so a table upstream created as indexed by iv(nli=4, quant=i) opens and answers.
  • b842187: DROP TABLE drops the IVF tables with IF EXISTS, as upstream does, so a table that lost one still drops.
  • 435381f: a scan or KNN query whose row the same connection deleted before reading its id or vector fails with SQLITE_ERROR naming the row, through the readers main fixed.
  • 64c3e67: nprobe=N states its rule, and the docs say how long it lasts.
  • eeef064: the IVF commands are unknown on a table with no IVF column, as upstream's are.
  • e50a923: each IVF statement that fails names what it read or wrote with SQLite's reason, and a point query or full scan keeps the message naming a corrupt row.
  • 52d0b18, 8a66fa3: a test that the IVF tables are shadow tables, and test-ivf-rename.py cites asg017/sqlite-vec@8b81f40, whose IVF tests test-rename.py now points to.
  • ab3bfc4 has no IVF counterpart, since every IVF table, upstream's and this one's, declares its id INTEGER PRIMARY KEY or uses the implicit rowid; nor has 1e0344d, since a later IVF option replaces an earlier one, as upstream's does, and no IVF doc says otherwise; nor have the DiskANN graph fixes, whose code IVF does not use.

Tests.

  • Fix ivf index bugs asg017/sqlite-vec#313's test-ivf.py, test-ivf-mutations.py, test-ivf-quantization.py and test-ivf-rename.py; upstream's eight IVF fuzz targets with their dictionary, built with the IVF index compiled in (upstream's fuzz Makefile builds them without it, so they create no IVF table), and with ivf-shadow-corrupt.c reading its n_vectors from the input unsigned, since upstream's abs() of an int is undefined for INT_MIN, which UBSan reports; and the IVF cases of Fix ivf index bugs asg017/sqlite-vec#313's test-unit.c, which call the functions through tests/sqlite-vec-internal.h. Fix ivf index bugs asg017/sqlite-vec#313's test-ann-embeddings.py needs an embedding server on localhost and is not ported.
  • New: a query reading every cell answers as a flat column does, constraint for constraint; the float vectors of a quantized column; a second connection after compute-centroids; a rename followed by a table under the old name; the refused columns and options; an interrupted command; the commands on every IVF column; the command errors; the compute-centroids options and seed; corrupt cells and vectors; INSERT OR REPLACE; VACUUM after a KNN query; optimize beside a flat column; and k-means in the unit tests. For main's review fixes: a table created with abbreviated options; DROP TABLE after losing each IVF table; rows deleted mid-read; nprobe=N through VACUUM; the commands on tables without IVF; each IVF table dropped or refusing a write, with the message each statement gives; and the IVF shadow tables. The tests of the messages fail against a build without them.
  • conftest.py skips the test-ivf files when vec_debug() names no IVF build flag, as upstream's does; make test-unit passes CPPFLAGS to the compiler, and a new CI job, build-linux-x86_64-ivf, compiles the IVF index in and runs every suite.
  • The last commit adds three IVF tables to test_interrupt_is_reported_as_interrupt: KNN queries over the centroids, on a rowid filter and on a quantized column, full scans, a point query, inserts into cells of a centroid, of the quantized column and of no centroid, an auxiliary column update, deletes, optimize, compute-centroids with and without options, clear-centroids, set-centroid and assign-vectors, each interrupted at every progress handler call. A build whose writes to the cells return SQLITE_ERROR in place of vec0_error_rc()'s code fails 12 of the 36 cases.

With CPPFLAGS=-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, as build-linux-x86_64-ivf runs them, make test-loadable passes 1004 tests with 3 skipped and make test-unit passes, and the same suite passes against an AddressSanitizer and UndefinedBehaviorSanitizer build of the extension with no report. Without the flag, make test-loadable passes 771 with 236 skipped and make test-unit passes. Both ran on SQLite 3.51.1, as scripts/vendor.sh vendors it. GCC 14.2 and Clang 19.1 with -Wall -Wextra print nothing for any combination of SQLITE_VEC_ENABLE_RESCORE, SQLITE_VEC_ENABLE_DISKANN and the IVF flag, nor with -DSQLITE_CORE, and make lint passes. ivf_shadow_corrupt and ivf_operations each ran 60 seconds without a sanitizer report; the other six IVF fuzz targets last ran, 90 seconds each without a finding, before the rebase onto f6a3244.

build-linux-x86_64-ivf follows build-linux-aarch64-extension in .github/workflows/test.yaml.

@mceachen

Copy link
Copy Markdown
Member

We'll review the index ports one at a time once #4 and #7 are in, starting with rescore (#8).

A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a
float vector column `indexed by ivf()`, which clusters its rows around nlist
centroids (128 by default) with k-means and keeps them in cells of up to 64
rows, each cell holding rows of one centroid. A KNN query reads the cells of
the nprobe centroids nearest its vector (10 by default) and the cells of the
rows no clustering has placed yet, so until compute-centroids runs it reads
every row. With quantizer=int8 or quantizer=binary, the cells and centroids
hold the vectors quantized, the float32 vectors are kept by rowid besides, and
a query ranks the k * oversample rows nearest by quantized distance by their
exact distances. The commands compute-centroids, compute-centroids:{...},
set-centroid:N, assign-vectors and clear-centroids build the index, and
nprobe=N sets the nprobe of a connection's queries until SQLite reconnects
the table. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN
and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream
names them. The prebuilt binaries are built without it.

Ported from asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main
has them at 04d28bd (merged in e2c38f3), with the refusal of a binary
quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector
UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6),
the rename of its shadow tables (8b81f40), the finalizing of every cached
statement on commit (fe94171), and the fixes of the unmerged asg017#313: an
insert takes the first free slot of a cell, where upstream takes the slot
after its count and overwrites a live row once a delete leaves a hole;
assign-vectors reads the centroids at the right stride and by their ids, which
set-centroid can leave apart; clear-centroids and set-centroid keep a
quantized column's cells and centroids quantized; and the int8 quantizer ranks
by the column's metric, where upstream ranks by L2.

Where this port differs from upstream:

- ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter
  ranks the rows it names by their exact distances, distance constraints
  filter the rows the query read, and MMR reranks them. Upstream reads only
  the query vector and k, so it returns rows outside a rowid filter or a
  distance constraint.
- A quantized column's point query and full scan read its float32 vectors
  from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point
  query reads a float32 vector's length past them.
- Whether a column has centroids is read from _info at each use. Upstream
  caches it per connection, so a connection that queried before another one
  ran compute-centroids reads only the cells of no centroid, which are gone,
  and finds no rows.
- The cells are found by centroid through the index of a
  UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY,
  which ALTER TABLE RENAME renames along with its table. Upstream creates an
  index named after the table, which keeps its name through a rename, so a
  table created later under the old name fails to create its own; nothing
  inside xRename can drop an index, since OP_Destroy refuses while the ALTER
  TABLE statement reads. Every statement on the cells names its columns and
  addresses a cell by rowid, so either implementation opens the other's
  tables.
- Only float columns can be indexed by ivf(). Upstream accepts int8 and bit
  columns and sizes their cells for float32 vectors, so it reads past an
  int8 vector and ranks a row's own vector at a distance such as 6.6e10.
- Errors reach the statement, so an interrupt stays an interrupt. Upstream
  ignores the results of its writes to the cells and of its blob reads and
  writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate,
  which SQLite refuses while the statement writes. An interrupted command
  fails with "interrupted" and the statement's rollback undoes it; upstream
  reports "SQL logic error", and the connection's KNN queries then find no
  rows. A statement that fails, as on a full database or a dropped shadow
  table, names the cell, row, centroid or vector it was reading or writing
  with SQLite's reason, as the DiskANN statements do, and a cell, centroid or
  vector blob whose size the column's options do not give fails with a
  message naming it.
- A cached statement is reset as soon as it has run, so VACUUM works after a
  KNN query.
- The commands apply to every IVF column of the table; upstream applies them
  to the first. compute-centroids:{...} is read with SQLite's json_each(),
  and its keys nlist, max_iterations and seed take integers in range; upstream
  finds them with strstr() and reads them with atoi(). The commands take
  digits only, so nprobe=5x fails with "nprobe must be a whole number from 1
  to 2147483647". A CREATE spells ivf and its options in full, as it does
  DiskANN's; a table upstream created with a prefix, as in `quant=int8`,
  still opens.
- vec0's xShadowName reports the four IVF tables, as it does the rescore and
  DiskANN tables; upstream's reports none of them.
- k-means indexes its arrays with size_t; upstream multiplies int indexes,
  which overflow past 2^31 floats.
- optimize, which upstream lacks, leaves an IVF column's rows, which are
  keyed by rowid, as they are.

The IVF code shares the DiskANN code's helpers, which take names for both:
diskann_stmt() becomes vec0_cached_stmt(), and vec0_parse_diskann_int()
vec0_parse_option_int().

The tests are asg017#313's test-ivf.py, test-ivf-mutations.py,
test-ivf-quantization.py and test-ivf-rename.py, upstream's IVF fuzz targets
with their dictionary, built with the IVF index compiled in (upstream's fuzz
Makefile builds them without it, so they create no IVF table) and with
ivf-shadow-corrupt.c reading its n_vectors from the input unsigned (upstream
takes abs() of an int, undefined for INT_MIN, which UBSan reports), and the IVF
cases of asg017#313's test-unit.c, which call the functions through
tests/sqlite-vec-internal.h. asg017#313's test-ann-embeddings.py needs an embedding
server on localhost and is not ported. New tests check that a query reading
every cell answers as a flat column does, constraint for constraint, the
float32 vectors of a quantized column, a second connection after
compute-centroids, a rename followed by a table under the old name, the
refused columns and options, a table created with abbreviated options, an
interrupted command, the commands on every IVF column and on tables without
one, the command errors, nprobe=N through VACUUM, the compute-centroids
options and seed, corrupt cells and vectors, each IVF table dropped or
refusing a write, DROP TABLE after losing an IVF table, rows deleted while a
query reads them, the IVF shadow tables, INSERT OR REPLACE, VACUUM after a
KNN query, optimize beside a flat column, and k-means in the unit tests.
conftest.py skips the test-ivf files when vec_debug() names no ivf build
flag, as upstream's does; make test-unit passes CPPFLAGS to the compiler, and
a CI job builds the IVF index in and runs every suite.
test_interrupt_is_reported_as_interrupt runs a statement on a copy of a
database, interrupted at each progress handler call, and expects
SQLITE_INTERRUPT. When the build compiles the IVF index in, three IVF tables
join its database: vi and viq with centroids and a free slot, viq quantized,
and viu with centroids set by hand and its rows in cells of no centroid. The
cases are KNN queries over the centroids, on a rowid filter and on the
quantized column, full scans, a point query, inserts into cells of a
centroid, of the quantized column and of no centroid, an auxiliary column
update, deletes, optimize, compute-centroids with and without options,
clear-centroids, set-centroid and assign-vectors, so the IVF code's internal
statements are interrupted too; a build without the IVF index skips them.
A build whose writes to the cells return SQLITE_ERROR in place of
vec0_error_rc()'s code fails 12 of the 36 cases.
@MayCXC
MayCXC force-pushed the ivf-index-photostructure branch from 2f2810d to c09c1b1 Compare November 18, 2026 08:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants