Repository navigation
Conversation
This was referenced Oct 23, 2026
MayCXC
force-pushed
the
ivf-index-photostructure
branch
from
October 23, 2026 17:03
cc009fd to
debd171
Compare
This was referenced Oct 23, 2026
Member
MayCXC
force-pushed
the
ivf-index-photostructure
branch
from
October 24, 2026 01:58
debd171 to
1ea66c8
Compare
MayCXC
force-pushed
the
ivf-index-photostructure
branch
from
November 18, 2026 08:49
1ea66c8 to
2f2810d
Compare
A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a
float vector column `indexed by ivf()`, which clusters its rows around nlist
centroids (128 by default) with k-means and keeps them in cells of up to 64
rows, each cell holding rows of one centroid. A KNN query reads the cells of
the nprobe centroids nearest its vector (10 by default) and the cells of the
rows no clustering has placed yet, so until compute-centroids runs it reads
every row. With quantizer=int8 or quantizer=binary, the cells and centroids
hold the vectors quantized, the float32 vectors are kept by rowid besides, and
a query ranks the k * oversample rows nearest by quantized distance by their
exact distances. The commands compute-centroids, compute-centroids:{...},
set-centroid:N, assign-vectors and clear-centroids build the index, and
nprobe=N sets the nprobe of a connection's queries until SQLite reconnects
the table. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN
and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream
names them. The prebuilt binaries are built without it.
Ported from asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main
has them at 04d28bd (merged in e2c38f3), with the refusal of a binary
quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector
UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6),
the rename of its shadow tables (8b81f40), the finalizing of every cached
statement on commit (fe94171), and the fixes of the unmerged asg017#313: an
insert takes the first free slot of a cell, where upstream takes the slot
after its count and overwrites a live row once a delete leaves a hole;
assign-vectors reads the centroids at the right stride and by their ids, which
set-centroid can leave apart; clear-centroids and set-centroid keep a
quantized column's cells and centroids quantized; and the int8 quantizer ranks
by the column's metric, where upstream ranks by L2.
Where this port differs from upstream:
- ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter
ranks the rows it names by their exact distances, distance constraints
filter the rows the query read, and MMR reranks them. Upstream reads only
the query vector and k, so it returns rows outside a rowid filter or a
distance constraint.
- A quantized column's point query and full scan read its float32 vectors
from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point
query reads a float32 vector's length past them.
- Whether a column has centroids is read from _info at each use. Upstream
caches it per connection, so a connection that queried before another one
ran compute-centroids reads only the cells of no centroid, which are gone,
and finds no rows.
- The cells are found by centroid through the index of a
UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY,
which ALTER TABLE RENAME renames along with its table. Upstream creates an
index named after the table, which keeps its name through a rename, so a
table created later under the old name fails to create its own; nothing
inside xRename can drop an index, since OP_Destroy refuses while the ALTER
TABLE statement reads. Every statement on the cells names its columns and
addresses a cell by rowid, so either implementation opens the other's
tables.
- Only float columns can be indexed by ivf(). Upstream accepts int8 and bit
columns and sizes their cells for float32 vectors, so it reads past an
int8 vector and ranks a row's own vector at a distance such as 6.6e10.
- Errors reach the statement, so an interrupt stays an interrupt. Upstream
ignores the results of its writes to the cells and of its blob reads and
writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate,
which SQLite refuses while the statement writes. An interrupted command
fails with "interrupted" and the statement's rollback undoes it; upstream
reports "SQL logic error", and the connection's KNN queries then find no
rows. A statement that fails, as on a full database or a dropped shadow
table, names the cell, row, centroid or vector it was reading or writing
with SQLite's reason, as the DiskANN statements do, and a cell, centroid or
vector blob whose size the column's options do not give fails with a
message naming it.
- A cached statement is reset as soon as it has run, so VACUUM works after a
KNN query.
- The commands apply to every IVF column of the table; upstream applies them
to the first. compute-centroids:{...} is read with SQLite's json_each(),
and its keys nlist, max_iterations and seed take integers in range; upstream
finds them with strstr() and reads them with atoi(). The commands take
digits only, so nprobe=5x fails with "nprobe must be a whole number from 1
to 2147483647". A CREATE spells ivf and its options in full, as it does
DiskANN's; a table upstream created with a prefix, as in `quant=int8`,
still opens.
- vec0's xShadowName reports the four IVF tables, as it does the rescore and
DiskANN tables; upstream's reports none of them.
- k-means indexes its arrays with size_t; upstream multiplies int indexes,
which overflow past 2^31 floats.
- optimize, which upstream lacks, leaves an IVF column's rows, which are
keyed by rowid, as they are.
The IVF code shares the DiskANN code's helpers, which take names for both:
diskann_stmt() becomes vec0_cached_stmt(), and vec0_parse_diskann_int()
vec0_parse_option_int().
The tests are asg017#313's test-ivf.py, test-ivf-mutations.py,
test-ivf-quantization.py and test-ivf-rename.py, upstream's IVF fuzz targets
with their dictionary, built with the IVF index compiled in (upstream's fuzz
Makefile builds them without it, so they create no IVF table) and with
ivf-shadow-corrupt.c reading its n_vectors from the input unsigned (upstream
takes abs() of an int, undefined for INT_MIN, which UBSan reports), and the IVF
cases of asg017#313's test-unit.c, which call the functions through
tests/sqlite-vec-internal.h. asg017#313's test-ann-embeddings.py needs an embedding
server on localhost and is not ported. New tests check that a query reading
every cell answers as a flat column does, constraint for constraint, the
float32 vectors of a quantized column, a second connection after
compute-centroids, a rename followed by a table under the old name, the
refused columns and options, a table created with abbreviated options, an
interrupted command, the commands on every IVF column and on tables without
one, the command errors, nprobe=N through VACUUM, the compute-centroids
options and seed, corrupt cells and vectors, each IVF table dropped or
refusing a write, DROP TABLE after losing an IVF table, rows deleted while a
query reads them, the IVF shadow tables, INSERT OR REPLACE, VACUUM after a
KNN query, optimize beside a flat column, and k-means in the unit tests.
conftest.py skips the test-ivf files when vec_debug() names no ivf build
flag, as upstream's does; make test-unit passes CPPFLAGS to the compiler, and
a CI job builds the IVF index in and runs every suite.
test_interrupt_is_reported_as_interrupt runs a statement on a copy of a database, interrupted at each progress handler call, and expects SQLITE_INTERRUPT. When the build compiles the IVF index in, three IVF tables join its database: vi and viq with centroids and a free slot, viq quantized, and viu with centroids set by hand and its rows in cells of no centroid. The cases are KNN queries over the centroids, on a rowid filter and on the quantized column, full scans, a point query, inserts into cells of a centroid, of the quantized column and of no centroid, an auxiliary column update, deletes, optimize, compute-centroids with and without options, clear-centroids, set-centroid and assign-vectors, so the IVF code's internal statements are interrupted too; a build without the IVF index skips them. A build whose writes to the cells return SQLITE_ERROR in place of vec0_error_rc()'s code fails 12 of the 36 cases.
MayCXC
force-pushed
the
ivf-index-photostructure
branch
from
November 18, 2026 08:57
2f2810d to
c09c1b1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports asg017's experimental IVF index, the third of its approximate-nearest-neighbour indexes, as asg017's main has it at 04d28bd. #8 (rescore) and #11 (DiskANN) have landed, so this is two commits on main at 5458604, and it carries the fixes their reviews made to rescore and DiskANN wherever IVF has the same code or behavior.
What it adds. A build compiled with
-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1can declare afloatvector columnindexed by ivf(), which clusters its rows aroundnlistcentroids (128 by default) with k-means and keeps them in cells of up to 64 rows, each cell holding rows of one centroid. A KNN query reads the cells of thenprobecentroids nearest its vector (10 by default) and the cells of the rows no clustering has placed yet, so untilcompute-centroidsruns it reads every row. Withquantizer=int8orquantizer=binary, the cells and centroids hold the vectors quantized, thefloatvectors are kept by rowid besides, and a query ranks thek * oversamplerows nearest by quantized distance by their exact distances.The commands
compute-centroids,compute-centroids:{"nlist": N, "max_iterations": N, "seed": N},set-centroid:N,assign-vectorsandclear-centroidsbuild the index, andnprobe=Nsets the nprobe of a connection's queries until SQLite reconnects the table, as afterVACUUM,ALTER TABLE ... RENAME, or another connection's schema change. Its tables are_ivf_centroidsNN,_ivf_cellsNN,_ivf_rowid_mapNNand_ivf_vectorsNN, with theivf_trained_Nrow of_info, named as upstream names them. The prebuilt binaries are built without it.What it ports. asg017's
sqlite-vec-ivf.candsqlite-vec-ivf-kmeans.cas its main has them at 04d28bd (merged in e2c38f3), with the refusal of a binary quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vectorUPDATE(9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6), the rename of its shadow tables (8b81f40), the finalizing of every cached statement on commit (fe94171), and the fixes of the unmerged asg017#313: an insert takes the first free slot of a cell, where upstream takes the slot after its count and overwrites a live row once a delete leaves a hole;assign-vectorsreads the centroids at the right stride and by their ids, whichset-centroidcan leave apart;clear-centroidsandset-centroidkeep a quantized column's cells and centroids quantized; and theint8quantizer ranks by the column's metric, where upstream ranks by L2.Where it differs from upstream.
ivf_knn()takesrescore_knn()'s arguments and results: a rowid filter ranks the rows it names by their exact distances, distance constraints filter the rows the query read, and MMR reranks them. Upstream reads only the query vector andk, so it returns rows outside a rowid filter or a distance constraint.floatvectors from_ivf_vectorsNN. Upstream returns the quantized bytes, and its point query reads afloatvector's length past them._infoat each use. Upstream caches it per connection, so a connection that queried before another one rancompute-centroidsreads only the cells of no centroid, which are gone, and finds no rows. Two connections to one file, 100 rows offloat[8]inivf(nlist=4, nprobe=4): a KNN on the first returns rows 7, 97 and 40; the second runscompute-centroids; the same KNN on the first then returns nothing on asg017's main, and 7, 97 and 40 here.UNIQUE (centroid_id, cell_id)constraint on acell_id INTEGER PRIMARY KEY, whichALTER TABLE RENAMErenames along with its table. Upstream creates an index named after the table, which keeps its name through a rename, so a table created later under the old name fails to create its own; nothing insidexRenamecan drop an index, sinceOP_Destroyrefuses while theALTER TABLEstatement reads. Every statement on the cells names its columns and addresses a cell by rowid, so either implementation opens the other's tables.floatcolumns can be indexed byivf(). Upstream acceptsint8andbitcolumns and sizes their cells forfloatvectors, so it reads past anint8vector and ranks a row's own vector at a distance such as 6.6e10.compute-centroidsin a SAVEPOINT it opens insidexUpdate, which SQLite refuses while the statement writes. An interrupted command fails withinterruptedand the statement's rollback undoes it; upstream reportsSQL logic error, and the connection's KNN queries then find no rows. A statement that fails, as on a full database or a dropped shadow table, names the cell, row, centroid or vector it was reading or writing with SQLite's reason, as main's DiskANN statements do, and a cell, centroid or vector blob whose size the column's options do not give fails with a message naming it.VACUUMworks after a KNN query.compute-centroids:{...}is read with SQLite'sjson_each(), and its keysnlist,max_iterationsandseedtake integers in range; upstream finds them withstrstr()and reads them withatoi(). The commands take digits only, sonprobe=5xfails withnprobe must be a whole number from 1 to 2147483647. A CREATE spellsivfand its options in full, as main's CREATE does for DiskANN's; a table upstream created with a prefix, as inquant=int8, still opens.vec0'sxShadowNamereports the four IVF tables, as it does the rescore and DiskANN tables; upstream's reports none of them.size_t; upstream multipliesintindexes, which overflow past 2^31 floats.optimize, which upstream lacks, leaves an IVF column's rows, which are keyed by rowid, as they are.The IVF code shares the DiskANN code's helpers, which take names for both:
diskann_stmt()becomesvec0_cached_stmt(), andvec0_parse_diskann_int()becomesvec0_parse_option_int(). The commit adds an### Addedentry under[Unreleased]in CHANGELOG.md and names the IVF tables in itsxShadowNameentry, and ARCHITECTURE.md and the README describe the index.The fixes main's reviews made, where IVF has the same code or behavior.
indexed by ivf(...)and its option names and quantizer values accept a prefix, as upstream's parser does, so a table upstream created asindexed by iv(nli=4, quant=i)opens and answers.DROP TABLEdrops the IVF tables withIF EXISTS, as upstream does, so a table that lost one still drops.SQLITE_ERRORnaming the row, through the readers main fixed.nprobe=Nstates its rule, and the docs say how long it lasts.INTEGER PRIMARY KEYor uses the implicit rowid; nor has 1e0344d, since a later IVF option replaces an earlier one, as upstream's does, and no IVF doc says otherwise; nor have the DiskANN graph fixes, whose code IVF does not use.Tests.
n_vectorsfrom the input unsigned, since upstream'sabs()of anintis undefined forINT_MIN, which UBSan reports; and the IVF cases of Fix ivf index bugs asg017/sqlite-vec#313's test-unit.c, which call the functions throughtests/sqlite-vec-internal.h. Fix ivf index bugs asg017/sqlite-vec#313's test-ann-embeddings.py needs an embedding server on localhost and is not ported.floatvectors of a quantized column; a second connection aftercompute-centroids; a rename followed by a table under the old name; the refused columns and options; an interrupted command; the commands on every IVF column; the command errors; thecompute-centroidsoptions and seed; corrupt cells and vectors;INSERT OR REPLACE;VACUUMafter a KNN query;optimizebeside a flat column; and k-means in the unit tests. For main's review fixes: a table created with abbreviated options;DROP TABLEafter losing each IVF table; rows deleted mid-read;nprobe=NthroughVACUUM; the commands on tables without IVF; each IVF table dropped or refusing a write, with the message each statement gives; and the IVF shadow tables. The tests of the messages fail against a build without them.vec_debug()names no IVF build flag, as upstream's does;make test-unitpassesCPPFLAGSto the compiler, and a new CI job,build-linux-x86_64-ivf, compiles the IVF index in and runs every suite.test_interrupt_is_reported_as_interrupt: KNN queries over the centroids, on a rowid filter and on a quantized column, full scans, a point query, inserts into cells of a centroid, of the quantized column and of no centroid, an auxiliary column update, deletes,optimize,compute-centroidswith and without options,clear-centroids,set-centroidandassign-vectors, each interrupted at every progress handler call. A build whose writes to the cells returnSQLITE_ERRORin place ofvec0_error_rc()'s code fails 12 of the 36 cases.With
CPPFLAGS=-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, asbuild-linux-x86_64-ivfruns them,make test-loadablepasses 1004 tests with 3 skipped andmake test-unitpasses, and the same suite passes against an AddressSanitizer and UndefinedBehaviorSanitizer build of the extension with no report. Without the flag,make test-loadablepasses 771 with 236 skipped andmake test-unitpasses. Both ran on SQLite 3.51.1, asscripts/vendor.shvendors it. GCC 14.2 and Clang 19.1 with-Wall -Wextraprint nothing for any combination ofSQLITE_VEC_ENABLE_RESCORE,SQLITE_VEC_ENABLE_DISKANNand the IVF flag, nor with-DSQLITE_CORE, andmake lintpasses.ivf_shadow_corruptandivf_operationseach ran 60 seconds without a sanitizer report; the other six IVF fuzz targets last ran, 90 seconds each without a finding, before the rebase onto f6a3244.build-linux-x86_64-ivffollowsbuild-linux-aarch64-extensionin.github/workflows/test.yaml.