Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,25 @@ jobs:
path: dist/*
# after the upload, so dist/test-unit stays out of the artifact
- run: make test-unit
# The IVF index is compiled only on request, so the other builds skip its
# tests; this one compiles it in and runs every suite.
build-linux-x86_64-ivf:
needs: memory-gate
runs-on: ubuntu-latest
env:
CPPFLAGS: -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
persist-credentials: false
- uses: astral-sh/setup-uv@681c641aba71e4a1c380be3ab5e12ad51f415867 # v7.1.6
with:
enable-cache: true
- run: ./scripts/vendor.sh
- run: make loadable static
- run: uv sync --locked --managed-python --directory tests
- run: make test-loadable python=./tests/.venv/bin/python
- run: make test-unit
build-macos-x86_64-extension:
runs-on: macos-15-intel
steps:
Expand Down
54 changes: 52 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,49 @@ A table whose every vector column is indexed by `diskann()` keeps no chunks:
its `xyz_chunks` stays empty, and its `xyz_rowids` rows have no `chunk_id` or
`chunk_offset`.

#### `xyz_ivf_centroidsNN`

The centroids of a column indexed by `ivf()`, held as its cells hold vectors:
`float32`, or quantized by the column's `quantizer`. The `ivf_trained_N` row of
`xyz_info` is 1 while the column has centroids.

- `centroid_id INTEGER`
- `centroid BLOB`

#### `xyz_ivf_cellsNN`

The rows of a column indexed by `ivf()`, in cells of up to 64. A cell holds
rows of one centroid, or, with `centroid_id` -1, rows no clustering has placed.
`validity` holds a bit per slot, set when the slot holds a row, `rowids` the
row's rowid as a 64-bit integer in the machine's byte order, and `vectors` its
vector as the centroids hold them. `n_vectors` counts the set bits. A query
finds the cells of a centroid through the index of the
`UNIQUE (centroid_id, cell_id)` constraint.

- `cell_id INTEGER`
- `centroid_id INTEGER`
- `n_vectors INTEGER`
- `validity BLOB`
- `rowids BLOB`
- `vectors BLOB`

#### `xyz_ivf_rowid_mapNN`

The cell and slot of each row of a column indexed by `ivf()`.

- `rowid INTEGER`
- `cell_id INTEGER`
- `slot INTEGER`

#### `xyz_ivf_vectorsNN`

The `float32` vectors of a column indexed by `ivf()` with a `quantizer`, one
row per row of the table. Without a quantizer, the cells hold the vectors and
the table does not exist.

- `rowid INTEGER`
- `vector BLOB`

#### `xyz_auxiliary`

- `rowid INTEGER`
Expand All @@ -106,11 +149,14 @@ The supported way to tell which tables belong to a `vec0` table is by name. A

- `xyz_info`, `xyz_chunks`, `xyz_rowids`: always
- `xyz_vector_chunksNN`: one per vector column, except those indexed by
`rescore()` or `diskann()`
`rescore()`, `diskann()` or `ivf()`
- `xyz_rescore_chunksNN`, `xyz_rescore_vectorsNN`: one each per vector column
indexed by `rescore()`
- `xyz_vectorsNN`, `xyz_diskann_nodesNN`, `xyz_diskann_bufferNN`: one each per
vector column indexed by `diskann()`
- `xyz_ivf_centroidsNN`, `xyz_ivf_cellsNN`, `xyz_ivf_rowid_mapNN`: one each per
vector column indexed by `ivf()`, and `xyz_ivf_vectorsNN` for each of those
with a `quantizer`
- `xyz_auxiliary`: only if there are auxiliary columns
- `xyz_metadatachunksNN`: one per metadata column
- `xyz_metadatatextNN`: one per text metadata column
Expand All @@ -132,6 +178,10 @@ join sqlite_master as s
or substr(s.name, length(v.name) + 2) glob 'vectors[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'diskann_nodes[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'diskann_buffer[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'ivf_centroids[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'ivf_cells[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'ivf_rowid_map[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'ivf_vectors[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'metadatachunks[0-9][0-9]'
or substr(s.name, length(v.name) + 2) glob 'metadatatext[0-9][0-9]')
where v.type = 'table' and v.rootpage = 0
Expand All @@ -149,7 +199,7 @@ relied on:
- Before SQLite 3.37 there is no `PRAGMA table_list` and no prefix marking (see
below), so the tables whose suffix has an underscore (`vector_chunksNN`,
`rescore_chunksNN`, `rescore_vectorsNN`, `diskann_nodesNN`,
`diskann_bufferNN`) are never shadow tables.
`diskann_bufferNN`, and the four `ivf_` tables) are never shadow tables.
- SQLite finds the virtual table that owns a newly created or newly parsed
table by splitting its name at the **last** underscore
([documented](https://www.sqlite.org/vtab.html#the_xshadowname_method)). For
Expand Down
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- A vector column definition accepts `indexed by flat()`, as in `embedding float[768] indexed by flat()`. `flat()` is the exhaustive search that a column without the clause gets. Ported from [asg017@0de765f](https://github.com/asg017/sqlite-vec/commit/0de765f) ([asg017#273](https://github.com/asg017/sqlite-vec/pull/273)).
- A `float` vector column can be declared `indexed by rescore(quantizer=bit)` or `indexed by rescore(quantizer=int8)`, with an optional `oversample=N` from 1 to 128 (8 by default). It also keeps its vectors quantized, in the new `_rescore_chunksNN` shadow table, and its `float` vectors by rowid in `_rescore_vectorsNN`. A KNN query scans the quantized vectors for the `k * oversample` nearest rows (`5 * k * oversample` with `mmr_lambda` below 1, at most 4096), then ranks those by the exact distances of their `float` vectors, to which distance constraints apply. The `oversample=N` command, as in `insert into t(t) values ('oversample=16')`, sets the oversample of the table's rescore columns for the connection's queries until SQLite reconnects the table, as after `VACUUM`, `ALTER TABLE ... RENAME`, or another connection's schema change. Rowid filters, MMR, auxiliary columns, `UPDATE`, `optimize`, and `ALTER TABLE ... RENAME` work on rescore columns; a table with one cannot have metadata or partition key columns. Ported from [asg017#276](https://github.com/asg017/sqlite-vec/pull/276) ([asg017@ba0db0b](https://github.com/asg017/sqlite-vec/commit/ba0db0b)) with its fixes [asg017@5e4c557](https://github.com/asg017/sqlite-vec/commit/5e4c557), [@82f4eb0](https://github.com/asg017/sqlite-vec/commit/82f4eb0), [@5522e86](https://github.com/asg017/sqlite-vec/commit/5522e86), [@b7fc459](https://github.com/asg017/sqlite-vec/commit/b7fc459), [@6e2c4c6](https://github.com/asg017/sqlite-vec/commit/6e2c4c6), the int8 quantizer of [@575371d](https://github.com/asg017/sqlite-vec/commit/575371d), and the distance constraints of the unmerged [asg017#309](https://github.com/asg017/sqlite-vec/pull/309). Unlike upstream, a point query for a rowid that a rescore table does not have returns no row instead of failing with `Could not fetch vector data`.
- A `float` or `int8` vector column can be declared `indexed by diskann(neighbor_quantizer=binary)` or `indexed by diskann(neighbor_quantizer=int8)`, with optional `n_neighbors=N` (72 by default), `search_list_size=N` (128) or `search_list_size_search=N` and `search_list_size_insert=N`, and `buffer_threshold=N`. The column keeps a graph of its rows in the new `_diskann_nodesNN` shadow table, its vectors by rowid in `_vectorsNN`, and its buffered inserts in `_diskann_bufferNN`, and a KNN query walks the graph instead of reading every vector. The commands `search_list_size=N`, `search_list_size_search=N` and `search_list_size_insert=N` set those sizes for the connection's queries and inserts until SQLite reconnects the table, as after `VACUUM`, `ALTER TABLE ... RENAME`, or another connection's schema change. Rowid filters, distance constraints, MMR, auxiliary columns, `optimize`, and `ALTER TABLE ... RENAME` work on DiskANN columns; a table with one cannot have metadata or partition key columns, and a DiskANN column's vector cannot be updated. A table whose every vector column is indexed by DiskANN keeps no chunks. Ported from [asg017@575371d](https://github.com/asg017/sqlite-vec/commit/575371d) with [asg017@9df59b4](https://github.com/asg017/sqlite-vec/commit/9df59b4), [@82f4eb0](https://github.com/asg017/sqlite-vec/commit/82f4eb0), [@b008654](https://github.com/asg017/sqlite-vec/commit/b008654), [@c36a995](https://github.com/asg017/sqlite-vec/commit/c36a995), [@01b4b2a](https://github.com/asg017/sqlite-vec/commit/01b4b2a), [@b7fc459](https://github.com/asg017/sqlite-vec/commit/b7fc459) and [@fe94171](https://github.com/asg017/sqlite-vec/commit/fe94171). Unlike upstream, an `int8` column's vectors are quantized as `int8` values (upstream reads an `int8` query as floats, past its end), a result's distance is always its exact one, a point query for a rowid that the table does not have returns no row instead of failing with `Could not fetch vector data`, the row every search starts from is replaced, when deleted, by a row of the graph rather than one still waiting in the buffer, a search of a table whose start row upstream left in the buffer, or deleted, starts from a row of the graph, where upstream returns the start row twice or fails, and the table's next delete or buffer flush replaces that start row, a search allocates memory for the rows it reaches rather than for its whole search list (upstream allocates room for `search_list_size` candidates, and a visited set four times that size, before each search), rowid filters, distance constraints and MMR apply, errors and interrupts reach the query instead of being skipped, `VACUUM` works after a KNN query, the commands set every DiskANN column of the table, a KNN query takes `k` as every vec0 column does, an integer from 0 to 4096 (upstream returns no rows for a negative `k` and has no limit), CREATE refuses an abbreviated option such as `n_neigh=16` or `neighbor_quantizer=bin` and a comma before `)`, which connect reads as upstream does, and a command value such as `search_list_size=12abc` and a number past an `int` are refused.
- A build compiled with `-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1` can declare a `float` vector column `indexed by ivf()`, with optional `nlist=N` (128 by default, or 0 to leave the centroids to `set-centroid`), `nprobe=N` (10, at most `nlist`), `quantizer=int8` or `quantizer=binary`, and `oversample=N` (1). The column keeps its rows in cells of up to 64 in the new `_ivf_cellsNN` shadow table, around the centroids in `_ivf_centroidsNN`, with each row's cell and slot in `_ivf_rowid_mapNN` and, with a quantizer, its `float` vector in `_ivf_vectorsNN`; a KNN query reads the cells of the `nprobe` centroids nearest its vector and the cells of rows no clustering has placed. The commands `compute-centroids`, `compute-centroids:{"nlist": N, "max_iterations": N, "seed": N}`, `set-centroid:N`, `assign-vectors` and `clear-centroids` build it, and the `nprobe=N` command sets the nprobe of a connection's queries until SQLite reconnects the table, as after `VACUUM`, `ALTER TABLE ... RENAME`, or another connection's schema change. Rowid filters, distance constraints, MMR, auxiliary columns, `optimize`, and `ALTER TABLE ... RENAME` work on IVF columns; a table with one cannot have metadata or partition key columns, and an IVF column's vector cannot be updated. The prebuilt binaries are built without it. Ported from [asg017@e2c38f3](https://github.com/asg017/sqlite-vec/commit/e2c38f3) with [asg017@4bee883](https://github.com/asg017/sqlite-vec/commit/4bee883), [@9df59b4](https://github.com/asg017/sqlite-vec/commit/9df59b4), [@b7fc459](https://github.com/asg017/sqlite-vec/commit/b7fc459), [@6e2c4c6](https://github.com/asg017/sqlite-vec/commit/6e2c4c6), [@8b81f40](https://github.com/asg017/sqlite-vec/commit/8b81f40) and [@fe94171](https://github.com/asg017/sqlite-vec/commit/fe94171), and the fixes of the unmerged [asg017#313](https://github.com/asg017/sqlite-vec/pull/313). Unlike upstream, a quantized column's point query and full scan return its `float` vectors (upstream returns the quantized bytes, or reads past them), a connection finds the centroids another connection computed (upstream caches the column's state per connection and finds no rows), rowid filters and distance constraints apply, a renamed table's cells are found through an index named after the new table, so a table created under the old name works, an `int8` or `bit` column is refused (upstream reads past an `int8` column's vectors), errors and interrupts reach the statement, so an interrupted command fails with `interrupted` and the connection's queries go on finding its rows (upstream reports `SQL logic error`, and its KNN queries then return no rows), `VACUUM` works after a KNN query, the commands apply to every IVF column of the table, and an abbreviated option, a command value such as `nprobe=5x`, and `compute-centroids` options that are not a JSON object of integers are refused.

### Changed

Expand All @@ -21,7 +22,7 @@
- A command that a `vec0` table does not know, as in `insert into t(t) values ('compact')`, now fails with `unknown vec0 command: 'compact'` instead of `SQL logic error`, as upstream [asg017@6e2c4c6](https://github.com/asg017/sqlite-vec/commit/6e2c4c6) does.
- `insert into t(t) values ('oversample=N')` now succeeds on every `vec0` table, as on upstream, and on a table without a rescore column changes nothing; it used to fail with `SQL logic error`.
- When the same connection deletes a row that a running full scan or KNN query has found but not yet read, reading that row's vector in a KNN query, or its `text` primary key or partition key in either query, now fails with `SQLITE_ERROR` instead of `SQLITE_EMPTY`, a code SQLite reserves for its own use. The primary key read fails with `Could not find a row with rowid N` and the partition key read with `SQL logic error`; both used to fail with `unknown error`.
- `vec0`'s `xShadowName` also reports the `_rescore_chunksNN` and `_rescore_vectorsNN` tables of rescore columns and the `_vectorsNN`, `_diskann_nodesNN` and `_diskann_bufferNN` tables of DiskANN columns, which upstream's does not. Under `SQLITE_DBCONFIG_DEFENSIVE`, a table with one of those names beside any `vec0` table is read-only once SQLite marks it as a shadow table, as after a `VACUUM`, and `CREATE TABLE t_vectors00` beside a table `t` fails with `object name reserved for internal use`.
- `vec0`'s `xShadowName` also reports the `_rescore_chunksNN` and `_rescore_vectorsNN` tables of rescore columns, the `_vectorsNN`, `_diskann_nodesNN` and `_diskann_bufferNN` tables of DiskANN columns, and in a build with IVF the `_ivf_centroidsNN`, `_ivf_cellsNN`, `_ivf_rowid_mapNN` and `_ivf_vectorsNN` tables of IVF columns, which upstream's does not. Under `SQLITE_DBCONFIG_DEFENSIVE`, a table with one of those names beside any `vec0` table is read-only once SQLite marks it as a shadow table, as after a `VACUUM`, and `CREATE TABLE t_vectors00` beside a table `t` fails with `object name reserved for internal use`.
- GitHub releases now wait for npm to serve the approved package, and carry these release notes.

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ test-loadable-watch:

test-unit: | $(prefix)
$(CC) tests/test-unit.c sqlite-vec.c vendor/sqlite3.c \
-I./ -Ivendor -DSQLITE_CORE -o $(prefix)/test-unit -lm
-I./ -Ivendor -DSQLITE_CORE $(EXT_CFLAGS) -o $(prefix)/test-unit -lm
$(prefix)/test-unit

test-all: test test-loadable test-unit
Expand Down
Loading