Skip to content

Reduce Memory Consumption - Streaming Coalescing in Quickwit - #6985

Open
PSeitz wants to merge 39 commits into
quickwit-oss:mainfrom
PSeitz:coalesce_reads
Open

PSeitz wants to merge 39 commits into
quickwit-oss:mainfrom
PSeitz:coalesce_reads

Conversation

@PSeitz

@PSeitz PSeitz commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Tantivy coalesces postings reads to reduce request count. For sparse matches, a query needing only ~1 MB of postings may end up downloading most of a ~250 MB postings file. This can cause OOMs in two stages:

  1. Download peak: merged ranges are buffered in full, including large gaps of unwanted postings.
  2. Cache retention: those merged buffers then enter the cache, retaining the unwanted bytes after the download finishes.

Streaming coalesced reads lets us extract the requested data and discard gaps as they arrive. The cache receives only the requested slices instead of retaining the merged downloads.

The batch API avoids a future per range from tantivy. We may receive one million and futures are very large.

Limitations

The streaming part only covers gaps introduced by coalescing.
A query that matches everything and downloads 250 MB without gaps still downloads and buffers a 250 MB chunk. It does not stream the postings lists. Full streaming of postings lists is future work.

What changes

  • Update Tantivy to use explicit batched reads for postings and dictionary warmup. Directory wrappers forward batches after removing cache hits.
  • Plan coalesced ranges in StorageDirectory, within one batch and one file, merging gaps up to 2 MiB. Preserve input order, duplicates, and empty reads.
  • Stream downloads that combine multiple requests. Discard gaps during reading and copy proper subranges so cached results cannot retain a larger overlapping-request buffer. Unmerged reads continue using buffered storage reads.
  • Support streaming through bundle offset translation and storage caches. Streamed cache misses bypass insertion of the merged response; the directory cache stores requested slices after a successful batch.
  • Apply timeouts to stream opening and body reads. Retries resume after emitted bytes and require immutable objects. Report S3 downloaded bytes in 8 MiB increments, flushing the remainder on drop.

Verification

  • 477 tests passed across quickwit-common, quickwit-storage, quickwit-directories, and quickwit-search.
  • Coverage includes recorded postings-range fixtures, batch ordering and ownership, chunk boundaries, gap disposal, cache behavior, cancellation, and timeout/retry handling.
  • Formatting, Clippy, and diff checks passed.

Full Docker/REST suites and live S3 memory/performance benchmarks were not run.

@PSeitz PSeitz added area:storage S3, GCS, Azure and local storage type:enhancement An improvement to something that exists, or its deprecation or removal area:search Root and leaf search, caches, sorting, pagination, Lambda offload labels Nov 3, 2026
@PSeitz
PSeitz force-pushed the coalesce_reads branch 15 times, most recently from 02eb51f to 49225b9 Compare November 6, 2026 10:46
@PSeitz PSeitz changed the title Coalesce concurrent split reads independently of caches Fix OOM - Move coalescing into Quickwit + streaming Nov 6, 2026
@PSeitz PSeitz changed the title Fix OOM - Move coalescing into Quickwit + streaming Fix OOM - Move coalescing into Quickwit + Streaming Nov 6, 2026
@PSeitz PSeitz changed the title Fix OOM - Move coalescing into Quickwit + Streaming Fix OOM - Streaming Coalescing in Quickwit Nov 6, 2026
@PSeitz
PSeitz force-pushed the coalesce_reads branch 2 times, most recently from d26f606 to 09e88c7 Compare November 10, 2026 17:01
@PSeitz PSeitz changed the title Fix OOM - Streaming Coalescing in Quickwit Reduce Memory Consumption - Streaming Coalescing in Quickwit Nov 17, 2026
@PSeitz
PSeitz marked this pull request as ready for review November 18, 2026 08:56
@PSeitz
PSeitz requested a review from a team as a code owner November 18, 2026 08:56
@datadog-official datadog-official Bot added the area:performance Speed, memory or cost is the point. Added on top of the code areas label Nov 18, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T09:05:30.816716Z 27da072 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Forward complete read batches through caches and bundle offsets. Collect concurrent reads for 1 ms and merge gaps up to 4 MiB, with five downloads per group. Stream useful slices and discard gaps, preserving cache ownership, retries, and cancellation.
Use the same helper for initial opens and retries after body timeouts.
Cover retry exhaustion and EOF timeouts without reopening empty ranges.
Document callback ordering and cache-hit handling. Assert storage ranges
directly in mocks instead of tracking reads through shared state.
Allow up to 32 concurrent asynchronous cache lookups instead of five.
Expand the test to verify 64 lookups complete in two waves while
preserving result order and forwarding cache misses together.
Stop probing for extra bytes after reading requested slices.
Use read_exact for Azure buffer reads and preserve short-read errors.
Update tests and documentation for the changed behavior.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27da072f48

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +258 to +259
let num_added_bytes = self.cache.put_slice(missing_range, bytes.clone());
read_reservation.keep(num_added_bytes);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Offload large-batch cache insertion from Tokio workers

For large regex/wildcard warmup batches, this loop performs one synchronous FileByteRangeCache::put_slice per requested range after the download finishes. Each call locks the cache and performs BTreeMap operations, potentially also copying overlapping buffers. With the 10k–1M-range batches targeted by this change, the entire population step runs without yielding on the leaf search's Tokio worker, delaying unrelated RPCs and I/O when several splits warm up concurrently. Move large-batch cache population onto run_cpu_intensive, following the Tokio/CPU-work guidance in AGENTS.md.

AGENTS.md reference: AGENTS.md:L87-L87

Useful? React with 👍 / 👎.

Comment on lines +287 to +289
let reader = storage
.get_slice_stream(path, download_chunk.clone())
.await?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve Azure body-error retries for merged downloads

For Azure-backed splits, switching merged downloads to this stream removes retries for transient body failures after the first chunk. AzureBlobStorage::get_slice previously used get_to_bytes, whose retry callback encloses the complete download_all; get_slice_stream ends its retry callback after obtaining the first chunk. Subsequent I/O errors therefore abort the whole collected batch, and TimeoutAndRetryStorage does not recover them because it propagates body errors immediately and retries only elapsed deadlines. Preserve Azure's body-error retry behavior with a resumable retrying stream.

Useful? React with 👍 / 👎.

@PSeitz
PSeitz requested review from trinity-1686a and a balanced review from Copilot November 18, 2026 10:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cache bypasses and redundant memory reservations can cause unnecessary remote reads and search failures, while valid empty ranges can panic in debug builds and Azure streaming reads lose body-error retries.

5 open issues
What changed in this PR

This PR reduces search-warmup memory use by moving read coalescing into Quickwit’s storage layer. Merged downloads are streamed so gaps can be discarded rather than retained in cached buffers.

Changes:

  • Updates Tantivy and forwards batched reads through directory, cache, and bundle wrappers.
  • Adds streaming coalescing with bounded download concurrency and requested-range extraction.
  • Extends stream timeouts, retry handling, S3 byte accounting, and regression coverage.
File Description
quickwit/​quickwit-storage/​test-data/​coalescing/​requested_ranges.json Adds recorded postings-range fixtures.
quickwit/​quickwit-storage/​test-data/​coalescing/​README.md Documents fixture provenance and reference measurements.
quickwit/​quickwit-storage/​src/​timeout_and_retry_storage.rs Adds stream deadlines and timeout-based resumption.
quickwit/​quickwit-storage/​src/​storage.rs Defines batch reads and revises buffer-fill behavior.
quickwit/​quickwit-storage/​src/​storage_cache/​storage.rs Adds batch handling to the shared storage cache.
quickwit/​quickwit-storage/​src/​object_storage/​s3_compatible_storage.rs Tracks streamed bytes and in-flight requests.
quickwit/​quickwit-storage/​src/​object_storage/​azure_blob_storage.rs Simplifies reading into caller-provided buffers.
quickwit/​quickwit-storage/​src/​lib.rs Exports coalescing storage and batch helpers.
quickwit/​quickwit-storage/​src/​error.rs Preserves underlying I/O error kinds.
quickwit/​quickwit-storage/​src/​coalescing_storage/​tests.rs Tests collection, sharing, and cancellation.
quickwit/​quickwit-storage/​src/​coalescing_storage/​stream_tests.rs Tests extraction, ownership, and stream failures.
quickwit/​quickwit-storage/​src/​coalescing_storage/​mod.rs Implements range collection, planning, and streaming extraction.
quickwit/​quickwit-storage/​src/​cache/​storage_with_cache.rs Adds generic cached batch reads.
quickwit/​quickwit-storage/​src/​cache/​mod.rs Exposes batch-cache utilities.
quickwit/​quickwit-storage/​src/​cache/​batch_read.rs Adds ordered cache lookup and miss-forwarding helpers.
quickwit/​quickwit-storage/​src/​bundle_storage.rs Translates batch offsets and validates file bounds.
quickwit/​quickwit-search/​src/​storage_cache_layer_tests.rs Tests batch forwarding through storage caches.
quickwit/​quickwit-search/​src/​leaf.rs Installs coalescing storage in search setup.
quickwit/​quickwit-directories/​src/​storage_directory.rs Forwards directory batches to storage.
quickwit/​quickwit-directories/​src/​hot_directory.rs Adds hot-cache-aware batch forwarding.
quickwit/​quickwit-directories/​src/​debug_proxy_directory.rs Tests batch-read instrumentation.
quickwit/​quickwit-directories/​src/​caching_directory.rs Adds batch warmup reservations and cache insertion.
quickwit/​quickwit-directories/​Cargo.toml Enables storage test utilities.
quickwit/​quickwit-config/​src/​node_config/​mod.rs Makes timeout iterator captures explicit.
quickwit/​Cargo.toml Updates the Tantivy revision.
quickwit/​Cargo.lock Records updated Tantivy dependencies.
LICENSE-3rdparty.csv Adds the new transitive dependency’s license entry.

Review effort: Balanced

Comment on lines +235 to +237
quickwit_storage::read_batch_with_cache(
requested_ranges,
|requested_range| self.cache.get_slice(requested_range),
Comment on lines +239 to +261
// Reserve every miss before starting I/O. Dropping these guards releases all
// reservations if any reservation or the download fails, or the future is dropped.
let mut read_reservations = Vec::with_capacity(ranges_to_fetch.len());
for missing_range in &ranges_to_fetch {
read_reservations.push(
self.memory_reserver
.try_reserve_memory(missing_range.len() as u64)?,
);
}
let downloaded = self
.underlying_filehandle
.read_bytes_batch_async(ranges_to_fetch.clone())
.await?;
assert_eq!(downloaded.len(), ranges_to_fetch.len());
for ((missing_range, bytes), read_reservation) in ranges_to_fetch
.into_iter()
.zip(&downloaded)
.zip(read_reservations)
{
let num_added_bytes = self.cache.put_slice(missing_range, bytes.clone());
read_reservation.keep(num_added_bytes);
}
Ok(downloaded)
Comment on lines +72 to +74
self.storage_directory
.storage
.get_slices(&self.path, requested_ranges)
Comment on lines +91 to +95
let skip_cache =
requested_ranges.len() > super::batch_read::MAX_NUM_RANGES_FOR_CACHE_LOOKUP;
super::read_batch_with_async_cache(
requested_ranges,
|requested_range| self.cache.get(path, requested_range),
Comment on lines +287 to +289
let reader = storage
.get_slice_stream(path, download_chunk.clone())
.await?;

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:performance Speed, memory or cost is the point. Added on top of the code areas area:search Root and leaf search, caches, sorting, pagination, Lambda offload area:storage S3, GCS, Azure and local storage type:enhancement An improvement to something that exists, or its deprecation or removal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants