Skip to content

fix: batch parent-folder queries during recursive listings - #394

Merged
shcheklein merged 4 commits into
iterative:mainfrom
kazeevn:fix/batch-parent-queries
Nov 21, 2026
Merged

shcheklein merged 4 commits into
iterative:mainfrom
kazeevn:fix/batch-parent-queries

Conversation

@kazeevn

@kazeevn kazeevn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Fixes #393.

Recursive listings currently combine every parent folder into one unbounded query. A real DVC data-cache listing with 251 subfolders fails with HTTP 400; querying the same parents in batches succeeds.

Limit each query to 50 parents and lazily yield all results before requesting the next batch. Empty inputs issue no requests. This preserves the existing pagination performed by _gdrive_list().

Validation:

  • 21 offline regression cases pass, covering batch boundaries, list/dict/iterator inputs, laziness, and a 251-folder recursive listing with and without details.
  • Substituting the original implementation produces 15 failures in the same tests.
  • Live read-only validation with batches of 50 returns all 852 cache files from the previously failing remote.
  • Flake8 and formatting checks of the changed code pass.

Related: #353, which was closed as a duplicate of a DVC run-cache issue. This reproduction affects regular data-cache traversal.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the parent-folder query batching change.

No open issues 🎉

What changed in this PR

This PR addresses #393 by batching parent-folder queries in recursive Google Drive listings, preventing oversized queries that caused HTTP 400 errors during DVC cache traversal.

Changes:

  • Limits queries to 50 parents and lazily yields each batch’s results while preserving existing pagination.
  • Avoids requests for empty inputs.
  • Adds offline regression tests for batch boundaries, iterable inputs, laziness, and 251-folder traversal with and without details.
File Description
pydrive2/​test/​test_fs_listing.py Adds regression coverage for batching and wide recursive listings.
pydrive2/​fs/​spec.py Implements lazy parent-query batching with an empty-input guard.

Review effort: Balanced

@shcheklein

Copy link
Copy Markdown
Member

/test-drive

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the Google Drive query-batching changes.

No open issues 🎉

Review effort: Balanced

@shcheklein

Copy link
Copy Markdown
Member

/test-drive

@shcheklein
shcheklein merged commit c904bc8 into iterative:main Nov 21, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch parent-folder queries to prevent HTTP 400 during recursive cache listings

3 participants