Skip to content

Batch parent-folder queries to prevent HTTP 400 during recursive cache listings #393

Closed

Description

@kazeevn

Problem

Recursive listings can fail with HTTP 400 when GDriveFileSystem.find() encounters a wide directory tree. _gdrive_list_ids() combines every parent ID into one unbounded OR query.

This also affects DVC's regular data-cache traversal, not only run-cache transfer. Related: #353 (closed as a duplicate of treeverse/dvc#10449).

Reproduction and isolation

Using DVC 3.67.1, dvc-gdrive 3.0.1 and PyDrive2 1.21.3 with an authenticated gdrive://appDataFolder remote:

  1. Run uv run dvc status -c -v (the same remote check used before pushing).
  2. DVC successfully checks individual objects and estimates remote size.
  3. Full traversal fails at pydrive2/fs/spec.py:find -> _gdrive_list_ids -> _gdrive_list with:
    HttpError 400: Bad Request
    domain: global
    reason: badRequest
    

Instrumenting the listing showed that listing the cache root succeeded and returned 251 subfolders. Listing those 251 parents in one query failed. Retrying exactly those parent IDs in batches of 10 succeeded and returned all 852 cache files. No credentials, permissions or DVC configuration changed.

The failing query has this shape:

('parent_1' in parents or ... or 'parent_251' in parents) and trashed=false

Authentication is working; requests for smaller groups succeed. Unlike #353, the API returned generic badRequest rather than queryTooComplex, so I cannot establish Google's exact limit from the response.

Expected behavior

Bound the number of parent clauses in each request and iterate through all batches and pages. A recursive listing should work for a wide directory tree without callers having to disable traversal.

Fix submitted in #394. The pinned fork also passes the full DVC cloud-status check against this remote without a runtime monkeypatch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions