You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Batch parent-folder queries to prevent HTTP 400 during recursive cache listings #393
Recursive listings can fail with HTTP 400 when GDriveFileSystem.find() encounters a wide directory tree. _gdrive_list_ids() combines every parent ID into one unbounded OR query.
This also affects DVC's regular data-cache traversal, not only run-cache transfer. Related: #353 (closed as a duplicate of treeverse/dvc#10449).
Reproduction and isolation
Using DVC 3.67.1, dvc-gdrive 3.0.1 and PyDrive2 1.21.3 with an authenticated gdrive://appDataFolder remote:
Run uv run dvc status -c -v (the same remote check used before pushing).
DVC successfully checks individual objects and estimates remote size.
Full traversal fails at pydrive2/fs/spec.py:find -> _gdrive_list_ids -> _gdrive_list with:
HttpError 400: Bad Request
domain: global
reason: badRequest
Instrumenting the listing showed that listing the cache root succeeded and returned 251 subfolders. Listing those 251 parents in one query failed. Retrying exactly those parent IDs in batches of 10 succeeded and returned all 852 cache files. No credentials, permissions or DVC configuration changed.
The failing query has this shape:
('parent_1' in parents or ... or 'parent_251' in parents) and trashed=false
Authentication is working; requests for smaller groups succeed. Unlike #353, the API returned generic badRequest rather than queryTooComplex, so I cannot establish Google's exact limit from the response.
Expected behavior
Bound the number of parent clauses in each request and iterate through all batches and pages. A recursive listing should work for a wide directory tree without callers having to disable traversal.
Fix submitted in #394. The pinned fork also passes the full DVC cloud-status check against this remote without a runtime monkeypatch.
Activity