Repository navigation
Fix fsspec data loss after flush and reduce stream buffering costs - #407
Open
shcheklein wants to merge 2 commits into
Open
shcheklein wants to merge 2 commits into
shcheklein wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the Drive fsspec write-handling and buffering changes.
No open issues 🎉
What changed in this PR
This PR fixes data loss when writing after flush() through the Drive fsspec integration and reduces stream buffering overhead.
Changes:
- Keep writes local until close, using temporary-file spooling for larger uploads and propagating upload errors.
- Avoid copying unread response tails and honor positive download
block_sizevalues. - Add regression coverage and document flush, close, and buffering behavior.
| File | Description |
|---|---|
pydrive2/test/test_fs.py |
Adds a flush between writes in the live Drive round-trip test. |
pydrive2/test/test_fs_unit.py |
Tests partial reads, peeks, download sizes, writer rollover, and upload-failure cleanup. |
pydrive2/fs/utils.py |
Uses memory views for unread data and preserves buffered bytes after peek failures. |
pydrive2/fs/spec.py |
Adds spooled writes, upload-on-close behavior, and configurable download request sizes. |
docs/fsspec.rst |
Documents download sizing and local buffering versus upload behavior. |
Review effort: Balanced
shcheklein
commented
Nov 22, 2026
shcheklein
left a comment
Member
Author
There was a problem hiding this comment.
🤖 AI-generated review
Overall this is a clean, well-scoped fix with strong test coverage and impressive memory/latency improvements. The flush()/close() lifecycle and IterStream memoryview slicing are much cleaner than main.
Three edge-case bugs found (see inline comments):
- Temp-file leak on
write()afterclose()inGDriveBufferedWriter.write()(pydrive2/fs/spec.py). TypeErroronblock_size="default"in_gdrive_open_file()(pydrive2/fs/spec.py), which is a standardfsspecsentinel.memoryviewexport left open on exception inIterStream.readinto()(pydrive2/fs/utils.py), preventing callers from resizing/clearing abytearrayinside anexceptblock.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
main, callingflush()on a Drive writer uploads the bytes written so far and marks the stream closed. A later binarywrite()still accepts bytes, butclose()skips the final upload. For example, writingb"first", flushing, writingb"last", and closing leavesb"first"on Drive.This change makes
flush()flush the local buffer andclose()upload once. Binary writes return their byte count. Small files stay in memory and larger files spill to a seekable temporary file. Upload errors propagate on close and the spool is released. Callers that need the file to be visible on Drive should close the writer or exit its context manager.Small reads previously copied the entire unread tail of the current Drive response. The reader now keeps a view of that tail. A positive
block_sizepassed tofs.open()controls the size of Drive download requests.Local measurements
On macOS with CPython 3.11.18, comparing this branch with
4798a5c. The read result is the median of seven runs. The write measurement usestracemallocaround the writer, reuses one input block, and excludes network traffic.Verification
The offline suite passed locally (138 passed, 2 skipped). The filesystem unit tests passed on fsspec 2021.7.0 and 2026.9.0. Coverage includes flush followed by another write in text and binary mode, disk rollover, cleanup after a failed upload, byte-range request sizes, and mixed partial reads and peeks. The existing live Drive round-trip test now flushes between two writes and checks the downloaded bytes. Ruff, mypy, and the Sphinx build passed.
Based on
main. Git verified a clean merge with the separate protocol-registration PR #406.