Repository navigation
Normalize protocol-prefixed paths before SFTP operations - #77
Conversation
|
I manually triggered the It looks like Would it make sense to switch to |
There was a problem hiding this comment.
🟡 Changes recommended
Byte-valued os.PathLike objects bypass the bytes guard and reach string-based URL parsing, causing a TypeError where equivalent raw bytes paths succeed.
1 open issue
What changed in this PR
This PR addresses #76 by normalizing remote paths across SSHFileSystem operations so plain paths and ssh:// or sftp:// URLs work consistently.
Changes:
- Normalize remote arguments before SFTP operations and remote shell commands.
- Add path-object conversion, preserve raw bytes paths, and document the protocol-prefix limitation with
path_encoding=None. - Parameterize regression tests across plain paths and both URL schemes, with normalized metadata expectations.
| File | Description |
|---|---|
tests/test_sshfs.py |
Expand path-format coverage and separate plain filesystem paths from remote URLs. |
sshfs/spec.py |
Apply remote-path normalization and document raw-byte path limitations. |
Review effort: Balanced
shcheklein
left a comment
There was a problem hiding this comment.
Thanks, this is the right fix. It matches how fsspec's local/ftp/sftp filesystems and s3fs/gcsfs normalize paths in every method. _strip_protocol gives the same result when applied twice, so the extra calls on paths fsspec already stripped don't change anything. I checked "", /, //, ssh://h, ssh://h//abs, ~/x, URLs with query/fragment, and the bytes and path-like cases. The fix for path-likes returning bytes looks good.
Main asks:
- Drop the redundant per-test
parametrize("remote_dir", ...)decorators (inline). - Add tests for
get_filewith a URL, and for themvfallback whenposix_renameis unsupported (monkeypatch it to raiseSFTPOpUnsupported). Optionally add one with a URL whose host differs from the connected one, to pin down that the host is ignored. - Reword the docstring note (inline).
For the release notes: before this PR, fs.makedirs("ssh://host/data") silently created ~/ssh:/host/data. asyncssh treats the URL as a relative path and creates each component. After the upgrade, the same call creates /data. Anyone with data in that tree should know.
Not blocking, and it was already like this before: bytes paths still work only in some methods. info/exists/isdir fail at path.rstrip("/"), rm(b"...") iterates over the bytes, _parent breaks put_file/pipe_file, and shlex.quote breaks cp_file/checksum.
|
Thanks for the updates. Everything from the review is addressed, and I've resolved those threads. The About CI: the Tests workflow is disabled in this repo (GitHub turned it off after 60 days of inactivity), so only pre-commit has run on this PR. Before merging, the workflow needs re-enabling and the Python 3.9 job needs a fix. |
Resolve the test import conflict: keep the extended asyncssh.sftp import and use entry_points from the stdlib importlib.metadata, as on main after the Python 3.11+ change (fsspec#82). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #77 +/- ##
==========================================
+ Coverage 94.30% 94.53% +0.23%
==========================================
Files 13 13
Lines 738 934 +196
==========================================
+ Hits 696 883 +187
- Misses 42 51 +9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
shcheklein
left a comment
There was a problem hiding this comment.
AI generated
Completed three review passes: correctness/bugs, simplification/unification, and test quality/idioms.
| # | Category | Summary |
|---|---|---|
| 1 | Bug / Unification | _rm only checks isinstance(path, str), so fs.rm(Path(...)) raises TypeError and fs.rm(b"...") iterates over byte ints; preserving list support in _strip_protocol lets _rm normalize path up front like the other methods |
| 2 | Bug / Simplification | Along with _info (already noted above), _put_file/_pipe_file (self._parent) and _ls (file_attr.filename in ["", ".", ".."]) still assume str when given bytes paths |
| 3 | Test fixture simplification | Create plain_remote_dir locally with tmp_path instead of fs.mkdir, and use plain_remote_dir directly in test_fsspec_url_parsing instead of indirect=True parametrization on remote_dir |
| 4 | Test simplification | Trim duplicate string PathLike assertions and repeated comment in test_strip_protocol |
| 5 | Test simplification / fixtures | Use fs.pipe_file and tmp_path in test_get_file instead of manual makedirs + NamedTemporaryFile |
| 6 | Test simplification | Use fs.open in test_cat_file_sync to avoid needing plain_remote_dir |
|
@xcRRRRt thanks! I've merged main (drops 3.9, fixes CI). I've done another round of review (with AI - argon). Please take a look, address / resolve if it is reasonable. |
Create plain_remote_dir locally instead of fsspec. Use plain_remote_dir directly in the URL parsing test.
- Strip protocols from list inputs. - Normalize paths in _rm and treat bytes paths as single targets. - Add test for list input.
76a73e4 to
a8ac543
Compare
Resolve the move-test conflict with fsspec#83: keep main's test_move_copy_fallback and test_move_copy_fallback_failure_keeps_source, and drop test_move_fallback_posix_rename_unsupported, which duplicated test_move_copy_fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Fixes #76
Normalize protocol-prefixed string paths consistently before remote operations, so plain paths,
ssh://URLs, andsftp://URLs resolve to the same remote path on an existing filesystem connection.stringify_pathto support path objects and preserve path-list handling.ssh://paths, andsftp://paths.posix_renameis unsupported, and URLs with a different host.Release note
Previously,
fs.makedirs("ssh://host/data")could create~/ssh:/host/databecause AsyncSSH interpreted the URL as a relative path under the remote home directory. After this change, the same call creates/dataon the server connected tofs. Users with files in the old directory tree should review their paths when upgrading.When a URL is passed to a method on an existing
SSHFileSysteminstance, its host, username, and port do not select a different connection; the operation uses the instance's current SSH connection.Validation
Local validation with Python 3.14.6:
python -m pytest --cov-report=xml --cov=. --reruns 1 --reruns-delay 2: 102 passed, 6 deprecation warnings.pre-commit run --all-files: Black, isort, mdformat, and codespell passed.git diff --check: passed.