Skip to content

Normalize protocol-prefixed paths before SFTP operations - #77

Merged
shcheklein merged 14 commits into
fsspec:mainfrom
xcRRRRt:fix/missing-handling-protocol-prefixed-url
Nov 17, 2026
Merged

shcheklein merged 14 commits into
fsspec:mainfrom
xcRRRRt:fix/missing-handling-protocol-prefixed-url

Conversation

@xcRRRRt

@xcRRRRt xcRRRRt commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #76

Normalize protocol-prefixed string paths consistently before remote operations, so plain paths, ssh:// URLs, and sftp:// URLs resolve to the same remote path on an existing filesystem connection.

  • Apply path normalization across SFTP operations and remote shell commands.
  • Use stringify_path to support path objects and preserve path-list handling.
  • Keep bytes paths out of URL parsing to avoid regressions.
  • Parameterize regression tests across plain paths, ssh:// paths, and sftp:// paths.
  • Add coverage for URL downloads, the move fallback when posix_rename is unsupported, and URLs with a different host.

Release note

Previously, fs.makedirs("ssh://host/data") could create ~/ssh:/host/data because AsyncSSH interpreted the URL as a relative path under the remote home directory. After this change, the same call creates /data on the server connected to fs. Users with files in the old directory tree should review their paths when upgrading.

When a URL is passed to a method on an existing SSHFileSystem instance, its host, username, and port do not select a different connection; the operation uses the instance's current SSH connection.

Validation

Local validation with Python 3.14.6:

  • python -m pytest --cov-report=xml --cov=. --reruns 1 --reruns-delay 2: 102 passed, 6 deprecation warnings.
  • pre-commit run --all-files: Black, isort, mdformat, and codespell passed.
  • git diff --check: passed.

@xcRRRRt

xcRRRRt commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

I manually triggered the test.yml workflow in my fork, but the Python 3.9 job failed during setup:

Error: The version '3.9' with architecture 'x64' was not found for Ubuntu 26.04.

It looks like ubuntu-latest now uses Ubuntu 26.04, for which actions/setup-python does not provide a Python 3.9 build.

Would it make sense to switch to uv for Python installation and test execution so we can retain Python 3.9 coverage on the latest Ubuntu runner?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Byte-valued os.PathLike objects bypass the bytes guard and reach string-based URL parsing, causing a TypeError where equivalent raw bytes paths succeed.

1 open issue
What changed in this PR

This PR addresses #76 by normalizing remote paths across SSHFileSystem operations so plain paths and ssh:// or sftp:// URLs work consistently.

Changes:

  • Normalize remote arguments before SFTP operations and remote shell commands.
  • Add path-object conversion, preserve raw bytes paths, and document the protocol-prefix limitation with path_encoding=None.
  • Parameterize regression tests across plain paths and both URL schemes, with normalized metadata expectations.
File Description
tests/​test_sshfs.py Expand path-format coverage and separate plain filesystem paths from remote URLs.
sshfs/​spec.py Apply remote-path normalization and document raw-byte path limitations.

Review effort: Balanced

Comment thread sshfs/spec.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking the remote-path normalization change.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is the right fix. It matches how fsspec's local/ftp/sftp filesystems and s3fs/gcsfs normalize paths in every method. _strip_protocol gives the same result when applied twice, so the extra calls on paths fsspec already stripped don't change anything. I checked "", /, //, ssh://h, ssh://h//abs, ~/x, URLs with query/fragment, and the bytes and path-like cases. The fix for path-likes returning bytes looks good.

Main asks:

  • Drop the redundant per-test parametrize("remote_dir", ...) decorators (inline).
  • Add tests for get_file with a URL, and for the mv fallback when posix_rename is unsupported (monkeypatch it to raise SFTPOpUnsupported). Optionally add one with a URL whose host differs from the connected one, to pin down that the host is ignored.
  • Reword the docstring note (inline).

For the release notes: before this PR, fs.makedirs("ssh://host/data") silently created ~/ssh:/host/data. asyncssh treats the URL as a relative path and creates each component. After the upgrade, the same call creates /data. Anyone with data in that tree should know.

Not blocking, and it was already like this before: bytes paths still work only in some methods. info/exists/isdir fail at path.rstrip("/"), rm(b"...") iterates over the bytes, _parent breaks put_file/pipe_file, and shlex.quote breaks cp_file/checksum.

Comment thread sshfs/spec.py Resolved
Comment thread sshfs/spec.py Resolved
Comment thread sshfs/spec.py Resolved
Comment thread sshfs/spec.py Resolved
Comment thread tests/test_sshfs.py Resolved
@shcheklein

Copy link
Copy Markdown
Collaborator

Thanks for the updates. Everything from the review is addressed, and I've resolved those threads. The _mv fallback thread stays open as a follow-up.

About CI: the Tests workflow is disabled in this repo (GitHub turned it off after 60 days of inactivity), so only pre-commit has run on this PR. Before merging, the workflow needs re-enabling and the Python 3.9 job needs a fix. ubuntu-latest is now Ubuntu 26.04, which has no 3.9 build, and with fail-fast: true that one failure would cancel the rest of the matrix.

Resolve the test import conflict: keep the extended asyncssh.sftp
import and use entry_points from the stdlib importlib.metadata, as on
main after the Python 3.11+ change (fsspec#82).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.53%. Comparing base (15aac74) to head (4339f17).
⚠️ Report is 18 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #77      +/-   ##
==========================================
+ Coverage   94.30%   94.53%   +0.23%     
==========================================
  Files          13       13              
  Lines         738      934     +196     
==========================================
+ Hits          696      883     +187     
- Misses         42       51       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Metadata operations still fail for bytes paths required by path_encoding=None because _info() uses string separators, leaving info() and modified() incompatible with those paths.

1 open issue

Review effort: Balanced

Comment thread sshfs/spec.py Resolved

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI generated

Completed three review passes: correctness/bugs, simplification/unification, and test quality/idioms.

# Category Summary
1 Bug / Unification _rm only checks isinstance(path, str), so fs.rm(Path(...)) raises TypeError and fs.rm(b"...") iterates over byte ints; preserving list support in _strip_protocol lets _rm normalize path up front like the other methods
2 Bug / Simplification Along with _info (already noted above), _put_file/_pipe_file (self._parent) and _ls (file_attr.filename in ["", ".", ".."]) still assume str when given bytes paths
3 Test fixture simplification Create plain_remote_dir locally with tmp_path instead of fs.mkdir, and use plain_remote_dir directly in test_fsspec_url_parsing instead of indirect=True parametrization on remote_dir
4 Test simplification Trim duplicate string PathLike assertions and repeated comment in test_strip_protocol
5 Test simplification / fixtures Use fs.pipe_file and tmp_path in test_get_file instead of manual makedirs + NamedTemporaryFile
6 Test simplification Use fs.open in test_cat_file_sync to avoid needing plain_remote_dir

Comment thread sshfs/spec.py Resolved
Comment thread sshfs/spec.py Resolved
Comment thread tests/test_sshfs.py Resolved
Comment thread tests/test_sshfs.py Resolved
Comment thread tests/test_sshfs.py Resolved
Comment thread tests/test_sshfs.py Resolved
@shcheklein

Copy link
Copy Markdown
Collaborator

@xcRRRRt thanks! I've merged main (drops 3.9, fixes CI). I've done another round of review (with AI - argon). Please take a look, address / resolve if it is reasonable.

Create plain_remote_dir locally instead of fsspec.
Use plain_remote_dir directly in the URL parsing test.
- Strip protocols from list inputs.
- Normalize paths in _rm and treat bytes paths as single targets.
- Add test for list input.
@xcRRRRt
xcRRRRt force-pushed the fix/missing-handling-protocol-prefixed-url branch from 76a73e4 to a8ac543 Compare November 17, 2026 17:47
Resolve the move-test conflict with fsspec#83: keep main's
test_move_copy_fallback and test_move_copy_fallback_failure_keeps_source,
and drop test_move_fallback_posix_rename_unsupported, which duplicated
test_move_copy_fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shcheklein
shcheklein merged commit 3825ddc into fsspec:main Nov 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing handling of protocol-prefixed paths in SSHFileSystem

4 participants