Skip to content

Keep the trailing separator in glob and rglob patterns - #576

Open
itzzdev09 wants to merge 1 commit into
fsspec:mainfrom
itzzdev09:fix/glob-trailing-separator
Open

itzzdev09 wants to merge 1 commit into
fsspec:mainfrom
itzzdev09:fix/glob-trailing-separator

Conversation

@itzzdev09

Copy link
Copy Markdown

A glob pattern ending in a separator matches only directories, in pathlib and in fsspec's glob. UPath.glob("*/") and UPath.rglob("*/") return files as well:

from upath import UPath

root = UPath("memory:///g")
(root / "d").mkdir(parents=True)
(root / "f.txt").write_text("x")

list(root.glob("*/"))  # [UPath('memory://g/d'), UPath('memory://g/f.txt')]

glob() and rglob() build the fsspec pattern with self.joinpath(pattern).path, and joinpath drops the trailing separator, so fsspec gets /g/* instead of /g/*/. This re-appends the separator when the pattern is a string that ends with one, in glob() and in both branches of rglob(). PathLike patterns are unchanged.

Tests: "*/" is added to the shared test_glob parametrization and test_rglob is parametrized over "*.txt" and "*/", so every backend is compared against pathlib. Without the change, 22 of the glob tests fail; with it, the suite passes locally (2485 passed, 1184 skipped; GitHub and HTTP tests deselected as their dependencies aren't installed here). black, isort and flake8 at the pre-commit versions pass.

"**/" also returns only directories now on memory and local paths, but it isn't added to the shared tests: for zip and tar, fsspec's archive glob("**/") doesn't include the archive root, which pathlib does, and that's unrelated to this change.

A pattern ending in a separator, such as "*/", matches only directories in
pathlib and in fsspec's glob. UPath.glob() and rglob() built the fsspec pattern
with joinpath(), which drops the trailing separator, so files were returned
as well.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant