Skip to content

path.iterdir() yields path itself as the first item (with S3) #226

Closed

Description

@danielgafni

Given the following objects on S3:

s3://my-bucket/my-directory/0.txt
s3://my-bucket/my-directory/1.txt

UPath("s3://my-bucket/my-directory").iterdir() yields:

s3://my-bucket/my-directory
s3://my-bucket/my-directory/0.txt
s3://my-bucket/my-directory/1.txt

The first item is wrong, right?

Activity

  1. ap-- commented on Jun 21, 2024

    @ap--
    Collaborator

    Hi @danielgafni

    Thank you for reporting! Could you check 3 things:

    • did this change with recent fsspec versions? (Just try installing the newest vs installing one from <2024)
    • if you list the contents of the bucket just using filesystem_spec (without upath) does it return an entry named my-bucket/my-directory ?
    • was this s3 bucket manually modified using the AWS webui ? (i.e. Files uploaded via the webui?)

    Cheers,
    Andreas 😊

  2. danielgafni commented on Jun 21, 2024

    @danielgafni
    Author

    Hey!

    I can answer to (1) and (3) right away:

    1. My fsspec version was 2024.3.1
    2. No, it wasn't

    Regarding (2), I will be able to check a bit later

  3. danielgafni commented on Jun 23, 2024

    @danielgafni
    Author

    Checked it, seems to be an issue with fsspec. It has this problem too.

  4. ap-- commented on Jun 23, 2024

    @ap--
    Collaborator

    I wouldn't call it a problem. There's something stored under that key in the bucket. We should just define / document behavior of upath in cases like these.

  5. danielgafni commented on Jun 23, 2024

    @danielgafni
    Author

    No object with this key exists in the bucket. Also, it happens with any "directory" (common path prefix) in the bucket, not just a specific one.

    Also, aws s3 ls doesn't contain the problematic key

  6. ap-- commented on Jun 23, 2024

    @ap--
    Collaborator

    Could you create a PR with a test case in the upath S3 tests that reproduces the issue? That would be super helpful for finding a solution

  7. ap-- commented on Oct 5, 2025

    @ap--
    Collaborator
  8. ap-- commented on Jan 13, 2026

    @ap--
    Collaborator

    @danielgafni could you please verify if #536 resolves your issue?

  9. danielgafni commented on Jan 13, 2026

    @danielgafni
    Author

    I wasn't aware of marker directories at the time of submitting this issue. This seems to be exactly what was going on, but I don't have access to an AWS environment at the moment to verify this.

    Nevertheless, I think your fix makes total sense, I'd say let's go for it.


    Edit: found the other article you've linked in your tests, nice

  10. ap-- commented on Jan 13, 2026

    @ap--
    Collaborator

    Okay. It makes sense for them to be omitted from .iterdir(). I'll merge the PR.

    Thanks again for reporting 🙏

  11. added this to the v0.4.0 milestone on Jan 31, 2026
  12. danielgafni commented on Nov 11, 2026

    @danielgafni
    Author

    Yay! 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug 🐛Something isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions