Skip to content

[9.1] feat (FileCatalogClient): add bulk method - #8875

Open
arrabito wants to merge 4 commits into
DIRACGrid:integrationfrom
arrabito:feat/add-method-to-fc
Open

arrabito wants to merge 4 commits into
DIRACGrid:integrationfrom
arrabito:feat/add-method-to-fc

Conversation

@arrabito

@arrabito arrabito commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

BEGINRELEASENOTES

*Resources/Catalog/FileCatalogClient
NEW: Add the getFileUserMetadataBulk method to the FileCatalogClient to make it compatible with the usage of RucioFileCatalogClient

ENDRELEASENOTES

@arrabito
arrabito force-pushed the feat/add-method-to-fc branch from 42aa343 to a7275fe Compare November 6, 2026 15:47

types_getFileUserMetadataBulk = [[list, dict, str]]

def export_getFileUserMetadataBulk(self, lfns):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between this function and getFileDetails (which is bulk already, I think?) Is getFileDetails missing the directory metadata?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andresailer I had never realized the existance of this method which returns user defined metadata as its name was not very informative.

If such a bulk method already exists then it's redundant with the proposed method and also with the existing non bulk one getFileUserMetadata.

For your information it does not seem to work though, as its decorator expects a return value with the {"Successful": successful, "Failed": failed} structure.

Then, I see that it returns not only user defined metadata but also the low-level metadata, such guid and so on.

So, in order to have a common FileCatalog interface for DIRAC and Rucio I think it would be good to have these 2 methods that just return user defined metadata :

getFileUserMetadata
getFileUserMetadataBulk

In order to not write redundant code, the getFileUserMetadataBulk method could call fileCatalogDB.getFileDetailsPublic, as in the method you pointed, but returning only user defined metadata with the expected return dict format.

@fstagni @andresailer what do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, one function calling the other would be sensible to unify interfaces.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at it in more detail there is actually a difference between the 2 methods.

getFileDetails returns only metadata associated to files and not those of their parent directories.

However we could still reuse the bulk method used to get file metadata that is used in getFileDetails and also add parent-directory metadata and return the correct dict.

I'm going to commit the proposed change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants