Skip to content

feature: advanced file conversion #54

Closed
Closed

Description

@Integer-Ctrl

Feature Request

Describe the feature you'd like
The on-the-fly conversion during download works, but could be improved for greater convenience:

  • Decompress files on download (from detected compression to file w/o compression)
  • Compress files without compression on download (from file w/o compression to specified compression)
  • Refactor file conversion logic to databusclient/extensions/file_converter.py
  • Currently the conversion feels slow. I didn't test whether this is reasonable or if this could be improved. An idea would be to add the conversion (and also checksum computation/validation) directly to the download step. Now the files is first downloaded completely and then converted. Maybe this can be squashed into one step.

Why is this feature important?
Explain the benefit or use case. Is your feature request related to a problem?
Follow up on #18. Not urgent, but useful for greater functionality

Describe alternatives you've considered
For now: manually

Activity

  1. added
    enhancementNew feature or request
    downloadIssue related to data download functionality from databus
    on Jan 22, 2026
  2. Dev10-sys commented on Jan 22, 2026

    @Dev10-sys

    This makes sense. Integrating conversion and checksum validation into the download step could improve performance.
    I’m currently focusing on small cleanups, but happy to help later once the approach is clarified.

  3. Tahoora-Tabassum commented on Jan 29, 2026

    @Tahoora-Tabassum

    One idea I had is to treat the download, (de)compression, and checksum as a small streaming pipeline rather than separate steps — so the download stream passes through optional processors (decompressor, compressor, checksum) before being written to disk.

    I could start with a minimal version (gzip + checksum only) to address this issue, and later this could be extended to support more formats or features if that fits the project’s direction. Would this approach make sense here?

  4. Integer-Ctrl commented on Sep 17, 2026

    @Integer-Ctrl
    ContributorAuthor

    This issue is deprecated due to the refactoring of the compression options as part of the GSoC 2026 project

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    downloadIssue related to data download functionality from databusenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions