Repository navigation
Conversation
Only Lightning's MLFlowLogger received mlflow_log_model; the end-of-fit metadata upload in _log_run_metadata_to_mlflow logged every file in <run_dir>/checkpoints/ regardless. Skip that upload when the flag is off. Fixes #543
jorisstrakeljahn
left a comment
Contributor
There was a problem hiding this comment.
I looked at the red jobs here. I think the change in fit.py is fine, and the CI fails for two other reasons.
First, Pyright runs without mlflow installed, so it cannot find mlflow.tracking in the new test. fit.py has the same kind of import and uses # type: ignore[import-untyped] there. I think the same comment would fix it here.
Second, the minimum extras and tabr jobs install mlflow 2.22.0 with SQLAlchemy 2.1. That combination already fails without this PR. As far as I can see, no test built a real MLflow logger before this one, so CI did not show it. I opened #624 for it and a fix in #625 and I think that then these jobs pass with that fix.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #543
Problem
ObservabilityConfig.mlflow_log_modelis documented as "Upload model checkpoints as MLflow artifacts", but only Lightning'sMLFlowLogger(log_model=...)received it. At the end offit(),_log_run_metadata_to_mlflowuploaded every file in<run_dir>/checkpoints/undercheckpoints/without checking the flag. Checkpoints therefore reached the tracking server even withmlflow_log_model=False.Fix
deeptab/models/_mixins/fit.py: the checkpoint upload in_log_run_metadata_to_mlflowruns only whenobs.mlflow_log_modelis true.config.yaml,config.json,lifecycle.jsonlandsummary.jsonare still logged as before.Tests
tests/integration/test_mlflow_artifacts.py(new file, skipped withoutmlflow) runstest_checkpoint_upload_follows_mlflow_log_model[True|False]. It fits with a local SQLite-backed MLflow tracker and lists the run's artifacts. WithTrue,checkpoints/best_model.ckptis uploaded. WithFalse, no.ckptartifact exists andsummary.jsonis still there. TheFalsecase fails onmainand passes with this change.Docs
No change needed. The flag's documentation already describes the intended behaviour.
Checks
just test: 3238 passed, 46 skipped, 28 xfailed; coverage 91.85%just docs: build succeeded (warnings treated as errors)just check: all hooks passed (ruff, prettier, pyright)