Skip to content

fix(models): honour mlflow_log_model for the checkpoint upload - #544

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/mlflow-log-model
Open

ChrisW09 wants to merge 1 commit into
mainfrom
fix/mlflow-log-model

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #543

Problem

ObservabilityConfig.mlflow_log_model is documented as "Upload model checkpoints as MLflow artifacts", but only Lightning's MLFlowLogger(log_model=...) received it. At the end of fit(), _log_run_metadata_to_mlflow uploaded every file in <run_dir>/checkpoints/ under checkpoints/ without checking the flag. Checkpoints therefore reached the tracking server even with mlflow_log_model=False.

Fix

deeptab/models/_mixins/fit.py: the checkpoint upload in _log_run_metadata_to_mlflow runs only when obs.mlflow_log_model is true. config.yaml, config.json, lifecycle.jsonl and summary.json are still logged as before.

Tests

tests/integration/test_mlflow_artifacts.py (new file, skipped without mlflow) runs test_checkpoint_upload_follows_mlflow_log_model[True|False]. It fits with a local SQLite-backed MLflow tracker and lists the run's artifacts. With True, checkpoints/best_model.ckpt is uploaded. With False, no .ckpt artifact exists and summary.json is still there. The False case fails on main and passes with this change.

Docs

No change needed. The flag's documentation already describes the intended behaviour.

Checks

  • just test: 3238 passed, 46 skipped, 28 xfailed; coverage 91.85%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

Only Lightning's MLFlowLogger received mlflow_log_model; the end-of-fit
metadata upload in _log_run_metadata_to_mlflow logged every file in
<run_dir>/checkpoints/ regardless. Skip that upload when the flag is off.

Fixes #543

@jorisstrakeljahn jorisstrakeljahn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the red jobs here. I think the change in fit.py is fine, and the CI fails for two other reasons.

First, Pyright runs without mlflow installed, so it cannot find mlflow.tracking in the new test. fit.py has the same kind of import and uses # type: ignore[import-untyped] there. I think the same comment would fix it here.

Second, the minimum extras and tabr jobs install mlflow 2.22.0 with SQLAlchemy 2.1. That combination already fails without this PR. As far as I can see, no test built a real MLflow logger before this one, so CI did not show it. I opened #624 for it and a fix in #625 and I think that then these jobs pass with that fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ObservabilityConfig(mlflow_log_model=False) is ignored: checkpoints are still uploaded to MLflow

2 participants