[VladimirShitov]

Describe your changes

Fixes the wrong Spearman scores and the three failed runs of 0.2.0, and adds a --seed to all methods, controls and run_benchmark.

correlation: Spearman scores of CSR matrices

dynutils < 1.0.13 ranks a CSR matrix along the wrong axis. In 0.2.0 this hit ss_opm, which writes CSR, on every dataset, and every method on GEX -> ATAC, whose solution is stored as CSR.

dynutils 1.0.13 fixes this, but the metric also converts both matrices to CSC, so the scores don't depend on the dynutils version in the image. A new component test (test_prediction_layout.py) scores a prediction as CSR and as CSC, and checks both against dense ranks.

Scoring the 176 predictions of the rerun with and without this fix, only mean_spearman_per_cell and mean_spearman_per_gene change:

changed	largest difference
ss_opm, every dataset	0.03 – 0.33
every other method, GEX -> ATAC only	≤ 7.2e-4
all other scores	0

guanlab_dengkw_pm: failed on both pbmc_multiome datasets

• ATAC -> GEX failed with LinAlgError: Internal potrf return info = [49313]. The kernel ridge regression is now solved with numpy.linalg.solve() instead of scipy.linalg.cho_factor(). Where the Cholesky works, the coefficients agree to 6e-13.
• The failure depends on the OpenBLAS kernel: cho_factor() fails at 66k cells with SkylakeX on the Cascade Lake workers of de.NBI, and succeeds with Haswell (on an AMD CPU, or with OPENBLAS_CORETYPE=Haswell on the same worker).
• numpy.linalg.solve() works on those same workers: at 66k cells on a Cascade Lake worker with SkylakeX, cho_factor() failed with the same 49313, and solve() took 270 s on 30 threads, with a largest residual of 9e-13.
• GEX -> ATAC hit the 8 hours of hightime. The component now asks for veryhightime and veryhighmem. In the rerun, ATAC -> GEX peaked at 112 GB and took 2.5 h, and GEX -> ATAC took 6.6 h on 30 cores.

simple_mlp: failed on pbmc_multiome ATAC -> GEX

split() densified whole folds, 143 GB for this dataset. The inputs and targets now stay sparse and only one batch at a time is densified, drawing the same batches as before. Training now takes 25 min.

Seeding

Rerunning the benchmark changed many scores by up to 0.02, because several methods had unseeded steps.

• --seed is part of the method and control method apis (default 0). Wrappers forward it to their train component.
• run_benchmark --seed passes it to every method and control. A seed in a paramset takes precedence.
• src/utils/seed.py seeds python, numpy, torch and tensorflow, and switches them to deterministic GPU algorithms.
• The default is 0 everywhere, since viash doesn't let a component override an argument from the api. This changes the seed of guanlab_dengkw_pm (was 1000), novel (1), ss_opm and senkin_tmp (42) and scbutterfly (19193), so their predictions differ from those in the rerun above.
• CPU methods reproduce exactly anywhere. GPU-trained models only reproduce exactly on the same GPU model: A100 and H100 differ by ~1e-3.
• --seed doesn't change the data splits and SVDs that keep a fixed seed: the KFold of senkin_tmp, the batch SVD of ss_opm, the LSI of novel and the validation split of scbutterfly.

Checklist before requesting a review

• I have performed a self-review of my code

• Check the correct box. Does this PR contain:

  • Breaking changes
  • New functionality
  • Major changes
  • Minor changes
  • Bug fixes

• Proposed changes are described in the CHANGELOG.md

• CI Tests succeed and look good!

• numpy.linalg.solve() works on pbmc_multiome-sized systems on a de.NBI worker