[xlancelottx]

Describe your changes

Adds the deg metric component with two metrics that check whether a prediction keeps the differences of gene/protein expression between cell populations. Both metrics use the same differential-expression (DE) setup:

1. The test cells are clustered with Leiden on the ground truth's normalized layer (resolution 1). The same clusters are applied to the prediction.
2. Each cluster with at least 30 cells is tested against the remaining cells with a one-vs-rest Wilcoxon rank-sum test (sc.tl.rank_genes_groups, tie-corrected, Benjamini-Hochberg). The test runs on the ground truth and on the prediction separately.

Two metrics are calculted:

• deg_f1 (0 to 1, higher is better): A feature is a DEG of a cluster when it is up-regulated, has an adjusted p-value below 0.05, and has a difference of means above the target modality's threshold. Averaged over the clusters that have ground-truth DEGs. A cluster scores 0 when the prediction calls no DEGs.
• spearman_z (−1 to 1, higher is better): The Spearman correlation of the ground-truth and predicted DE z-scores over all features, averaged over the clusters. It compares each cluster's whole DE profile, not just the significant DEGs.

Checklist before requesting a review

• I have performed a self-review of my code

• Check the correct box. Does this PR contain:

  • Breaking changes
  • New functionality
  • Major changes
  • Minor changes
  • Bug fixes

• Proposed changes are described in the CHANGELOG.md

• CI Tests succeed and look good!

[rcannood]

Thanks for this! The core of the metric works well but it crashes on some of the method outputs.

I ran the metric in its image on all 8 test sets of the benchmark (1000 cells each), with predictions I generated myself. deg_f1 / spearman_z:

prediction	GEX	ADT	ATAC
solution	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
solution / 2	0.38-0.60 / 1.00	0.39-0.62 / 1.00	1.00 / 1.00
shuffled cells	0.00 / -0.00-0.04	0.00 / -0.03--0.02	0.00-0.02 / 0.05-0.11
mean per gene	0.00 / 0.00	0.00 / 0.00	0.00 / 0.00
zeros	0.00 / 0.00	0.00 / 0.00	0.00 / 0.00

So the good and bad predictions end up where they should, and at most 4.5 GB and under a minute per prediction fits the labels fine.

Two small things outside of the diff:

• typo in the PR description: "calculted"
• the changelog will conflict with #83, whose 0.2.1 entry should end up below 0.3.0

Once the crash is fixed, it would be great to see the scores on real method outputs, to check that they differ usefully between methods.

[rcannood]

This crashes with a KeyError on predictions without names, and file_prediction.yaml doesn't require any. lm, knnr_r, novel, simple_mlp and random_predict all write theirs without, so these would go missing from the results, and on de.NBI the run can still report success. CI doesn't catch it because its test prediction has names. mse and correlation pair by position, so how about only reordering when the names match?

Suggested change

	if not (ad_pred.obs_names.equals(ad_sol.obs_names) and ad_pred.var_names.equals(ad_sol.var_names)):
	if set(ad_pred.obs_names) == set(ad_sol.obs_names) and set(ad_pred.var_names) == set(ad_sol.var_names):

With this change a real random_predict output gets a score, and a shuffled solution that keeps its names still gets 1.0 / 1.0.

[rcannood]

Leftover from before the rename:

Suggested change

	raise ValueError(f"DEG recall needs at least 2 clusters with {par['min_group_size']} cells or more, found {len(groups)}")
	raise ValueError(f"The DEG metric needs at least 2 clusters with {par['min_group_size']} cells or more, found {len(groups)}")

[rcannood]

The cells of small clusters also get removed from the rest group, while the PR description says each cluster is tested against the remaining cells. On bmmc_cite/swap this drops 7 of 18 clusters and 172 of 1000 cells, and process_dataset deliberately favours rare cell types in the test set.

Passing groups= to rank_genes_groups skips testing them but keeps their cells in the rest (I checked this against a manual Mann-Whitney z-score). The effect calculation would need to use the same rest group.

[rcannood]

The p-value is two-sided and the effect is a difference of means, so a feature that is significantly lower by rank but higher on average counts as up-regulated. None of the 31k DEGs in the real solutions do this, so it's only a cheap safeguard, also for the predictions:

Suggested change

	deg_sol = (padj_sol < par["padj_threshold"]) & (effect_sol > effect_threshold)
	deg_sol = (padj_sol < par["padj_threshold"]) & (effect_sol > effect_threshold) & (z_sol > 0)
	deg_pred = (padj_pred < par["padj_threshold"]) & (effect_pred > effect_threshold) & (z_pred > 0)

[rcannood]

Skipping clusters without ground-truth DEGs means that DEGs a prediction calls in those clusters are never penalised. This happens for 5 clusters across 3 of the 8 test sets, all ADT or ATAC. Is that intended? If so, I'd mention it in the metric description.

[rcannood]

With a threshold of 0, ATAC F1 ignores the scale of the prediction: dividing the solution by 2 scores 1.00 on ATAC, but 0.38-0.62 on GEX and ADT. That contradicts "the prediction must be on the scale of the normalized layer" in the description. Either set a threshold for ATAC, or document that ATAC is the exception.

[rcannood]

Suggested change

	* `deg`: Added the `deg_f1` and `spearman_z` metrics, which compare one-vs-rest Wilcoxon DE results of Leiden clusters of the ground truth between the ground truth and the prediction (PR #XX).
	* `deg`: Added the `deg_f1` and `spearman_z` metrics, which compare one-vs-rest Wilcoxon DE results of Leiden clusters of the ground truth between the ground truth and the prediction (PR #84).