CompStats is an open-source Python package for the statistical analysis of the results of supervised-learning systems evaluated on a shared held-out test set, the setting of collaborative competitions (also called challenges). From the gold standard and the predictions of each system, it estimates by bootstrap the standard error and the confidence interval of every performance measure, tests whether the difference between the best system and each of the others is statistically significant, and corrects those tests for multiple comparisons. It implements the methodology of the article Analysis of systems' performance in natural language processing competitions (Pattern Recognition Letters, 2024), which complements the standard selection of a winner, usually based only on the ranking of the scores.
- Using the package: https://ingeotec.github.io/CompStats/
- Extending the package (developer reference): https://ingeotec.github.io/CompStats/dev/
The package is installed from PyPI with
pip install CompStatsor from conda-forge with
conda install --channel conda-forge CompStatsThe following example compares three classifiers on the digits problem.
from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import LinearSVC
from CompStats.metrics import f1_score
X, y = load_digits(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0)
svm = LinearSVC(random_state=0).fit(X_train, y_train)
score = f1_score(y_val, svm.predict(X_val), average='macro', name='Linear SVM')
score(RandomForestClassifier(random_state=0).fit(X_train, y_train).predict(X_val),
name='Random Forest')
score(GaussianNB().fit(X_train, y_train).predict(X_val), name='Naive Bayes')
score # point estimate and standard error of every system
score.ci # 95% percentile bootstrap confidence intervals
diff = score.difference() # paired differences against the best system
diff.p_value(correction='holm') # p-values adjusted for multiple comparisons
score.plot() # performance with confidence intervalsThe variable score holds the point estimate and the standard error of every system, and
score.ci returns the percentile confidence intervals obtained from the bootstrap
distribution. The call score.difference() computes the paired differences between the best
system and each of the others, and diff.p_value(correction='holm') returns the p-values of
those differences adjusted for multiple comparisons with Holm's method. Finally,
score.plot() draws the performance of the systems together with their confidence intervals.
When CompStats is used in academic work, the first reference below, which describes the methodology, should be cited. The second is the work it extends.
@article{Nava-Munoz2024,
author = {Nava-Mu{\~n}oz, Sergio and Graff, Mario and Escalante, Hugo Jair},
title = {Analysis of systems' performance in natural language processing competitions},
journal = {Pattern Recognition Letters},
volume = {186},
pages = {346--353},
year = {2024},
doi = {10.1016/j.patrec.2024.03.010},
url = {https://doi.org/10.1016/j.patrec.2024.03.010}
}
@inproceedings{Nava-Munoz2023,
author = {Nava-Mu{\~n}oz, Sergio and Graff Guerrero, Mario and Escalante, Hugo Jair},
title = {Comparison of Classifiers in Challenge Scheme},
booktitle = {Pattern Recognition},
series = {Lecture Notes in Computer Science},
volume = {13902},
pages = {89--98},
year = {2023},
publisher = {Springer Nature Switzerland},
doi = {10.1007/978-3-031-33783-3_9},
url = {https://doi.org/10.1007/978-3-031-33783-3_9}
}CompStats is distributed under the Apache License 2.0. See the LICENSE file.