Repository navigation
Description
Context
This issue builds on #49: the site workflow added here builds the Sphinx developer reference with
the strict command #49 makes pass, and it extends the test file CompStats/tests/test_docs.py
that #49 creates. It also assumes the workflows of #48 (test.yml, publish-pypi.yml) exist.
CompStats implements the methodology published in Analysis of systems' performance in natural
language processing competitions (Nava-Muñoz, Graff and Escalante, Pattern Recognition Letters,
2024, https://doi.org/10.1016/j.patrec.2024.03.010, preprint https://arxiv.org/abs/2403.04693),
which extends Comparison of Classifiers in Challenge Scheme (MCPR 2023,
https://doi.org/10.1007/978-3-031-33783-3_9). The user-facing documentation must therefore be
formal and must describe the methodology exactly as published.
The documentation has two halves that share one GitHub Pages site, the layout already used by
the maintainer's sibling projects scjn and dofjson:
- the Quarto user site at the root,
https://ingeotec.github.io/CompStats/(source:
quarto/), and - the Sphinx developer reference under
/dev/,
https://ingeotec.github.io/CompStats/dev/(source:docs/source/, made strict-clean by Make the Sphinx developer reference formal and strict-build clean #49).
State today:
quarto/_quarto.ymldefines a website with four pages, all Jupyter notebooks:
index.ipynb,pages/sklearn_users.ipynb,pages/competition_organizers.ipynb,
pages/multiple_testing.ipynb.execute: freeze: autois set andquarto/_freeze/is
committed. The navbar links "API Docs" to ReadTheDocs, which is being dropped.- The committed freeze is stale for all four pages. Quarto stores, in
quarto/_freeze/<page>/execute-results/html.json, ahashequal to the MD5 of the page's
source file; the last commit that touched the notebooks (adding the "Open in Colab" cells) did
not refresh it. A render without Python therefore fails ("Jupyter is not available"). - Each tutorial notebook starts with a code cell
!pip install -q CompStatsmeant for Colab. It
also runs when the site is rendered locally, which is an unwanted side effect. pages/sklearn_users.ipynbhas no random seed, so its outputs change on every execution, and
it stops before the comparison against the best system..github/workflows/publish.ymlis a manual (workflow_dispatch) job that builds a conda
environment and runsquarto publish gh-pages. It has never been set up: the repository has no
gh-pagesbranch and GitHub Pages is not enabled.README.rstis a long tutorial with stale output, a ReadTheDocs badge, a Colab badge and two
images that point at another branch (docs).pyproject.toml[project.urls]hasHomepage = "https://compstats.readthedocs.io".docs/CompStats.ipynbanddocs/CompStats_metrics.ipynbare tutorials of the previous site;
they point at ReadTheDocs and are superseded by the notebooks underquarto/pages/.- Open issues Documentar cómo previsualizar el sitio Quarto y la documentación de docs/ desde develop #45 (document how to preview both sites) and Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46 (automate publishing with a
freeze-verification script and agh-pagesbranch) describe earlier plans for this same work;
see them for history only. This issue supersedes both.
Facts verified while planning (do not re-derive them; they were tested on a scratch copy):
- Quarto 1.10.20 is installed at
/usr/local/bin/quarto. .venv/bin/python(Python 3.13) hasjupyter,nbformat,nbclient,ipykernel,docutils
and every runtime dependency. It has nopip. Do not install anything.- Rendering with execution works and takes about 75 seconds:
QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto(run from the repository root).
pages/competition_organizers.ipynbdownloads one small file fromraw.githubusercontent.com,
so this needs network. - Once the freeze is fresh,
quarto render quartoneeds no Python at all (about 5 seconds)
and does not modifyquarto/_freeze/. This is what CI relies on. - A first line
#| eval: falsein a code cell makes Quarto show the cell without executing it;
the line is an ordinary comment in Colab, where the cell still runs.
Scope
In scope:
quarto/:_quarto.yml, the home page (converted fromindex.ipynbtoindex.qmd), three new
prose pages (pages/methodology.qmd,pages/install.qmd,pages/citing.qmd), small edits to
the three tutorial notebooks, a refreshedquarto/_freeze/.README.rst(rewritten, short and formal).pyproject.toml[project.urls]only..github/workflows/website.yml(new) and removal of.github/workflows/publish.yml.- Removal of
docs/CompStats.ipynbanddocs/CompStats_metrics.ipynb. - New tests appended to
CompStats/tests/test_docs.py. CLAUDE.md: a## Documentationsection.- One short subsection added to the Development section of
docs/source/index.rst(how to
preview the user site).
Out of scope:
- Any change under
CompStats/other thanCompStats/tests/test_docs.py. - Enabling GitHub Pages, creating branches on the remote, publishing the site, tagging a release.
These are manual steps for the maintainer (listed under Decisions). - Renaming
quarto/towebsite/, changing the Quarto theme, restylingquarto/style.scss
beyond adding a rule a new page needs. - Tutorials that use the DataFrame-based functions of
CompStats.performance
(performance,difference,plot_difference,performance_multiple_metrics, ...). Every
code example on the user site and in the README uses thePerf-based syntax of this branch
(CompStats.metrics.<measure>(y_true, y_pred, name=...), calling the instance to add a system,
.difference(),.p_value(correction=...),.plot()). The LaTeX papers contain examples in
the older functional syntax; do not copy those. - Changing the code cells of
pages/competition_organizers.ipynband
pages/multiple_testing.ipynbbeyond the#| eval: falseline (their prose quotes numbers
produced by their fixed seeds). - A script that verifies the freeze (the pytest test below replaces it), a
gh-pagesbranch, a
CONTRIBUTINGfile. environment.yml,.devcontainer/,docs/source/*.png.
Implementation plan
Run every command from the repository root. Do the steps in order; step 6 (render) must come after
every edit under quarto/.
1. quarto/_quarto.yml
Change only the website.navbar block (keep project, lang, execute, site-url,
repo-url, page-footer, format exactly as they are):
navbar:
logo: images/ingeotec.png
left:
- href: index.qmd
text: Home
- href: pages/install.qmd
text: Install
- href: pages/methodology.qmd
text: Methodology
- href: pages/sklearn_users.ipynb
text: scikit-learn Users
- href: pages/competition_organizers.ipynb
text: Competition Organizers
- href: pages/multiple_testing.ipynb
text: Multiple Testing
- href: pages/citing.qmd
text: Citing
- href: https://ingeotec.github.io/CompStats/dev/
text: Developer docs
right:
- icon: github
href: https://github.com/INGEOTEC/CompStats
aria-label: GitHub2. Home page: quarto/index.qmd replaces quarto/index.ipynb
The home page has no code, so it becomes a plain-text .qmd file (easier to edit and review, and
it needs no freeze entry).
-
git rm quarto/index.ipynbandgit rm -r quarto/_freeze/index. -
Create
quarto/index.qmd. Front matter:--- title: "CompStats" subtitle: "Bootstrap-based statistical comparison of supervised-learning systems evaluated on a shared test set" toc: false number-sections: false ---
-
Body, in this order. The CSS classes
.home-lead,.pillars/.pillarand
.installs/.installalready exist inquarto/style.scss; use them with Quarto fenced divs
(::: {.home-lead}...:::), as the current notebook does (read its markdown cells before
deleting it and reuse what is still accurate).-
Lead (
.home-lead, one paragraph): what CompStats is. Content: an open-source Python
package for the statistical analysis of the results of supervised-learning systems evaluated
on a shared held-out test set (the setting of collaborative competitions, also called
challenges); from the gold standard and the predictions of each system it estimates, by
bootstrap, the standard error and the confidence interval of every performance measure, tests
whether the difference between the best system and each of the others is statistically
significant, and corrects those tests for multiple comparisons. -
Motivation (one or two paragraphs, formal, third person). Content: competitions rank
systems by a point estimate computed on one test set; differences between consecutive systems
are often in the hundredths or thousandths, so a higher score does not imply that a system
would remain better on the population the test set was sampled from; organizers have only the
predictions (not the algorithms), a single dataset of fixed size and few submissions, which
rules out the classical tests that require retraining or several datasets. CompStats
implements the methodology of the Pattern Recognition Letters article (cite it with its DOI
link and the arXiv link), which was validated on eight natural language processing
competitions covering classification and regression tasks; the methodology itself is not
specific to NLP. -
Three pillars (
.pillars): Follows the scikit-learn metrics API (a function of
CompStats.metricstakes(y_true, y_pred)like itssklearn.metricsnamesake and returns
aPerfobject instead of a float); Uncertainty for every measure (standard error and
percentile confidence interval from the bootstrap distribution); Paired significance
testing (every system is evaluated on the same bootstrap samples, so differences are paired;
p-values come from the bootstrap distribution of the difference, with optional Bonferroni,
Holm or Benjamini-Hochberg correction). -
Quickstart: the two install commands in one sentence linking to
Install, then this example in a non-executed fenced block (```python):from sklearn.datasets import load_digits from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.naive_bayes import GaussianNB from sklearn.svm import LinearSVC from CompStats.metrics import f1_score X, y = load_digits(return_X_y=True) X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0) svm = LinearSVC(random_state=0).fit(X_train, y_train) score = f1_score(y_val, svm.predict(X_val), average='macro', name='Linear SVM') score(RandomForestClassifier(random_state=0).fit(X_train, y_train).predict(X_val), name='Random Forest') score(GaussianNB().fit(X_train, y_train).predict(X_val), name='Naive Bayes') score # point estimate and standard error of every system score.ci # 95% percentile bootstrap confidence intervals diff = score.difference() # paired differences against the best system diff.p_value(correction='holm') # p-values adjusted for multiple comparisons score.plot() # performance with confidence intervals
Before writing it into the page, save it to a temporary file outside the repository and run
it withMPLBACKEND=Agg .venv/bin/python <file>; it must finish without an exception. -
Where to go next: one paragraph linking Install, Methodology, the three tutorials
(say that these three pages are Jupyter notebooks that can be opened in Colab from the badge
at their top), Citing, and the developer docs (https://ingeotec.github.io/CompStats/dev/). -
Colophon (
## Colophon {.unnumbered}): keep the existing paragraph.
-
3. New prose pages (no executable code, plain .qmd)
Formal register in all three: impersonal, present tense, definitions before use, every formula
defined, no exclamation marks, no promotional adjectives. Use $...$/$$...$$ for mathematics.
Code shown on these pages goes in non-executed fenced blocks (```python, never
```{python}).
quarto/pages/install.qmd (title: "Install"): requirements (Python; the runtime
dependencies exactly as listed in pyproject.toml: numpy, scikit-learn >= 1.3.0, pandas,
seaborn >= 0.13.0, statsmodels; tqdm is optional and only adds progress bars); installing from
PyPI (pip install CompStats); from conda-forge (conda install --channel conda-forge CompStats); from source for development (git clone https://github.com/INGEOTEC/CompStats.git,
pip install -e ".[test]"); how to check the installation
(python -c "import CompStats; print(CompStats.__version__)").
quarto/pages/methodology.qmd (title: "Methodology", subtitle "The statistical basis of
CompStats"). Sections, each with its definition, its formula and the CompStats call that computes
it. This is the content; write it as connected prose, not as a bullet list:
-
Setting. A competition provides a gold standard
$y$ of size$n$ kept by the organizers and
the predictions$\hat{y}_1, \ldots, \hat{y}_m$ of$m$ systems; a performance measure
$s(y, \hat{y})$ ranks them. The quantity of interest is the performance$\theta_j$ of system
$j$ on the population from which the test set is assumed to be a random sample; the value
computed on the test set is an estimate of it. Constraints of the setting: one dataset of
fixed size, few submissions, access to predictions only. -
Bootstrap. Bootstrapping draws conclusions about the sampling distribution of a statistic by
resampling the data with replacement (Chernick and LaBudde, 2011; Efron and Tibshirani, 1994).
$B$ samples of$n$ indices are drawn with replacement; the measure is computed on each sample
for every system, giving$s_j^{*1}, \ldots, s_j^{*B}$ . The article uses$B = 10{,}000$ ; the
package default isnum_samples=500and it is a parameter of every function of
CompStats.metricsand ofPerf. -
Standard error and confidence interval. The standard error is the standard deviation of the
bootstrap distribution (Perf.se). The$1-\alpha$ confidence interval is the percentile
interval $[q_{\alpha/2}(s_j^{}),, q_{1-\alpha/2}(s_j^{})]$ with$\alpha = 0.05$ by default
(Perf.ci). -
Comparison through independent samples. If the confidence intervals of two systems do not
overlap, their performances differ significantly; if they overlap, the comparison is
inconclusive, which motivates the paired comparison.Perf.plot()draws the ordered
intervals. -
Comparison through paired samples. Every bootstrap sample contains the gold standard and the
prediction of every system for the same elements, so the difference in performance can be
computed per sample (the paired bootstrap): $\delta^{} = s^{}(y, \hat{y}_A) - s^{*}(y,
\hat{y}_B)$. If the confidence interval of the difference contains zero, the hypothesis that
both systems perform equally cannot be rejected.Perf.difference()returns the differences
against the best system (or against the system named bywrt), andDifference.plot()draws
their intervals. -
Statistical significance. For systems
$A$ and$B$ with observed difference
$\delta(x) = \theta_A(x) - \theta_B(x)$ on the test set$x$ , the null hypothesis$H_0$ states
that$A$ is not better than$B$ in the population. The p-value is
$p(\delta(X) > \delta(x) \mid H_0, x)$ , where$X$ ranges over test sets of size$n$ . Following
Berg-Kirkpatrick, Burkett and Klein (2012), it is estimated as the fraction of bootstrap
samples in which the difference exceeds$2\delta(x)$ : the bootstrap distribution is centred on
$\delta(x)$ , not on zero, because the samples are drawn from$x$ .Difference.p_value()
returns these p-values. -
Multiple comparisons. Comparing the best system with each of the other
$m-1$ tests the
hypotheses$H_0^j: \theta_1 \le \theta_j$ against$H_1^j: \theta_1 > \theta_j$ for
$j = 2, \ldots, m$ simultaneously, which inflates the probability of a Type I error. Three
corrections, all of which adjust the p-values: Bonferroni (conservative; multiplies each
p-value by the number of comparisons), Holm's step-down procedure (adjusts the p-values
sequentially; Holm, 1979) and Benjamini-Hochberg (controls the false discovery rate; Benjamini
and Hochberg, 1995). In CompStats:Difference.p_value(correction='bonferroni'),'holm',
'fdr_bh', or any other method accepted bystatsmodels.stats.multitest.multipletests; the
samecorrectionargument is accepted byPerf.plotandDifference.plot. With several
measures in onePerf, each measure is corrected as its own family. -
Competitiveness of a competition. The article characterizes a competition with: the number
of systems statistically tied with the winner; the coefficient of variation of the systems'
performance,$CV = 100, s_x / \bar{x}$ (smaller means more competitive); the distance between
the winner and the system in the middle of the table,$|win - med|$ ; and the possible
percentage improvement,$PPI = 100,(1 - x_{winner})$ for bigger-is-better scores bounded by 1
(a low value means little room for improvement). State that these indicators are computed by
the earlier DataFrame-based interface (CompStats.performance.performance_multiple_metrics)
and link to the developer docs for it; show no code for it. -
Scope and assumptions. The inference assumes the test set can be treated as a random sample
of the population of interest; intervals and tests must be read in light of how the test set
was built. The methodology applies to any measure computed from a gold standard and
predictions, for classification and regression; measures where smaller is better (for example
mean absolute error) are handled through theBiB(bigger is better) flag, which every
function ofCompStats.metricssets for its measure. - A closing "References" list: the two CompStats papers and the five references listed in the
Citing section ofdocs/source/index.rst(copy the entries from there).
quarto/pages/citing.qmd (title: "Citing"): one sentence asking users to cite the Pattern
Recognition Letters article when CompStats is used in academic work, then the two BibTeX entries
copied verbatim from the Citing section of docs/source/index.rst (do not add fields), each
introduced by one sentence (the first is the methodology; the second is the conference paper it
extends), then the authors and affiliations (Sergio Nava-Muñoz, CIMAT, Aguascalientes, Mexico;
Mario Graff, INFOTEC, Aguascalientes, Mexico; Hugo Jair Escalante, INAOE, Puebla, Mexico), and the
license (Apache 2.0).
4. Tutorial notebooks: small scripted edits
Notebook files are JSON; do not edit them by hand. Write a throwaway Python script (outside the
repository, do not commit it), run it with .venv/bin/python, and let it assert every change it
makes. It must do exactly this:
import nbformat
from pathlib import Path
PAGES = Path("quarto/pages")
# (a) The Colab-only install cell is shown but not executed when the site is rendered.
for path in sorted(PAGES.glob("*.ipynb")):
nb = nbformat.read(path, as_version=4)
cells = [c for c in nb.cells if c.cell_type == "code" and "!pip install" in c.source]
assert len(cells) == 1, path
if not cells[0].source.startswith("#| eval: false"):
cells[0].source = "#| eval: false\n" + cells[0].source
nbformat.validate(nb)
nbformat.write(nb, path)
# (b) scikit-learn Users: reproducible, and it ends with the comparison against the best.
path = PAGES / "sklearn_users.ipynb"
nb = nbformat.read(path, as_version=4)
def replace_once(old, new):
hits = [c for c in nb.cells if c.cell_type == "code" and old in c.source]
assert len(hits) == 1 and hits[0].source.count(old) == 1, old
hits[0].source = hits[0].source.replace(old, new)
replace_once("from sklearn.svm import LinearSVC",
"import numpy as np\nfrom sklearn.svm import LinearSVC")
replace_once("train_test_split(X, y, test_size=0.3)",
"train_test_split(X, y, test_size=0.3, random_state=0)")
replace_once("LinearSVC()", "LinearSVC(random_state=0)")
replace_once("RandomForestClassifier()", "RandomForestClassifier(random_state=0)")
replace_once("HistGradientBoostingClassifier()", "HistGradientBoostingClassifier(random_state=0)")
replace_once("score = f1_score(", "np.random.seed(0)\nscore = f1_score(")
last = nb.cells.pop() # the closing "see also" markdown cell
assert last.cell_type == "markdown"
nb.cells += [
nbformat.v4.new_markdown_cell(SEED_NOTE),
nbformat.v4.new_markdown_cell(DIFFERENCE_INTRO),
nbformat.v4.new_code_cell("diff = score.difference()\ndiff"),
nbformat.v4.new_markdown_cell(P_VALUE_INTRO),
nbformat.v4.new_code_cell("diff.p_value(correction='holm')"),
nbformat.v4.new_markdown_cell(PLOT_INTRO),
nbformat.v4.new_code_cell("diff.plot()"),
last,
]
nbformat.validate(nb)
nbformat.write(nb, path)Define the four text constants yourself (plain markdown strings, formal register):
SEED_NOTE: one sentence saying that the data split, the classifiers and NumPy's global random
state (from which the bootstrap samples are drawn) are seeded so the page is reproducible.DIFFERENCE_INTRO: heading## Comparison against the best systemand two sentences:
Perf.differencecomputes, on the same bootstrap samples, the difference between the best
system and each of the others, and the object it returns prints the p-value of each difference.P_VALUE_INTRO: heading## Correction for multiple comparisonsand two sentences: three
systems are compared with the best at once, so the p-values are adjusted, here with Holm's
method; link to[Multiple Testing](multiple_testing.ipynb).PLOT_INTRO: one sentence:Difference.plotdraws the confidence interval of each difference;
an interval that contains zero means the system is not statistically different from the best.
Do not write, in any markdown cell of this notebook, which classifier wins or any number: the
text must stay true whatever the outputs are.
After running the script: git diff --stat quarto/pages must list the three notebooks, and
pages/competition_organizers.ipynb and pages/multiple_testing.ipynb must differ from develop
by exactly one added source line each (#| eval: false).
5. README.rst, pyproject.toml, legacy notebooks
README.rst — replace the whole file. reStructuredText, about 80-120 lines, formal:
- Title
CompStatsand badges: tests (actions/workflows/test.yml), Coveralls, PyPI, the three
conda-forge badges (copy these six from the current file). Drop the ReadTheDocs badge and
the Colab badge. - One paragraph: the same definition as the home page lead, plus the sentence that it implements
the methodology of the Pattern Recognition Letters article (link the DOI). Documentation: two bullets, "Using the package: https://ingeotec.github.io/CompStats/" and
"Extending the package (developer reference):
https://ingeotec.github.io/CompStats/dev/".Installation: thepipand thecondacommands.Quickstart: the same example as the home page, in a.. code:: pythonblock, followed by
one sentence per result (score,score.ci,diff.p_value(...),score.plot()). No printed
outputs and no images.Citing: the two BibTeX entries copied verbatim fromdocs/source/index.rst, in a
.. code:: bibtexblock.License: Apache License 2.0, pointing at theLICENSEfile.
The README is the long description on PyPI, so it must be valid reStructuredText:
.venv/bin/python -m docutils --halt=warning README.rst /dev/nullmust exit 0 with no output.
pyproject.toml — replace the [project.urls] table with:
[project.urls]
Homepage = "https://ingeotec.github.io/CompStats/"
Documentation = "https://ingeotec.github.io/CompStats/"
"Developer documentation" = "https://ingeotec.github.io/CompStats/dev/"
Repository = "https://github.com/INGEOTEC/CompStats"
Issues = "https://github.com/INGEOTEC/CompStats/issues"Legacy notebooks — git rm docs/CompStats.ipynb docs/CompStats_metrics.ipynb.
Afterwards git grep -il readthedocs must print nothing (if a file still matches, fix that file;
quarto/_freeze is regenerated in the next step).
6. Refresh the freeze and render
QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quartoThis executes the three tutorial notebooks (about 75 seconds, needs network for one small
download) and writes the site to quarto/_site/ (git-ignored). It prints one [n/7] line per
page and ends with Output created: _site/index.html. If it stops with an error in a notebook
cell, the error is in an edit of step 4: fix the notebook through the script, not the freeze.
Then confirm that the freeze is complete and that CI will not need Python:
env QUARTO_PYTHON=/nonexistent/python quarto render quarto
git status --porcelain quarto/_freezeThe first command must end with Output created: _site/index.html and must not print
Starting python3 kernel (it prints two harmless WARN lines about QUARTO_PYTHON); if it tries
to start a kernel, a page's freeze is stale: run the first render command again. Record the state
with git add -A quarto/_freeze quarto/pages quarto/index.qmd quarto/_quarto.yml (-A matters:
figure files are renamed when cells are added, and the old ones must be removed from the index).
Run the second command once more after git add: git status --porcelain quarto/_freeze must
then show only staged changes (no line starting with a space or ??).
Open nothing in a browser; instead check the output exists:
ls quarto/_site/index.html quarto/_site/pages/methodology.html quarto/_site/pages/install.html quarto/_site/pages/citing.html quarto/_site/pages/sklearn_users.html.
Never commit quarto/_site/ or quarto/.quarto/.
7. .github/workflows/website.yml (new) and removal of publish.yml
git rm .github/workflows/publish.yml, then create .github/workflows/website.yml with exactly:
name: Publish website
# One GitHub Pages site, two sources: the Quarto user site (quarto/) at the
# root and the Sphinx developer reference (docs/) under /dev/. Pull requests
# and pushes to develop only build both; pushes to main also deploy them.
#
# The Quarto pages that are notebooks are never executed here: Quarto reads
# their results from the committed quarto/_freeze/ cache. If a notebook was
# changed without refreshing the cache, the render step fails ("Jupyter is
# not available"); the fix is to render locally and commit quarto/_freeze/.
on:
pull_request:
paths:
- "quarto/**"
- "docs/**"
- "CompStats/**"
- "pyproject.toml"
- ".github/workflows/website.yml"
push:
branches: [main, develop]
paths:
- "quarto/**"
- "docs/**"
- "CompStats/**"
- "pyproject.toml"
- ".github/workflows/website.yml"
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: quarto-dev/quarto-actions/setup@v2
with:
# Keep equal to the version used to produce quarto/_freeze/.
version: 1.10.20
- run: quarto render quarto
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install docs and package dependencies
run: |
pip install -r docs/requirements.txt
# Not --no-deps: conf.py imports CompStats, and autodoc imports its
# dependencies with it.
pip install -e .
- name: Build the developer docs strictly into the site
run: python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev
- uses: actions/upload-pages-artifact@v3
with:
path: quarto/_site
deploy:
if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
concurrency:
group: website
cancel-in-progress: true
steps:
- id: deployment
uses: actions/deploy-pages@v4(actions/configure-pages is deliberately absent: it fails on a repository where Pages is not
enabled yet, and nothing in the build needs its outputs.)
Check locally that the two halves fit together, exactly as the workflow does it (after step 6, so
quarto/_site/ exists):
.venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev
ls quarto/_site/dev/index.html8. Documentation for contributors (absorbs #45)
CLAUDE.md — add a ## Documentation section after ## Releasing stating:
- the two halves and their URLs (user site from
quarto/, developer reference from
docs/source/published under/dev/), both deployed by.github/workflows/website.ymlon
pushes tomain; - user-facing examples use the
Perf-based API only; - preview commands:
QUARTO_PYTHON="$PWD/.venv/bin/python" quarto preview quarto(live reload)
andQUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto(static, into
quarto/_site/); - the freeze rule: CI never executes notebooks; after changing any
quarto/**/*.ipynb, render
locally and commitquarto/_freeze/withgit add -A quarto/_freezein the same commit;
CompStats/tests/test_docs.pyfails when a notebook and its freeze entry disagree; - notebooks are edited through
nbformat(or Jupyter), never by hand-editing JSON; - the Colab-only install cell keeps
#| eval: falseas its first line.
Also update the performance.py and utils.py bullets if they mention ReadTheDocs (they do not
today; just make sure git grep -i readthedocs is empty).
docs/source/index.rst — in the Development section that #49 added, append a short
"User site" paragraph with the two preview commands and the freeze rule (three or four
sentences). Rebuild the developer docs strictly afterwards.
9. Commit message
End the commit message body with the two lines Closes #45 and Closes #46 (this issue
supersedes them), in addition to whatever reference to this issue the run normally adds.
Acceptance criteria
-
quarto/index.ipynbandquarto/_freeze/index/no longer exist;quarto/index.qmd,
quarto/pages/install.qmd,quarto/pages/methodology.qmdandquarto/pages/citing.qmd
exist and are in the navbar, together with a "Developer docs" link to
https://ingeotec.github.io/CompStats/dev/. -
env QUARTO_PYTHON=/nonexistent/python quarto render quartosucceeds without starting a
Jupyter kernel and leavesgit status --porcelain quarto/_freezefree of unstaged changes. - For every
quarto/**/*.ipynbpage, thehashin
quarto/_freeze/<path without .ipynb>/execute-results/html.jsonequals the MD5 of the
notebook file. - Each tutorial notebook has exactly one
!pip installcell and it starts with
#| eval: false. -
pages/sklearn_users.ipynbis seeded and ends with the comparison against the best system
(score.difference(),diff.p_value(correction='holm'),diff.plot()), and its prose
names no winner and quotes no number. - The methodology page has the nine sections of step 3 with their formulas, names
Berg-Kirkpatrick et al. (2012) for the p-value estimator and Bonferroni, Holm and
Benjamini-Hochberg for the corrections, and shows no code ofCompStats.performance. - No code example on the user site or in the README imports from
CompStats.performanceor
callsperformance(,difference(,plot_difference(as free functions. -
README.rstpasses.venv/bin/python -m docutils --halt=warning README.rst /dev/null,
links both documentation URLs, contains the DOI10.1016/j.patrec.2024.03.010, and has no
image and no Colab or ReadTheDocs badge. -
git grep -il readthedocsprints nothing. -
pyproject.toml[project.urls]matches step 5; nothing else in the file changed. -
.github/workflows/publish.ymlis gone;.github/workflows/website.ymlexists with the
content above. -
docs/CompStats.ipynbanddocs/CompStats_metrics.ipynbare gone. -
.venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/devends
withbuild succeeded. -
quarto/_site/,quarto/.quarto/anddocs/build/are not committed. -
CLAUDE.mdhas a## Documentationsection with the preview commands and the freeze rule. -
.venv/bin/python -m pytest CompStatsis green.
Tests
Append to CompStats/tests/test_docs.py (created by #49; keep its existing tests and its
module-level skip). Add near the top:
import hashlib
import json
SITE = ROOT / "quarto"
_SKIPPED_DIRS = {"_site", "_freeze", ".quarto"}
def _site_files(suffix):
"""Source pages of the Quarto site with the given suffix"""
return sorted(p for p in SITE.rglob(f"*{suffix}")
if not _SKIPPED_DIRS & set(p.relative_to(SITE).parts))Required test functions (skip each with pytest.mark.skipif(not SITE.is_dir(), ...)):
test_freeze_matches_notebooks: for every_site_files(".ipynb"), the file
SITE / "_freeze" / <relative path without suffix> / "execute-results" / "html.json"exists
and its"hash"equalshashlib.md5(notebook.read_bytes()).hexdigest(). The assertion message
must say how to fix it: render locally with
QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quartoand commitquarto/_freeze/.
Assert as well that there is at least one notebook (so the test cannot pass vacuously).test_no_orphan_freeze_entries: every directory underquarto/_freezethat contains an
execute-results/html.jsoncorresponds to an existing notebook (this catches the leftover
quarto/_freeze/index).test_install_cells_are_not_executed: in every notebook (parse it withjson, no extra
dependency), exactly one code cell contains!pip install, and its source starts with
#| eval: false.test_navbar_pages_exist: everyhref:value inquarto/_quarto.yml(extract them with a
regular expression, do not importyaml) that does not start withhttpnames an existing
file underquarto/; andindex.qmd,pages/install.qmd,pages/methodology.qmd,
pages/citing.qmdand the three notebooks are all among them.test_user_docs_use_the_perf_api: no.qmdpage, no notebook code cell and notREADME.rst
containsfrom CompStats.performanceorfrom CompStats import performance.test_methodology_page_content:pages/methodology.qmdcontains (case-insensitive) each of
bootstrap,confidence interval,paired,p-value,Bonferroni,Holm,
Benjamini,Berg-Kirkpatrick,coefficient of variation,PPI, and the string
10.1016/j.patrec.2024.03.010.test_no_readthedocs_anywhere: list tracked files withgit ls-files -z(skip the test if
gitis unavailable orROOTis not a checkout), read each as UTF-8 (skip files that are not
text), and assert none contains the forbidden word, built as"readthe" + "docs",
case-insensitive.test_readme:README.rstcontainshttps://ingeotec.github.io/CompStats/,
https://ingeotec.github.io/CompStats/dev/and10.1016/j.patrec.2024.03.010, contains no
.. image::directive that is not a badge (simplest check: it does not contain
raw/docs/docs), and renders without warnings:pytest.importorskip("docutils"), then
docutils.core.publish_string(text, writer_name="html", settings_overrides={"halt_level": 2, "report_level": 2})must not raise.test_site_workflow:.github/workflows/website.ymlexists,publish.ymldoes not; the
workflow text containsquarto render quarto,
python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev,
actions/deploy-pagesand does not containcondaorgh-pages.test_project_urls:pytest.importorskip("tomllib");[project.urls]inpyproject.toml
hasHomepageandDocumentationequal tohttps://ingeotec.github.io/CompStats/.
Commands that must be green before committing, in this order:
env QUARTO_PYTHON=/nonexistent/python quarto render quarto
.venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev
.venv/bin/python -m docutils --halt=warning README.rst /dev/null
.venv/bin/python -m pytest CompStatsDurations: about 5 seconds, about 10 seconds plus inventory downloads, instant, and about
2.5 minutes (ends with a line like 99 passed). The one slow step of this issue is the executing
render of step 6 (about 75 seconds); it is finished when it prints
Output created: _site/index.html.
Decisions
- One GitHub Pages site, deployed with
actions/deploy-pages(as inscjn/dofjson), not a
gh-pagesbranch: this supersedes the plan of Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46. - CI renders from the committed freeze and never executes notebooks; a stale freeze fails the
build loudly, and a pytest test (test_freeze_matches_notebooks) catches it earlier. This
replaces the verification script proposed in Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46. - Deploy only from
main; build on pull requests and ondevelop:developis the
integration branch, and the public site must reflect released code. actions/configure-pagesomitted: it fails while Pages is not enabled, which would turn
every pull request red before the maintainer enables it.- The home page becomes
index.qmd: it has no code, so a notebook only added a freeze entry
and a Colab badge with nothing to run. - New prose pages are
.qmdwithout executable code: nothing to execute, nothing to freeze. sklearn_users.ipynbis seeded and extended with the comparison against the best: a formal
tutorial must be reproducible and must reach the significance test; its prose avoids naming
winners or numbers so it cannot disagree with its outputs.#| eval: falseon the Colab install cell: rendering the site must not runpipon the
author's machine; Colab ignores the comment.- User-facing examples use the
PerfAPI only (maintainer's choice); the DataFrame-based
functions appear only in the developer reference and, by name, in the competitiveness section. - Directory stays
quarto/(not renamed towebsite/): the Colab links and existing history
use that path. - README stays reStructuredText and becomes short: tutorials live on the site; a README that
duplicates them goes stale, as the current one did. - Legacy notebooks under
docs/are deleted: they document the previous API and site; they
remain available in the git history and on thedocsbranch. - Quarto pinned to 1.10.20 in CI: the version installed locally, which produced the freeze.
- Manual steps left to the maintainer (not part of this issue): enable GitHub Pages with
source "GitHub Actions" in the repository settings before the first push tomain; redirect or
retire the ReadTheDocs project; tag and publish 0.2.0 to PyPI, which the Colab install cells
need because the tutorials use the API of this branch.
Planned against develop at d2c6feade9eac5eb039b405ac272f70fc66458ce.
Activity
mgraffg commentedon Nov 19, 2026
Implemented on branch
issues-48-49-50, commit 8608cbb (the message ends withCloses #45/Closes #46).index.qmd(replacesindex.ipynband its freeze), newinstall.qmd,methodology.qmd,citing.qmd;#| eval: falseon the Colab install cells;sklearn_users.ipynbseeded and extended withdifference(),p_value(correction="holm"),plot()(prose names no winner and quotes no number); the two other notebooks differ by exactly one added line.env QUARTO_PYTHON=/nonexistent/python quarto render quartosucceeded without starting a kernel andgit status --porcelain quarto/_freezeshows only staged changes.README.rstrewritten (121 lines, validates with docutils--halt=warning),pyproject.tomlURLs updated, legacydocs/*.ipynbandpublish.ymlremoved,website.ymladded exactly as specified.CLAUDE.md## Documentationsection and a "User site" paragraph indocs/source/index.rst;git grep -il readthedocsis empty.quarto/_site/devsucceeded;.venv/bin/python -m pytest CompStatsgreen (exit 0); 10 new tests appended toCompStats/tests/test_docs.py.test_no_legacy_hosting_in_developer_docs,test_no_legacy_hosting_anywhere) and one docstring was reworded.mgraffg commentedon Nov 20, 2026
Review fix-1 (commit f4b315a, branch
issues-48-49-50): wording corrected to agree with the article (no "developed by" attribution, footer names the authors, no "validated" claim, correct argument for why classical tests do not apply, purpose of the methodology, hypotheses and 0.05 decision rule, three aspects of competitiveness plus PPI separately, reasons for multiple-comparison correction, three added references) inquarto/index.qmd,quarto/_quarto.yml,quarto/pages/methodology.qmdandREADME.rst; new testtest_site_follows_the_article. Render from the freeze without Python succeeds (freeze unchanged); full suite: 100 passed (run with MPLBACKEND=Agg).mgraffg commentedon Nov 20, 2026
Review fix-2 (commit 5c4f897, branch
issues-48-49-50): the user site now cites through Quarto's bibliography (quarto/references.bib,bibliography: references.bib); every citation is a link to its complete reference, volume and pages of the 2024 article were added to the BibTeX inciting.qmdandREADME.rst, andtest_site_bibliographywas added. Render from the freeze without Python succeeds withquarto/_freezeunchanged; full suite: 101 passed (MPLBACKEND=Agg).mgraffg commentedon Nov 20, 2026
Review fix-3 (commit d7b6a3e, branch
issues-48-49-50): the preprint link of the 2024 article now lives in its bibliography entry (notefield) and is printed by a small site citation style,quarto/compstats.csl, instead of being typed in the home page prose; Efron1994 publisher fixed with double braces. Render from the freeze without Python succeeds withquarto/_freezeunchanged; full suite: 101 passed (MPLBACKEND=Agg).mgraffg commentedon Nov 20, 2026
Review fix-4 (commit 95fe5b3, branch
issues-48-49-50): removedcode-fold: truefromquarto/_quarto.ymlso the code of the tutorial pages is always visible; addedtest_site_code_is_visible. Render from the freeze without Python succeeds withquarto/_freezeunchanged; full suite: 102 passed (MPLBACKEND=Agg).mgraffg commentedon Nov 20, 2026
Review fix-5 (commit 0763e70, branch
issues-48-49-50): the Colab install cell of the three notebooks now guards withtry: import CompStats / except ImportErrorand installs withuv pip install(pip fallback), still#| eval: false;sklearn_users.ipynbexplainsstatistic/seand the figure of 'Rest of the classifiers' and gains a Regression section;quarto/_freezeregenerated; test and CLAUDE.md updated. Inserted sentences checked against the regenerated outputs. Render without Python leaves the freeze unchanged; full suite: 103 passed (MPLBACKEND=Agg).mgraffg commentedon Nov 20, 2026
Review fix-6 (commit 5abc121, branch
issues-48-49-50): the prose of the user site and the README was rewritten in the punctuation style of the article (semicolons and dashes replaced by separate sentences, connectors or subordination, colons only before enumerations). Only markdown cells of the notebooks changed (code cells verified identical),quarto/_freezeregenerated, andtest_site_prose_styleplus a rule in CLAUDE.md added. The notebook edits kept the maintainer's later commit a57a565. The test helper now ignores.ipynb_checkpoints, which a local Jupyter session had created. Render without Python leaves the freeze unchanged; full suite: 104 passed (MPLBACKEND=Agg).Formalize documentation, CI and PyPI publishing (#48, #49, #50) (#51)
mgraffg commentedon Nov 20, 2026
Implemented in #51, merged into develop. Commit 8608cbb.
Develop (#52)
Develop (#53)