Skip to content

Formalize the Quarto user site and README, and publish both doc sites to GitHub Pages #50

Closed

Description

@mgraffg

Context

This issue builds on #49: the site workflow added here builds the Sphinx developer reference with
the strict command #49 makes pass, and it extends the test file CompStats/tests/test_docs.py
that #49 creates. It also assumes the workflows of #48 (test.yml, publish-pypi.yml) exist.

CompStats implements the methodology published in Analysis of systems' performance in natural
language processing competitions
(Nava-Muñoz, Graff and Escalante, Pattern Recognition Letters,
2024, https://doi.org/10.1016/j.patrec.2024.03.010, preprint https://arxiv.org/abs/2403.04693),
which extends Comparison of Classifiers in Challenge Scheme (MCPR 2023,
https://doi.org/10.1007/978-3-031-33783-3_9). The user-facing documentation must therefore be
formal and must describe the methodology exactly as published.

The documentation has two halves that share one GitHub Pages site, the layout already used by
the maintainer's sibling projects scjn and dofjson:

State today:

  • quarto/_quarto.yml defines a website with four pages, all Jupyter notebooks:
    index.ipynb, pages/sklearn_users.ipynb, pages/competition_organizers.ipynb,
    pages/multiple_testing.ipynb. execute: freeze: auto is set and quarto/_freeze/ is
    committed. The navbar links "API Docs" to ReadTheDocs, which is being dropped.
  • The committed freeze is stale for all four pages. Quarto stores, in
    quarto/_freeze/<page>/execute-results/html.json, a hash equal to the MD5 of the page's
    source file; the last commit that touched the notebooks (adding the "Open in Colab" cells) did
    not refresh it. A render without Python therefore fails ("Jupyter is not available").
  • Each tutorial notebook starts with a code cell !pip install -q CompStats meant for Colab. It
    also runs when the site is rendered locally, which is an unwanted side effect.
  • pages/sklearn_users.ipynb has no random seed, so its outputs change on every execution, and
    it stops before the comparison against the best system.
  • .github/workflows/publish.yml is a manual (workflow_dispatch) job that builds a conda
    environment and runs quarto publish gh-pages. It has never been set up: the repository has no
    gh-pages branch and GitHub Pages is not enabled.
  • README.rst is a long tutorial with stale output, a ReadTheDocs badge, a Colab badge and two
    images that point at another branch (docs).
  • pyproject.toml [project.urls] has Homepage = "https://compstats.readthedocs.io".
  • docs/CompStats.ipynb and docs/CompStats_metrics.ipynb are tutorials of the previous site;
    they point at ReadTheDocs and are superseded by the notebooks under quarto/pages/.
  • Open issues Documentar cómo previsualizar el sitio Quarto y la documentación de docs/ desde develop #45 (document how to preview both sites) and Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46 (automate publishing with a
    freeze-verification script and a gh-pages branch) describe earlier plans for this same work;
    see them for history only. This issue supersedes both.

Facts verified while planning (do not re-derive them; they were tested on a scratch copy):

  • Quarto 1.10.20 is installed at /usr/local/bin/quarto.
  • .venv/bin/python (Python 3.13) has jupyter, nbformat, nbclient, ipykernel, docutils
    and every runtime dependency. It has no pip. Do not install anything.
  • Rendering with execution works and takes about 75 seconds:
    QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto (run from the repository root).
    pages/competition_organizers.ipynb downloads one small file from raw.githubusercontent.com,
    so this needs network.
  • Once the freeze is fresh, quarto render quarto needs no Python at all (about 5 seconds)
    and does not modify quarto/_freeze/. This is what CI relies on.
  • A first line #| eval: false in a code cell makes Quarto show the cell without executing it;
    the line is an ordinary comment in Colab, where the cell still runs.

Scope

In scope:

  • quarto/: _quarto.yml, the home page (converted from index.ipynb to index.qmd), three new
    prose pages (pages/methodology.qmd, pages/install.qmd, pages/citing.qmd), small edits to
    the three tutorial notebooks, a refreshed quarto/_freeze/.
  • README.rst (rewritten, short and formal).
  • pyproject.toml [project.urls] only.
  • .github/workflows/website.yml (new) and removal of .github/workflows/publish.yml.
  • Removal of docs/CompStats.ipynb and docs/CompStats_metrics.ipynb.
  • New tests appended to CompStats/tests/test_docs.py.
  • CLAUDE.md: a ## Documentation section.
  • One short subsection added to the Development section of docs/source/index.rst (how to
    preview the user site).

Out of scope:

  • Any change under CompStats/ other than CompStats/tests/test_docs.py.
  • Enabling GitHub Pages, creating branches on the remote, publishing the site, tagging a release.
    These are manual steps for the maintainer (listed under Decisions).
  • Renaming quarto/ to website/, changing the Quarto theme, restyling quarto/style.scss
    beyond adding a rule a new page needs.
  • Tutorials that use the DataFrame-based functions of CompStats.performance
    (performance, difference, plot_difference, performance_multiple_metrics, ...). Every
    code example on the user site and in the README uses the Perf-based syntax of this branch
    (CompStats.metrics.<measure>(y_true, y_pred, name=...), calling the instance to add a system,
    .difference(), .p_value(correction=...), .plot()). The LaTeX papers contain examples in
    the older functional syntax; do not copy those.
  • Changing the code cells of pages/competition_organizers.ipynb and
    pages/multiple_testing.ipynb beyond the #| eval: false line (their prose quotes numbers
    produced by their fixed seeds).
  • A script that verifies the freeze (the pytest test below replaces it), a gh-pages branch, a
    CONTRIBUTING file.
  • environment.yml, .devcontainer/, docs/source/*.png.

Implementation plan

Run every command from the repository root. Do the steps in order; step 6 (render) must come after
every edit under quarto/.

1. quarto/_quarto.yml

Change only the website.navbar block (keep project, lang, execute, site-url,
repo-url, page-footer, format exactly as they are):

  navbar:
    logo: images/ingeotec.png
    left:
      - href: index.qmd
        text: Home
      - href: pages/install.qmd
        text: Install
      - href: pages/methodology.qmd
        text: Methodology
      - href: pages/sklearn_users.ipynb
        text: scikit-learn Users
      - href: pages/competition_organizers.ipynb
        text: Competition Organizers
      - href: pages/multiple_testing.ipynb
        text: Multiple Testing
      - href: pages/citing.qmd
        text: Citing
      - href: https://ingeotec.github.io/CompStats/dev/
        text: Developer docs
    right:
      - icon: github
        href: https://github.com/INGEOTEC/CompStats
        aria-label: GitHub

2. Home page: quarto/index.qmd replaces quarto/index.ipynb

The home page has no code, so it becomes a plain-text .qmd file (easier to edit and review, and
it needs no freeze entry).

  1. git rm quarto/index.ipynb and git rm -r quarto/_freeze/index.

  2. Create quarto/index.qmd. Front matter:

    ---
    title: "CompStats"
    subtitle: "Bootstrap-based statistical comparison of supervised-learning systems evaluated on a shared test set"
    toc: false
    number-sections: false
    ---
  3. Body, in this order. The CSS classes .home-lead, .pillars/.pillar and
    .installs/.install already exist in quarto/style.scss; use them with Quarto fenced divs
    (::: {.home-lead} ... :::), as the current notebook does (read its markdown cells before
    deleting it and reuse what is still accurate).

    • Lead (.home-lead, one paragraph): what CompStats is. Content: an open-source Python
      package for the statistical analysis of the results of supervised-learning systems evaluated
      on a shared held-out test set (the setting of collaborative competitions, also called
      challenges); from the gold standard and the predictions of each system it estimates, by
      bootstrap, the standard error and the confidence interval of every performance measure, tests
      whether the difference between the best system and each of the others is statistically
      significant, and corrects those tests for multiple comparisons.

    • Motivation (one or two paragraphs, formal, third person). Content: competitions rank
      systems by a point estimate computed on one test set; differences between consecutive systems
      are often in the hundredths or thousandths, so a higher score does not imply that a system
      would remain better on the population the test set was sampled from; organizers have only the
      predictions (not the algorithms), a single dataset of fixed size and few submissions, which
      rules out the classical tests that require retraining or several datasets. CompStats
      implements the methodology of the Pattern Recognition Letters article (cite it with its DOI
      link and the arXiv link), which was validated on eight natural language processing
      competitions covering classification and regression tasks; the methodology itself is not
      specific to NLP.

    • Three pillars (.pillars): Follows the scikit-learn metrics API (a function of
      CompStats.metrics takes (y_true, y_pred) like its sklearn.metrics namesake and returns
      a Perf object instead of a float); Uncertainty for every measure (standard error and
      percentile confidence interval from the bootstrap distribution); Paired significance
      testing
      (every system is evaluated on the same bootstrap samples, so differences are paired;
      p-values come from the bootstrap distribution of the difference, with optional Bonferroni,
      Holm or Benjamini-Hochberg correction).

    • Quickstart: the two install commands in one sentence linking to
      Install, then this example in a non-executed fenced block (```python):

      from sklearn.datasets import load_digits
      from sklearn.ensemble import RandomForestClassifier
      from sklearn.model_selection import train_test_split
      from sklearn.naive_bayes import GaussianNB
      from sklearn.svm import LinearSVC
      from CompStats.metrics import f1_score
      
      X, y = load_digits(return_X_y=True)
      X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0)
      
      svm = LinearSVC(random_state=0).fit(X_train, y_train)
      score = f1_score(y_val, svm.predict(X_val), average='macro', name='Linear SVM')
      score(RandomForestClassifier(random_state=0).fit(X_train, y_train).predict(X_val),
            name='Random Forest')
      score(GaussianNB().fit(X_train, y_train).predict(X_val), name='Naive Bayes')
      
      score                 # point estimate and standard error of every system
      score.ci              # 95% percentile bootstrap confidence intervals
      diff = score.difference()            # paired differences against the best system
      diff.p_value(correction='holm')      # p-values adjusted for multiple comparisons
      score.plot()                         # performance with confidence intervals

      Before writing it into the page, save it to a temporary file outside the repository and run
      it with MPLBACKEND=Agg .venv/bin/python <file>; it must finish without an exception.

    • Where to go next: one paragraph linking Install, Methodology, the three tutorials
      (say that these three pages are Jupyter notebooks that can be opened in Colab from the badge
      at their top), Citing, and the developer docs (https://ingeotec.github.io/CompStats/dev/).

    • Colophon (## Colophon {.unnumbered}): keep the existing paragraph.

3. New prose pages (no executable code, plain .qmd)

Formal register in all three: impersonal, present tense, definitions before use, every formula
defined, no exclamation marks, no promotional adjectives. Use $...$/$$...$$ for mathematics.
Code shown on these pages goes in non-executed fenced blocks (```python, never
```{python}).

quarto/pages/install.qmd (title: "Install"): requirements (Python; the runtime
dependencies exactly as listed in pyproject.toml: numpy, scikit-learn >= 1.3.0, pandas,
seaborn >= 0.13.0, statsmodels; tqdm is optional and only adds progress bars); installing from
PyPI (pip install CompStats); from conda-forge (conda install --channel conda-forge CompStats); from source for development (git clone https://github.com/INGEOTEC/CompStats.git,
pip install -e ".[test]"); how to check the installation
(python -c "import CompStats; print(CompStats.__version__)").

quarto/pages/methodology.qmd (title: "Methodology", subtitle "The statistical basis of
CompStats"). Sections, each with its definition, its formula and the CompStats call that computes
it. This is the content; write it as connected prose, not as a bullet list:

  1. Setting. A competition provides a gold standard $y$ of size $n$ kept by the organizers and
    the predictions $\hat{y}_1, \ldots, \hat{y}_m$ of $m$ systems; a performance measure
    $s(y, \hat{y})$ ranks them. The quantity of interest is the performance $\theta_j$ of system
    $j$ on the population from which the test set is assumed to be a random sample; the value
    computed on the test set is an estimate of it. Constraints of the setting: one dataset of
    fixed size, few submissions, access to predictions only.
  2. Bootstrap. Bootstrapping draws conclusions about the sampling distribution of a statistic by
    resampling the data with replacement (Chernick and LaBudde, 2011; Efron and Tibshirani, 1994).
    $B$ samples of $n$ indices are drawn with replacement; the measure is computed on each sample
    for every system, giving $s_j^{*1}, \ldots, s_j^{*B}$. The article uses $B = 10{,}000$; the
    package default is num_samples=500 and it is a parameter of every function of
    CompStats.metrics and of Perf.
  3. Standard error and confidence interval. The standard error is the standard deviation of the
    bootstrap distribution (Perf.se). The $1-\alpha$ confidence interval is the percentile
    interval $[q_{\alpha/2}(s_j^{}),, q_{1-\alpha/2}(s_j^{})]$ with $\alpha = 0.05$ by default
    (Perf.ci).
  4. Comparison through independent samples. If the confidence intervals of two systems do not
    overlap, their performances differ significantly; if they overlap, the comparison is
    inconclusive, which motivates the paired comparison. Perf.plot() draws the ordered
    intervals.
  5. Comparison through paired samples. Every bootstrap sample contains the gold standard and the
    prediction of every system for the same elements, so the difference in performance can be
    computed per sample (the paired bootstrap): $\delta^{} = s^{}(y, \hat{y}_A) - s^{*}(y,
    \hat{y}_B)$. If the confidence interval of the difference contains zero, the hypothesis that
    both systems perform equally cannot be rejected. Perf.difference() returns the differences
    against the best system (or against the system named by wrt), and Difference.plot() draws
    their intervals.
  6. Statistical significance. For systems $A$ and $B$ with observed difference
    $\delta(x) = \theta_A(x) - \theta_B(x)$ on the test set $x$, the null hypothesis $H_0$ states
    that $A$ is not better than $B$ in the population. The p-value is
    $p(\delta(X) &gt; \delta(x) \mid H_0, x)$, where $X$ ranges over test sets of size $n$. Following
    Berg-Kirkpatrick, Burkett and Klein (2012), it is estimated as the fraction of bootstrap
    samples in which the difference exceeds $2\delta(x)$: the bootstrap distribution is centred on
    $\delta(x)$, not on zero, because the samples are drawn from $x$. Difference.p_value()
    returns these p-values.
  7. Multiple comparisons. Comparing the best system with each of the other $m-1$ tests the
    hypotheses $H_0^j: \theta_1 \le \theta_j$ against $H_1^j: \theta_1 &gt; \theta_j$ for
    $j = 2, \ldots, m$ simultaneously, which inflates the probability of a Type I error. Three
    corrections, all of which adjust the p-values: Bonferroni (conservative; multiplies each
    p-value by the number of comparisons), Holm's step-down procedure (adjusts the p-values
    sequentially; Holm, 1979) and Benjamini-Hochberg (controls the false discovery rate; Benjamini
    and Hochberg, 1995). In CompStats: Difference.p_value(correction='bonferroni'), 'holm',
    'fdr_bh', or any other method accepted by statsmodels.stats.multitest.multipletests; the
    same correction argument is accepted by Perf.plot and Difference.plot. With several
    measures in one Perf, each measure is corrected as its own family.
  8. Competitiveness of a competition. The article characterizes a competition with: the number
    of systems statistically tied with the winner; the coefficient of variation of the systems'
    performance, $CV = 100, s_x / \bar{x}$ (smaller means more competitive); the distance between
    the winner and the system in the middle of the table, $|win - med|$; and the possible
    percentage improvement, $PPI = 100,(1 - x_{winner})$ for bigger-is-better scores bounded by 1
    (a low value means little room for improvement). State that these indicators are computed by
    the earlier DataFrame-based interface (CompStats.performance.performance_multiple_metrics)
    and link to the developer docs for it; show no code for it.
  9. Scope and assumptions. The inference assumes the test set can be treated as a random sample
    of the population of interest; intervals and tests must be read in light of how the test set
    was built. The methodology applies to any measure computed from a gold standard and
    predictions, for classification and regression; measures where smaller is better (for example
    mean absolute error) are handled through the BiB (bigger is better) flag, which every
    function of CompStats.metrics sets for its measure.
  10. A closing "References" list: the two CompStats papers and the five references listed in the
    Citing section of docs/source/index.rst (copy the entries from there).

quarto/pages/citing.qmd (title: "Citing"): one sentence asking users to cite the Pattern
Recognition Letters article when CompStats is used in academic work, then the two BibTeX entries
copied verbatim from the Citing section of docs/source/index.rst (do not add fields), each
introduced by one sentence (the first is the methodology; the second is the conference paper it
extends), then the authors and affiliations (Sergio Nava-Muñoz, CIMAT, Aguascalientes, Mexico;
Mario Graff, INFOTEC, Aguascalientes, Mexico; Hugo Jair Escalante, INAOE, Puebla, Mexico), and the
license (Apache 2.0).

4. Tutorial notebooks: small scripted edits

Notebook files are JSON; do not edit them by hand. Write a throwaway Python script (outside the
repository, do not commit it), run it with .venv/bin/python, and let it assert every change it
makes. It must do exactly this:

import nbformat
from pathlib import Path

PAGES = Path("quarto/pages")

# (a) The Colab-only install cell is shown but not executed when the site is rendered.
for path in sorted(PAGES.glob("*.ipynb")):
    nb = nbformat.read(path, as_version=4)
    cells = [c for c in nb.cells if c.cell_type == "code" and "!pip install" in c.source]
    assert len(cells) == 1, path
    if not cells[0].source.startswith("#| eval: false"):
        cells[0].source = "#| eval: false\n" + cells[0].source
    nbformat.validate(nb)
    nbformat.write(nb, path)

# (b) scikit-learn Users: reproducible, and it ends with the comparison against the best.
path = PAGES / "sklearn_users.ipynb"
nb = nbformat.read(path, as_version=4)


def replace_once(old, new):
    hits = [c for c in nb.cells if c.cell_type == "code" and old in c.source]
    assert len(hits) == 1 and hits[0].source.count(old) == 1, old
    hits[0].source = hits[0].source.replace(old, new)


replace_once("from sklearn.svm import LinearSVC",
             "import numpy as np\nfrom sklearn.svm import LinearSVC")
replace_once("train_test_split(X, y, test_size=0.3)",
             "train_test_split(X, y, test_size=0.3, random_state=0)")
replace_once("LinearSVC()", "LinearSVC(random_state=0)")
replace_once("RandomForestClassifier()", "RandomForestClassifier(random_state=0)")
replace_once("HistGradientBoostingClassifier()", "HistGradientBoostingClassifier(random_state=0)")
replace_once("score = f1_score(", "np.random.seed(0)\nscore = f1_score(")

last = nb.cells.pop()                      # the closing "see also" markdown cell
assert last.cell_type == "markdown"
nb.cells += [
    nbformat.v4.new_markdown_cell(SEED_NOTE),
    nbformat.v4.new_markdown_cell(DIFFERENCE_INTRO),
    nbformat.v4.new_code_cell("diff = score.difference()\ndiff"),
    nbformat.v4.new_markdown_cell(P_VALUE_INTRO),
    nbformat.v4.new_code_cell("diff.p_value(correction='holm')"),
    nbformat.v4.new_markdown_cell(PLOT_INTRO),
    nbformat.v4.new_code_cell("diff.plot()"),
    last,
]
nbformat.validate(nb)
nbformat.write(nb, path)

Define the four text constants yourself (plain markdown strings, formal register):

  • SEED_NOTE: one sentence saying that the data split, the classifiers and NumPy's global random
    state (from which the bootstrap samples are drawn) are seeded so the page is reproducible.
  • DIFFERENCE_INTRO: heading ## Comparison against the best system and two sentences:
    Perf.difference computes, on the same bootstrap samples, the difference between the best
    system and each of the others, and the object it returns prints the p-value of each difference.
  • P_VALUE_INTRO: heading ## Correction for multiple comparisons and two sentences: three
    systems are compared with the best at once, so the p-values are adjusted, here with Holm's
    method; link to [Multiple Testing](multiple_testing.ipynb).
  • PLOT_INTRO: one sentence: Difference.plot draws the confidence interval of each difference;
    an interval that contains zero means the system is not statistically different from the best.

Do not write, in any markdown cell of this notebook, which classifier wins or any number: the
text must stay true whatever the outputs are.

After running the script: git diff --stat quarto/pages must list the three notebooks, and
pages/competition_organizers.ipynb and pages/multiple_testing.ipynb must differ from develop
by exactly one added source line each (#| eval: false).

5. README.rst, pyproject.toml, legacy notebooks

README.rst — replace the whole file. reStructuredText, about 80-120 lines, formal:

  1. Title CompStats and badges: tests (actions/workflows/test.yml), Coveralls, PyPI, the three
    conda-forge badges (copy these six from the current file). Drop the ReadTheDocs badge and
    the Colab badge.
  2. One paragraph: the same definition as the home page lead, plus the sentence that it implements
    the methodology of the Pattern Recognition Letters article (link the DOI).
  3. Documentation: two bullets, "Using the package: https://ingeotec.github.io/CompStats/" and
    "Extending the package (developer reference):
    https://ingeotec.github.io/CompStats/dev/".
  4. Installation: the pip and the conda commands.
  5. Quickstart: the same example as the home page, in a .. code:: python block, followed by
    one sentence per result (score, score.ci, diff.p_value(...), score.plot()). No printed
    outputs and no images.
  6. Citing: the two BibTeX entries copied verbatim from docs/source/index.rst, in a
    .. code:: bibtex block.
  7. License: Apache License 2.0, pointing at the LICENSE file.

The README is the long description on PyPI, so it must be valid reStructuredText:

.venv/bin/python -m docutils --halt=warning README.rst /dev/null

must exit 0 with no output.

pyproject.toml — replace the [project.urls] table with:

[project.urls]
Homepage = "https://ingeotec.github.io/CompStats/"
Documentation = "https://ingeotec.github.io/CompStats/"
"Developer documentation" = "https://ingeotec.github.io/CompStats/dev/"
Repository = "https://github.com/INGEOTEC/CompStats"
Issues = "https://github.com/INGEOTEC/CompStats/issues"

Legacy notebooks — git rm docs/CompStats.ipynb docs/CompStats_metrics.ipynb.

Afterwards git grep -il readthedocs must print nothing (if a file still matches, fix that file;
quarto/_freeze is regenerated in the next step).

6. Refresh the freeze and render

QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto

This executes the three tutorial notebooks (about 75 seconds, needs network for one small
download) and writes the site to quarto/_site/ (git-ignored). It prints one [n/7] line per
page and ends with Output created: _site/index.html. If it stops with an error in a notebook
cell, the error is in an edit of step 4: fix the notebook through the script, not the freeze.

Then confirm that the freeze is complete and that CI will not need Python:

env QUARTO_PYTHON=/nonexistent/python quarto render quarto
git status --porcelain quarto/_freeze

The first command must end with Output created: _site/index.html and must not print
Starting python3 kernel (it prints two harmless WARN lines about QUARTO_PYTHON); if it tries
to start a kernel, a page's freeze is stale: run the first render command again. Record the state
with git add -A quarto/_freeze quarto/pages quarto/index.qmd quarto/_quarto.yml (-A matters:
figure files are renamed when cells are added, and the old ones must be removed from the index).
Run the second command once more after git add: git status --porcelain quarto/_freeze must
then show only staged changes (no line starting with a space or ??).

Open nothing in a browser; instead check the output exists:
ls quarto/_site/index.html quarto/_site/pages/methodology.html quarto/_site/pages/install.html quarto/_site/pages/citing.html quarto/_site/pages/sklearn_users.html.

Never commit quarto/_site/ or quarto/.quarto/.

7. .github/workflows/website.yml (new) and removal of publish.yml

git rm .github/workflows/publish.yml, then create .github/workflows/website.yml with exactly:

name: Publish website

# One GitHub Pages site, two sources: the Quarto user site (quarto/) at the
# root and the Sphinx developer reference (docs/) under /dev/. Pull requests
# and pushes to develop only build both; pushes to main also deploy them.
#
# The Quarto pages that are notebooks are never executed here: Quarto reads
# their results from the committed quarto/_freeze/ cache. If a notebook was
# changed without refreshing the cache, the render step fails ("Jupyter is
# not available"); the fix is to render locally and commit quarto/_freeze/.
on:
  pull_request:
    paths:
      - "quarto/**"
      - "docs/**"
      - "CompStats/**"
      - "pyproject.toml"
      - ".github/workflows/website.yml"
  push:
    branches: [main, develop]
    paths:
      - "quarto/**"
      - "docs/**"
      - "CompStats/**"
      - "pyproject.toml"
      - ".github/workflows/website.yml"
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: quarto-dev/quarto-actions/setup@v2
        with:
          # Keep equal to the version used to produce quarto/_freeze/.
          version: 1.10.20

      - run: quarto render quarto

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install docs and package dependencies
        run: |
          pip install -r docs/requirements.txt
          # Not --no-deps: conf.py imports CompStats, and autodoc imports its
          # dependencies with it.
          pip install -e .

      - name: Build the developer docs strictly into the site
        run: python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev

      - uses: actions/upload-pages-artifact@v3
        with:
          path: quarto/_site

  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
    needs: build
    runs-on: ubuntu-latest
    permissions:
      pages: write
      id-token: write
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    concurrency:
      group: website
      cancel-in-progress: true
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4

(actions/configure-pages is deliberately absent: it fails on a repository where Pages is not
enabled yet, and nothing in the build needs its outputs.)

Check locally that the two halves fit together, exactly as the workflow does it (after step 6, so
quarto/_site/ exists):

.venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev
ls quarto/_site/dev/index.html

8. Documentation for contributors (absorbs #45)

CLAUDE.md — add a ## Documentation section after ## Releasing stating:

  • the two halves and their URLs (user site from quarto/, developer reference from
    docs/source/ published under /dev/), both deployed by .github/workflows/website.yml on
    pushes to main;
  • user-facing examples use the Perf-based API only;
  • preview commands: QUARTO_PYTHON="$PWD/.venv/bin/python" quarto preview quarto (live reload)
    and QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto (static, into
    quarto/_site/);
  • the freeze rule: CI never executes notebooks; after changing any quarto/**/*.ipynb, render
    locally and commit quarto/_freeze/ with git add -A quarto/_freeze in the same commit;
    CompStats/tests/test_docs.py fails when a notebook and its freeze entry disagree;
  • notebooks are edited through nbformat (or Jupyter), never by hand-editing JSON;
  • the Colab-only install cell keeps #| eval: false as its first line.

Also update the performance.py and utils.py bullets if they mention ReadTheDocs (they do not
today; just make sure git grep -i readthedocs is empty).

docs/source/index.rst — in the Development section that #49 added, append a short
"User site" paragraph with the two preview commands and the freeze rule (three or four
sentences). Rebuild the developer docs strictly afterwards.

9. Commit message

End the commit message body with the two lines Closes #45 and Closes #46 (this issue
supersedes them), in addition to whatever reference to this issue the run normally adds.

Acceptance criteria

  • quarto/index.ipynb and quarto/_freeze/index/ no longer exist; quarto/index.qmd,
    quarto/pages/install.qmd, quarto/pages/methodology.qmd and quarto/pages/citing.qmd
    exist and are in the navbar, together with a "Developer docs" link to
    https://ingeotec.github.io/CompStats/dev/.
  • env QUARTO_PYTHON=/nonexistent/python quarto render quarto succeeds without starting a
    Jupyter kernel and leaves git status --porcelain quarto/_freeze free of unstaged changes.
  • For every quarto/**/*.ipynb page, the hash in
    quarto/_freeze/<path without .ipynb>/execute-results/html.json equals the MD5 of the
    notebook file.
  • Each tutorial notebook has exactly one !pip install cell and it starts with
    #| eval: false.
  • pages/sklearn_users.ipynb is seeded and ends with the comparison against the best system
    (score.difference(), diff.p_value(correction='holm'), diff.plot()), and its prose
    names no winner and quotes no number.
  • The methodology page has the nine sections of step 3 with their formulas, names
    Berg-Kirkpatrick et al. (2012) for the p-value estimator and Bonferroni, Holm and
    Benjamini-Hochberg for the corrections, and shows no code of CompStats.performance.
  • No code example on the user site or in the README imports from CompStats.performance or
    calls performance(, difference(, plot_difference( as free functions.
  • README.rst passes .venv/bin/python -m docutils --halt=warning README.rst /dev/null,
    links both documentation URLs, contains the DOI 10.1016/j.patrec.2024.03.010, and has no
    image and no Colab or ReadTheDocs badge.
  • git grep -il readthedocs prints nothing.
  • pyproject.toml [project.urls] matches step 5; nothing else in the file changed.
  • .github/workflows/publish.yml is gone; .github/workflows/website.yml exists with the
    content above.
  • docs/CompStats.ipynb and docs/CompStats_metrics.ipynb are gone.
  • .venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev ends
    with build succeeded.
  • quarto/_site/, quarto/.quarto/ and docs/build/ are not committed.
  • CLAUDE.md has a ## Documentation section with the preview commands and the freeze rule.
  • .venv/bin/python -m pytest CompStats is green.

Tests

Append to CompStats/tests/test_docs.py (created by #49; keep its existing tests and its
module-level skip). Add near the top:

import hashlib
import json

SITE = ROOT / "quarto"
_SKIPPED_DIRS = {"_site", "_freeze", ".quarto"}


def _site_files(suffix):
    """Source pages of the Quarto site with the given suffix"""
    return sorted(p for p in SITE.rglob(f"*{suffix}")
                  if not _SKIPPED_DIRS & set(p.relative_to(SITE).parts))

Required test functions (skip each with pytest.mark.skipif(not SITE.is_dir(), ...)):

  1. test_freeze_matches_notebooks: for every _site_files(".ipynb"), the file
    SITE / "_freeze" / <relative path without suffix> / "execute-results" / "html.json" exists
    and its "hash" equals hashlib.md5(notebook.read_bytes()).hexdigest(). The assertion message
    must say how to fix it: render locally with
    QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto and commit quarto/_freeze/.
    Assert as well that there is at least one notebook (so the test cannot pass vacuously).
  2. test_no_orphan_freeze_entries: every directory under quarto/_freeze that contains an
    execute-results/html.json corresponds to an existing notebook (this catches the leftover
    quarto/_freeze/index).
  3. test_install_cells_are_not_executed: in every notebook (parse it with json, no extra
    dependency), exactly one code cell contains !pip install, and its source starts with
    #| eval: false.
  4. test_navbar_pages_exist: every href: value in quarto/_quarto.yml (extract them with a
    regular expression, do not import yaml) that does not start with http names an existing
    file under quarto/; and index.qmd, pages/install.qmd, pages/methodology.qmd,
    pages/citing.qmd and the three notebooks are all among them.
  5. test_user_docs_use_the_perf_api: no .qmd page, no notebook code cell and not README.rst
    contains from CompStats.performance or from CompStats import performance.
  6. test_methodology_page_content: pages/methodology.qmd contains (case-insensitive) each of
    bootstrap, confidence interval, paired, p-value, Bonferroni, Holm,
    Benjamini, Berg-Kirkpatrick, coefficient of variation, PPI, and the string
    10.1016/j.patrec.2024.03.010.
  7. test_no_readthedocs_anywhere: list tracked files with git ls-files -z (skip the test if
    git is unavailable or ROOT is not a checkout), read each as UTF-8 (skip files that are not
    text), and assert none contains the forbidden word, built as "readthe" + "docs",
    case-insensitive.
  8. test_readme: README.rst contains https://ingeotec.github.io/CompStats/,
    https://ingeotec.github.io/CompStats/dev/ and 10.1016/j.patrec.2024.03.010, contains no
    .. image:: directive that is not a badge (simplest check: it does not contain
    raw/docs/docs), and renders without warnings: pytest.importorskip("docutils"), then
    docutils.core.publish_string(text, writer_name="html", settings_overrides={"halt_level": 2, "report_level": 2}) must not raise.
  9. test_site_workflow: .github/workflows/website.yml exists, publish.yml does not; the
    workflow text contains quarto render quarto,
    python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev,
    actions/deploy-pages and does not contain conda or gh-pages.
  10. test_project_urls: pytest.importorskip("tomllib"); [project.urls] in pyproject.toml
    has Homepage and Documentation equal to https://ingeotec.github.io/CompStats/.

Commands that must be green before committing, in this order:

env QUARTO_PYTHON=/nonexistent/python quarto render quarto
.venv/bin/python -m sphinx -n -W --keep-going -b html docs/source quarto/_site/dev
.venv/bin/python -m docutils --halt=warning README.rst /dev/null
.venv/bin/python -m pytest CompStats

Durations: about 5 seconds, about 10 seconds plus inventory downloads, instant, and about
2.5 minutes (ends with a line like 99 passed). The one slow step of this issue is the executing
render of step 6 (about 75 seconds); it is finished when it prints
Output created: _site/index.html.

Decisions

  • One GitHub Pages site, deployed with actions/deploy-pages (as in scjn/dofjson), not a
    gh-pages branch: this supersedes the plan of Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46.
  • CI renders from the committed freeze and never executes notebooks; a stale freeze fails the
    build loudly, and a pytest test (test_freeze_matches_notebooks) catches it earlier. This
    replaces the verification script proposed in Automate Quarto site publishing via GitHub Actions using a render/freeze verification script #46.
  • Deploy only from main; build on pull requests and on develop: develop is the
    integration branch, and the public site must reflect released code.
  • actions/configure-pages omitted: it fails while Pages is not enabled, which would turn
    every pull request red before the maintainer enables it.
  • The home page becomes index.qmd: it has no code, so a notebook only added a freeze entry
    and a Colab badge with nothing to run.
  • New prose pages are .qmd without executable code: nothing to execute, nothing to freeze.
  • sklearn_users.ipynb is seeded and extended with the comparison against the best: a formal
    tutorial must be reproducible and must reach the significance test; its prose avoids naming
    winners or numbers so it cannot disagree with its outputs.
  • #| eval: false on the Colab install cell: rendering the site must not run pip on the
    author's machine; Colab ignores the comment.
  • User-facing examples use the Perf API only (maintainer's choice); the DataFrame-based
    functions appear only in the developer reference and, by name, in the competitiveness section.
  • Directory stays quarto/ (not renamed to website/): the Colab links and existing history
    use that path.
  • README stays reStructuredText and becomes short: tutorials live on the site; a README that
    duplicates them goes stale, as the current one did.
  • Legacy notebooks under docs/ are deleted: they document the previous API and site; they
    remain available in the git history and on the docs branch.
  • Quarto pinned to 1.10.20 in CI: the version installed locally, which produced the freeze.
  • Manual steps left to the maintainer (not part of this issue): enable GitHub Pages with
    source "GitHub Actions" in the repository settings before the first push to main; redirect or
    retire the ReadTheDocs project; tag and publish 0.2.0 to PyPI, which the Colab install cells
    need because the tutorials use the API of this branch.

Planned against develop at d2c6feade9eac5eb039b405ac272f70fc66458ce.

Activity

  1. mgraffg commented on Nov 19, 2026

    @mgraffg
    MemberAuthor

    Implemented on branch issues-48-49-50, commit 8608cbb (the message ends with Closes #45 / Closes #46).

    • Quarto: new navbar, index.qmd (replaces index.ipynb and its freeze), new install.qmd, methodology.qmd, citing.qmd; #| eval: false on the Colab install cells; sklearn_users.ipynb seeded and extended with difference(), p_value(correction="holm"), plot() (prose names no winner and quotes no number); the two other notebooks differ by exactly one added line.
    • Freeze refreshed with the executing render; then env QUARTO_PYTHON=/nonexistent/python quarto render quarto succeeded without starting a kernel and git status --porcelain quarto/_freeze shows only staged changes.
    • README.rst rewritten (121 lines, validates with docutils --halt=warning), pyproject.toml URLs updated, legacy docs/*.ipynb and publish.yml removed, website.yml added exactly as specified.
    • CLAUDE.md ## Documentation section and a "User site" paragraph in docs/source/index.rst; git grep -il readthedocs is empty.
    • Tests: strict Sphinx build into quarto/_site/dev succeeded; .venv/bin/python -m pytest CompStats green (exit 0); 10 new tests appended to CompStats/tests/test_docs.py.
    • Decision: to satisfy the "no readthedocs anywhere" check, the two tests whose names contained that word were renamed (test_no_legacy_hosting_in_developer_docs, test_no_legacy_hosting_anywhere) and one docstring was reworded.
    • Manual steps left to the maintainer: enable GitHub Pages (source: GitHub Actions), retire the ReadTheDocs project, tag/publish 0.2.0.
  2. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-1 (commit f4b315a, branch issues-48-49-50): wording corrected to agree with the article (no "developed by" attribution, footer names the authors, no "validated" claim, correct argument for why classical tests do not apply, purpose of the methodology, hypotheses and 0.05 decision rule, three aspects of competitiveness plus PPI separately, reasons for multiple-comparison correction, three added references) in quarto/index.qmd, quarto/_quarto.yml, quarto/pages/methodology.qmd and README.rst; new test test_site_follows_the_article. Render from the freeze without Python succeeds (freeze unchanged); full suite: 100 passed (run with MPLBACKEND=Agg).

  3. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-2 (commit 5c4f897, branch issues-48-49-50): the user site now cites through Quarto's bibliography (quarto/references.bib, bibliography: references.bib); every citation is a link to its complete reference, volume and pages of the 2024 article were added to the BibTeX in citing.qmd and README.rst, and test_site_bibliography was added. Render from the freeze without Python succeeds with quarto/_freeze unchanged; full suite: 101 passed (MPLBACKEND=Agg).

  4. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-3 (commit d7b6a3e, branch issues-48-49-50): the preprint link of the 2024 article now lives in its bibliography entry (note field) and is printed by a small site citation style, quarto/compstats.csl, instead of being typed in the home page prose; Efron1994 publisher fixed with double braces. Render from the freeze without Python succeeds with quarto/_freeze unchanged; full suite: 101 passed (MPLBACKEND=Agg).

  5. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-4 (commit 95fe5b3, branch issues-48-49-50): removed code-fold: true from quarto/_quarto.yml so the code of the tutorial pages is always visible; added test_site_code_is_visible. Render from the freeze without Python succeeds with quarto/_freeze unchanged; full suite: 102 passed (MPLBACKEND=Agg).

  6. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-5 (commit 0763e70, branch issues-48-49-50): the Colab install cell of the three notebooks now guards with try: import CompStats / except ImportError and installs with uv pip install (pip fallback), still #| eval: false; sklearn_users.ipynb explains statistic/se and the figure of 'Rest of the classifiers' and gains a Regression section; quarto/_freeze regenerated; test and CLAUDE.md updated. Inserted sentences checked against the regenerated outputs. Render without Python leaves the freeze unchanged; full suite: 103 passed (MPLBACKEND=Agg).

  7. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-6 (commit 5abc121, branch issues-48-49-50): the prose of the user site and the README was rewritten in the punctuation style of the article (semicolons and dashes replaced by separate sentences, connectors or subordination, colons only before enumerations). Only markdown cells of the notebooks changed (code cells verified identical), quarto/_freeze regenerated, and test_site_prose_style plus a rule in CLAUDE.md added. The notebook edits kept the maintainer's later commit a57a565. The test helper now ignores .ipynb_checkpoints, which a local Jupyter session had created. Render without Python leaves the freeze unchanged; full suite: 104 passed (MPLBACKEND=Agg).

  8. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Implemented in #51, merged into develop. Commit 8608cbb.

  9. added 2 commits that reference this issue on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions