Skip to content

Replace legacy CI with pytest and a tag-driven PyPI publish workflow #48

Closed

Description

@mgraffg

Context

CompStats is a pure-Python package (CompStats/, version read from CompStats.__version__,
currently 0.2.0; PyPI has 0.1.15). Its continuous integration is legacy and no longer matches
how the project is developed:

  • .github/workflows/test.yaml builds a conda environment and runs nosetests, only on pushes to
    develop. Day-to-day development uses pytest CompStats (see CLAUDE.md).
  • .github/workflows/pip.yaml publishes to PyPI from a conda matrix (Linux uploads the sdist,
    Windows uploads the wheel, --skip-existing). Nothing checks that the pushed tag matches
    __version__, so a stale tag silently re-publishes nothing, or publishes the wrong version.
  • pyproject.toml has no [build-system] table and no optional test dependencies.

Two sibling projects of the same maintainer (scjn and dofjson) use a simpler, stricter layout:
a test.yml workflow running pytest on a Python matrix plus a version gate, and a
publish-pypi.yml workflow triggered by a v* tag that first verifies the tag against the package
version. This issue brings that layout to CompStats. The complete file contents are given below, so
nothing has to be looked up in those repositories.

Facts already verified while planning (do not re-derive them):

  • The repository secret used for the upload is named TWINE and already exists.
  • pytest CompStats passes today: 74 tests, about 2.5 minutes.
  • .venv/bin/python is Python 3.13 and has pytest, packaging, build and coverage
    installed. It does not have pip or twine. Do not install anything into it.

Scope

In scope:

  • Replace .github/workflows/test.yaml with .github/workflows/test.yml (pytest).
  • Replace .github/workflows/pip.yaml with .github/workflows/publish-pypi.yml (tag-driven).
  • Add scripts/check_package_versions.py (version gate against PyPI).
  • Add [build-system] and a test extra to pyproject.toml.
  • Add CompStats/tests/test_packaging.py.
  • Update the test badge URL (test.yaml -> test.yml) wherever it appears, and the CI notes in
    CLAUDE.md.

Out of scope:

  • Publishing anything: do not create or push a git tag, do not run twine upload, do not bump
    __version__ (0.2.0 is already exactly one minor release ahead of PyPI's 0.1.15, which is what
    the gate requires).
  • Any documentation work (docs/, quarto/, the body of README.rst): that is done in follow-up
    issues. Only the badge URL changes here.
  • .github/workflows/publish.yml (the Quarto site workflow): leave it untouched.
  • Adding requires-python, changing dependencies, changing classifiers, conda-forge, Windows or
    macOS CI, PyPI trusted publishing.
  • Any change to the behaviour of the code under CompStats/ (only a new test file is added there).

Implementation plan

1. pyproject.toml

Add, at the very top of the file:

[build-system]
# 77 is the first setuptools that accepts the SPDX string form `license = "Apache-2.0"`
# used below.
requires = ["setuptools>=77.0.3"]
build-backend = "setuptools.build_meta"

Add, after the dependencies = [...]/dynamic = ['version'] lines and before classifiers
(any position inside the document is fine as long as it is its own table):

[project.optional-dependencies]
# `packaging` is only needed by scripts/check_package_versions.py and its test.
test = ["pytest>=7.0", "packaging"]

Careful with TOML table order: [project.optional-dependencies] starts a new table, so every key
that belongs to [project] (name, description, readme, license, dependencies, dynamic,
classifiers) must appear before it. The safest edit is to put the new table right before
[tool.setuptools.dynamic]. Leave every existing key and value unchanged.

Verify with:

.venv/bin/python -c "import tomllib; d = tomllib.load(open('pyproject.toml','rb')); print(d['build-system'], d['project']['optional-dependencies'], d['project']['classifiers'][0])"

2. scripts/check_package_versions.py (new file, new scripts/ directory)

Create it with exactly this content:

#!/usr/bin/env python3
"""Check the package's local version against PyPI.

``publish-pypi.yml`` only checks, at the moment a tag is pushed, that the tag
matches ``CompStats.__version__``. Nothing checks *before* that whether the
local version is actually ahead of what PyPI has, or whether it jumped more
than one release past it. This script reports PyPI's latest version, the
local version (read from the attribute ``pyproject.toml`` declares as the
dynamic version, without importing the package, so none of its dependencies
need to be installed), and whether the local version is a valid single-step
jump ahead of PyPI: the next patch, or the next minor with patch reset to 0.
A package PyPI has never published always passes.

Usage::

    python scripts/check_package_versions.py
"""

import ast
import json
import sys
import tomllib
import urllib.error
import urllib.request
from pathlib import Path

from packaging.version import Version

ROOT = Path(__file__).resolve().parents[1]

#: Name of the distribution on PyPI.
PACKAGE = "CompStats"


def pypi_latest_version(package: str = PACKAGE):
    """Return the latest version PyPI reports for `package`, or `None` if it
    has never been published (a 404 from PyPI's JSON API)."""
    url = f"https://pypi.org/pypi/{package}/json"
    try:
        with urllib.request.urlopen(url, timeout=10) as response:
            return json.load(response)["info"]["version"]
    except urllib.error.HTTPError as error:
        if error.code == 404:
            return None
        raise


def local_version(package_dir: Path = ROOT) -> str:
    """Return the version ``pyproject.toml`` resolves dynamically, read
    straight out of the source file its ``attr`` points at (no import)."""
    with open(package_dir / "pyproject.toml", "rb") as fpt:
        pyproject = tomllib.load(fpt)
    attr = pyproject["tool"]["setuptools"]["dynamic"]["version"]["attr"]
    module_name, attr_name = attr.rsplit(".", 1)
    module_path = package_dir / module_name.replace(".", "/") / "__init__.py"
    tree = ast.parse(module_path.read_text(encoding="utf-8"),
                     filename=str(module_path))
    for node in ast.walk(tree):
        if not isinstance(node, ast.Assign):
            continue
        if any(isinstance(t, ast.Name) and t.id == attr_name
               for t in node.targets):
            return ast.literal_eval(node.value)
    raise ValueError(f"{attr} not found in {module_path}")


def next_versions(version: Version):
    """Return the two versions immediately ahead of `version`: the next
    patch, and the next minor with patch reset to 0."""
    major, minor, patch = (tuple(version.release) + (0, 0, 0))[:3]
    return (Version(f"{major}.{minor}.{patch + 1}"),
            Version(f"{major}.{minor + 1}.0"))


def check_package(package: str = PACKAGE, package_dir: Path = ROOT):
    """Return ``(package, pypi_version, local_version, ok, detail)``."""
    pypi = pypi_latest_version(package)
    local = local_version(package_dir)
    if pypi is None:
        return package, pypi, local, True, "never published"
    local_v, pypi_v = Version(local), Version(pypi)
    if local_v <= pypi_v:
        return package, pypi, local, False, f"local is not ahead of PyPI ({pypi})"
    next_patch, next_minor = next_versions(pypi_v)
    if local_v not in (next_patch, next_minor):
        return (package, pypi, local, False,
                f"jumps more than one release past PyPI ({pypi}); "
                f"expected {next_patch} or {next_minor}")
    return package, pypi, local, True, "ok"


def main() -> int:
    """Print the verdict and return the process exit status."""
    package, pypi, local, ok, detail = check_package()
    print(f"{'package':<12}  {'pypi':<10}  {'local':<10}  verdict")
    verdict = "OK" if ok else "FAIL"
    print(f"{package:<12}  {pypi or '-':<10}  {local:<10}  {verdict}: {detail}")
    return 0 if ok else 1


if __name__ == "__main__":
    sys.exit(main())

Run it once (it makes one HTTPS request to pypi.org):

.venv/bin/python scripts/check_package_versions.py

Expected output: a line ending in OK: ok with pypi = 0.1.15 and local = 0.2.0, exit
status 0. If the only failure is that the network is unreachable, say so in the commit message body
and continue; any other failure must be fixed.

3. .github/workflows/test.yml (new) and removal of test.yaml

git rm .github/workflows/test.yaml, then create .github/workflows/test.yml with exactly:

name: test

on:
  push:
    branches: [main, develop]
  pull_request:

env:
  # The suite draws seaborn figures; the runner has no display.
  MPLBACKEND: Agg

jobs:
  # The whole suite: no test touches the network.
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install package and test dependencies
        run: pip install -e ".[test]"
      - name: Run tests
        run: pytest CompStats

  # Coverage is collected once (not per matrix cell) and sent to Coveralls,
  # which the README badge reads. `.coveragerc` omits the tests package.
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install package, test and coverage dependencies
        run: |
          pip install -e ".[test]"
          pip install coverage coveralls
      - name: Run tests with coverage
        run: coverage run -m pytest CompStats
      - name: Report
        run: coverage report
      - name: Coveralls
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: coveralls --service=github

  # Hard gate on the single-step-ahead rule: __version__ may not be behind,
  # equal to, or more than one release past what is published on PyPI.
  check-package-versions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install checker dependencies
        run: pip install packaging
      - name: Check package versions against PyPI
        run: python scripts/check_package_versions.py

4. .github/workflows/publish-pypi.yml (new) and removal of pip.yaml

git rm .github/workflows/pip.yaml, then create .github/workflows/publish-pypi.yml with exactly:

name: Publish to PyPI

on:
  push:
    tags:
      - "v*"
  workflow_dispatch:

concurrency:
  group: publish-${{ github.ref_name }}
  cancel-in-progress: false

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install version-check dependencies
        if: github.event_name != 'workflow_dispatch'
        run: pip install packaging

      # Tags are the release trigger, so a stale tag (forgot to bump
      # __version__) must fail loudly instead of publishing the wrong
      # version. `version` is declared `dynamic` in pyproject.toml, so it is
      # read the same way scripts/check_package_versions.py reads it.
      - name: Verify tag matches the package version
        if: github.event_name != 'workflow_dispatch'
        run: |
          tag_version="${GITHUB_REF_NAME#v}"
          pkg_version=$(python -c "import sys; sys.path.insert(0, 'scripts'); from check_package_versions import local_version; print(local_version())")
          if [ "$tag_version" != "$pkg_version" ]; then
            echo "Tag version $tag_version does not match CompStats.__version__ $pkg_version" >&2
            exit 1
          fi

      - name: Install build tooling
        run: pip install build twine

      # CompStats is pure Python: one sdist and one universal wheel, built once.
      - name: Build sdist and wheel
        run: python -m build

      - name: Check package metadata
        run: twine check dist/*

      # `TWINE` is a repository secret holding a PyPI API token.
      - name: Upload to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.TWINE }}
        run: twine upload dist/*

5. Check that the package still builds

.venv/bin/python -m build --outdir "$(mktemp -d)"

This downloads setuptools into an isolated environment (needs network, about 30 seconds) and must
end with Successfully built compstats-0.2.0.tar.gz and compstats-0.2.0-py3-none-any.whl (the
exact file-name casing may differ). It creates build/ and CompStats.egg-info/ in the
repository; both are already git-ignored, do not commit them. If it fails only because the network
is unreachable, say so in the commit message body and continue.

6. Badge URL

Replace actions/workflows/test.yaml with actions/workflows/test.yml in every tracked file that
has it. Today that is two occurrences (badge image and its target) in each of README.rst,
docs/source/index.rst and docs/source/metrics_api.rst. Find them with:

git grep -n "test.yaml"

After the edit that command must print nothing except, possibly, lines inside this issue's own
text that you may have pasted somewhere (there should be none).

7. CLAUDE.md

In the ## Commands section, replace the sentence in parentheses that says CI still invokes
nosetests and the final Note: paragraph (the one starting ".github/workflows/test.yaml (CI)
builds the environment with conda and runs nosetests") so that they state the truth after this
change: CI (.github/workflows/test.yml) runs pytest CompStats on Python 3.10-3.12, collects
coverage for Coveralls, and runs scripts/check_package_versions.py.

Add a new ## Releasing section right after ## Commands with this content (reword freely, keep
the facts):

  • The version lives in one place: __version__ in CompStats/__init__.py.
  • scripts/check_package_versions.py (run by CI on every push and pull request) fails unless
    __version__ is exactly one patch or one minor release ahead of the latest version on PyPI.
  • To release: make sure __version__ is the version to publish, merge to main, then create and
    push the tag v<__version__> (for example git tag v0.2.0 && git push origin v0.2.0).
    .github/workflows/publish-pypi.yml verifies that the tag equals __version__, builds the
    sdist and the wheel, runs twine check and uploads with the TWINE repository secret.
  • After a release, bump __version__ before the next change is merged, otherwise the version gate
    fails ("local is not ahead of PyPI").

Acceptance criteria

  • .github/workflows/test.yaml and .github/workflows/pip.yaml no longer exist;
    .github/workflows/test.yml and .github/workflows/publish-pypi.yml exist with the content
    above.
  • No tracked file mentions nosetests or test.yaml (git grep -n -e nosetests -e test.yaml
    prints nothing).
  • scripts/check_package_versions.py exists and .venv/bin/python scripts/check_package_versions.py exits 0 printing OK: ok (network permitting).
  • pyproject.toml parses, has [build-system] with setuptools>=77.0.3 and a test extra
    containing pytest>=7.0 and packaging; every pre-existing key is unchanged.
  • .venv/bin/python -m build --outdir "$(mktemp -d)" succeeds (network permitting).
  • CompStats.__version__ is still '0.2.0'; no tag was created.
  • CLAUDE.md no longer describes CI as legacy/nosetests and has a ## Releasing section.
  • .github/workflows/publish.yml is byte-for-byte unchanged.
  • .venv/bin/python -m pytest CompStats is green.

Tests

Add CompStats/tests/test_packaging.py (pytest style, like the other files in that directory;
English names and docstrings). The tests package is shipped inside the wheel, so the file must skip
cleanly when it runs outside a repository checkout. Start it with:

import importlib.util
from pathlib import Path

import pytest

ROOT = Path(__file__).resolve().parents[2]
SCRIPT = ROOT / "scripts" / "check_package_versions.py"
WORKFLOWS = ROOT / ".github" / "workflows"

pytest.importorskip("tomllib")    # Python 3.11+
pytest.importorskip("packaging")
pytestmark = pytest.mark.skipif(
    not SCRIPT.is_file(), reason="needs the repository checkout (scripts/)")


def _load_script():
    """Import scripts/check_package_versions.py as a module"""
    spec = importlib.util.spec_from_file_location("check_package_versions", SCRIPT)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

Required test functions (no test may touch the network: replace pypi_latest_version with
monkeypatch.setattr(module, "pypi_latest_version", lambda package=None: "<value>")):

  1. test_local_version_matches_package: module.local_version() equals CompStats.__version__.
  2. test_next_versions: next_versions(Version("0.1.15")) is (Version("0.1.16"), Version("0.2.0")).
  3. test_check_package_never_published: with PyPI returning None, ok is True.
  4. test_check_package_single_step: with PyPI returning a version whose next patch or next minor
    equals the local version, ok is True. Derive the fake PyPI version from the local one so the
    test does not break at the next version bump, e.g. monkeypatch local_version to return
    "1.4.0" and pypi_latest_version to return "1.3.7" (next minor) and then "1.3.9" with
    local "1.3.10" (next patch).
  5. test_check_package_not_ahead: PyPI equal to local, and PyPI greater than local, both give
    ok is False.
  6. test_check_package_jump: local "1.5.0" with PyPI "1.3.7" gives ok is False and the
    detail mentions 1.3.8 and 1.4.0.
  7. test_workflows: test.yml and publish-pypi.yml exist in WORKFLOWS; test.yaml and
    pip.yaml do not; the text of test.yml contains pytest CompStats and
    scripts/check_package_versions.py and does not contain nosetests; the text of
    publish-pypi.yml contains secrets.TWINE and twine check.
  8. test_pyproject_build_system_and_test_extra: load pyproject.toml with tomllib and assert
    build-system.build-backend == "setuptools.build_meta" and that the test extra contains an
    entry starting with pytest.

The command that must be green before committing (about 2.5 minutes; it prints a progress line per
test file and ends with a line like 82 passed):

.venv/bin/python -m pytest CompStats

Decisions

  • Ubuntu only, Python 3.10-3.12: same matrix as the sibling projects; the package is pure
    Python, so the old Windows cell only existed to upload a wheel.
  • One build job for both sdist and wheel: a pure-Python wheel is platform independent; the old
    split across operating systems was unnecessary.
  • No --skip-existing on upload: with the tag/version check a duplicate upload is a mistake
    and must fail loudly.
  • Coveralls kept, as a single separate job, because the README badge depends on it.
  • requires-python not added: declaring a minimum Python changes what users can install and
    was not requested.
  • No version bump: 0.2.0 is already the next minor after PyPI's 0.1.15.
  • Tests run on pushes to main and develop and on every pull request: develop is the
    integration branch here, unlike the sibling projects.
  • The version-gate script lives in scripts/ and its test in CompStats/tests/: the project's
    test command is pytest CompStats, so a test outside the package would never run.
  • Tagging and publishing 0.2.0 is a manual step for the maintainer, after the work reaches
    main.

Planned against develop at d2c6feade9eac5eb039b405ac272f70fc66458ce.

Activity

  1. mgraffg commented on Nov 19, 2026

    @mgraffg
    MemberAuthor

    Implemented on branch issues-48-49-50, commit 7830a8a.

    • Replaced test.yaml/pip.yaml with test.yml (pytest on 3.10-3.12, Coveralls, version gate) and publish-pypi.yml (tag-driven) exactly as specified.
    • Added scripts/check_package_versions.py, [build-system] + test extra in pyproject.toml, CompStats/tests/test_packaging.py (8 tests), badge URLs, and CLAUDE.md updates (new ## Releasing).
    • Tests: .venv/bin/python -m pytest CompStats green (exit 0); version script prints OK: ok (pypi 0.1.15, local 0.2.0).
    • Decisions: python -m build is not installed in .venv (and nothing may be installed), so I verified the build by calling setuptools.build_meta.build_sdist/build_wheel on a scratch copy: compstats-0.2.0.tar.gz and compstats-0.2.0-py3-none-any.whl built. I also removed the nosetests.xml line from .gitignore so git grep nosetests is empty, as the acceptance criteria require.
  2. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Review fix-1 (commit f4b315a, branch issues-48-49-50): CompStats/tests/test_packaging.py no longer contains the strings nosetests/test.yaml literally (built from fragments), so git grep -e nosetests -e test.yaml is empty as the acceptance criteria require. Full suite: 100 passed.

  3. mgraffg commented on Nov 20, 2026

    @mgraffg
    MemberAuthor

    Implemented in #51, merged into develop. Commit 7830a8a.

  4. added 2 commits that reference this issue on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions