Skip to content

fix: install Poetry with pipx and bump psycopg2-binary and oauthlib - #149

Merged
tkislan merged 3 commits into
mainfrom
tk/install-poetry-with-pipx
Nov 10, 2026
Merged

tkislan merged 3 commits into
mainfrom
tk/install-poetry-with-pipx

Conversation

@tkislan

@tkislan tkislan commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Makes CI green again. This PR does three things:

  1. CI: installs Poetry with pipx instead of snok/install-poetry, whose download from install.python-poetry.org is now blocked with HTTP 403.
  2. psycopg2-binary: 2.9.11 → 2.9.13, merged in from fix(deps): bump psycopg2-binary to 2.9.13 #148.
  3. oauthlib: 4.0.0 → 4.0.1, for CVE-2026-49272, which fails the Production and Server audits.

1. Install Poetry with pipx

Why it broke

  • snok/install-poetry downloads the installer from https://install.python-poetry.org/ with Python's urllib, which sends the default Python-urllib/3.x User-Agent.
  • That host is behind Cloudflare, which now rejects that User-Agent with error code: 1010 (a block on the client's signature). curl and browser User-Agents still get 200:
    $ curl -s -A "Python-urllib/3.11" https://install.python-poetry.org/
    error code: 1010
    
  • It started between the 2026-11-05 runs, which got past this step, and 2026-11-10.
  • The action's latest main.sh makes the same request, so a newer version of the action wouldn't help.
  • Every job that installs Poetry failed within about 8 seconds, including CD's Publish Python package. Only jobs that restored a cached Poetry install got through.

The change

All 8 uses of the action (7 in ci.yml, 1 in cd.yml) become:

      - name: Install Poetry
        run: pipx install --python "$(command -v python)" "poetry==${POETRY_VERSION}"
  • Where Poetry comes from: PyPI, through the pipx the runner image preinstalls (1.16.7 on ubuntu-latest). dockerfiles/test.Dockerfile already installs Poetry this way.
  • Settings the action applied: the action ran poetry config for virtualenvs.create, virtualenvs.in-project and installer.parallel. Both workflows already set these as POETRY_* env vars, which Poetry reads ahead of its config file.
  • --python: keeps Poetry on the setup-python interpreter, as the action did.
  • Cached jobs: the runner image sets PIPX_HOME=/opt/pipx and PIPX_BIN_DIR=/opt/pipx_bin (install-python.sh). The two jobs that cache ~/.local for their Poetry install (CI Test and CD Publish) therefore redirect pipx into ~/.local. Without that, the cache would save nothing, and a later cache hit would leave poetry missing. Existing cache entries in the old layout still restore and work.

2. and 3. Dependency bumps

Audit

  • Command: poetry export <matrix export-args> -f requirements.txt, then pypa/gh-action-pip-audit with require-hashes: true, disable-pip: true, ignoring PYSEC-2024-161 and PYSEC-2026-113, on Python 3.11 (.github/workflows/ci.yml, audit job).
  • Exports: Production (--without dev), Server (--extras server), Dev (--only dev).
  • Before: Production and Server failed on oauthlib 4.0.0 (CVE-2026-49272), on main too.
  • Now: all three are clean.

Precedent

psycopg2-binary 2.9.11 → 2.9.13 (from #148)

  • Advisories: none published. The bump was done the way an audit fix would be. Fixes from the changelog:
    • 2.9.12: infinite loop on a malformed interval.
    • 2.9.13: parsing of malformed bytea input, and of malformed int64 input in arrays.
    • 2.9.13 also adds Python 3.15 and drops 3.9.
  • Path: direct production dependency.
  • Decision: floor 2.9.13 on every Python.
    • The old split existed only because 2.9.10 was the first release with Python 3.13 wheels. 2.9.13 has wheels for 3.10–3.14, so the two lines collapse into one, keeping the <3 ceiling.
    • On 3.10–3.12 the ceiling widens from <2.10 to <3, which changes nothing today because no 2.10 exists.
  • Effect on user environments: when DEEPNOTE_ENFORCE_PIP_CONSTRAINTS is on, the published constraints file now rejects user pins below 2.9.13.
-    "psycopg2-binary>=2.9.4,<2.10; python_version < '3.13'",
-    "psycopg2-binary>=2.9.10,<3; python_version>='3.13'",
+    "psycopg2-binary>=2.9.13,<3",  # 2.9.12 and 2.9.13 fix parsing of malformed interval, bytea and int8-array input

Verified

  • The lock block is byte-identical to what Poetry generates, and all 67 hashes match PyPI.
  • Unit suite unchanged.
  • Round-trip against Postgres 17 with the toolkit's wait_select callback: interval, bytea and extreme int8 values in an array parse correctly, and statement_timeout cancellation works.

oauthlib 4.0.0 → 4.0.1

  • Advisory: CVE-2026-49272 / GHSA-3jp2-3mrg-2vxc (CVSS 3.1 AV:N/AC:L/PR:L/UI:N/S:C/C:L/I:H/A:N). The RFC 8628 device-flow token endpoint did not validate device_code.
  • Why it fails now: fix(deps): resolve pip-audit failures for pymongo, dnspython, oauthlib and pylint #145 treated 4.0.0 as the fix. The advisory was revised on 2026-11-05 to name 4.0.1, which was released on 2026-10-31.
  • Path: transitive, production. It comes in only through databricks-sql-connector (oauthlib>=3.1.0,<5.0.0); the toolkit never imports oauthlib.
  • Decision: raise the existing floor in the transitive security block in place, as fix(deps): resolve pip-audit failures for tornado and pyasn1 #146 did for pyasn1. The comment now names only the CVE that 4.0.1 fixes. PYSEC-2026-4114 is still covered, because it was fixed in 4.0.0.
  • Risk: 4.0.1 changes only the server-side device flow, adds a device_code default on Request, and rewords one error message. The connector uses oauthlib as a client (WebApplicationClient, OAuth2Error), so none of its calls changed and the toolkit was never exploitable. Still, the fix is a patch release, so this follows the precedent of bumping rather than adding an ignore.
-    "oauthlib>=4.0.0,<5",  # 4.0.0 fixes CVE-2026-49272, PYSEC-2026-4114 (transitive via databricks-sql-connector)
+    "oauthlib>=4.0.1,<5",  # 4.0.1 fixes CVE-2026-49272 (transitive via databricks-sql-connector)

Verified

  • The lock block is byte-identical to what Poetry generates, and its hashes match PyPI.
  • databricks.sql.auth.oauth imports, and the connector's client calls give the same output on 4.0.0 and 4.0.1.
  • Unit suite unchanged.

Decided with the user

Checks

Check Result
actionlint 1.7.7 + shellcheck 0.11.0 same 16 pre-existing info findings as main
poetry check --lock (Poetry 2.2.0) exit 0
Lock compared block-by-block with base and with a fresh poetry lock only psycopg2-binary, oauthlib and content-hash differ from main
pip-audit 2.10.1 on Python 3.11 with CI's flags: Production, Server, Dev all clean
pip-audit on an all-extras + dev export, Python 3.11–3.14 clean
pytest tests/unit on Python 3.13 1462 passed, 4 skipped, before and after
CI on ab9f127 all jobs pass, including every audit; Test 3.11 and 3.14 took the cache-miss pipx path
CD on ab9f127 all jobs pass, including Publish Python package and the artifact pushes

Not fixed

The CI audits run on Python 3.11 only. An all-extras + dev export audited on Python 3.10 still shows pillow, dnspython, tornado, filelock and virtualenv. These are the exposures accepted in #143–#146, because none of their fixed versions supports Python 3.10. This PR leaves them unchanged.

Every job that installs Poetry through snok/install-poetry fails with
HTTP 403. The action downloads https://install.python-poetry.org/ with
Python's urllib, and the Cloudflare front of that host now rejects the
Python-urllib User-Agent with error 1010. The action's latest release
makes the same request, so bumping it does not help.

Poetry is installed from PyPI with the runner's preinstalled pipx
instead, as dockerfiles/test.Dockerfile already does. The settings the
action applied are already set by the workflow-level POETRY_* env vars.

The runner image points pipx at /opt, so the two jobs that cache
~/.local for their Poetry install redirect pipx there; otherwise the
cache would save nothing and a later cache hit would leave poetry
missing.
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The CI and CD workflows now install Poetry with pipx and associate it with the active Python interpreter. Cache-miss installation steps set pipx home and binary directories under ~/.local. The workflows remove the previous Poetry setup action and its virtualenv and installer configuration. pyproject.toml updates the psycopg2-binary and oauthlib version requirements.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: m1so

Merge Risk: Minimal · up to ab9f1

Poetry now installs through pipx. No concrete merge-blocking regression was identified; the full CI outcome remains unconfirmed.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Updates Docs Passed The PR changes CI/CD installation and dependency versions. It does not implement a feature that requires documentation updates. The private roadmap is unavailable; please check it if a roadmap update …
Title check Passed The title clearly summarizes the Poetry installation change and dependency updates.
Full details: Updates Docs

Explanation

The PR changes CI/CD installation and dependency versions. It does not implement a feature that requires documentation updates. The private roadmap is unavailable; please check it if a roadmap update is needed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: b8918f26-f9da-4add-a22b-0f305f8ef3e4
Commits

Reviewing files that changed from the base of the PR and between b6b7dbd and ab9f127.

Files ignored due to path filters: 1

  • poetry.lock is excluded by !**/*.lock
Files selected for processing: 1

  • pyproject.toml

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

📦 Python package built successfully!

  • Version: 2.9.1.dev4+ce5d9ce
  • Wheel: deepnote_toolkit-2.9.1.dev4+ce5d9ce-py3-none-any.whl
  • Install:
    pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.9.1.dev4%2Bce5d9ce/deepnote_toolkit-2.9.1.dev4%2Bce5d9ce-py3-none-any.whl"

coderabbitai[bot]
coderabbitai Bot previously approved these changes Nov 10, 2026
@deepnote-bot

deepnote-bot commented Nov 10, 2026 •

Copy link
Copy Markdown

🚀 Review App Deployment Started

📝 Description 🌐 Link / Info
🌍 Review application ra-149
🔑 Sign-in URL Click to sign-in
📊 Application logs View logs
🔄 Actions Click to redeploy
🚀 ArgoCD deployment View deployment
⏰ Last deployed 2026-11-10 14:51:12 (UTC)
📜 Deployed commit e79b719bd6e7995c165ce430847eed8588bbbe63
🛠️ Toolkit version ce5d9ce

psycopg2-binary 2.9.12 and 2.9.13 fix an infinite loop on a malformed
interval and the parsing of malformed bytea and int64-array input. No
advisory is published for them yet; the floor is raised the way past
audit fixes raise one.

2.9.13 requires Python 3.10 and ships wheels for 3.10 to 3.14, so the
per-Python split, which existed only because 2.9.10 was the first
release with 3.13 wheels, collapses into one constraint with the newer
line's <3 ceiling, as was done for pymongo and pylint.

Only the psycopg2-binary block and content-hash were spliced into the
lock.
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.43%. Comparing base (3122452) to head (ab9f127).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #149   +/-   ##
=======================================
  Coverage   78.43%   78.43%           
=======================================
  Files         121      121           
  Lines        6920     6920           
  Branches     1005     1005           
=======================================
  Hits         5428     5428           
  Misses       1182     1182           
  Partials      310      310           
Flag Coverage Δ
combined 78.43% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

CVE-2026-49272 (GHSA-3jp2-3mrg-2vxc) was revised on 2026-11-05 to name
4.0.1 as its fix, so pip-audit fails Production and Server on 4.0.0.
4.0.1 changes only the server-side device flow; databricks-sql-connector,
the only consumer, uses oauthlib as a client and allows <5. The floor in
the transitive security block is raised in place, as for pyasn1.

Only the oauthlib block and content-hash were spliced into the lock.
@tkislan tkislan changed the title fix(ci): install Poetry with pipx instead of snok/install-poetry fix: install Poetry with pipx and bump psycopg2-binary and oauthlib Nov 10, 2026
@tkislan
tkislan marked this pull request as ready for review November 10, 2026 15:00
@tkislan
tkislan requested a review from a team as a code owner November 10, 2026 15:00
@tkislan
tkislan requested a review from m1so November 10, 2026 15:00
@tkislan
tkislan merged commit 6523327 into main Nov 10, 2026
42 of 44 checks passed
@tkislan
tkislan deleted the tk/install-poetry-with-pipx branch November 10, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants