Skip to content

Add SQLAlchemy 2.1.x support and SQLAlchemy test suite integration - #206

Merged
bimalkjha merged 2 commits into
ibmdb:masterfrom
bchoudhary6415:ibmdbsa_test
Nov 18, 2026
Merged

bimalkjha merged 2 commits into
ibmdb:masterfrom
bchoudhary6415:ibmdbsa_test

Conversation

@bchoudhary6415

Copy link
Copy Markdown
Collaborator

Summary

This PR adds support for SQLAlchemy 2.1.x to ibm_db_sa. It also lets
SQLAlchemy's pytest-based test suite for third-party dialects run against
ibm_db_sa, so the dialect can be tested with different SQLAlchemy versions.

Changes

  • SQLAlchemy 2.1.x support: ibm_db_sa now supports SQLAlchemy 1.4.x through 2.1.x.
  • Testing:
    • conftest.py registers the Db2 dialect and loads SQLAlchemy's pytest plugin.
    • setup.cfg holds the test settings and database URLs.
    • ibm_db_sa/requirements.py lists the features Db2 supports.
    • The suite also includes Db2-specific tests.
    • To test another SQLAlchemy version, install it and rerun the tests; no test changes are needed.
  • Packaging:
    • The SQLAlchemy requirement in setup.py is updated, and the nose extra is replaced with a pytest extra.
    • New requirements-test.txt.
    • MANIFEST.in now includes the test files.
  • Docs: New Testing section in README.md and a full guide in README.unittests.md.

How to test

pip install -e .
pip install -r requirements-test.txt
pytest test/test_dialect.py                       # no database needed
pytest --dburi "ibm_db_sa://user:pass@host:port/DBNAME" test/   # against Db2

Signed-off-by: Balram Choudhary <bchoudhary@rocketsoftware.com>
@bchoudhary6415 bchoudhary6415 self-assigned this Nov 17, 2026

@bimalkjha bimalkjha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this. I read the full diff and ran the PR in a temporary worktree against SQLAlchemy 2.0.46. I did not run it against a live Db2, so the dialect-behaviour comments come from reading the code. Requesting changes mainly for items 1, 2, 4 and 6.

Blocking

1. conftest.py overrides SQLAlchemy's pytest_collection_modifyitems

  • from sqlalchemy.testing.plugin.pytestplugin import * imports SQLAlchemy's pytest_collection_modifyitems. conftest.py then defines a function with the same name later in the file, which replaces it.
  • SQLAlchemy's hook expands __backend__ classes per database and applies __only_on__, __requires__ and the backend markers.
  • Confirmed by collecting test/test_suite.py:
    • With the PR's hook, classes appear as CTETest, ArgSignatureTest and so on, with no backend suffix, and pytest warns about an unknown pytest.mark.backend.
    • With the PR's hook renamed so SQLAlchemy's runs, classes appear as BinaryTest_sqlite+pysqlite_3_50_4.
  • So the documented test names (NumericTest_ibm_db_sa+ibm_db_sa_DB2/...) will not appear, and backend gating is silently off.
  • Suggested fix: call SQLAlchemy's hook first, then add the skip markers.
    from sqlalchemy.testing.plugin import pytestplugin as _sa_plugin
    
    def pytest_collection_modifyitems(session, config, items):
        _sa_plugin.pytest_collection_modifyitems(session, config, items)
        ...
    Alternatively, skip these cases through a requirement or exclusion instead of by node ID.
  • -p no:warnings in addopts in setup.cfg hides the symptom.

2. The default [db] URL is SQLite, so a plain pytest runs the compliance suite against SQLite

  • Running pytest test/test_suite.py with no options fails on SQLite (DeprecatedCompoundSelectTest::test_limit_offset_selectable_in_unions failed first). That is misleading, because the PR body and docs present pytest test/ as the full run.
  • Suggested fix: make the Db2-requiring modules skip or error clearly unless the backend is Db2 (for example __only_on__ = "db2" on those classes, or a collection guard in conftest), or have the default fail fast with a clear message.

Should fix

3. Packaging and docs are inconsistent

  • setup.py now says sqlalchemy>=1.4,<2.2 and the README says 1.4–2.1. requirements-test.txt and the docs only test 2.0 and 2.1, so 1.4 is claimed but untested.
  • README says "Python 3.11 to 3.15", but python_requires is still >=3.9. Please align them.
  • Dropping 1.3 in setup.py while base.py and ibm_db.py still carry 1.3 and 1.4 shims is a breaking change. It needs a CHANGES.md entry and a version bump. Renaming the dev extra to test also breaks pip install ibm_db_sa[dev].
  • I did not check whether _de_clone exists in 2.1. The try/except Exception fallback silently turns it into a no-op, which would hide a break rather than surface it.

4. base.py: the OFFSET rewrite is a regression

  • The old regex was anchored to the end (\s+OFFSET ...\s*$). The new loop takes the last \bOFFSET\b anywhere in the SQL and cuts everything after it.
  • A statement with no trailing OFFSET but one in a subquery or CTE (or in a literal or identifier) would be truncated and corrupted.
  • Please keep an end anchor, or check that the captured tail is a single offset token or bind.
  • FETCH FIRST/LIMIT text after the OFFSET would also end up in offset_text.

5. base.py: CTAS and type DDL changes

  • visit_create_table_as ignores element.temporary and element.if_not_exists.
  • visit_unicode_text now emits DBCLOB instead of LONG VARGRAPHIC. That is a DDL change for every user and is not mentioned in the PR description.
  • visit_enum emits VARCHAR(n CODEUNITS32), which is only valid on newer LUW and on z/OS. Please document the minimum server versions, or gate it.

6. base.py: visit_BOOLEAN and visit_enum read DB2Dialect.serverType (class-level global state)

  • It is set by whichever dialect connected last. Two engines in one process (for example LUW and AS400) would render the wrong DDL.
  • Prefer self.dialect.dbms_name, as visit_empty_set_op_expr already does.

7. reflection.py: caching and normalization

  • @reflection.cache on has_table means an Inspector reused across CREATE and DROP returns stale results until clear_cache(). The new _ensure_table_exists also adds an extra query to every reflection call.
  • normalize_name no longer calls _requires_quotes. Uppercase names that need quoting now become lowercase, and mixed-case names become quoted_name(quote=True). This is a behaviour change for existing users and needs a regression test.
  • A genuine unique index created with EXCLUDE NULL KEYS is now indistinguishable from a UNIQUE constraint. That is acceptable, but please document it.

8. ibm_db.py: _homogenize_numeric_columns

  • It silently converts int, float and Decimal values within a column in executemany.
  • On SQLAlchemy 1.4, float(Decimal) and float(int) lose precision for large values.
  • It runs on every executemany and scans every row and column. It should be scoped to Numeric/Float columns.
  • The supports_unicode_binds comment is cut off mid-sentence ("...pre-encodes to UTF-8 bytes and").

9. visit_bindparam and render_literal_value

  • visit_bindparam wraps BigInteger binds in CAST(... AS BIGINT) in every clause. This changes the SQL for every statement that uses a BigInteger bind, and the except Exception swallows errors. Please restrict it to the places that need it and narrow the exception.
  • render_literal_value returns UX'...' for any non-ASCII string. It is untested, and unverified for z/OS and AS400.

Tests and cleanup

  • test/test_dialect.py has 7 weak tests. test_limit_renders and test_offset_uses_rownumber only check for substrings.
  • None of the new compiler logic is covered: the truediv/floordiv casts, the bind CAST, the empty-IN rewrite, the CTAS rendering, the Enum and Boolean AS400 branches, render_literal_value, the OFFSET capture, and the normalize_name and denormalize_name changes.
  • These are all compile-only and can use AssertsCompiledSQL without a database.
  • test_dialect.py calls registry.register at import time, which duplicates conftest.py and is not needed.
  • Nits:
    • quote_identifier is now indented 10 spaces in a 3-space file, and its docstring was removed.
    • Some comments are one-off test notes (for example test_get_view_definition_does_not_exist) and will rot.
    • requirements-test.txt has a stray trailing blank line.
    • README.unittests.md should mention that a plain pytest runs against SQLite (see item 2).

…ect server type

Signed-off-by: Balram Choudhary <bchoudhary@rocketsoftware.com>
@bchoudhary6415

Copy link
Copy Markdown
Collaborator Author

Thanks for the work on this. I read the full diff and ran the PR in a temporary worktree against SQLAlchemy 2.0.46. I did not run it against a live Db2, so the dialect-behaviour comments come from reading the code. Requesting changes mainly for items 1, 2, 4 and 6.

Blocking

1. conftest.py overrides SQLAlchemy's pytest_collection_modifyitems

  • from sqlalchemy.testing.plugin.pytestplugin import * imports SQLAlchemy's pytest_collection_modifyitems. conftest.py then defines a function with the same name later in the file, which replaces it.

  • SQLAlchemy's hook expands __backend__ classes per database and applies __only_on__, __requires__ and the backend markers.

  • Confirmed by collecting test/test_suite.py:

    • With the PR's hook, classes appear as CTETest, ArgSignatureTest and so on, with no backend suffix, and pytest warns about an unknown pytest.mark.backend.
    • With the PR's hook renamed so SQLAlchemy's runs, classes appear as BinaryTest_sqlite+pysqlite_3_50_4.
  • So the documented test names (NumericTest_ibm_db_sa+ibm_db_sa_DB2/...) will not appear, and backend gating is silently off.

  • Suggested fix: call SQLAlchemy's hook first, then add the skip markers.
    from sqlalchemy.testing.plugin import pytestplugin as _sa_plugin

    def pytest_collection_modifyitems(session, config, items):
    _sa_plugin.pytest_collection_modifyitems(session, config, items)
    ...
    Alternatively, skip these cases through a requirement or exclusion instead of by node ID.

  • -p no:warnings in addopts in setup.cfg hides the symptom.

2. The default [db] URL is SQLite, so a plain pytest runs the compliance suite against SQLite

  • Running pytest test/test_suite.py with no options fails on SQLite (DeprecatedCompoundSelectTest::test_limit_offset_selectable_in_unions failed first). That is misleading, because the PR body and docs present pytest test/ as the full run.
  • Suggested fix: make the Db2-requiring modules skip or error clearly unless the backend is Db2 (for example __only_on__ = "db2" on those classes, or a collection guard in conftest), or have the default fail fast with a clear message.

Should fix

3. Packaging and docs are inconsistent

  • setup.py now says sqlalchemy>=1.4,<2.2 and the README says 1.4–2.1. requirements-test.txt and the docs only test 2.0 and 2.1, so 1.4 is claimed but untested.
  • README says "Python 3.11 to 3.15", but python_requires is still >=3.9. Please align them.
  • Dropping 1.3 in setup.py while base.py and ibm_db.py still carry 1.3 and 1.4 shims is a breaking change. It needs a CHANGES.md entry and a version bump. Renaming the dev extra to test also breaks pip install ibm_db_sa[dev].
  • I did not check whether _de_clone exists in 2.1. The try/except Exception fallback silently turns it into a no-op, which would hide a break rather than surface it.

4. base.py: the OFFSET rewrite is a regression

  • The old regex was anchored to the end (\s+OFFSET ...\s*$). The new loop takes the last \bOFFSET\b anywhere in the SQL and cuts everything after it.
  • A statement with no trailing OFFSET but one in a subquery or CTE (or in a literal or identifier) would be truncated and corrupted.
  • Please keep an end anchor, or check that the captured tail is a single offset token or bind.
  • FETCH FIRST/LIMIT text after the OFFSET would also end up in offset_text.

5. base.py: CTAS and type DDL changes

  • visit_create_table_as ignores element.temporary and element.if_not_exists.
  • visit_unicode_text now emits DBCLOB instead of LONG VARGRAPHIC. That is a DDL change for every user and is not mentioned in the PR description.
  • visit_enum emits VARCHAR(n CODEUNITS32), which is only valid on newer LUW and on z/OS. Please document the minimum server versions, or gate it.

6. base.py: visit_BOOLEAN and visit_enum read DB2Dialect.serverType (class-level global state)

  • It is set by whichever dialect connected last. Two engines in one process (for example LUW and AS400) would render the wrong DDL.
  • Prefer self.dialect.dbms_name, as visit_empty_set_op_expr already does.

7. reflection.py: caching and normalization

  • @reflection.cache on has_table means an Inspector reused across CREATE and DROP returns stale results until clear_cache(). The new _ensure_table_exists also adds an extra query to every reflection call.
  • normalize_name no longer calls _requires_quotes. Uppercase names that need quoting now become lowercase, and mixed-case names become quoted_name(quote=True). This is a behaviour change for existing users and needs a regression test.
  • A genuine unique index created with EXCLUDE NULL KEYS is now indistinguishable from a UNIQUE constraint. That is acceptable, but please document it.

8. ibm_db.py: _homogenize_numeric_columns

  • It silently converts int, float and Decimal values within a column in executemany.
  • On SQLAlchemy 1.4, float(Decimal) and float(int) lose precision for large values.
  • It runs on every executemany and scans every row and column. It should be scoped to Numeric/Float columns.
  • The supports_unicode_binds comment is cut off mid-sentence ("...pre-encodes to UTF-8 bytes and").

9. visit_bindparam and render_literal_value

  • visit_bindparam wraps BigInteger binds in CAST(... AS BIGINT) in every clause. This changes the SQL for every statement that uses a BigInteger bind, and the except Exception swallows errors. Please restrict it to the places that need it and narrow the exception.
  • render_literal_value returns UX'...' for any non-ASCII string. It is untested, and unverified for z/OS and AS400.

Tests and cleanup

  • test/test_dialect.py has 7 weak tests. test_limit_renders and test_offset_uses_rownumber only check for substrings.

  • None of the new compiler logic is covered: the truediv/floordiv casts, the bind CAST, the empty-IN rewrite, the CTAS rendering, the Enum and Boolean AS400 branches, render_literal_value, the OFFSET capture, and the normalize_name and denormalize_name changes.

  • These are all compile-only and can use AssertsCompiledSQL without a database.

  • test_dialect.py calls registry.register at import time, which duplicates conftest.py and is not needed.

  • Nits:

    • quote_identifier is now indented 10 spaces in a 3-space file, and its docstring was removed.
    • Some comments are one-off test notes (for example test_get_view_definition_does_not_exist) and will rot.
    • requirements-test.txt has a stray trailing blank line.
    • README.unittests.md should mention that a plain pytest runs against SQLite (see item 2).

Thanks @bimalkjha for the detailed review. All points are addressed in the latest commit. Summary below.

Blocking

1. conftest.py hook: Fixed. pytest_collection_modifyitems now calls sqlalchemy.testing.plugin.pytestplugin.pytest_collection_modifyitems first and then adds the Db2 DECIMAL-limit skips. Test ids now show the backend suffix (e.g. NumericTest_ibm_db_sa+ibm_db_sa_DB2/LINUXX8664_12_01_0600), and __only_on__ / __requires__ apply again. Removed -p no:warnings and registered SQLAlchemy's pytest markers in setup.cfg.

2. Plain pytest runs on SQLite: Fixed. conftest.py now skips test/test_suite.py with a clear reason ("Requires a Db2 database; run with --db db2 or --dburi ") unless the backend is Db2. The other Db2-only classes already use __only_on__. README.unittests.md now states that a plain pytest runs against SQLite.

4. OFFSET rewrite: Fixed. limit_clause records the exact OFFSET text it renders, and visit_select strips only that trailing text. OFFSETs in subqueries/CTEs, literals and identifiers are left untouched. Expression offsets (e.g. 1 + 2) work, and FETCH FIRST is removed before the capture. Added offline tests for subquery OFFSET and expression OFFSET; FetchLimitOffsetTest passes fully on Db2.

6. Class-level serverType: Fixed. The type compiler and do_execute now read self.dialect.dbms_name, so two engines (e.g. LUW and Db2 for i) in one process render their own DDL. Added a test with two dialects side by side. DB2Dialect.serverType is still set for backward compatibility but no longer read.

Should fix

3. Packaging and docs:

  • requirements-test.txt and the docs now cover SQLAlchemy 1.4, 2.0 and 2.1.
  • Python is aligned to >=3.10 in both setup.py and README (SQLAlchemy 2.1 requires 3.10+).
  • dev extra is kept as an alias of test, so pip install ibm_db_sa[dev] still works.
  • _de_clone: it does exist in 2.0 and 2.1 (compiler.py imports it from sql.base). The fallback now only catches ImportError, so a real break would surface.

5. CTAS and type DDL:

  • visit_create_table_as now raises CompileError for temporary=True and if_not_exists=True.
  • UnicodeText → DBCLOB is intentional: LONG VARGRAPHIC is deprecated in Db2. Documented in README.
  • Enum: Db2 for z/OS does not accept CODEUNITS32 in column DDL, so it is now gated: VARCHAR(n CODEUNITS32) on LUW (10.5+), VARCHAR(n) on z/OS, VARCHAR(n) CCSID 1208 on Db2 for i. Documented in README.

7. reflection.py:

  • @reflection.cache on has_table is standard Inspector behaviour (same as other dialects); Inspector.clear_cache() refreshes it. The cache also absorbs the extra _ensure_table_exists check, so it runs once per table per Inspector.
  • normalize_name: Db2 stores unquoted names in uppercase, so uppercase → lowercase and anything else → quoted_name(quote=True) gives a lossless round trip with denormalize_name. The _requires_quotes path broke reserved words such as REF. Added regression tests for uppercase, reserved, mixed-case and lowercase names, including round trip.
  • EXCLUDE NULL KEYS unique index reflected as a UNIQUE constraint: documented in README known limitations.

8. _homogenize_numeric_columns: When bind types are known, it now only touches Numeric/Float columns; other columns are skipped. DECIMAL columns are unified to Decimal (no float precision loss); only FLOAT columns use float. Rows are copied only when a column actually needs coercion. Fixed the cut-off supports_unicode_binds comment. Added unit tests.

9. visit_bindparam / render_literal_value:

  • The BIGINT CAST is kept in all clauses: without it Db2 types the parameter marker as INTEGER and large values overflow (SQL0302N). The exception handler is narrowed to CompileError / NotImplementedError.
  • UX'...' literals are now covered by compile tests, and the literal_non_ascii suite tests pass on Db2. UX Unicode constants are documented for Db2 for z/OS and Db2 for i.

Tests and cleanup

  • test/test_dialect.py now uses exact assert_compile checks and covers truediv/floordiv, BIGINT bind CAST, empty-IN on Db2 for i, CTAS, Enum/Boolean/UnicodeText per server type, UX literals, OFFSET capture, name normalization and numeric homogenization.
  • Removed the duplicate registry.register from test_dialect.py.

Nits

  • quote_identifier re-indented to 3 spaces with docstring restored.
  • One-off test-name comments removed.
  • Trailing blank line in requirements-test.txt removed.
  • README.unittests.md updated for item 2.

@bimalkjha
bimalkjha merged commit 92e6ae1 into ibmdb:master Nov 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants