[henryiii]

Only build one wheel for 3.12+ and 3.15t+ per platform and support all future versions of Python.

Implemented with Fable, added abi3t, got numpy working, got chrono working, iterated with linked Claude sessions on several downstream projects (finding both fixes and performance optimizations to keep this from being to costly), simplified with Fable, and reviewed with Astra, which found five issues. This was originally following #6104, with a commit for each "PR" in that roadmap, but there was quite a bit of iteration, so I opened this as one PR. I'm thinking it might be preferred to review this way these days anyway.

This supports abi3t on 3.15+ and abi3 on 3.12+. Original performance was pretty bad, but optimized a bit and some parts are faster than the unstable ABI (though those optimizations can of course be applied there too later). It depends very much on what you are doing (some things are slower with stable ABI, like attribute access with no call), but for most projects, this should be performant enough. awkward showed a 1-3% drop for one microbenchmark, otherwise unmeasurable. After fixes, iminuit shows no measurable slowdown (one microbenchmark is 17% faster due to an optimization to method-descriptor calls we could also apply to non-stable ABI builds).

Questions:

• Should we auto-detect and use scikit-build-core's cmake variables?

🤖 AI text below 🤖

Description

Adds support for the CPython stable ABI (Py_LIMITED_API >= 0x030C0000, abi3) and the free-threaded stable ABI (abi3t, CPython 3.15+, PEP 803). One *.abi3.so per platform then runs on every CPython from the targeted version on.

Draft: opened so that others can build their projects against it. Tried so far on boost-histogram, iminuit, and awkward. PyTorch can't use it, due to custom code calling C-API functions, but checked to make sure it still works with the mode turned off.

How to try it:

• CMake: pybind11_add_module(example STABLE_ABI example.cpp) or -DPYBIND11_STABLE_ABI=ON (CMake 3.26+, FindPython mode). PYBIND11_STABLE_ABI_VERSION selects the targeted version (default 3.12; 3.15 on free-threaded Python).
• setuptools: Pybind11Extension(..., py_limited_api=True).
• Any other build: define Py_LIMITED_API=0x030C0000 and name the module *.abi3.so.

What changes:

• Type objects are created from PyType_Spec with PyType_FromMetaclass(). This path is also available to regular builds on 3.12+ as the PYBIND11_TYPE_CREATION_VIA_SPEC opt-in. (Edit: now opt-out)
• Field reads of CPython structs go through public functions (PyType_GetSlot(), PyObject_GetTypeData(), checked tuple/list accessors, __mro__/__dict__ lookups).
• Methods are bound through a pybind11 instancemethod type, which is a vectorcall method descriptor, so obj.method(...) does not create a bound method.
• chrono.h uses the datetime types by attribute; numpy.h under abi3t needs NumPy 2.5+.
• Not available under the stable ABI: embed.h, subinterpreter.h, py::custom_type_setup, GIL-held assertions; py::metaclass() is restricted. See docs/advanced/stable_abi.rst for the full table.
• Stable-ABI and regular modules in one process do not share internals (the internals ID gets a _stable / _stable_ft tag); pybind11_conduit_v1 bridges them.

CI builds and runs the test suite for abi3 on 3.12 to 3.14 and abi3t on 3.15t, and runs the 3.12 abi3 binaries on 3.13 and 3.14 and the 3.15t abi3t binaries on GIL-enabled 3.15.

Suggested changelog entry:

• Support the CPython stable ABI (abi3, Py_LIMITED_API >= 0x030C0000) and the free-threaded stable ABI (abi3t, CPython 3.15+). Enable with STABLE_ABI in pybind11_add_module(), py_limited_api=True in Pybind11Extension, or by defining Py_LIMITED_API. Type creation from PyType_Spec is also available to regular builds as the PYBIND11_TYPE_CREATION_VIA_SPEC opt-in.
• Type objects are now created with PyType_FromMetaclass() by default on CPython 3.12+. Define PYBIND11_TYPE_CREATION_VIA_SPEC=0 (CMake -DPYBIND11_TYPE_CREATION_VIA_SPEC=OFF) to restore the previous path.

---

📚 Documentation preview 📚: https://pybind11--6198.org.readthedocs.build/

[henryiii]

If this is better in smaller parts, there are some good candidates for that, including two bug fixes and a nice performance speedup:

🤖 AI text below 🤖

Yes. Beyond the method-descriptor patch, there are two real bug fixes and one large feature that do not depend on Py_LIMITED_API, plus a pile of prep refactors.

Bug fixes (strongest candidates)

• type_is_managed_by_our_internals() cross-module fix (ad10b98, include/pybind11/detail/cpp_conduit.h). Master compares tp_new against this module's pybind11_object_new, which fails for types whose base came from another module. The branch switches to the registry lookup PyPy already used. No test was added on the branch, so it needs one before it ships alone.
• pybind11_traverse/pybind11_clear under PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET on 3.13+ (part of 4d65cb2). Master unconditionally calls PyObject_VisitManagedDict/PyObject_ClearManagedDict on 3.13+, even when that macro puts the dict at tp_dictoffset and the type has no Py_TPFLAGS_MANAGED_DICT. The dict is then never visited or cleared by the GC, at best. I did not run this on master, so verify with a 3.13 build and that macro before filing.

Feature, independent of the stable ABI

• Spec-based type creation / PyType_FromMetaclass default on 3.12+ (fbc9d96, 8f47ff2, c05d08d, a5f341a, 04459a5, e669ac1, 4d65cb2). PYBIND11_TYPE_CREATION_VIA_SPEC is a separate switch from Py_LIMITED_API and already has its own CI job. It is big, but it is the natural first PR: the stable ABI PR then shrinks to "the spec path, plus limited-API gating".

Method descriptor / instancemethod (the one you named)

• 73cd296 and 721fabc give pybind11's own instancemethod type vectorcall and Py_TPFLAGS_METHOD_DESCRIPTOR. On the branch that type exists only under Py_LIMITED_API; master uses PyInstanceMethod_Type, which lacks the flag. A standalone PR has to make the own type unconditional. If you do that, 973e9ef comes along for free: the metaclass tp_getattro can be dropped everywhere, which removes the slow MRO walk on class attribute reads, and the interned-name type_lookup() speedup applies to regular builds too.

Prep refactors, low value alone but they shrink the main diff

• Traceback walk by attribute in error_already_set::what() (683f376).
• PyThreadState_GetInterpreter() instead of tstate->interp, with the PyPy fallback (5c50c3d, part of e0bc493).
• get_bases()/get_mro()/get_tp_name()/tp_name_is_one_of() helpers (d1af605, 1865f0b) and the PyTuple_GET_ITEM-style macro replacements (cdbc9b4).
• value_and_holder::type_is_exact() replacing the repeated Py_TYPE(v_h.inst) == v_h.type->type in init.h and pybind11.h.
• The cross-module test modules factored into an exec function (test-only).

Not separable

• The chrono per-TU fix (88fb22a) fixes helpers the branch itself introduced; master inlines the macros, so it does not apply.
• SABIModule export (573d161), eval.h, numpy.h, complex.h, the TLS/critical-section changes, and the conftest compile-time version logic are stable-ABI-only.

[rwgk]

@henryiii this is the first pass reviewing with

codex GPT-6.1-Sol ultra:

Rechecked against 75be228. All six findings remain after the rebase:

1. [P1] Pin registry candidates before acquiring the internals lock.
   get_object_handle() retains registry iterators and raw instances across values_and_holders(), which acquires the internals lock through all_type_info(). Under abi3t, contention suspends the outer registry critical section, allowing another thread to erase entries or rehash the shard. py::get_override() can then resume with invalid iterators or freed wrappers. Please collect strong candidates under the registry lock and inspect them afterward, as find_registered_python_instance() already does, retaining the selected wrapper through override lookup.

2. [P2] Keep dictionary-offset helpers available for non-limited builds.
   Defining PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET in a normal CPython 3.12+ build now makes including pybind11.h fail to compile. The new spec path calls type_dictoffset and type_basicsize, but both helpers exist only under Py_LIMITED_API. I reproduced this with CPython 3.14.4 headers; the same include compiles without that macro. Please provide non-limited equivalents or retain direct field access for this configuration.

3. [P2] Check base metaclasses before selecting the spec path.
   The fallback condition checks only the requested/default metaclass. Binding Base with py::metaclass(Meta), where Python defines class Meta(type): pass, selects the legacy path. Subsequently binding py::class_<Derived, Base>(m, "Derived") selects the spec path with the default pybind11_type. PyType_FromMetaclass() then raises a metaclass conflict; the previous PyType_Ready path accepts this binding. Please consider the base metaclasses when deciding whether to fall back.

4. [P2] Recognize Windows free-threaded Python suffixes.
   The fallback detection matches cpython-…t, whereas Windows Python 3.15t uses cp315t-win_amd64 / .cp315t-win_amd64.pyd. Since Python_FREE_THREADED was added in CMake 4.4, CMake 3.30–4.3 misdetects this interpreter as GIL-enabled. Default STABLE_ABI consequently targets 3.12 and fails compilation because free-threaded stable ABI requires 3.15.

5. [P2] Preserve FindPython paths for Windows abi3t consumers.
   With add_subdirectory(pybind11) and Python discovery inside that subdirectory, Python_INCLUDE_DIRS and Python_SABI_LIBRARY_DIRS remain scoped there. A caller using GIL-enabled Windows Python 3.15+, PYBIND11_ABI3T=ON, and pybind11_add_module(... STABLE_ABI ...) reaches the manual setup with empty paths. find_library(... NO_DEFAULT_PATH REQUIRED) fails even when python3t.lib is installed. Please obtain these paths from the global imported target or preserve them for callers.

6. [P2] Include the second chrono translation unit when selecting chrono tests.
   -DPYBIND11_TEST_OVERRIDE=test_chrono excludes the independently listed test_chrono_second_tu.cpp, while test_chrono.py unconditionally imports its submodule. The extension builds, but pytest fails collection with ImportError. Please include the helper translation unit whenever chrono is selected.

Finding 2 was reproduced through compilation at this revision. The others are based on code inspection; no test suites were run.

[henryiii]

I'm going to revisit this soon, I don't agree with all the review fixes I applied. "Pin registry candidates before acquiring the internals lock" specifically is wrong, we should not be returning an object instead of a handle from get_object_handle.

[Copilot]

🟡 Changes recommended

Unresolved critical issues can cause use-after-free during free-threaded override lookup and deadlocks during concurrent initialization of Python-importing caches.

8 open issues

• Avoid GIL and static-guard deadlock during chrono initialization · New
• Avoid GIL and static-guard deadlock importing types · New
• Return and retain an owning wrapper object · New
• Bind complex descriptors before calling · New
• Encode the limited-API version in guard symbols · New
• Read structural type attributes through original descriptors · New
• Derive _abi3t from final ABI macros · New
• Set importable module naming independently of ABI tags · New

What changed in this PR

Adds CPython stable-ABI support to pybind11, allowing downstream extensions to reuse binaries across Python versions, including free-threaded builds.

Changes:

• Adds abi3 and abi3t build options, naming, and compatibility checks.
• Adapts type creation, object access, conversions, and internal state for the limited API.
• Documents supported configurations and expands regression and cross-version CI coverage.

File	Description
tools/​pybind11Tools.cmake	Rejects stable ABI in classic discovery mode.
tools/​pybind11NewTools.cmake	Configures stable-ABI targets, linkage, and suffixes.
tools/​pybind11Common.cmake	Supports stable-ABI precompiled libraries.
tools/​check_inl_headers.py	Recognizes new ABI configuration macros.
tests/​test_thread.py	Skips unavailable native-thread lock tests.
tests/​test_thread.cpp	Guards native-thread lock fixtures.
tests/​test_pytypes.cpp	Adapts iterator tests for limited API.
tests/​test_multiple_interpreters.py	Skips unsupported subinterpreter tests.
tests/​test_multiple_inheritance.py	Tests multiple dynamic-attribute bases.
tests/​test_multiple_inheritance.cpp	Adds dynamic-attribute inheritance fixtures.
tests/​test_modules.cpp	Avoids opaque module definitions under abi3t.
tests/​test_methods_and_attributes.py	Tests method descriptors and metaclass behavior.
tests/​test_methods_and_attributes.cpp	Adds metaclass-conflict fixtures.
tests/​test_kwargs_and_defaults.cpp	Uses public tuple accessors.
tests/​test_custom_type_setup.py	Skips unavailable custom-type fixtures.
tests/​test_cross_module_rtti/​CMakeLists.txt	Keeps embedding bindings outside stable ABI.
tests/​test_cmake_build/​test.py	Checks expected stable-ABI filenames.
tests/​test_cmake_build/​subdirectory_stable_abi/​CMakeLists.txt	Adds subdirectory stable-ABI integration tests.
tests/​test_cmake_build/​CMakeLists.txt	Registers and configures ABI build tests.
tests/​test_class_release_gil_before_calling_cpp_dtor.py	Skips unavailable GIL-check fixtures.
tests/​test_chrono.py	Tests subclass fields and translation-unit initialization.
tests/​test_chrono_second_tu.cpp	Adds a second chrono translation unit.
tests/​test_call_policies.py	Conditions GIL tests on fixture availability.
tests/​test_call_policies.cpp	Guards unsupported GIL-state checks.
tests/​test_builtin_casters.py	Tests complex-conversion edge cases.
tests/​pybind11_tests.cpp	Exposes compile-time Python and ABI metadata.
tests/​extra_setuptools/​test_setuphelper.py	Tests stable-ABI builds and version selection.
tests/​exo_planet_c_api.cpp	Adds the abi3t module export hook.
tests/​env.py	Exposes limited-API test configuration.
tests/​cross_module_interleaved_error_already_set.cpp	Supports abi3t module initialization.
tests/​cross_module_gil_utils.cpp	Supports abi3t GIL-test module exports.
tests/​conftest.py	Normalizes annotations using the build Python version.
tests/​CMakeLists.txt	Configures stable-ABI test selection.
pybind11/​setup_helpers.py	Adds ABI selection, filenames, and export handling.
include/​pybind11/​trampoline_self_life_support.h	Uses opaque-safe instance access.
include/​pybind11/​subinterpreter.h	Uses public interpreter-state accessors.
include/​pybind11/​stl.h	Adapts iterable checks and list access.
include/​pybind11/​pytypes.h	Adds limited-API object and type helpers.
include/​pybind11/​pytypes-inl.h	Removes direct traceback and code-field access.
include/​pybind11/​pybind11.h	Adds abi3t exports and opaque-safe operations.
include/​pybind11/​pybind11-inl.h	Adapts dispatch, overrides, and module initialization.
include/​pybind11/​numpy.h	Uses NumPy’s abi3t data accessors.
include/​pybind11/​gil_simple.h	Guards unavailable GIL assertions.
include/​pybind11/​functional.h	Uses a compatible function-self accessor.
include/​pybind11/​eval.h	Adds limited-API evaluation paths.
include/​pybind11/​embed.h	Rejects unsupported stable-ABI embedding.
include/​pybind11/​detail/​value_and_holder.h	Adds opaque-safe holder access.
include/​pybind11/​detail/​typeid.h	Handles incomplete PyObject types.
include/​pybind11/​detail/​type_caster_base.h	Adapts instance and holder conversions.
include/​pybind11/​detail/​type_caster_base-inl.h	Adds weak-reference-based abi3t instance lookup.
include/​pybind11/​detail/​internals.h	Adds ABI isolation, opaque storage, and locking.
include/​pybind11/​detail/​internals-inl.h	Uses critical sections for registry counting.
include/​pybind11/​detail/​init.h	Adapts constructor and pickle instance access.
include/​pybind11/​detail/​function_record_pyobject.h	Supports opaque function-record storage.
include/​pybind11/​detail/​cpp_conduit.h	Removes an obsolete forward declaration.
include/​pybind11/​detail/​cpp_conduit-inl.h	Uses registry and descriptor lookup helpers.
include/​pybind11/​detail/​common.h	Defines ABI gates, exports, and instance-layout helpers.
include/​pybind11/​detail/​class.h	Declares stable-ABI type and method helpers.
include/​pybind11/​detail/​class-inl.h	Implements spec-based types and method descriptors.
include/​pybind11/​complex.h	Adds limited-API complex conversion.
include/​pybind11/​chrono.h	Adds descriptor-based datetime conversion.
include/​pybind11/​cast.h	Adapts casts, tuple writes, and GIL checks.
include/​pybind11/​attr.h	Guards unsupported custom type setup.
include/​pybind11/​attr-inl.h	Detects inherited dynamic attributes without struct access.
docs/​index.rst	Links the stable-ABI guide.
docs/​compiling.rst	Documents build options and precompile constraints.
docs/​advanced/​stable_abi.rst	Explains setup, limitations, and ABI isolation.
CMakeLists.txt	Exposes the type-creation mode override.
.github/​workflows/​ci.yml	Adds stable-ABI, cross-version, and legacy-path coverage.

_{Review effort: Balanced}

[rwgk]

codex GPT-6 on behalf of Ralf:

I prepared ten suggested-fix commits atop 1ea0ea0, one per issue, each with regression coverage. The clean series is in #6225, an open DO NOT MERGE PR used for CI validation.

1. 80ba2963fdd6 — Keep the selected wrapper owned throughout override lookup; cover classic and smart-holder lifetimes.
2. 585af4233d22 — Bypass metaclass overrides when reading structural type metadata.
3. 9621441ff9f7 — Preserve inherited dictionaries and Python-slot GC traversal, including cycles.
4. cd1fee7d2bc4 — Keep bound and fallback class layouts consistent in dictionary-offset compatibility mode.
5. 5791328f55a2 — Initialize datetime and MethodType caches with GIL-safe once handling.
6. e0cc3e72515e — Bind __complex__ descriptors with Python special-method semantics.
7. def221569a2f — Expose Python headers through the public CMake target, including Windows abi3t consumers.
8. 1f513f5e0016 — Keep WITHOUT_SOABI modules importable by applying Python prefix/suffix rules.
9. 3416a32beaa8 — Derive filenames and Windows export symbols from the effective explicit ABI macros.
10. 61f504dc713e — Encode the numeric limited-API version in the link guard and reject incompatible precompiled CMake target reuse.

Full CI is green on both initial validation #6224 and clean series #6225: each has 97 passing checks, two expected skips, and all three external status contexts passing.

The clean series is tree-identical to the initial validation PR #6224, with CI follow-ups folded into the corresponding commits. Local full default/free-threaded and abi3 suites passed, as did C++11 precompiled checks and the same abi3t binary on GIL-enabled and free-threaded Python 3.15. Independent review found no remaining significant findings.

These commits are offered for adoption or adaptation in the listed order. Neither validation PR is intended for merging.

[henryiii]

1. Converts get_object_handle from retuning a handle to returning an object! It's literally in the name. I think this is just the AI being pedantic; it's a really slim edge case with multiple threads and one of them dropping a vector while the other reads it. If we really want to plug the hole, we need to introduce a new function. The test here is circular. I already reverted Claude's attempt to add this earlier.
2. Another technically correct but pedantic change, with a costly solution; this is a huge cost on every lookup for no real gain. CPython reads tp_bases and non-limited pybind11 does too.
3. This is a bug fix for master, not related to stable ABI. Also a bit costly, adds a registry lookup per base.
4. Looks good, though it can be made a little simpler.
5. Good.
6. Correct though minimal. No cost other than more lines, so let's take it.
7. Yep.
8. Good, and 6 lines.
9. Overbuilt, but core fix is good. Tests is more expensive than needed.
10. We don't need an 80+ line CMake test for this, but core is good.

So I'll adopt 4-10, with some modifications.

[henryiii]

🤖 AI text below 🤖

Cherry-picked unchanged (5 to 8) with Ralf's authorship. I left the <pybind11/gil_safe_call_once.h> include as is since class.h already uses angle brackets.

Rewritten (4, 9, 10), each noting the source commit in the message:

• 4: common.h now defaults PYBIND11_TYPE_CREATION_VIA_SPEC to 0 when PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET is user-defined, and errors if both are set explicitly. The class-inl.h condition is untouched. Kept the docs sentence and the test. Note the test passes on every path, so it only guards the layout mix, not the macro case. No CI job defines that macro.
• 9: the explicit macros are a list, validated in a loop, with abi3t set if any is Py_TARGET_ABI3T. Kept the unit test; dropped the extra real-build parametrization.
• 10: header, docs, and CMake same-tree check adopted. The 83-line precompile_guard.cmake test and its subproject were dropped. I verified by hand that the mangled guard symbol differs between 0x030C0000 and 0x030D0000 and is identical for 0x030C0000 and 0x030c0000.

[henryiii]

🤖 AI text below 🤖

Downstream testing

I built 12 downstream projects against this branch (macOS arm64) in three ways:

1. A regular build. If tests failed, I compared with the project's pinned or released pybind11.
2. An abi3 module, built on CPython 3.12. I installed the same wheel or module on 3.12, 3.13, 3.14 and 3.15 and ran the project's tests.
3. An abi3t module, built on free-threaded 3.15.0rc3t. I ran it on 3.15t and on GIL-enabled 3.15.

I also ran abi3audit on each module and found no violations.

Project	Build path	Regular	abi3	abi3t	Changes needed in the project
boost-histogram	scikit-build-core	pass, same as 3.0.4	pass	pass	CMake snippet from the docs, wheel.py-api
iminuit	scikit-build-core	pass	pass	pass, GIL stays off	CMake snippet from the docs, wheel.py-api
awkward-cpp	scikit-build-core	pass	pass	pass, GIL stays off	Also add ${SKBUILD_SABI_COMPONENT} to its own find_package(Python)
correctionlib	scikit-build-core	pass	pass	pass, GIL stays off	CMake snippet from the docs, wheel.py-api. Its C++ library in the same wheel gets no Python flags.
contourpy	meson-python	pass	pass	pass, GIL stays off	limited-api = true, and limited_api set to '3.15' on free-threaded Python
matplotlib	meson-python	no regressions	pass	pass, GIL stays off	Same as contourpy. _macosx uses the raw C API, so it was not built.
hnswlib	setuptools	pass	pass	pass	Pybind11Extension(py_limited_api=...). Change PYBIND11_PLUGIN to PYBIND11_MODULE for abi3t.
pikepdf 10.5.1	setuptools	no regressions	pass	pass	setup.py only
cvxpy (pybind11 module)	setuptools	no regressions	pass	pass	None for pybind11. SWIG cannot target abi3t.
dlib	plain CMake	pass, same as 3.0.4	pass	pass, also with PYBIND11_ABI3T=ON	Its own use of CO_VARARGS
OpenTimelineIO	plain CMake	pass	pass	pass, also with PYBIND11_ABI3T=ON	Add Development.SABIModule to its own find_package(Python)
pyhepmc	setup.py that runs CMake	pass	pass	pass	Its own PyByteArray_AS_STRING and Py_SET_SIZE. Pass -DPython_EXECUTABLE.

"No regressions" means some tests failed, but the same tests also failed with released pybind11. The causes were outside pybind11: Hypothesis, solver versions, and LaTeX or Tk in the environment.

Some test runs had one or two extra limits:

• hnswlib's abi3 wheel did not run on 3.13.
• For cvxpy, several solvers have no 3.15 or 3.15t wheels, so some tests could not run. The failures were the same as with a regular build in the same environment.

Other checks:

• pybind11's own Eigen tests passed under abi3, abi3t and PYBIND11_ABI3T=ON (159 tests each).
• Under abi3t, weakref.getweakrefcount() is one higher, as the docs say. No downstream test depended on it.
• Multithreaded stress tests passed on 3.15t with the GIL off for iminuit, matplotlib, OpenTimelineIO, cvxpy and dlib.

What I changed on this branch:

• PYBIND11_PLUGIN now fails under abi3t with a clear static_assert. Before, users got a confusing overload error.
• An empty PYBIND11_STABLE_ABI_VERSION now means the default. scikit-build-core before 0.11 does not set SKBUILD_SABI_VERSION, so the documented snippet gave an empty value.
• Doc notes:
  • Development.SABIModule when a project calls find_package(Python) itself;
  • the meson-python setup;
  • the bdist_wheel tag for setuptools;
  • the .abi3-<platform>.so / .abi3t-<platform>.so suffixes;
  • build abi3 wheels with the Python version you target, and abi3t wheels on free-threaded Python;
  • pass Python_EXECUTABLE when a setup.py runs CMake;
  • every module in a stable-ABI wheel must use the stable ABI;
  • numpy>=2.5 for abi3t;
  • export PyModExport_* when you restrict exported symbols;
  • Py_GIL_DISABLED is always defined under abi3t.

I also found a problem that this PR did not cause: Pybind11Extension changes the caller's argument lists in place. The fix is in #6226.

[rwgk]

@henryiii I'm dumping the codex findings here. I haven't yet looked carefully myself.

Feeding back codex finding is something I can easily do on the side, to keep the ball rolling here. Looking carefully myself will have to wait until the weekend. I hope that's OK.

codex GPT-6.1-Sol ultra:

I re-reviewed 6198 at 3c6c309 against current master, accounting for merged PRs 6228–6231. I found two new issues I’d fix before approval:

1. [P1] Generator cache initialization can deadlock.
   stl.h:88–89 imports types inside a C++ static initializer. If that import releases the GIL, another converting thread can hold the GIL while waiting for the static guard. Neither can proceed. Ordinary cold imports reproduced this 6/6 times; using the existing gil_safe_call_once_and_store helper passed 6/6 controls.

2. [P2] abi3t NumPy scalar conversion can return a dangling pointer.
   numpy.h:1862–1868 accepts numpy.void subclasses, obtains their overridable buffer, then releases it before using its address. A scalar storing (1, 2) converted as (10, 20) under abi3t while the regular build returned (1, 2). A weak-reference probe confirmed the temporary backing storage was destroyed before the C++ function used the pointer.

[rwgk]

New codex finding. I still haven't look carefully myself.

At 6f94025, the generator deadlock is fixed: all six cold-import reproductions now complete. One P2 remains in the NumPy fix.

numpy.h:1872 calls generic PyBuffer_Release() after explicitly acquiring through NumPy’s base implementation. Release dispatches through the subclass, calling its __release_buffer__ even though its __buffer__ was bypassed.

I reproduced this against the new head: conversion returns the correct (1, 2), but the subclass records a release without an acquisition. A hook expecting matching acquisition state raises an unraisable exception. The fix should pair the base getter with the corresponding release handling, and the regression should cover both hooks.

I’d address that remaining release mismatch before approval.

[rwgk]

@henryiii I added a small commit to fix the codex finding.
I'll still want to work through the changes manually.

[rwgk]

This is a huge change:

$ git diff master --stat | tail -n 1
 71 files changed, 3443 insertions(+), 482 deletions(-)
$ git diff master --stat -- include | tail -n 1
 28 files changed, 1777 insertions(+), 413 deletions(-)
$ git diff master --stat -- tests | tail -n 1
 34 files changed, 757 insertions(+), 48 deletions(-)

I'm approaching the review like this:

My prompt

Could you please walk me through the changes in 6198?

Questions on my mind:

What is the "Stable ABI"? — I have ideas about it, but I never got to looking / understanding systematically.

Why does using the Stable ABI (apparently) impact runtime performance?

With respect to the large-scale code changes: can we crystallize out a few major themes that explain large swaths of the changes?

What are "interesting special cases" in the pile of changes that deserve dedicated attention?

---

My time budget: 90 minutes

codex GPT-6.1-Sol ultra

codex response

The central change is this: pybind11 gains a way to produce extension binaries that continue working across CPython releases, by giving up assumptions about CPython’s internal layouts. Much of the diff implements the consequences of that decision.

I examined the current PR head, bdcd3c34: 71 changed files. For your 90 minutes, I suggest this route. The explanation below supplies the map; the linked source supplies a manageable selection of code to read.

Time	Focus
0–15 min	API, ABI, and the compatibility promise
15–30 min	Why performance changes
30–50 min	Type creation and object storage
50–65 min	Method binding and wrapper lifetime
65–80 min	Datetime and NumPy special cases
80–90 min	Restrictions, packaging, and compatibility tests

The starting point is API versus ABI. An API describes how source code uses a library: declarations, functions, types, and their meanings. An ABI describes what already-compiled machine code expects: exported symbols, calling conventions, and relevant memory layouts.

Suppose an extension reads a field from a CPython object. The compiler turns that into something like “load the pointer at this offset.” If a later CPython release moves that field, recompiling the source might produce a working extension—but the existing binary still uses the old offset. That is an ABI problem. Changes to object layouts were a central motivation for the original Stable ABI proposal, PEP 384.

The next distinction is Limited API versus Stable ABI:

Term	Meaning
Limited API	The subset of CPython’s C API that extension source agrees to use
Stable ABI	The binary compatibility contract supported by that subset

You select the restrictions when compiling, usually by defining something such as:

#define Py_LIMITED_API 0x030C0000  // Target the Python 3.12 Limited API

CPython’s headers then hide facilities that the extension must avoid. The resulting binary can use the corresponding Stable ABI. The distinction between the source-facing API and binary-facing ABI is described explicitly in PEP 652.

For this PR, the useful compatibility matrix is:

Build mode	Intended CPython runtimes
Ordinary, version-specific build	Matching Python version/build
abi3, targeting 3.12	GIL-enabled CPython 3.12 and later
abi3t, targeting 3.15	Both GIL-enabled and free-threaded CPython 3.15 and later

Stable ABI itself dates back to Python 3.2. The 3.12 minimum is a requirement of this pybind11 implementation, notably its use of PyType_FromMetaclass(). The new free-threading-compatible ABI, abi3t, is defined by PEP 803.

“One binary” still means one binary per suitable platform. This does not make a Linux extension run on Windows, make incompatible C++ toolchains compatible, or remove dependencies on other native libraries.

---

The runtime cost comes from moving work across that compatibility boundary. A direct field access embeds knowledge in the extension. A stable function call lets the running CPython interpreter supply that knowledge.

A particularly good first source example is the tuple/list helper block:

// Ordinary build:
return PyTuple_GET_ITEM(tup, i);

// Stable ABI build:
return PyTuple_GetItem(tup, i);

The macro can become a direct array access. The function provides checked access without the extension knowing the tuple layout. CPython documents this loss of direct access and inlining as a potential Stable ABI performance cost.

Two qualifications matter:

• A C function call does not automatically allocate a Python object. Both operations above return a borrowed reference.
• Other replacements involve substantially more work than one function call.

For example, stable-ABI tuple/list iteration cannot simply advance a pointer through CPython’s element array. Its fallback obtains elements through the sequence API, with owning references to manage. Some private type lookups become walks through the method-resolution order and class dictionaries.

The impact therefore depends on the workload:

runtime = C++ computation + Python-facing dispatch and conversion work

A substantial C++ computation can amortize additional boundary costs. Thousands of tiny calls, repeated attribute access, or element-by-element conversion expose those costs much more clearly.

As an illustration—not a benchmark—if boundary work occupies 10% of a program and becomes 20% more expensive, total runtime increases by roughly 2%.

There is also an interesting countervailing optimization in this PR. CPython’s PyInstanceMethod_Type is unavailable in the Stable ABI, so pybind11 supplies a replacement. That replacement supports an immediate-call optimization:

C.method          # Access through the class
saved = obj.method  # Obtain a bound method that remembers obj
obj.method(...)   # Call immediately

The last expression can pass obj directly into the call, avoiding a temporary bound-method object. The implementation uses the method-descriptor flag and vectorcall. See the replacement descriptor.

That helps explain Henry’s reported downstream results: small or unmeasurable differences in several workloads, and an improvement in one method-call microbenchmark. Those are workload-specific measurements reported in the PR, rather than a general overhead guarantee.

---

Five themes organize most of the code changes.

Theme	What it explains
Replace layout-dependent access	Many small changes in pytypes.h, casters, argument handling, and STL conversion
Describe types through supported APIs	Changes to class creation, slots, allocation, dynamic attributes, and metaclasses
Accommodate opaque object headers	Many pointer conversions and the additional abi3t machinery
Recreate unavailable facilities while preserving behavior	Method objects, datetime conversion, complex conversion, evaluation
Make binary reuse work end to end	Internals isolation, CMake, setuptools, filenames, wheel metadata, and CI

The small helper changes are a good place to begin. For example:

type->tp_alloc(type, 0)

becomes, on the restricted path, a slot lookup followed by a call. The allocation/free helpers show both implementations together.

Many ordinary builds retain their direct-access implementation behind these helpers. However, some refactoring and correctness changes affect ordinary builds too; the entire PR is not isolated behind one switch.

Type creation is the next major step. Historically, pybind11 could allocate a CPython heap-type structure and populate its fields. The restricted implementation instead supplies a recipe:

• The type’s name and flags.
• Its required storage.
• Numbered slots containing callbacks for operations such as initialization, destruction, and buffer access.

CPython constructs the actual type from that recipe.

An important baseline detail: the default PyType_Spec / PyType_FromMetaclass() path for ordinary CPython 3.12+ builds already landed through #6231. It is essential background for understanding #6198, but the whole mechanism is not new in the remaining diff. #6198 requires that path for Stable ABI builds and adds the restricted implementations.

Read class creation with that distinction in mind.

Two consequences deserve attention here.

First, the Stable ABI does not provide the managed-dictionary facility used by ordinary builds. The PR reserves one dictionary pointer in every stable-ABI instance, including instances whose classes do not enable dynamic attributes. This keeps instance sizes compatible for multiple inheritance. Reserving the pointer does not eagerly allocate a dictionary.

Second, ordinary builds can fall back to the legacy type-creation path for certain customizations. Stable builds cannot. This produces genuine restrictions on custom metaclasses and makes py::custom_type_setup unavailable.

---

abi3t goes further: even the Python object header becomes opaque. This is the key to understanding many cast replacements.

In the traditional layout, pybind11’s instance begins with PyObject_HEAD. The Python-object pointer and pybind11-instance pointer identify the same starting address.

With abi3t, pybind11 cannot embed a header whose size it does not know:

Traditional:
    [Python header | pybind11 payload]
     ^
     PyObject* and instance*

abi3t:
    [opaque Python-owned portion | pybind11 payload]
     ^                             ^
     PyObject*                     instance*

CPython provides a mechanism for requesting extra storage without knowing the base object’s layout. That is the purpose of PEP 697’s type-data facilities.

The PR introduces a pair of explicit conversions:

get_instance(python_object)
instance_object(pybind11_instance)

On traditional builds these reduce to the familiar casts. Under abi3t, they account for the payload’s offset. See the conversion helpers.

This explains changes spread across constructors, holders, casters, and trampolines. For example, reference counting must act on the Python object, not a pointer into its interior payload.

The opaque header also changes module initialization. PyModuleDef cannot be instantiated when it is incomplete, so abi3t uses the new PyModExport_<name> entry point and returns a description built from slots. Binding authors still write PYBIND11_MODULE. See the macro implementation and PEP 793.

---

Within those themes, I would give these special cases dedicated attention.

Method binding is both a semantics problem and a performance opportunity. The replacement must preserve class access, subclass access, aliases, __self__, __func__, and overloaded calls while supporting the immediate-call optimization described earlier.

The new method test is an unusually useful reading aid: it makes the required behavior concrete before you inspect the descriptor implementation.

The abi3t instance registry has a consequential lifetime problem. The registry lets pybind11 find an existing Python wrapper for a C++ address. A raw pointer to that wrapper does not keep it alive. Without the GIL, another thread may be destroying it during lookup.

The ordinary free-threaded implementation uses facilities such as PyUnstable_TryIncRef(), which are unavailable in the Stable ABI. The new path maintains a weak reference and asks CPython to obtain a strong reference safely through PyWeakref_GetRef().

There is a second complication: Python critical sections can be suspended while acquiring another critical section. The code therefore gathers strong references under the registry lock, releases that lock, and then examines types. It must avoid keeping registry iterators across operations that could suspend the protection.

This is where I would spend the most careful concurrency-reading time: wrapper lookup. The additional weak reference is observable through Python’s weakref API.

Datetime conversion shows why “replace the field access with an attribute access” can be wrong. The datetime C API is outside the Stable ABI. A tempting replacement would be:

value.year

But a datetime subclass can override year. The existing C macros read the actual stored year, regardless of that override.

The PR preserves that behavior by obtaining the base datetime type’s field descriptor and invoking it on the instance. The implementation also illustrates a real performance tradeoff: stored-field access becomes a cached descriptor call and conversion.

There is a separate C++ wrinkle affecting ordinary builds: PyDateTimeAPI is static per translation unit. The helpers need suitable linkage so initialization through one translation unit cannot be paired with another unit’s still-null API pointer. That explains the static inline helpers and second-translation-unit regression.

NumPy introduces another ABI boundary. Its array and dtype layouts are separate from CPython’s ABI. pybind11 has historically used proxy structures describing those layouts. Under abi3t, it cannot assume a fixed Python header inside those proxies.

The new path uses accessors supplied by NumPy 2.5 and rejects older versions at runtime. Consequently, a project using numpy.h in an abi3t wheel needs an appropriate NumPy runtime requirement. CPython ABI stability does not remove that dependency.

The scalar-buffer conversion is particularly instructive—and is the final commit at this PR head. An np.void subclass can override __buffer__ to return unrelated or temporary memory. Conversion to a C++ record needs the scalar’s actual storage.

The implementation therefore calls the base NumPy type’s buffer-acquisition slot. It must pair that with the same base type’s release slot: generic PyBuffer_Release() could dispatch to the subclass’s release hook even though its acquisition hook was bypassed. Read the short conversion block.

The different meanings of “ABI” matter inside pybind11 too. Ordinary, abi3, and abi3t modules can coexist, but their private pybind11 structures differ. They receive distinct internals identifiers so they do not blindly share incompatible registries.

The C++ platform ABI identifier remains separate, allowing the conduit protocol to bridge compatible C++ objects across those internals boundaries. The internals-tag comment captures this distinction well.

---

The final ten minutes should connect the implementation to its compatibility promise.

There are four separate things to keep aligned:

1. The Python headers used while compiling.
2. The minimum API version the binary is permitted to use.
3. The extension’s filename and exported entry point.
4. The wheel’s compatibility metadata.

For example, compiling with 3.14 headers while targeting the 3.12 Stable ABI does not permit references to functions introduced in 3.14. The new PYBIND11_API_VERSION_HEX expresses the permitted API floor. This explains many preprocessor-condition changes in the diff.

Likewise, an .abi3 filename alone does not make a binary compatible. The CMake and setuptools changes arrange the compiler definitions, linking, suffixes, and export symbols. The packaging caveats are worth reading because a compatible binary in an incorrectly tagged wheel is still a broken distribution.

A few capability boundaries also remain explicit:

• PYBIND11_SIMPLE_GIL_MANAGEMENT is forced because the other implementation reads thread-state internals. You will recognize that setting from our earlier investigation.
• Embedding, subinterpreter support, and custom heap-type setup are unavailable in this mode.
• abi3t compatibility does not automatically make user C++ code thread-safe.
• Under abi3t, compile-time Py_GIL_DISABLED describes compilation requirements; it does not tell you whether the running interpreter currently has the GIL enabled.

Finally, the forward-compatibility CI jobs test the important claim directly:

• Build with 3.12, then run the same binaries on 3.13 and 3.14.
• Build abi3t with free-threaded 3.15, then run those binaries on GIL-enabled 3.15.
• Build abi3t with GIL-enabled 3.15, then run those binaries on free-threaded 3.15.

That is stronger evidence for the feature than independently rebuilding successfully on every interpreter.

I retained the revision and source map so we can resume at any of these examples.

[rwgk]

My prompt

What values for Py_LIMITED_API do the CI tests supply? Is it always 0x030C0000, or is it dynamic somehow?

codex GPT-6.1-Sol ultra

codex response

It’s selected at build time, with three API floors exercised by the main CI suite:

CI configuration	Effective API target
Most abi3 jobs—including builds using Python 3.13 or 3.14	0x030C0000 — 3.12
One explicit newer-target job, using Python 3.14	0x030E0000 — 3.14
abi3t jobs	0x030F0000 — 3.15

The CI matrix deliberately includes this override:

-DPYBIND11_STABLE_ABI_VERSION=3.14

Otherwise, PYBIND11_STABLE_ABI_VERSION defaults to 3.12. CMake automatically raises it to 3.15 when targeting abi3t. The version-selection logic also converts the version to its hexadecimal representation.

For abi3t, some build paths define Py_TARGET_ABI3T=0x030F0000 rather than directly defining Py_LIMITED_API; CPython’s headers arrange the corresponding API restrictions.

The important distinction is header version versus target version. A job compiling with Python 3.14 headers can deliberately restrict itself to the 3.12 API. That tests whether our code correctly avoids newer facilities despite their presence in those headers.

Once compiled, the target is fixed. Loading that binary into a newer interpreter does not change its Py_LIMITED_API value or reselect its conditional compilation paths.

[rwgk]

— codex GPT-6 on behalf of Ralf —

An observation while reading type_alloc() / type_free(): the shared PYBIND11_HAS_DIRECT_STRUCT_ACCESS guard routes both PyPy and Stable ABI builds through PyType_GetSlot(). PyPy actually supports some direct field access; allocation used type->tp_alloc directly before this PR, including on PyPy.

I agree with sharing the public-API path here. More granular PyPy-specific direct-access paths might improve performance, but would add code complexity. I’d revisit individual paths only if PyPy users identify them as performance bottlenecks.

[rwgk]

To close the loop here, while reading

An important baseline detail: the default PyType_Spec / PyType_FromMetaclass() path for ordinary CPython 3.12+ builds already landed through #6231.

above it crossed my mind to ask about the performance impact of #6231. Codex jumped straight to a basic benchmark experiment. The results are shown here:

• feat: create types with PyType_FromMetaclass on CPython 3.12+ #6231 (comment)

[rwgk]

— codex GPT-6 on behalf of Ralf —

One scope observation, separate from the type-creation switch already landed in #6231: the remaining changes in this PR also affect some ordinary CPython builds, even with the Stable ABI disabled.

Many existing fast paths are retained, but there are shared implementation changes. For example:

• Bytearray-to-std::string conversion now uses PyByteArray_AsString() and PyByteArray_Size() instead of their direct-access macros, without a Limited API conditional.
• Base-type slot forwarding uses cached PyType_GetSlot() results on ordinary CPython 3.10+ too. The lookup happens once per helper; subsequent calls use the cached function pointer.

This is a source-level observation; I haven’t benchmarked these changes. If users report noticeable slowdowns, we can optimize the affected paths in follow-up PRs.

[henryiii]

What is the "Stable ABI"? — I have ideas about it, but I never got to looking / understanding systematically.

It's a limited subset of CPython's API (Limited API). If you use that, you can build a ABI that CPython commits to keeping in future versions (Stable ABI). Practical (massive) benefit: No longer have to build wheels for every version of Python. Can reduce a massive build matrix into a small one. Less storage on PyPI (which projects often have to request size increases for!).

Why does using the Stable ABI (apparently) impact runtime performance?

It is more abstracted from the implementation so it can be stable across future implementations. Most applications don't spend a lot of time in the binding layer, so this is fine most of the time; I'd recommend projects measure. I didn't see any measurable regressions in the projects I tested (at least after a couple of rounds of optimizations on this PR). And it's even not always negative, as in this PR we take a different, not quite backward compatible path to get better performance in one case with stable ABI than without. :)

The other two comments were probably better answered by the AI.

[henryiii]

Here's a Fable 5.2 review:

🤖 AI text below 🤖

The review found 10 issues in origin/master...HEAD (61 commits, 72 files). Nothing was built or run. I read the code for each finding: 9 match the code, and I did not confirm the CMake one (8).

Behavior changes and bugs

1. include/pybind11/pybind11.h:870: install_buffer_funcs now checks PyType_GetSlot(type, Py_bf_getbuffer) and not tp_as_buffer. On Python 3.9, PyType_GetSlot reads the type's own as_buffer, not the inherited pointer. A Derived class that calls def_buffer without py::buffer_protocol(), with a Base that has the annotation, now throws at import. It worked before, 3.10+ is not affected, and no test covers it.
2. include/pybind11/complex.h:88: the stable-ABI caster calls interned_name("__complex__") on each load and puts the new reference in a non-owning handle. Each conversion of a non-complex object leaks one reference. All other callers keep the result in a function-local static.
3. include/pybind11/detail/internals.h:243: under Py_LIMITED_API, get_thread_state_unchecked() returns PyGILState_GetThisThreadState(). That value is non-null when the GIL is not held, and it can be from a different interpreter. The callers are gil-inl.h:48,61, subinterpreter.h:402 and internals.h:261. PyInterpreterState_Get() is in the stable ABI and gives the current interpreter.
4. include/pybind11/pytypes.h:548: stable-ABI get_tp_name() adds the module if type_is_managed_by_our_internals() is true. That is also true for Python subclasses after all_type_info() caches them. The name of class D(m.Base) changes from D to __main__.D after the first D(). Each call on a heap type also takes the internals lock.
5. include/pybind11/detail/class-inl.h:1136: under Py_LIMITED_API, set_published_type_names() does not restore tp_name after it sets __name__. The type.__name__ setter overwrites tp_name, so the comment there is wrong for names that contain a dot. py::class_<T>(m, "a.b") gets tp_name == "a.b" with no module.

Lower risk

6. include/pybind11/pytypes.h:514: get_tp_name() builds a std::string from tp_name, and the null checks in error_fetch_and_normalize are gone. A null tp_name is now undefined behavior and not a pybind11_fail. This needs an exception type that is not ready, which is rare.
7. include/pybind11/pytypes.h:1354: get_staticmethod_type() reads staticmethod from the current frame's builtins (PyEval_GetBuiltins()) and caches it in a plain static. A frame with a custom __builtins__ can make the lookup fail or cache the wrong object. The review says the other three type caches use gil_safe_call_once_and_store and a module import.
8. tools/pybind11NewTools.cmake:471 (not confirmed): the default STABLE_ABI MODULE path links pybind11::module in addition to USE_SABI. The review says this puts python3.lib and python3XY.lib on the Windows link line, and only link order keeps the .pyd version-independent. The review also says CI runs the abi3 forward test only on Linux.

Efficiency and cleanup

9. include/pybind11/pytypes.h:2752: stable-ABI type_lookup() uses PyObject_GetItem for targets below 3.13. Each miss raises and clears a KeyError, one for each MRO class. A call to the mappingproxy get method prevents the exceptions.
10. include/pybind11/detail/class.h:57-59: static_property_dict_ptr, pybind11_static_property_traverse and pybind11_static_property_clear are declared but have no definition and no caller. Remove them.

Local origin/master is stale, so the reviewed diff also contains merged commit #6248.

[henryiii]

Applied fixes:
🤖 AI text below 🤖

#	Change	Regression test
1	install_buffer_funcs reads tp_as_buffer->bf_getbuffer in regular builds; PyType_GetSlot only under the stable ABI	test_def_buffer_in_derived_class; import failed on 3.9 before
2	__complex__ name kept in a function-local static	None; the string is immortal on 3.12+, so no leak shows
3	get_interpreter_state_unchecked() uses PyInterpreterState_Get() under the stable ABI; get_thread_state_unchecked() no longer exists there	None
4	New type_is_registered_in_our_internals() for the name helpers; Python subclasses keep the bare name	test_obj_class_name_python_subclass; failed under the stable ABI before
5	Comment corrected only	None
6	get_tp_name() returns an empty string for a null tp_name; the two pybind11_fail checks are back	test_error_already_set_without_tp_name; crashed before
7	get_staticmethod_type() imports builtins through gil_safe_call_once_and_store, with a helper shared with get_bound_method_type()	None
8	Stable-ABI modules and the stable-ABI precompiled library no longer link Python::Module	Configure-time link check in tests/CMakeLists.txt; failed before, with and without precompile
9	type_lookup() tests membership before PyObject_GetItem on targets below 3.13	None
10	Three dead declarations removed	None

• Finding 5 has no real fix: the stable ABI cannot write tp_name, so the difference for names with a dot stays.
• Finding 7 still uses PyImport_ImportModule: I first used an import that bypasses builtins.__import__, but tests/test_cmake_build/test.py needs the types import to go through that hook. A frame whose builtins have no __import__ still makes the lookup fail.
• Finding 8, precompiled library: it now calls _pybind11_stable_abi_setup(), the same setup as a stable-ABI module. This changes the Windows link line, so Windows CI must confirm it.