Skip to content

Fix persistent namespaces on Python 3.13+ (dbm.sqlite3 cross-thread error) - #2707

Open
nvmassen wants to merge 1 commit into
AppDaemon:devfrom
nvmassen:fix/persistent-dict-sqlite-threads
Open

nvmassen wants to merge 1 commit into
AppDaemon:devfrom
nvmassen:fix/persistent-dict-sqlite-threads

Conversation

@nvmassen

@nvmassen nvmassen commented Nov 6, 2026

Copy link
Copy Markdown

Problem

Starting with Python 3.13, dbm creates new databases with dbm.sqlite3. That backend's sqlite3 connection uses the default check_same_thread=True. PersistentDict is opened on the main thread in State.add_persistent_namespace. The first time a sync app reads or writes the namespace from its worker thread, it fails with:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread.
The object was created in thread id ... and this is thread id ...

This affects every new persistent namespace on 3.13+ when the Python build doesn't have _gdbm. That includes the Home Assistant add-on image (python-build-standalone via uv). It also affects any SQLite store already created by 4.5.13 on 3.13.

A related problem: if a namespace file exists in a format the running Python can't open (for example a GDBM file, or a GDBM-written .pag/.dir pair, on a Python without _gdbm), dbm doesn't fail. It creates a separate empty store next to the file, so the namespace silently comes up empty. I ran into this after the add-on moved to Python 3.13.

Changes

  • appdaemon/utils/state.py
    • PersistentDict.__init__ replaces the dbm.sqlite3 connection with one opened on the same file with check_same_thread=False (and WAL mode, like dbm.sqlite3 itself). PersistentDict.rlock already serializes access, and Python's sqlite3 module is built in serialized threading mode. Other backends (gnu, ndbm, dumb) open exactly as before. File format, file names and backend selection don't change, so existing stores of every kind keep working.
    • New find_unreadable_store_files(path) sniffs the header of the store file (and of its .pag file) for SQLite/GDBM magic. It reports files whose backend isn't available in the running Python. For a GDBM .pag, it only reports the file if dbm.ndbm actually fails to open it.
  • appdaemon/state.py: add_persistent_namespace logs a warning for each such file before opening the namespace. Behaviour is otherwise unchanged (it still starts empty).
  • tests/unit/test_persistent_dict.py: covers reads and writes from another thread (both writeback modes), concurrent writers, reopening an existing dbm.sqlite3 store, an existing dbm.dumb store still opening with its own backend, and the unreadable-file detection.

Notes for review

  • dbm.sqlite3 has no public way to pass check_same_thread, so the fix swaps the private _Database._cx attribute. That attribute and _Database.__init__ are identical in CPython 3.13, 3.14 and 3.15. The swap only runs when the shelf's database is a dbm.sqlite3._Database, and the file path comes from PRAGMA database_list instead of rebuilding the URI. If CPython changes these internals, the new tests will fail.
  • The alternative was to pick a non-SQLite backend explicitly for new files. That wouldn't fix SQLite stores that 4.5.13 has already created on 3.13, and it would make the file format depend on which C modules the build has.

Testing

  • pytest tests/unit passes on 3.12 (SQLite test skipped), 3.13 and 3.14 (uv / python-build-standalone builds, no _gdbm). tests/functional/test_namespaces.py and test_state.py pass on 3.13.
  • With the fix disabled, the 4 cross-thread tests fail with the ProgrammingError above.
  • Checked detection against a real GDBM file written by a Python that has _gdbm: it is reported on 3.13 without _gdbm. A GDBM-compat ndbm store that dbm.ndbm can open is not reported.

Possibly related, not changed here

  • remove_persistent_namespace deletes files whose stem equals filepath.stem. On 3.13+ the store is ns.db, so ndbm/dumb files such as ns.db.db, ns.db.dat or ns.db.dir (stem ns.db) are left behind.
  • The 3.13 change from ns to ns.db file names means a GDBM store written by AppDaemon on ≤3.12 (file ns) isn't picked up after upgrading to 3.13. The namespace starts empty and there's no warning.

🤖 Generated with Claude Code

On Python 3.13+, dbm creates new files with dbm.sqlite3, whose connection
is bound to the thread that opened it. PersistentDict is opened on the
main thread in State.add_persistent_namespace, so the first access from an
app worker thread raised "SQLite objects created in a thread can only be
used in that same thread".

Reopen the dbm.sqlite3 connection with check_same_thread=False; access is
already serialized by PersistentDict.rlock. Other backends (gnu, ndbm,
dumb) open exactly as before, and existing SQLite stores are fixed too.

Also warn when an existing namespace file is in a format the running
Python can't open (e.g. GDBM on a Python built without _gdbm), instead of
silently starting the namespace empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant