Skip to content

fix(mysql): run scheduler transactions at READ COMMITTED so the claim stops blocking publish() (#129) - #139

Open
rawo wants to merge 2 commits into
mainfrom
fix/129-mysql-claim-read-committed
Open

rawo wants to merge 2 commits into
mainfrom
fix/129-mysql-claim-read-committed

Conversation

@rawo

@rawo rawo commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Closes #129.

Problem

Under MySQL's default REPEATABLE READ, the processor's claim (FOR UPDATE SKIP LOCKED on idx_okapi_outbox_status_created_at) takes gap locks, because SKIP LOCKED only skips record locks. When the backlog is smaller than the batch, the gap runs to the end of the index, which is exactly where every publish() inserts. Business transactions then wait for the entire HTTP/Kafka delivery and can fail with Lock wait timeout exceeded.

Fix

On MySQL/MariaDB, the scheduler (processor and purger) transaction now runs at READ COMMITTED, which takes no gap locks. Other databases are untouched (Postgres already defaults to READ COMMITTED).

  • Spring Boot: okapiTransactionRunner pins READ COMMITTED only when the store is MysqlOutboxStore and its TransactionTemplate leaves isolation at DEFAULT.
    • It works on a copy, so the user's or Boot's shared template bean is never mutated, and the copy keeps its PTM, timeout, propagation and read-only settings.
    • A template with an explicit isolation level is still used verbatim.
    • Non-MySQL stores get the template verbatim, exactly as before (no extra isolation round trip per tick).
    • Pinning is skipped with a startup WARN when the PTM rejects custom isolation levels: JtaTransactionManager (no public getter for allowCustomIsolationLevels, so always treated as unsupported) and JpaTransactionManager without the Hibernate/EclipseLink JpaDialect. Otherwise every tick would throw InvalidIsolationLevelException. The WARN points to an explicit-isolation TransactionTemplate or server-side transaction_isolation.
  • Exposed: ExposedTransactionRunner(database) uses READ COMMITTED when database.dialect is MySQL/MariaDB (resolved lazily on first use), the database default otherwise. A new ExposedTransactionRunner(database, transactionIsolation) sets the level explicitly (null = database default).

Behaviour changes to review

  • On MySQL, the Spring runner's template is a READ COMMITTED copy rather than the user's/Boot's template by reference (reference identity is kept for explicit-isolation templates and for every non-MySQL store).
  • On MySQL/MariaDB, ExposedTransactionRunner(database) overrides a configured DatabaseConfig.defaultIsolationLevel for scheduler transactions (noted under CHANGELOG Changed).
  • README: MySQL at READ COMMITTED requires binlog_format=ROW (default since 5.7.7) or MIXED.

Tests

  • MysqlClaimDoesNotBlockPublishersTest (Testcontainers MySQL) runs a real OutboxProcessor.processNext() tick with a deliverer that blocks mid-delivery, and publishes from another connection with innodb_lock_wait_timeout=3. Going through the processor keeps it on whatever claim query the processor runs (e.g. after Claim only delivery types supported by each worker #120).
  • OutboxAutoConfigurationIsolationTest: auto-built template, user template at DEFAULT (copied, not mutated), explicit isolation (verbatim), non-MySQL store (verbatim), and JtaTransactionManager over a fake UserTransaction (premise: RC throws; okapi doesn't pin, ticks run, WARN logged once).
  • ExposedTransactionRunnerTest: database default on non-MySQL, explicit and null isolation.
  • ./gradlew ktlintCheck build test pass locally.

🤖 Generated with Claude Code

… stops blocking publish()

Under MySQL's default REPEATABLE READ the claim's FOR UPDATE SKIP LOCKED takes
gap locks on (status, created_at); SKIP LOCKED only skips record locks. With a
backlog smaller than the batch the gap runs to the end of the index, where every
publish() inserts, so business transactions waited for the whole delivery and
could fail with "Lock wait timeout exceeded".

- Spring: okapiTransactionRunner pins READ COMMITTED when its TransactionTemplate
  leaves isolation at DEFAULT, using a copy so shared template beans are not
  mutated; explicit isolation levels are still honoured verbatim.
- Exposed: ExposedTransactionRunner gains transactionIsolation, defaulting to
  READ_COMMITTED (null = database default).
- MysqlClaimDoesNotBlockPublishersTest reproduces the lock at REPEATABLE READ
  (control) and proves both runners no longer block a concurrent publish.

Closes #129

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@rawo rawo left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review focused on transactions, deadlocks/lock waits, performance and structure. The diagnosis is right: SKIP LOCKED only skips record locks, not gap or next-key locks. READ COMMITTED is the standard fix, and the RR control test proves the reproduction. The copy-not-mutate handling of the shared TransactionTemplate is careful. One finding I'd treat as blocking, plus a few notes.

Cross-PR note (#120): #120 changes the processor to claim through RouteAwareOutboxStore.claimPending(types, limit) with FORCE INDEX (status, delivery_type, created_at, id). Under RR that query gap-locks the end of each type's range, which is again exactly where publish() of that type inserts. So #129 still applies after #120, and this PR still fixes it. But MysqlClaimDoesNotBlockPublishersTest calls the legacy claimPending(10), so once #120 lands it would stop exercising the query the processor actually runs. Whichever PR merges second should switch the test to processor.processNext() (or the route-aware claim).

…ccepts it (#129)

Addresses review on #139:
- Spring: pin READ COMMITTED only when the store is MysqlOutboxStore. Postgres already
  defaults to it, so other stores keep the template verbatim (no extra round trip, no
  behaviour change). Skip pinning with a startup WARN for PTMs that reject custom
  isolation levels (JtaTransactionManager, JpaTransactionManager without the
  Hibernate/EclipseLink JpaDialect); otherwise every tick would throw
  InvalidIsolationLevelException.
- Exposed: ExposedTransactionRunner(database) uses READ COMMITTED only on MySQL/MariaDB
  dialects; ExposedTransactionRunner(database, level) sets it explicitly.
- MysqlClaimDoesNotBlockPublishersTest drives a real OutboxProcessor tick with a blocking
  deliverer instead of calling claimPending directly, so it follows whatever claim query
  the processor runs (e.g. after #120). Adds Hibernate-JPA and DefaultJpaDialect cases.
- New OutboxAutoConfigurationIsolationTest, including a JtaTransactionManager case.
- CHANGELOG: behaviour-change note for Exposed overriding defaultIsolationLevel on MySQL.
  README: binlog_format=ROW/MIXED requirement and the JTA/JPA fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MySQL: processor claim gap-locks okapi_outbox under REPEATABLE READ, blocking every publish() for the duration of delivery

1 participant