[clebertsuconic]

Motivation:

Artemis's append-only journal delivers outstanding throughput and
is the ideal choice when fast local storage is available. However,
many cloud and containerised deployments already operate managed
relational databases and would benefit from an alternative
persistence path that leverages that existing infrastructure —
simplifying operations, backup, and disaster recovery without
giving up the broker's asynchronous, non-blocking persistence
contract.

Modifications:

Introduce a new artemis-database-storage module that implements
the full broker persistence layer on top of JDBC, wired through
the existing StorageManager / PagingManager SPIs so no
protocol or core-broker code needs to change.

Key design choices:

• Group-commit / write-coalescing: DataManager buffers incoming
  writes and a scheduled timer flushes them in a single JDBC
  transaction, amortising the most expensive part (network
  round-trip + fsync) across many operations — the same technique
  database engines use internally (e.g. InnoDB group commit).

• Command pattern with heterogeneous batching: each DBData
  subclass (messages, references, addresses, queues, prepared-TX,
  generic KV) dispatches itself to the correct
  BatchableStatement; a single transaction can therefore mix
  inserts and deletes across different tables atomically.

• Worker pool with dedicated connections: multiple DataWorker
  instances, each owning its own JDBC Connection and full set of
  PreparedStatements, cycle through a pool. BorrowedWorker
  adds a lease-based model for callers that need a connection for
  an extended sequence of operations.

• Credit-based flow control: producers are throttled via a credit
  window so the broker cannot out-run the database.

• Async write-behind with IOCompletion callbacks: callers never
  block on commit; each data item carries storeLineUp() /
  done() / onError() callbacks, preserving the same contract
  the journal already exposes.

• Retry with reconnect: transient SQLExceptions are retried up
  to five times with connection re-establishment between attempts.

• Database paging: DatabasePagingManager and
  DatabasePagingStoreImpl replace file-based paging when this
  store type is selected. AddressSizeLimiter is extracted from
  PagingStoreImpl so both the file and database paths share
  address-level flow control (max size, max messages, blocking,
  page-full rejection).

• DatabaseStorageMessageReader provides cursor-based message
  reading from the database with configurable prefetch for queue
  consumption.

• Six database dialects are supported out of the box: PostgreSQL,
  MySQL, Oracle, DB2, MS SQL Server, and HSQLDB, each with a
  dedicated SQLProvider handling type mappings and DDL
  differences.

• AbstractStorageManager is extracted from
  AbstractJournalStorageManager so both the journal-backed and
  database-backed managers share common bookkeeping without
  duplication.

Tests cover every layer: statement-level (insert/delete/update per
entity), worker-level (retry, flow control, borrowed-worker
lifecycle), full integration (StorageManager round-trips, real
broker boot, cross-protocol, XA / prepared transactions), and
stress (concurrent producers under sustained load).

Result:

Operators now have a choice between the journal — which remains
the highest-throughput option for local storage — and a relational
database backend that integrates naturally with managed database
services, broadening the set of environments where Artemis can be
deployed with minimal operational overhead.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

[grepjava]

I ran the transactional send/consume test on 5206d7e, PostgreSQL 16 against the file journal, with everything on one machine: broker, client and PostgreSQL on the same ext4 disk (WSL2) sharing 4 CPUs, fsync and synchronous_commit on. Same build, default artemis create settings. Persistent 1 KiB messages on one queue, auto-ack consumers, perf client --commit-interval N, 45 s measured after 15 s warmup, median of 3 runs:

• 1 producer/1 consumer, commit every 100: NIO 23.6k msg/s, AIO 25.0k, PostgreSQL 9.2k (39% of NIO); p99 transfer 8.1 ms vs 17.9 ms
• 1/1, commit every 10: NIO 5.6k, AIO 5.6k, PostgreSQL 1.7k (31%); p99 3.8 ms vs 8.2 ms
• 4/4, commit every 100: NIO 50.7k, AIO 52.5k, PostgreSQL 25.4k (50%); p99 15.7 ms vs 31.0 ms
• Same 4/4 for 5 minutes, one run each: NIO 53.6k, AIO 52.4k, PostgreSQL 25.2k (47%), with autovacuum running 7 times and no lasting drop

All runs finished, every message was received, and there were no broker warnings. PostgreSQL shared its 4 CPUs with the broker and client, so a separate database host may well do better.

Most of the gap at small transactions is database-flush-period-nanos. Lowering it from the 3.33 ms default (mean of 2 runs):

• 1/1, commit every 10: 1.8k msg/s at 3.33 ms, 3.0k at 1 ms, 4.4k at 0.1 ms (79% of NIO); median transfer 5.5 ms down to 2.1 ms
• 4/4, commit every 100: 26.9k, 31.9k, 33.1k (65% of NIO)

I didn't measure the CPU cost of a period that short, so I can't say what the default should be, but it may be worth a look.

One PostgreSQL run (1/1, commit every 100) decayed steadily from about 5k to 1.2k msg/s with nothing in the broker log. It didn't happen again in the other two repetitions or in two more runs on a fresh cluster, and I don't have a cause.

Two perf tool problems got in the way, and they're on main too. --tx on the consumer opens a transacted session but never commits, so a paging backlog blocks it. And a producer with --commit-interval doesn't commit its last partial transaction on close, so the client waits for those messages forever; I patched that locally for these runs. I'll open Jiras for both.

Separately, artemis create writes a & from --jdbc-connection-url into broker.xml unescaped, so I had to fix that by hand.

[clebertsuconic]

Did you setup to use the new database stuff?

[clebertsuconic]

I plan to improve the create of the Db

[grepjava]

Yes, it's the new storage: created with artemis create --database-storage, with <database-storage>true</database-storage> in broker.xml, and the data goes to the db_messages and db_references tables. database-flush-period-nanos changes the numbers too, which only the new storage reads.

Everything else was what create generates: database-connections 6, database-max-read-connections 3, database-page-join-fetch false, and the 3.33 ms flush period. If you have settings you normally test with, tell me and I'll rerun with them, or I can rerun once the create changes are in.

[clebertsuconic]

you could increase the number of connections. One thing I have been playing with is that I think it scales a lot better with the database.

if you add paging to the mix (not fitting the memory) the new relational database is a lot better than the current JDBC version.

I will merge your PRs for the table and benchmark improvements. thanks for those.

[clebertsuconic]

You could increase the number of connections.

I think the new database is going to be very fast, but not as fast as the journal...

But if you think about this should be very compelling for mos users. Especially with many independent producers as you could have multiple writers (multiple connections).

[grepjava]

Thanks, I ran both on the same setup as before: WSL ext4, 4 shared CPUs, PostgreSQL 16, defaults unless stated, 1 KB persistent messages, producers committing every 100 unless stated.

Paging: you're right, it's a big difference. global-max-size 64 MB, 500,000 message backlog, msg/s:

	Journal (AIO)	New storage	Old JDBC
Fill the backlog	24.2k	16.4k	6.6k
Drain it, nothing producing	49.0k	23.7k	1.1k
Producing and consuming on the backlog, sent	26.5k	18.0k	176
Same, received	26.5k	24.7k	1.1k

Means of 5 or 6 fills, 2 drains, and 3 producing and consuming runs of about 60 s. The journal's second producing and consuming run dipped to 20.3k (the others were 31.3k and 28.0k) and I don't know why. Every drain got all 500,000 back. Old JDBC stalled at 0 msg/s for some seconds in each producing and consuming run. The new storage drains about 20x faster than old JDBC, and about half as fast as the journal.

The new storage logged AMQ224127 (dispatch from paging blocked) once in each drain and in 2 of the 3 producing and consuming runs. It carried on each time.

Connections: they made no difference here, at the default flush period. 16 queues, one producer and one consumer each, received msg/s, mean of 2 runs:

	Journal NIO	Journal AIO	6 connections	12	24
Commit every 100	54.6k	57.3k	24.3k	23.7k	23.9k
Commit every 10	16.3k	15.7k	5.5k	5.9k	5.7k

PostgreSQL did get the connections (8, 14 and 26 open), but sampling pg_stat_activity found at most 1 to 4 active, with 40 to 60% CPU idle. I think that's DataManager.flush(): it's a single loop handing one batch to one worker per pass, paced by database-flush-period-nanos, so the writers in use are about batch time / flush period whatever the pool size.

To check, I set the flush period to 0.1 ms:

	6 connections, 3.33 ms	6 connections, 0.1 ms	24 connections, 0.1 ms
Commit every 100	24.3k	28.2k	29.7k
Commit every 10	5.5k	10.5k	9.7k

Active connections now reached 6 to 9, and commit every 10 nearly doubled, to about 65% of the journal. But 24 connections still did no better than 6, and mean active stayed around 2.5. So the flush period is what limits this workload here. This is one box with 4 shared CPUs and PostgreSQL on the same disk, so it can't show what several connections do on a real database server, where I'd expect more room.

One thing from the tests: perf client can't consume messages sent by perf producer. perf client uses System.nanoTime() and perf producer wall-clock micros, so the latency comes out negative, the histogram throws in onMessage, and messages redeliver forever. I used perf producer and perf consumer side by side instead.

[grepjava]

@clebertsuconic the perf client problem from my last comment is ARTEMIS-6355, with the fix in #6873. It's small and merges cleanly with #6867 and #6870.

[grepjava]

@clebertsuconic here are the transactional consumer numbers I offered on #6873, now that #6870 is in. Same setup as before: 5206d7e with #6867 and #6870 added to the perf client, WSL ext4, 4 shared CPUs, PostgreSQL 16, defaults, so 6 connections. 16 queues, one producer and one consumer each, perf client --persistent --tx --commit-interval N --num-destinations 16 --producers 1 --consumers 1 --threads 4 --duration 60 --warmup 15. Received msg/s, 3 runs each:

	Run 1	Run 2	Run 3
Commit every 100	20.7k	23.3k	23.4k
Commit every 10	5.7k	5.5k	5.6k

Every run ended with sent equal to received, no exceptions and nothing left in db_references.

That's about the same as the auto-ack runs I posted (24.3k and 5.5k), so batching the consumers' acknowledgements didn't change the result here. The host was also running some unrelated VMs this time, so the small differences don't mean much. I kept to 6 connections and the default flush period, so this doesn't add anything on scaling with more connections, and the limit from one box with shared CPUs still applies.

[clebertsuconic]

Auto ack does a lot of batching.

[grepjava]

I tried 247536e with many queues paging at once and saw the read connections go over database-max-read-connections.

Setup: PostgreSQL 16, default database-connections (6), database-max-read-connections 3, 16 queues each filled with 10,000 × 1 KB messages past a 10 MB global-max-size, then one slow consumer per queue (--sleep 100).

• Readers held 6 connections at once (counted from pg_stat_activity, sessions whose query is the paged read), against a limit of 3. With consumers that stall, it was 5.
• AMQ232002 was logged 2,802 times in about a minute (82 with stalled consumers), at INFO, each with the table of borrowed connections.

From the code, borrowWorker doesn't check the read limit. It's only checked later, in demandBorrowedWorkersBack, which sets needConnectionBack, and only readBatch looks at that flag. So the limit decides when to ask for connections back, not how many readers can take. The warning's advice to raise database-max-read-connections wouldn't help, because the readers ignore it either way.

I expected this to stall producers when readers hold every connection, but it didn't. A probe sending persistent messages every 50 ms on another queue never took longer than 123 ms. So as far as I can tell, the problems are the limit and the noisy log, not throughput.

Possible fixes: cap borrowing at database-max-read-connections, and log AMQ232002 at DEBUG or rate-limit it.

[clebertsuconic]

Oh wow!!! Man you're great.

To me the borrowed cursor was an idea I just introduced. I mean I was still wondering if it was the right idea. And you already found improvements to the concept.

Sure we could / should limit the borrow limit. Perhaps by closing previous ones.