[brandur]

This one's a huge grab gag of Ruby optimizations. I've had Codex try to
write them up in tabulated form here (all timing in microseconds):

Operation	ActiveRecord before -> after	Sequel before -> after	Statements
Claim 50 jobs	5,871 -> 3,691 (1.59x)	5,599 -> 4,304 (1.30x)	53 -> 1
Claim + complete	734 -> 449 (1.63x)	676 -> 427 (1.58x)	8 -> 2
Merge metadata	297 -> 156 (1.91x)	268 -> 145 (1.86x)	4 -> 1
Update job	231 -> 154 (1.49x)	216 -> 146 (1.48x)	2 -> 1
Insert pending job	343 -> 272 (1.26x)	282 -> 220 (1.28x)	3 -> 1
Upsert queue	164 -> 143 (1.15x)	157 -> 128 (1.23x)	2 -> 1

And a couple additional measurements:

• ActiveRecord connection reclamation: thread/update/reap benchmark
  fell from 6.33 ms to 0.274 ms, 23.1× faster.

• SQLite batch claiming: ActiveRecord 2.22 → 1.31 ms, 1.70×;
  Sequel 2.88 → 1.59 ms, 1.81×.

• Sequel/Postgres leader acquisition: 153.5 → 127.0 µs, 1.21×,
  with 5 → 4 statements.

• PG 17 planner fix: with stale statistics on Postgres 17,
  completion SQL fell from 174.4 → 80.6 µs, 2.16×. Whole-driver
  completion improved 261 → 176 µs for ActiveRecord and 267 → 168
  µs for Sequel. Postgres 18 already selected the primary key and
  remained essentially unchanged at 121.4 µs.

Here's a list of miscellaneous fixes:

• Replace ActiveRecord connection leases with with_connection so
  they're checked back in after use and don't need reset.

• Decode ActiveRecord query results directly, avoiding model allocation
  for job reads and lists.

• Invalidate ActiveRecord's query cache for writes using RETURNING.

• Return full jobs directly from claim, completion, cancellation, retry,
  metadata merge, state transition, update, and delete operations.

• Claim entire batches with one statement, eliminating per-job refetch.

• Return queue upsert/update results directly.

• Remove explicit transactions around single-statement claims,
  completions, metadata merges, and resignation requests.

• Skip insert transactions when no notification is required.

• Delete finalized jobs with one bounded statement (protection against
  concurrent retries is preserved).

• Use RETURNING for successful Postgres leader acquisition.

• Decode SQLite insert metadata once instead of twice

• Keep ID-targeted running job mutations on efficient primary key query
  plan when Postgres stats are stale.

• Prevent ordinary worker wakeups from bypassing maintenance error
  backoff.

[brandur]

Most of the code here is from a new benchmark that Codex added. IDK if committing these is always a good idea or not, but it doesn't seem to be too harmful, so I just left it in.

[brandur]

This also does repro the headline figure from the other PR:

Metric	Baseline	PR #1566	This branch
Ruby CPU/job	2.362 ms	0.567 ms	0.567 ms
Postgres CPU/job	1.601 ms	0.392 ms	0.336 ms
Total CPU/job	3.963 ms	0.958 ms	0.904 ms
Jobs/second	2,063	8,358	8,843
SQL statements/job	15.643	1.291	1.288
Connection resets/run	19,894	0	0
Follow-up job reads/run	40,000	0	0

Total CPU/job fell from 3.963 to 0.904 ms, so 77.2% lower, and 5.6% less than #1566 as we found a couple additional (albeit much smaller) improvements.