Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ Or you can also set the environment variable `SOLID_QUEUE_SUPERVISOR_MODE` to `a

**The recommended and default mode is `fork`. Only use `async` if you know what you're doing and have strong reasons to**

This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (`threads: N`) or as fibers on a single fiber reactor thread (`fibers: N`).
This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (`threads: N`), in a thread pool with a reactor per thread (`reactor: true`), or as fibers on a single fiber reactor thread (`fibers: N`).

Because these are separate concerns, you can combine the default `fork` supervisor mode with fiber workers. In that setup, each worker process gets its own fiber reactor and bounded fiber count.

Expand Down Expand Up @@ -289,9 +289,10 @@ Here's an overview of the different options:

- `threads`: configures a worker to execute jobs in a thread pool of this size. By default, workers use `threads: 3`. Only workers have this setting, and it can't be combined with `fibers`.
It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker uses connections for polling and heartbeat and thread mode may use additional connections for job execution.
- `reactor`: when `true`, a thread worker runs each of its threads' jobs inside an [Async](https://github.com/socketry/async) reactor that lives as long as the thread, one job at a time per thread. See [reactor thread workers](#reactor-thread-workers). Requires the `async` gem, and can't be combined with `fibers`.
- `fibers`: configures a worker to execute jobs as fibers on a single fiber reactor thread, with this value as the maximum number of in-flight jobs. It can't be combined with `threads`.
Fiber workers require fiber-scoped isolated execution state. In Rails apps, set `config.active_support.isolation_level = :fiber` before using `fibers`. Solid Queue refuses to boot fiber workers when isolation remains thread-scoped. On Rails 7.2 and later, a practical starting point is usually `3-5` queue database connections per worker process rather than matching the `fibers` value, because ordinary Active Record query paths can release connections between non-blocking waits. On Rails 7.1, size the queue database pool more conservatively, as in-flight fiber jobs may still retain connections roughly in proportion to `fibers`.
- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with both `threads` and `fibers` workers as long as the supervisor is running in the default `fork` mode. **Note**: this option is ignored only when the supervisor itself is [running in `async` mode](#fork-vs-async-mode).
- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with `threads`, `reactor` and `fibers` workers as long as the supervisor is running in the default `fork` mode. **Note**: this option is ignored only when the supervisor itself is [running in `async` mode](#fork-vs-async-mode).
- `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
- `batch_maintenance`: whether the dispatcher will sweep stalled [batches](#batch-jobs) as part of its maintenance work, on the same timer as concurrency maintenance (see [batch maintenance](#batch-maintenance)). This is `true` by default; disable it if you don't use batches, or if you run multiple dispatchers and want only some of them doing maintenance work.

Expand Down Expand Up @@ -383,7 +384,7 @@ queues: back*

### Threads, processes, and signals

By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Workers can also be configured with `fibers`, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.
By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Workers can also be configured with `reactor`, in which case each of those threads runs its jobs inside a reactor, or with `fibers`, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.

Fiber worker execution is best suited for cooperative, mostly I/O-bound jobs. Blocking or CPU-heavy work still blocks the single reactor thread, so it should not be expected to outperform thread mode for every workload.

Expand Down Expand Up @@ -414,6 +415,20 @@ This means finished and failed jobs still follow the normal Solid Queue lifecycl
In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.


#### Reactor thread workers

Jobs on a thread worker run without a reactor, so a client built on Async, such as `async-http`, starts one for every call and pays for a new connection and TLS handshake each time. Fiber workers avoid that, but they run every job on one thread, so a job that blocks without yielding, such as a C extension decoding an image, stalls the others. Setting `reactor: true` on a thread worker keeps the thread pool and gives each thread a reactor that lives as long as the thread, so those clients reuse their connections across calls and across jobs:

```yml
workers:
- queues: [ api_sync, webhooks ]
threads: 5
reactor: true
```

Each thread still runs one job at a time, so database pool sizing is the same as for `threads`, and fiber-scoped isolated execution state isn't required.


### Database configuration

You can configure the database used by Solid Queue via the `config.solid_queue.connects_to` option in the `config/application.rb` or `config/environments/production.rb` config files. By default, a single database is used for both writing and reading called `queue` to match the database configuration you set up during the install.
Expand Down
18 changes: 18 additions & 0 deletions lib/solid_queue/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ class Configuration
validate :ensure_configured_processes, :ensure_valid_recurring_tasks
validate :ensure_valid_worker_execution_options
validate :ensure_fiber_workers_have_required_dependency, :ensure_fiber_workers_use_supported_isolation_level
validate :ensure_reactor_workers_have_required_dependency
validate :warn_about_incorrectly_sized_database_pool, :warn_about_missing_config_files

before_validation { warnings.clear }
Expand Down Expand Up @@ -130,6 +131,10 @@ def ensure_valid_worker_execution_options
if options.key?(:threads) && options.key?(:fibers)
errors.add(:base, "Workers can specify either `threads` or `fibers`, but not both.")
end

if options[:reactor] && fiber_worker?(options)
errors.add(:base, "`reactor` applies to thread workers. Fiber workers already run their jobs on a reactor.")
end
end
end

Expand All @@ -143,6 +148,15 @@ def ensure_fiber_workers_have_required_dependency
"Add `gem \"async\"` to your Gemfile to configure workers with `fibers`.")
end

def ensure_reactor_workers_have_required_dependency
return unless workers_options.any? { |options| reactor_thread_worker?(options) }

require "async"
rescue LoadError
errors.add(:base, "Reactor thread workers require the `async` gem. " \
"Add `gem \"async\"` to your Gemfile to configure workers with `reactor: true`.")
end

def ensure_fiber_workers_use_supported_isolation_level
return unless workers_options.any? { |options| fiber_worker?(options) }

Expand Down Expand Up @@ -319,6 +333,10 @@ def fiber_worker?(options)
options.key?(:fibers)
end

def reactor_thread_worker?(options)
options[:reactor] && !fiber_worker?(options)
end

def worker_defaults_for(options)
if fiber_worker?(options)
WORKER_DEFAULTS.except(:threads)
Expand Down
138 changes: 138 additions & 0 deletions lib/solid_queue/reactor_thread_pool.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# frozen_string_literal: true

module SolidQueue
class ReactorThreadPool < Pool
def initialize(size, on_idle: nil, on_unrecoverable_error: nil)
super

@state_mutex = Mutex.new
@shutdown = false
@fatal_error = nil
@pending_executions = Thread::Queue.new
@reactor_threads = []
end

def post(execution)
raise_if_fatal_error!
raise RuntimeError, "Execution pool is shutting down" if shutdown?

super
end

def available_capacity
raise_if_fatal_error!
super
end

def shutdown
state_mutex.synchronize do
next false if @shutdown

@shutdown = true
end.tap do |shut_down|
# Each thread drains the executions already queued, then its blocked
# pop returns nil and its reactor exits
pending_executions.close if shut_down
end
end

def shutdown?
state_mutex.synchronize { @shutdown }
end

def wait_for_termination(timeout)
deadline = Timer.monotonic_time_now + timeout.to_f
threads = state_mutex.synchronize { reactor_threads.dup }

threads.map { |thread| terminated?(thread, deadline) }.all?
end

private
attr_reader :pending_executions, :reactor_threads, :state_mutex

def name
@name ||= "solid_queue-reactor-thread-pool-#{object_id}"
end

# A thread that died raises its error again from +join+. It was reported
# when the thread died, and the other threads still have to be waited for
def terminated?(thread, deadline)
thread.join([ deadline - Timer.monotonic_time_now, 0 ].max)
rescue Exception
raise unless thread.status.nil?
true
end

def schedule(execution)
start_reactor_threads_if_needed
pending_executions << execution
end

# Threads are started lazily, like the fiber pool's reactor, so the pool
# can be built in the supervisor before it forks.
def start_reactor_threads_if_needed
state_mutex.synchronize do
next if reactor_threads.any?

require "async"
size.times { reactor_threads << start_reactor_thread }
end
end

# Each thread runs its jobs one at a time on a reactor that lives as long
# as the thread, so async clients keep their connections across jobs
def start_reactor_thread
create_thread do
Sync do |task|
while execution = pending_executions.pop
task.async(execution) do |_execution_task, scheduled_execution|
perform_execution(scheduled_execution)
end.wait
end
end
rescue Exception => error
register_fatal_error(error)
raise
end
end

def perform_execution(execution)
wrap_in_app_executor do
execution.perform
ensure
stop_tasks_left_running
end
rescue Async::Stop => error
handle_thread_error(error)
register_fatal_error(error)
rescue Exception => error
handle_unrecoverable_error(error)
handle_thread_error(error)
ensure
restore_capacity
end

# A job runs in its own task, so the tasks it started and didn't wait for
# are that task's children. They're stopped when the job ends, so they
# don't run alongside the next job or hold up shutdown. Transient tasks,
# such as the ones async clients keep their connections open with, stay
def stop_tasks_left_running
Async::Task.current.children&.each do |child|
child.stop unless child.transient?
end
end

def register_fatal_error(error)
state_mutex.synchronize do
@fatal_error ||= error
end

on_idle&.call
end

def raise_if_fatal_error!
error = state_mutex.synchronize { @fatal_error }
raise error if error
end
end
end
8 changes: 7 additions & 1 deletion lib/solid_queue/worker.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,13 @@ class Worker < Processes::Poller
attr_reader :queues, :pool

def initialize(**options)
execution_pool_type = options.key?(:fibers) ? :fiber : :thread
execution_pool_type = if options.key?(:fibers)
:fiber
elsif options[:reactor]
:reactor_thread
else
:thread
end

options = options.dup.with_defaults(SolidQueue::Configuration::WORKER_DEFAULTS)
execution_pool_size = execution_pool_type == :fiber ? options[:fibers] : options[:threads]
Expand Down
130 changes: 130 additions & 0 deletions test/integration/reactor_processes_lifecycle_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# frozen_string_literal: true

require "test_helper"

class ReactorProcessesLifecycleTest < ActiveSupport::TestCase
self.use_transactional_tests = false

setup do
@pid = fork do
SolidQueue::Supervisor.start(skip_recurring: true, workers: [ { queues: :background, threads: 3, reactor: true, polling_interval: 0.1 } ])
end

wait_for_registered_processes(2, timeout: 3.second)
assert_registered_workers_for(:background, supervisor_pid: @pid)
end

teardown do
terminate_process(@pid) if process_exists?(@pid)
end

test "process jobs in reactor threads in a forked worker" do
worker = find_processes_registered_as("Worker").first
assert_metadata worker, pool_type: "reactor_thread", pool_size: 3

6.times { |i| enqueue_store_result_job("job_#{i}") }

wait_for_jobs_to_finish_for(3.seconds)

assert_equal 6, skip_active_record_query_cache { JobResult.count }
6.times { |i| assert_completed_job_results("job_#{i}", :background) }

terminate_process(@pid)
assert_clean_termination
end

test "term supervisor while there are jobs in-flight" do
no_pause = enqueue_store_result_job("no pause")
pause = enqueue_store_result_job("pause", pause: 1.second)

# Wait for the "pause" job to start, which means both jobs have been
# claimed, before sending TERM, so the worker finishes them while
# shutting down instead of stopping before it gets to claim them
wait_while_with_timeout(3.seconds) { !JobResult.exists?(status: "started", value: "pause") }

signal_process(@pid, :TERM, wait: 0.1.second)
wait_for_jobs_to_finish_for(3.seconds)

assert_completed_job_results("no pause")
assert_completed_job_results("pause")

assert_job_status(no_pause, :finished)
assert_job_status(pause, :finished)

wait_for_process_termination_with_timeout(@pid, timeout: 1.second)
assert_clean_termination
end

test "quit supervisor while there are jobs in-flight" do
no_pause = enqueue_store_result_job("no pause")
# long enough pause to make sure it doesn't finish before QUIT arrives
pause = enqueue_store_result_job("pause", pause: 60.seconds)

# Wait for the "no pause" job to finish and the "pause" job to start
# before sending QUIT, so it always arrives while "pause" is in-flight
wait_for_jobs_to_finish_for(3.seconds, except: pause)
wait_while_with_timeout(3.seconds) { !JobResult.exists?(status: "started", value: "pause") }

signal_process(@pid, :QUIT, wait: 0.1.second)

wait_while_with_timeout(2.seconds) { process_exists?(@pid) }
assert_not process_exists?(@pid)

assert_completed_job_results("no pause")
assert_job_status(no_pause, :finished)

assert_started_job_result("pause")
# Workers were shutdown without a chance to terminate orderly, but
# since they're linked to the supervisor, the supervisor deregistering
# also deregistered them and released claimed jobs
assert_job_status(pause, :ready)
assert_clean_termination
end

private
def assert_clean_termination
wait_for_registered_processes 0, timeout: 0.2.second
assert_no_registered_processes
assert_no_claimed_jobs
assert_not process_exists?(@pid)
end

def assert_registered_workers_for(*queues, supervisor_pid: nil)
workers = find_processes_registered_as("Worker")
registered_queues = workers.map { |process| process.metadata["queues"] }.compact
assert_equal queues.map(&:to_s).sort, registered_queues.sort
if supervisor_pid
assert_equal [ supervisor_pid ], workers.map { |process| process.supervisor.pid }.uniq
end
end

def enqueue_store_result_job(value, queue_name = :background, **options)
StoreResultJob.set(queue: queue_name).perform_later(value, **options)
end

def assert_completed_job_results(value, queue_name = :background, count = 1)
actual = skip_active_record_query_cache {
JobResult.where(queue_name: queue_name, status: "completed", value: value).count
}
assert_equal count, actual
end

def assert_started_job_result(value, queue_name = :background, count = 1)
actual = skip_active_record_query_cache {
JobResult.where(queue_name: queue_name, status: "started", value: value).count
}
assert_equal count, actual
end

def assert_job_status(active_job, status)
# Make sure we skip AR query cache. Otherwise the queries done here
# might be cached and since we haven't done any non-SELECT queries
# after they were cached on the connection used in the test, the cache
# will still apply, even though the data returned by the cached queries
# might have been deleted in the forked processes.
skip_active_record_query_cache do
job = SolidQueue::Job.find_by(active_job_id: active_job.job_id)
assert job.public_send("#{status}?")
end
end
end
Loading
Loading