diff --git a/README.md b/README.md index ac408f7a..45ef48b5 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ Or you can also set the environment variable `SOLID_QUEUE_SUPERVISOR_MODE` to `a **The recommended and default mode is `fork`. Only use `async` if you know what you're doing and have strong reasons to** -This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (`threads: N`) or as fibers on a single fiber reactor thread (`fibers: N`). +This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (`threads: N`), in a thread pool with a reactor per thread (`reactor: true`), or as fibers on a single fiber reactor thread (`fibers: N`). Because these are separate concerns, you can combine the default `fork` supervisor mode with fiber workers. In that setup, each worker process gets its own fiber reactor and bounded fiber count. @@ -289,9 +289,10 @@ Here's an overview of the different options: - `threads`: configures a worker to execute jobs in a thread pool of this size. By default, workers use `threads: 3`. Only workers have this setting, and it can't be combined with `fibers`. It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker uses connections for polling and heartbeat and thread mode may use additional connections for job execution. +- `reactor`: when `true`, a thread worker runs each of its threads' jobs inside an [Async](https://github.com/socketry/async) reactor that lives as long as the thread, one job at a time per thread. See [reactor thread workers](#reactor-thread-workers). Requires the `async` gem, and can't be combined with `fibers`. - `fibers`: configures a worker to execute jobs as fibers on a single fiber reactor thread, with this value as the maximum number of in-flight jobs. It can't be combined with `threads`. Fiber workers require fiber-scoped isolated execution state. In Rails apps, set `config.active_support.isolation_level = :fiber` before using `fibers`. Solid Queue refuses to boot fiber workers when isolation remains thread-scoped. On Rails 7.2 and later, a practical starting point is usually `3-5` queue database connections per worker process rather than matching the `fibers` value, because ordinary Active Record query paths can release connections between non-blocking waits. On Rails 7.1, size the queue database pool more conservatively, as in-flight fiber jobs may still retain connections roughly in proportion to `fibers`. -- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with both `threads` and `fibers` workers as long as the supervisor is running in the default `fork` mode. **Note**: this option is ignored only when the supervisor itself is [running in `async` mode](#fork-vs-async-mode). +- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with `threads`, `reactor` and `fibers` workers as long as the supervisor is running in the default `fork` mode. **Note**: this option is ignored only when the supervisor itself is [running in `async` mode](#fork-vs-async-mode). - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else. - `batch_maintenance`: whether the dispatcher will sweep stalled [batches](#batch-jobs) as part of its maintenance work, on the same timer as concurrency maintenance (see [batch maintenance](#batch-maintenance)). This is `true` by default; disable it if you don't use batches, or if you run multiple dispatchers and want only some of them doing maintenance work. @@ -383,7 +384,7 @@ queues: back* ### Threads, processes, and signals -By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Workers can also be configured with `fibers`, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling. +By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Workers can also be configured with `reactor`, in which case each of those threads runs its jobs inside a reactor, or with `fibers`, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling. Fiber worker execution is best suited for cooperative, mostly I/O-bound jobs. Blocking or CPU-heavy work still blocks the single reactor thread, so it should not be expected to outperform thread mode for every workload. @@ -414,6 +415,20 @@ This means finished and failed jobs still follow the normal Solid Queue lifecycl In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation. +#### Reactor thread workers + +Jobs on a thread worker run without a reactor, so a client built on Async, such as `async-http`, starts one for every call and pays for a new connection and TLS handshake each time. Fiber workers avoid that, but they run every job on one thread, so a job that blocks without yielding, such as a C extension decoding an image, stalls the others. Setting `reactor: true` on a thread worker keeps the thread pool and gives each thread a reactor that lives as long as the thread, so those clients reuse their connections across calls and across jobs: + +```yml +workers: + - queues: [ api_sync, webhooks ] + threads: 5 + reactor: true +``` + +Each thread still runs one job at a time, so database pool sizing is the same as for `threads`, and fiber-scoped isolated execution state isn't required. + + ### Database configuration You can configure the database used by Solid Queue via the `config.solid_queue.connects_to` option in the `config/application.rb` or `config/environments/production.rb` config files. By default, a single database is used for both writing and reading called `queue` to match the database configuration you set up during the install. diff --git a/lib/solid_queue/configuration.rb b/lib/solid_queue/configuration.rb index 18f833ab..c9d26c78 100644 --- a/lib/solid_queue/configuration.rb +++ b/lib/solid_queue/configuration.rb @@ -8,6 +8,7 @@ class Configuration validate :ensure_configured_processes, :ensure_valid_recurring_tasks validate :ensure_valid_worker_execution_options validate :ensure_fiber_workers_have_required_dependency, :ensure_fiber_workers_use_supported_isolation_level + validate :ensure_reactor_workers_have_required_dependency validate :warn_about_incorrectly_sized_database_pool, :warn_about_missing_config_files before_validation { warnings.clear } @@ -130,6 +131,10 @@ def ensure_valid_worker_execution_options if options.key?(:threads) && options.key?(:fibers) errors.add(:base, "Workers can specify either `threads` or `fibers`, but not both.") end + + if options[:reactor] && fiber_worker?(options) + errors.add(:base, "`reactor` applies to thread workers. Fiber workers already run their jobs on a reactor.") + end end end @@ -143,6 +148,15 @@ def ensure_fiber_workers_have_required_dependency "Add `gem \"async\"` to your Gemfile to configure workers with `fibers`.") end + def ensure_reactor_workers_have_required_dependency + return unless workers_options.any? { |options| reactor_thread_worker?(options) } + + require "async" + rescue LoadError + errors.add(:base, "Reactor thread workers require the `async` gem. " \ + "Add `gem \"async\"` to your Gemfile to configure workers with `reactor: true`.") + end + def ensure_fiber_workers_use_supported_isolation_level return unless workers_options.any? { |options| fiber_worker?(options) } @@ -319,6 +333,10 @@ def fiber_worker?(options) options.key?(:fibers) end + def reactor_thread_worker?(options) + options[:reactor] && !fiber_worker?(options) + end + def worker_defaults_for(options) if fiber_worker?(options) WORKER_DEFAULTS.except(:threads) diff --git a/lib/solid_queue/reactor_thread_pool.rb b/lib/solid_queue/reactor_thread_pool.rb new file mode 100644 index 00000000..ae7ba0e0 --- /dev/null +++ b/lib/solid_queue/reactor_thread_pool.rb @@ -0,0 +1,138 @@ +# frozen_string_literal: true + +module SolidQueue + class ReactorThreadPool < Pool + def initialize(size, on_idle: nil, on_unrecoverable_error: nil) + super + + @state_mutex = Mutex.new + @shutdown = false + @fatal_error = nil + @pending_executions = Thread::Queue.new + @reactor_threads = [] + end + + def post(execution) + raise_if_fatal_error! + raise RuntimeError, "Execution pool is shutting down" if shutdown? + + super + end + + def available_capacity + raise_if_fatal_error! + super + end + + def shutdown + state_mutex.synchronize do + next false if @shutdown + + @shutdown = true + end.tap do |shut_down| + # Each thread drains the executions already queued, then its blocked + # pop returns nil and its reactor exits + pending_executions.close if shut_down + end + end + + def shutdown? + state_mutex.synchronize { @shutdown } + end + + def wait_for_termination(timeout) + deadline = Timer.monotonic_time_now + timeout.to_f + threads = state_mutex.synchronize { reactor_threads.dup } + + threads.map { |thread| terminated?(thread, deadline) }.all? + end + + private + attr_reader :pending_executions, :reactor_threads, :state_mutex + + def name + @name ||= "solid_queue-reactor-thread-pool-#{object_id}" + end + + # A thread that died raises its error again from +join+. It was reported + # when the thread died, and the other threads still have to be waited for + def terminated?(thread, deadline) + thread.join([ deadline - Timer.monotonic_time_now, 0 ].max) + rescue Exception + raise unless thread.status.nil? + true + end + + def schedule(execution) + start_reactor_threads_if_needed + pending_executions << execution + end + + # Threads are started lazily, like the fiber pool's reactor, so the pool + # can be built in the supervisor before it forks. + def start_reactor_threads_if_needed + state_mutex.synchronize do + next if reactor_threads.any? + + require "async" + size.times { reactor_threads << start_reactor_thread } + end + end + + # Each thread runs its jobs one at a time on a reactor that lives as long + # as the thread, so async clients keep their connections across jobs + def start_reactor_thread + create_thread do + Sync do |task| + while execution = pending_executions.pop + task.async(execution) do |_execution_task, scheduled_execution| + perform_execution(scheduled_execution) + end.wait + end + end + rescue Exception => error + register_fatal_error(error) + raise + end + end + + def perform_execution(execution) + wrap_in_app_executor do + execution.perform + ensure + stop_tasks_left_running + end + rescue Async::Stop => error + handle_thread_error(error) + register_fatal_error(error) + rescue Exception => error + handle_unrecoverable_error(error) + handle_thread_error(error) + ensure + restore_capacity + end + + # A job runs in its own task, so the tasks it started and didn't wait for + # are that task's children. They're stopped when the job ends, so they + # don't run alongside the next job or hold up shutdown. Transient tasks, + # such as the ones async clients keep their connections open with, stay + def stop_tasks_left_running + Async::Task.current.children&.each do |child| + child.stop unless child.transient? + end + end + + def register_fatal_error(error) + state_mutex.synchronize do + @fatal_error ||= error + end + + on_idle&.call + end + + def raise_if_fatal_error! + error = state_mutex.synchronize { @fatal_error } + raise error if error + end + end +end diff --git a/lib/solid_queue/worker.rb b/lib/solid_queue/worker.rb index 3947b05d..ae36f8aa 100644 --- a/lib/solid_queue/worker.rb +++ b/lib/solid_queue/worker.rb @@ -11,7 +11,13 @@ class Worker < Processes::Poller attr_reader :queues, :pool def initialize(**options) - execution_pool_type = options.key?(:fibers) ? :fiber : :thread + execution_pool_type = if options.key?(:fibers) + :fiber + elsif options[:reactor] + :reactor_thread + else + :thread + end options = options.dup.with_defaults(SolidQueue::Configuration::WORKER_DEFAULTS) execution_pool_size = execution_pool_type == :fiber ? options[:fibers] : options[:threads] diff --git a/test/integration/reactor_processes_lifecycle_test.rb b/test/integration/reactor_processes_lifecycle_test.rb new file mode 100644 index 00000000..e3e558e1 --- /dev/null +++ b/test/integration/reactor_processes_lifecycle_test.rb @@ -0,0 +1,130 @@ +# frozen_string_literal: true + +require "test_helper" + +class ReactorProcessesLifecycleTest < ActiveSupport::TestCase + self.use_transactional_tests = false + + setup do + @pid = fork do + SolidQueue::Supervisor.start(skip_recurring: true, workers: [ { queues: :background, threads: 3, reactor: true, polling_interval: 0.1 } ]) + end + + wait_for_registered_processes(2, timeout: 3.second) + assert_registered_workers_for(:background, supervisor_pid: @pid) + end + + teardown do + terminate_process(@pid) if process_exists?(@pid) + end + + test "process jobs in reactor threads in a forked worker" do + worker = find_processes_registered_as("Worker").first + assert_metadata worker, pool_type: "reactor_thread", pool_size: 3 + + 6.times { |i| enqueue_store_result_job("job_#{i}") } + + wait_for_jobs_to_finish_for(3.seconds) + + assert_equal 6, skip_active_record_query_cache { JobResult.count } + 6.times { |i| assert_completed_job_results("job_#{i}", :background) } + + terminate_process(@pid) + assert_clean_termination + end + + test "term supervisor while there are jobs in-flight" do + no_pause = enqueue_store_result_job("no pause") + pause = enqueue_store_result_job("pause", pause: 1.second) + + # Wait for the "pause" job to start, which means both jobs have been + # claimed, before sending TERM, so the worker finishes them while + # shutting down instead of stopping before it gets to claim them + wait_while_with_timeout(3.seconds) { !JobResult.exists?(status: "started", value: "pause") } + + signal_process(@pid, :TERM, wait: 0.1.second) + wait_for_jobs_to_finish_for(3.seconds) + + assert_completed_job_results("no pause") + assert_completed_job_results("pause") + + assert_job_status(no_pause, :finished) + assert_job_status(pause, :finished) + + wait_for_process_termination_with_timeout(@pid, timeout: 1.second) + assert_clean_termination + end + + test "quit supervisor while there are jobs in-flight" do + no_pause = enqueue_store_result_job("no pause") + # long enough pause to make sure it doesn't finish before QUIT arrives + pause = enqueue_store_result_job("pause", pause: 60.seconds) + + # Wait for the "no pause" job to finish and the "pause" job to start + # before sending QUIT, so it always arrives while "pause" is in-flight + wait_for_jobs_to_finish_for(3.seconds, except: pause) + wait_while_with_timeout(3.seconds) { !JobResult.exists?(status: "started", value: "pause") } + + signal_process(@pid, :QUIT, wait: 0.1.second) + + wait_while_with_timeout(2.seconds) { process_exists?(@pid) } + assert_not process_exists?(@pid) + + assert_completed_job_results("no pause") + assert_job_status(no_pause, :finished) + + assert_started_job_result("pause") + # Workers were shutdown without a chance to terminate orderly, but + # since they're linked to the supervisor, the supervisor deregistering + # also deregistered them and released claimed jobs + assert_job_status(pause, :ready) + assert_clean_termination + end + + private + def assert_clean_termination + wait_for_registered_processes 0, timeout: 0.2.second + assert_no_registered_processes + assert_no_claimed_jobs + assert_not process_exists?(@pid) + end + + def assert_registered_workers_for(*queues, supervisor_pid: nil) + workers = find_processes_registered_as("Worker") + registered_queues = workers.map { |process| process.metadata["queues"] }.compact + assert_equal queues.map(&:to_s).sort, registered_queues.sort + if supervisor_pid + assert_equal [ supervisor_pid ], workers.map { |process| process.supervisor.pid }.uniq + end + end + + def enqueue_store_result_job(value, queue_name = :background, **options) + StoreResultJob.set(queue: queue_name).perform_later(value, **options) + end + + def assert_completed_job_results(value, queue_name = :background, count = 1) + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "completed", value: value).count + } + assert_equal count, actual + end + + def assert_started_job_result(value, queue_name = :background, count = 1) + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "started", value: value).count + } + assert_equal count, actual + end + + def assert_job_status(active_job, status) + # Make sure we skip AR query cache. Otherwise the queries done here + # might be cached and since we haven't done any non-SELECT queries + # after they were cached on the connection used in the test, the cache + # will still apply, even though the data returned by the cached queries + # might have been deleted in the forked processes. + skip_active_record_query_cache do + job = SolidQueue::Job.find_by(active_job_id: active_job.job_id) + assert job.public_send("#{status}?") + end + end +end diff --git a/test/unit/configuration_test.rb b/test/unit/configuration_test.rb index 2a969a93..95a61211 100644 --- a/test/unit/configuration_test.rb +++ b/test/unit/configuration_test.rb @@ -91,6 +91,38 @@ class ConfigurationTest < ActiveSupport::TestCase end end + test "reactor configures thread workers to run each thread's jobs on a reactor" do + configuration = SolidQueue::Configuration.new( + workers: [ { queues: "api*", threads: 3, reactor: true } ], + dispatchers: [], + skip_recurring: true + ) + + assert configuration.valid? + assert_processes configuration, :worker, 1, threads: 3, reactor: true + end + + test "workers reject reactor with fibers" do + with_execution_isolation(:fiber) do + configuration = SolidQueue::Configuration.new( + workers: [ { queues: "llm*", fibers: 10, reactor: true } ], + dispatchers: [], + skip_recurring: true + ) + + assert_not configuration.valid? + assert_match /`reactor` applies to thread workers/, configuration.errors.full_messages.first + end + end + + test "reactor thread workers require the async gem" do + configuration = SolidQueue::Configuration.new(skip_recurring: true, dispatchers: [], workers: [ { threads: 3, reactor: true } ]) + configuration.expects(:require).with("async").raises(LoadError.new("cannot load such file -- async")) + + assert_not configuration.valid? + assert_match /Reactor thread workers require the `async` gem/, configuration.errors.full_messages.first + end + test "fiber worker size inflates required database pool size on Rails 7.1" do skip if fiber_workers_release_connections_between_queries? diff --git a/test/unit/reactor_thread_pool_test.rb b/test/unit/reactor_thread_pool_test.rb new file mode 100644 index 00000000..ac345ee7 --- /dev/null +++ b/test/unit/reactor_thread_pool_test.rb @@ -0,0 +1,279 @@ +require "test_helper" + +class ReactorThreadPoolTest < Minitest::Test + Execution = Struct.new(:started, :results, :pause) do + def perform + started << true if started + sleep(pause) if pause + results << [ Thread.current.object_id, Async::Task.current.reactor.object_id ] if results + end + end + + CancelledExecution = Struct.new(:started) do + def perform + started << true if started + raise Async::Stop.new + end + end + + class FailingExecution + def perform + raise "job failed" + end + end + + CallableExecution = Struct.new(:callable) do + def perform + callable.call + end + end + + def test_builds_a_reactor_thread_pool + pool = mock + + SolidQueue::ReactorThreadPool.expects(:new).with(5, on_idle: nil, on_unrecoverable_error: nil).returns(pool) + + assert_equal pool, SolidQueue::Pool.build(type: :reactor_thread, size: 5) + end + + def test_executes_jobs_concurrently_on_separate_threads_each_inside_a_reactor + pool = SolidQueue::ReactorThreadPool.new(2) + started = Thread::Queue.new + results = Thread::Queue.new + + pool.post Execution.new(started, results, 0.1) + pool.post Execution.new(started, results, 0.1) + + 2.times { Timeout.timeout(1.second) { started.pop } } + entries = 2.times.map { Timeout.timeout(1.second) { results.pop } } + + assert_equal 2, entries.map(&:first).uniq.count + assert_equal 2, entries.map(&:last).uniq.count + wait_for_capacity(pool, 2) + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_keeps_the_same_reactor_across_jobs_on_a_thread + pool = SolidQueue::ReactorThreadPool.new(1) + results = Thread::Queue.new + + pool.post Execution.new(nil, results, nil) + first = Timeout.timeout(1.second) { results.pop } + wait_for_capacity(pool, 1) + pool.post Execution.new(nil, results, nil) + second = Timeout.timeout(1.second) { results.pop } + + assert_equal first, second + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_waits_for_in_flight_executions_during_shutdown + pool = SolidQueue::ReactorThreadPool.new(1) + started = Thread::Queue.new + + pool.post Execution.new(started, nil, 0.1) + Timeout.timeout(1.second) { started.pop } + + pool.shutdown + + assert_equal false, pool.wait_for_termination(0.01) + assert pool.wait_for_termination(1.second) + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_shutdown_runs_executions_queued_before_it + pool = SolidQueue::ReactorThreadPool.new(2) + results = Thread::Queue.new + + pool.post Execution.new(nil, results, 0.05) + pool.post Execution.new(nil, results, 0.05) + pool.shutdown + + assert pool.wait_for_termination(1.second) + assert_equal 2, results.size + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_shutdown_stops_idle_threads + pool = SolidQueue::ReactorThreadPool.new(2) + results = Thread::Queue.new + + pool.post Execution.new(nil, results, nil) + Timeout.timeout(1.second) { results.pop } + + pool.shutdown + + assert pool.wait_for_termination(1.second) + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_refuses_executions_after_shutdown + pool = SolidQueue::ReactorThreadPool.new(1) + pool.shutdown + + error = assert_raises(RuntimeError) { pool.post Execution.new(nil, nil, nil) } + assert_equal "Execution pool is shutting down", error.message + assert_equal 1, pool.available_capacity + ensure + pool&.wait_for_termination(1.second) + end + + def test_keeps_running_jobs_on_the_same_reactor_after_a_job_raises + reported_errors = Thread::Queue.new + original_on_thread_error = SolidQueue.on_thread_error + SolidQueue.on_thread_error = ->(error) { reported_errors << error.message } + + pool = SolidQueue::ReactorThreadPool.new(1) + results = Thread::Queue.new + + pool.post Execution.new(nil, results, nil) + first = Timeout.timeout(1.second) { results.pop } + wait_for_capacity(pool, 1) + + pool.post FailingExecution.new + assert_equal "job failed", Timeout.timeout(1.second) { reported_errors.pop } + wait_for_capacity(pool, 1) + + pool.post Execution.new(nil, results, nil) + second = Timeout.timeout(1.second) { results.pop } + + assert_equal first, second + ensure + SolidQueue.on_thread_error = original_on_thread_error + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_stops_a_task_a_job_left_running_before_the_next_job + pool = SolidQueue::ReactorThreadPool.new(1) + children = Thread::Queue.new + finished = Thread::Queue.new + + pool.post CallableExecution.new(-> { children << Async { sleep 30 } }) + child = Timeout.timeout(1.second) { children.pop } + wait_for_capacity(pool, 1) + + pool.post CallableExecution.new(-> { finished << child.finished? }) + + assert Timeout.timeout(1.second) { finished.pop } + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_a_task_a_job_left_running_does_not_hold_up_shutdown + pool = SolidQueue::ReactorThreadPool.new(1) + children = Thread::Queue.new + + pool.post CallableExecution.new(-> { children << Async { sleep 30 } }) + Timeout.timeout(1.second) { children.pop } + wait_for_capacity(pool, 1) + + pool.shutdown + + assert pool.wait_for_termination(1.second) + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_keeps_a_transient_task_running_across_jobs + pool = SolidQueue::ReactorThreadPool.new(1) + children = Thread::Queue.new + finished = Thread::Queue.new + + pool.post CallableExecution.new(-> { children << Async(transient: true) { sleep 30 } }) + child = Timeout.timeout(1.second) { children.pop } + wait_for_capacity(pool, 1) + + pool.post CallableExecution.new(-> { finished << child.finished? }) + + assert_equal false, Timeout.timeout(1.second) { finished.pop } + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_waits_for_the_other_threads_when_one_has_died + original_on_thread_error = SolidQueue.on_thread_error + SolidQueue.on_thread_error = ->(error) { } + + pool = SolidQueue::ReactorThreadPool.new(2) + started = Thread::Queue.new + results = Thread::Queue.new + + pool.post CallableExecution.new(-> { started << Thread.current; sleep 0.2; results << true }) + busy_thread = Timeout.timeout(1.second) { started.pop } + + idle_thread = pool.send(:reactor_threads).find { |thread| thread != busy_thread } + idle_thread.report_on_exception = false + idle_thread.raise RuntimeError, "thread died" + Timeout.timeout(1.second) { sleep 0.01 while idle_thread.alive? } + + pool.shutdown + + assert pool.wait_for_termination(1.second) + assert_equal 1, results.size + ensure + SolidQueue.on_thread_error = original_on_thread_error + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_starts_threads_lazily_so_the_pool_can_be_built_before_forking + pool = SolidQueue::ReactorThreadPool.new(1) + + pid = fork do + results = Thread::Queue.new + pool.post Execution.new(nil, results, nil) + Timeout.timeout(1.second) { results.pop } + pool.shutdown + pool.wait_for_termination(1.second) + exit!(0) + end + + _, status = Process.waitpid2(pid) + assert_equal 0, status.exitstatus + ensure + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + def test_marks_the_pool_as_fatal_when_an_execution_is_cancelled + notifications = Thread::Queue.new + started = Thread::Queue.new + reported_errors = [] + original_on_thread_error = SolidQueue.on_thread_error + SolidQueue.on_thread_error = ->(error) { reported_errors << error.class.name } + + pool = SolidQueue::ReactorThreadPool.new(1, on_idle: -> { notifications << :changed }) + + pool.post CancelledExecution.new(started) + Timeout.timeout(1.second) { started.pop } + Timeout.timeout(1.second) { notifications.pop } + + error = assert_raises(Async::Stop) { pool.available_capacity } + assert_equal [ error.class.name ], reported_errors + ensure + SolidQueue.on_thread_error = original_on_thread_error + pool&.shutdown + pool&.wait_for_termination(1.second) + end + + private + # A job hands back its result before the pool restores its capacity + def wait_for_capacity(pool, expected) + Timeout.timeout(1.second) { sleep 0.01 until pool.available_capacity == expected } + assert_equal expected, pool.available_capacity + end +end diff --git a/test/unit/worker_test.rb b/test/unit/worker_test.rb index 77f41edf..a74875a4 100644 --- a/test/unit/worker_test.rb +++ b/test/unit/worker_test.rb @@ -13,6 +13,10 @@ class WorkerTest < ActiveSupport::TestCase { name: "fiber", options: { fibers: 3 } + }, + { + name: "reactor_thread", + options: { threads: 3, reactor: true } } ].freeze