Repository navigation
Keep the message that hits the limit in the queue - #340
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #340 +/- ##
============================================
+ Coverage 94.58% 94.59% +0.01%
- Complexity 373 375 +2
============================================
Files 59 59
Lines 978 981 +3
============================================
+ Hits 925 928 +3
Misses 53 53 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Pushed a follow-up: the fix left |
`queue:run --limit=1` over two queued messages reported one processed and left the queue empty: `InMemoryAdapter::runExisting()` removed the message that tripped the limit even though the callback returned `false` without handling it. Both shipped adapters already treat `false` as "not consumed" — `queue-redis` deletes only when the callback returned `true`, `queue-amqp` nacks with requeue otherwise — so the stub was the outlier. Spell that contract out on `runExisting()` and `subscribe()`.
9a65e0f to
a258088
Compare
QueueConsumer::run() hits the limit
samdark
left a comment
There was a problem hiding this comment.
The limit fix itself works, but two things need changing:
-
A handled message is processed again after a stop.
QueueConsumer::handle()runs the worker and then returnscanContinue(). When the loop stops after a message was handled (SIGTERM, memory limit), the callback returnsfalse, andInMemoryAdapternow keeps that message, so the next run handles it again:[1, 2]becomes[1, 1, 2].testMessageIsNotConsumedWhenLoopCannotContinue()asserts 0 processed while the handler did run once; assertingexecutionTimesshows it. Checking the loop before processing a message (#346) fixes this, so I'd land both together. -
The new
AdapterInterfacecontract doesn't hold yet. It saysfalsemeans "not consumed, keep it", butQueueConsumerreturnsfalsefor a consumed message on stop, and queue-amqp'ssubscribe()acks regardless of the callback result. The old "Returns false if it cannot continue handling messages" sentence is still there next to it. "FIFO order" also isn't true for queue-db, which orders by priority, then id.
A handled message was kept and handled again when the loop stopped right after it. The callback now returns false only for a message it left untouched, which is what the adapter contract says. Drop the FIFO claim and the contract sentence on subscribe(): queue-db orders by priority then id, and queue-amqp's subscribe() acks regardless of the callback result.
The pause between iterations ran even when the loop had already stopped, so SIGTERM waited out the full --pause before the command exited. The limit option is read once, and both commands return Command::SUCCESS.
Should #346 close with this, or do you still want the callback to return an enum? |
The pause line lost its only (accidental) coverage when the command stopped sleeping on shutdown.
samdark
left a comment
There was a problem hiding this comment.
Please merge this together with yiisoft/queue-amqp#143: with only one of the two, a worker stopping on its memory limit either loses the current message (this PR first) or processes it twice (queue-amqp#143 first). Also, please restore the line in AdapterInterface::subscribe() saying that returning false means the message was not consumed.
|
What does this PR do?
queue:run --limit=1over two queued messages printsMessages processed: 1.and leaves the queue empty. The second message is discarded without ever reaching a handler.Cause
QueueConsumer::run()returnsfalsefrom the callback for the message that trips the limit, without handling it.InMemoryAdapter::runExisting()unsets the message anyway, before it looks at what the callback returned:Both shipped adapters already read
falseas "not consumed":queue-redis—if ($result) { $this->provider->delete(...); }(Adapter.php:35)queue-amqp—if ($this->messageConsumed = $callback($message)) { basic_ack(); } else { basic_nack($tag, false, true); }(ExistingMessagesConsumer.php:42)So the stub is the outlier, and
AdapterInterfacenever said which reading was intended — it only said "Returns false if it cannot continue handling messages".Fix
InMemoryAdapterkeeps a message the callback refused, andrunExisting()/subscribe()state the contract the other adapters already implement.QueueConsumeris unchanged.Not addressed here
One bit cannot carry both "was this message consumed?" and "should I continue?". When the loop stops after a message was handled (
SignalLooponSIGTERM), the callback returnsfalsefor a message that was processed, so it is kept and processed again on the next run. Every adapter behaves this way today; splitting the two signals is anAdapterInterfacechange and belongs in its own issue: #346.Validation
Both new assertions fail without the fix (
actual size 0 matches expected size 1,actual size 1 matches expected size 2) and pass with it. Full suite: 280 tests, 516 assertions, OK. Psalm and PHP CS Fixer clean.