Repository navigation
Wake idle workers once a named queue is released - #653
Conversation
Since completion became fire-and-forget in 0.17.0, a worker asks for its next job before the completion of a job in a named queue has been committed, finds the queue still locked, and waits a full pollInterval. Nothing else wakes it, so once inserts stop a backlog in a named queue advances only once per poll. Nudge the pool after completing or failing jobs that belong to a named queue, the same way a jobs:insert notification does. Fixes graphile#650
benjie
left a comment
There was a problem hiding this comment.
Thanks for the PR! This follows pretty much the pattern I was intending to implement for this: count the number of jobs that were released that had queues, and then nudge that many workers (via the pool). I think we should move a few parts of the logic to more central locations, but other than that this looks good 👍 (I've not yet reviewed the test/changeset)
Check that the pool is active and not shutting down in WorkerPool.nudge rather than in each caller, and have a worker ignore a nudge once it is no longer active instead of asserting, since a released worker can still be in the pool until its promise settles.
|
Thanks for the review! Both done in d875b75:
The full suite still passes, and the two new tests still fail on |
d34cc67 to
4fc9ddc
Compare
benjie
left a comment
There was a problem hiding this comment.
I went with a slightly different approach that eliminates additional loops and promises and reduces the amount of code needed; I also trimmed down some of the verbosity of the LLM. WDYT?
|
Thanks, that's much cleaner, and having the nudge right after the query that releases the lock makes more sense. Sorry for leaving the unused import behind, and thanks for fixing it. |
Fixes #650
Since 0.17.0
completeJobandfailJobare fire-and-forget, so a worker that finishes a job in a named queue asks for its next job before the completion has been committed. The queue is still locked,getJobreturns nothing and the worker sleeps forpollInterval. Onlyjobs:insertnudges idle workers, so once inserts stop the queue advances once per poll.This change nudges the pool after completing or failing jobs that belong to a named queue has been committed, once per released queue, in the same way a
jobs:insertnotification does. It covers the batched and unbatched completion paths, and withlocalQueuethe nudge becomes apulse(). It does nothing while the pool is shutting down. Jobs without a queue are unaffected.The nudge is local to the pool that completed the job. Other worker processes still wait for their poll or a notification, the same as before 0.17.0. I kept it local because sending a
pg_notifyper completion seemed too heavy for what is a wake-up hint, but I'm happy to change that if you'd prefer.Tests: added a
test.eachtomain.runTaskList.test.tsthat inserts 5 jobs into one named queue in a single statement and runs a pool withpollInterval: 60_000, with and withoutlocalQueue. Both cases time out onmain(sleepUntil5 s) and finish in about 200 ms with this change. The full suite passes (23 suites, 107 tests) on Node 22 and PostgreSQL 17, and eslint and prettier are clean on the changed files. I added a patch changeset.