Skip to content

Wake idle workers once a named queue is released - #653

Merged
benjie merged 6 commits into
graphile:mainfrom
efeesrefinal:fix/650-wake-workers-after-queue-unlock
Nov 5, 2026
Merged

benjie merged 6 commits into
graphile:mainfrom
efeesrefinal:fix/650-wake-workers-after-queue-unlock

Conversation

@efeesrefinal

Copy link
Copy Markdown
Contributor

Fixes #650

Since 0.17.0 completeJob and failJob are fire-and-forget, so a worker that finishes a job in a named queue asks for its next job before the completion has been committed. The queue is still locked, getJob returns nothing and the worker sleeps for pollInterval. Only jobs:insert nudges idle workers, so once inserts stop the queue advances once per poll.

This change nudges the pool after completing or failing jobs that belong to a named queue has been committed, once per released queue, in the same way a jobs:insert notification does. It covers the batched and unbatched completion paths, and with localQueue the nudge becomes a pulse(). It does nothing while the pool is shutting down. Jobs without a queue are unaffected.

The nudge is local to the pool that completed the job. Other worker processes still wait for their poll or a notification, the same as before 0.17.0. I kept it local because sending a pg_notify per completion seemed too heavy for what is a wake-up hint, but I'm happy to change that if you'd prefer.

Tests: added a test.each to main.runTaskList.test.ts that inserts 5 jobs into one named queue in a single statement and runs a pool with pollInterval: 60_000, with and without localQueue. Both cases time out on main (sleepUntil 5 s) and finish in about 200 ms with this change. The full suite passes (23 suites, 107 tests) on Node 22 and PostgreSQL 17, and eslint and prettier are clean on the changed files. I added a patch changeset.

Since completion became fire-and-forget in 0.17.0, a worker asks for its
next job before the completion of a job in a named queue has been
committed, finds the queue still locked, and waits a full pollInterval.
Nothing else wakes it, so once inserts stop a backlog in a named queue
advances only once per poll.

Nudge the pool after completing or failing jobs that belong to a named
queue, the same way a jobs:insert notification does.

Fixes graphile#650

@benjie benjie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! This follows pretty much the pattern I was intending to implement for this: count the number of jobs that were released that had queues, and then nudge that many workers (via the pool). I think we should move a few parts of the logic to more central locations, but other than that this looks good 👍 (I've not yet reviewed the test/changeset)

Comment thread src/main.ts Resolved
Comment thread src/main.ts Resolved
Check that the pool is active and not shutting down in WorkerPool.nudge
rather than in each caller, and have a worker ignore a nudge once it is
no longer active instead of asserting, since a released worker can
still be in the pool until its promise settles.
@efeesrefinal

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Both done in d875b75:

  • The _active / _shuttingDown check now lives in WorkerPool.nudge, so the helper only counts the released queues, and the try/catch is gone.
  • worker.nudge() no longer asserts: if the worker is no longer active it logs at debug level and returns false, so the pool moves on to the next worker.

The full suite still passes, and the two new tests still fail on main.

@benjie
benjie force-pushed the fix/650-wake-workers-after-queue-unlock branch from d34cc67 to 4fc9ddc Compare November 4, 2026 16:42
benjie
benjie previously approved these changes Nov 4, 2026

@benjie benjie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with a slightly different approach that eliminates additional loops and promises and reduces the amount of code needed; I also trimmed down some of the verbosity of the LLM. WDYT?

@efeesrefinal

Copy link
Copy Markdown
Contributor Author

Thanks, that's much cleaner, and having the nudge right after the query that releases the lock makes more sense. Sorry for leaving the unused import behind, and thanks for fixing it.

@benjie
benjie merged commit b4f1b41 into graphile:main Nov 5, 2026
14 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A named queue runs one job per pollInterval once inserts stop (since 0.17.0)

2 participants