[mrhoribu]

Fixes #766.

Problem

On Windows, SQLite's own sleeps release the GVL and can deliver interrupts while SQLite holds a connection's mutex. That contradicts Database#busy_timeout's documentation ("holds the GVL") and can deadlock the whole process:

• Sharing: another Ruby thread uses the same connection during the sleep. It blocks on SQLite's mutex while holding the GVL, and the sleeping thread can never get the GVL back.
• Interrupts: a Thread#raise, Thread#kill or Timeout lands in the sleep and unwinds through SQLite's C frames, leaving the mutex locked for the next caller.

We hit the first in production: whole-process freezes at 0% CPU in a multi-process app. A full-memory dump showed the GVL owned by a thread in EnterCriticalSection on a connection mutex, and that mutex owned by a thread in rb_w32_Sleep, inside SQLite's winLock retry, waiting for the GVL. The write-up is in #766 and in elanthia-online/lich-5#2254.

Cause

• SQLite's Windows VFS sleeps through the "Sleep" entry of its system-call table (osSleep) while holding db->mutex:
  • winLock's PENDING-lock retry (osSleep(1) between 3 attempts), which happens whenever another process holds the lock, even with busy_timeout(0);
  • winRetryIoerr;
  • the default busy handler behind busy_timeout(N) (sqliteDefaultBusyCallback → winSleep);
  • winClose's retry.
• With SQLite linked into the extension, mkmf links Ruby's import library before kernel32. The Ruby DLL exports a Sleep symbol that forwards to rb_w32_Sleep: win32/mkexports.rb turns #define Sleep(msec) rb_w32_Sleep(msec) in include/ruby/win32.h into an export. So that "Sleep" slot resolves to rb_w32_Sleep.
  • Every precompiled Windows gem I checked imports Sleep from the Ruby DLL: 13 builds from 1.3.13 to 2.10.2.
  • Cross-linking with MinGW shows the same for a vendored source build. A build with SQLite as a separate DLL (--enable-system-libraries) gets kernel32's Sleep.
• rb_w32_Sleep (thread_win32.c) runs the sleep in BLOCKING_REGION, which releases the GVL and, through unblock_function_set, runs RUBY_VM_CHECK_INTS.

Fix

In Init_sqlite3_native, on Windows, after sqlite3_initialize(), look up kernel32's Sleep with GetProcAddress and install it in SQLite's system-call table through xSetSystemCall.

• Why GetProcAddress: inside the extension, writing Sleep would give you the ruby/win32.h macro and the same Ruby import, so the address has to be looked up at run time.
• Which VFSes: os_win.c's VFSes (win32, win32-longpath, win32-none, win32-longpath-none) share one table. Each is tried in case some are compiled out.
• When SQLite is a separate DLL: this is harmless, because that DLL's Sleep already is kernel32's.
• Guards: #if defined(_WIN32) && SQLITE_VERSION_NUMBER >= 3007006, since xSetSystemCall needs VFS iVersion 3. Nothing changes on other platforms.

Behaviour change

This applies only to builds with SQLite linked into the extension: the precompiled gems and the default build. Builds with --enable-system-libraries (which SQLCipher builds imply) already used kernel32's Sleep. That includes CI's mswin job, but a vendored mswin build would be affected like the default build, and the fix runs there too. On those Windows builds, SQLite's waits now hold the GVL, as documented and as on Linux and macOS. Upgraders will notice:

• Same-process lock holders can't commit. Other Ruby threads don't run during a busy_timeout wait. If another connection in the same process holds the lock (a connection pool, say), its thread can't run to release it, so the wait ends in BusyException after the full timeout.
• Interrupts wait. Timeout.timeout, Thread#raise, Thread#kill and Ctrl-C take effect only once the wait ends. A Timeout shorter than the busy timeout ends in BusyException rather than Timeout::Error.
• VFS retries hold the GVL too. The Windows VFS's own retries on sharing or lock violations (antivirus, indexers) now hold the GVL whatever busy handler is set: up to about 1.4 s per failing I/O call (winRetryIoerr), and up to 200 ms when closing (winClose).
• busy_handler_timeout= doesn't help shared connections. It lets other threads run during the wait only if they don't use the waiting connection. SQLite runs the Ruby handler while holding the connection's mutex, so a thread that calls into the connection meanwhile hangs the process, on every platform. If threads share a connection, keep busy_timeout, or give each thread its own connection.

These are in the CHANGELOG, in Database#busy_timeout's rdoc (GVL and interrupts), and in Database#busy_handler_timeout='s rdoc (the shared-connection caveat).

Alternatives considered

1. Runtime override (this PR). At load time, look up kernel32's Sleep and install it with xSetSystemCall. It doesn't depend on link order or toolchain.
   • Cost: xSetSystemCall is documented as a testing aid, and the set of overridable calls may change between SQLite releases. Mitigations: the return codes are checked, and the extension warns at load time if no VFS accepted the override. test/test_sqlite_sleep.rb also fails on the Windows CI jobs if the override stops working, which the tests-only run demonstrated.
2. Fix the link order. Make sure kernel32 is searched before Ruby's import library for SQLite's objects, for example by appending -lkernel32 right after the vendored libsqlite3.a in extconf.rb. A MinGW cross-link test showed this binds kernel32's Sleep.
   • Cost: it depends on mkmf's link-line details and on each toolchain (rake-compiler-dock, MinGW, mswin). It would also silently regress if the vendored library moves in the link line.
3. Build SQLite as a separate DLL (as --enable-system-libraries does). Its Sleep comes from kernel32.
   • Cost: it changes how the precompiled gems are packaged (another DLL to ship and locate) for the sake of one import.

Option 1 can also be combined with 2 later, if the maintainers prefer to fix the link and keep the override as a guard.

Tests (test/test_sqlite_sleep.rb)

1. A busy_timeout wait doesn't let another Ruby thread run (a ticker thread counts at most a tick or two over a 300 ms wait).
2. A thread using a connection while another thread waits on it doesn't deadlock.
3. A Thread#raise aimed at a thread in a busy_timeout wait leaves the connection usable from another thread.

Tests 2 and 3 run in a child process with a 30 s timeout, because a deadlock stops every thread of the process. They use spawn rather than fork (unlike assert_connection_usable_from_another_thread), so they run on Windows too.

Checking that the tests catch the bug. The bug can't be reproduced natively on Linux, so I used a small test-only LD_PRELOAD shim. It routes nanosleep calls made from sqlite3_native.so through rb_thread_call_without_gvl, which releases the GVL and checks interrupts, as rb_w32_Sleep does. With it, all three tests fail as they would on Windows:

• test 1: "another thread ran 59 times during the busy wait";
• tests 2 and 3: the child deadlocked and was killed after 30 s.

Without the shim, all three pass.

Test 1 is skipped on non-CRuby. TruffleRuby has no GVL, so other threads legitimately run during the wait. The two subprocess tests run everywhere.

Results:

• Linux (Ruby 4.0.9, vendored SQLite 3.54.1): full suite 431 runs, 0 failures, 1 skip (baseline 428 runs, 0 failures, 1 skip). The new tests pass 5 out of 5. RuboCop clean.
• Windows, on this repo's own CI workflow, run on my fork. One run had only the test commit (tests only), the other the fix commit on top (tests + fix):

Windows job	Tests only	Tests + fix
test (windows, 3.2–4.0, disable) (vendored SQLite)	3 failures: the new tests	pass
test source (windows, 3.2–4.0, disable)	3 failures	pass
x64-mingw-ucrt 3.2–4.0 (native gem tests)	3 failures	pass
test (windows, *, enable), mswin (both --enable-system-libraries: SQLite as a separate DLL)	pass	pass

On the Ruby 4.0 Windows job without the fix, the failures were:

• test 1: "another thread ran 49 times during the busy wait";
• tests 2 and 3: the child process deadlocked.

Skip counts are the same in both runs, so the tests ran rather than being skipped. Jobs that link SQLite as a separate DLL pass either way, because that DLL's Sleep comes from kernel32.

On my fork, at 15e4a37 (one more docs-only commit since): 131 jobs pass and 3 fail, none from this change.

• sqlcipher (windows, mingw): the MSYS2 package install fails, the same as on main.
• sqlcipher (windows, 3.3): TestCollation#test_replacing_a_collation_releases_the_previous_comparator, which fails the same way on main at this PR's base commit and on the v2.10.2 tag.
• arm64-darwin 3.3: an artifact download timed out before any test ran. It passed on re-run.

On the Ruby 4.0 Windows job, the log has no "could not point SQLite's Sleep" warning, so the override installed.

One earlier run also had a single failure of the existing test_busy_timeout wall-clock check (0.996 >= 1.0) on x86_64-linux-gnu 4.0.

Notes

• The Ruby DLL also exports a forwarding getpid (rb_w32_getpid), which the extension imports too. It's harmless and left alone.
• The same link-order effect makes MinGW's DLL startup code bind its Sleep to the Ruby export as well. That's outside SQLite and not touched here.