fix: point SQLite's Sleep at kernel32 on Windows

CHANGELOG.md

@@ -2,12 +2,21 @@
 
 ## next / unreleased
 
+### Security / Stability
+
+- Fix a deadlock on Windows when SQLite, linked into the extension (the precompiled gems and the default build), waits for a lock. SQLite's sleeps (the default busy handler installed by `Database#busy_timeout`, and the Windows VFS's lock, I/O and close retries) resolved to the Ruby DLL's `Sleep`, which forwards to `rb_w32_Sleep`. That released the GVL and checked for interrupts while SQLite held the connection's mutex, so another thread using the same connection could block forever with the GVL held, and a `Thread#raise` or `Thread#kill` during the wait could leave the connection unusable. The extension now points SQLite's `Sleep` at kernel32's at load time. Builds that use SQLite as a separate DLL (`--enable-system-libraries`, which SQLCipher builds imply) already used kernel32's `Sleep` and are unaffected. #766, #771 @mrhoribu
+
 ### Added
 
 - Experimental Ractor support. On Ruby 3.4 and later, when SQLite is compiled thread-safe, the extension declares itself Ractor-safe, and each Ractor can open and use its own connections. `SQLite3.ractor_safe?` reports whether the extension made the declaration. See [RACTOR.md](RACTOR.md) for the limits on sharing connections between Ractors and on fork safety. We welcome bug reports, but we recommend caution before using Ractor support in production applications. #749 @rubys @sandlerr @tenderlove @flavorjones
 
 ### Changed
 
+- On Windows, when SQLite is linked into the extension (the precompiled gems and the default build), SQLite's waits now hold the GVL, as documented and as on other platforms. #766, #771 @mrhoribu
+  - Other Ruby threads don't run while a connection waits in `Database#busy_timeout`. If the lock is held by another connection in the same process (for example a connection pool), that thread can't run to release it, so the wait ends in `SQLite3::BusyException` after the full timeout.
+  - `Timeout.timeout`, `Thread#raise`, `Thread#kill` and Ctrl-C take effect only once the wait ends. A `Timeout` shorter than the busy timeout now ends in `SQLite3::BusyException` rather than `Timeout::Error`.
+  - The Windows VFS's own retries on sharing or lock violations (for example from antivirus or indexing software) now hold the GVL too, whatever busy handler is set: up to about 1.4 s per failing I/O call, and up to 200 ms when closing.
+  - For a lock wait that lets other threads run, use `Database#busy_handler_timeout=`, but only if no other thread uses the waiting connection: a thread that calls into it during the wait blocks with the GVL held and the process hangs. If threads share a connection, keep `Database#busy_timeout`, or give each thread its own connection.
 - `SQLite3::VERSION`, `SQLite3::VERSION_INFO`, `SQLite3::SQLITE_VERSION` and `SQLite3::SQLITE_LOADED_VERSION` are frozen, so that any Ractor can read them. Code that mutates them now raises `FrozenError`. #749 @rubys @tenderlove
 
 ## 2.10.2 / 2026-10-26

ext/sqlite3/database.c

@@ -784,6 +784,14 @@ set_authorizer(VALUE self, VALUE authorizer)
  * busy resources. To restore the default behavior, send 0 as the
  * +ms+ parameter.
  *
+ * The calling thread holds the GVL while SQLite sleeps and retries: other
+ * Ruby threads don't run during the wait, and interrupts such as
+ * Thread#raise, Thread#kill, Timeout.timeout or Ctrl-C take effect only once
+ * the wait ends. In particular, another thread of the same process that holds
+ * the lock can't run to release it, so the wait ends in SQLite3::BusyException
+ * after the full timeout. See #busy_handler_timeout= for a wait that lets
+ * other threads run.
+ *
  * See also the mutually exclusive #busy_handler.
  */
 static VALUE

ext/sqlite3/sqlite3.c

@@ -202,6 +202,66 @@ init_sqlite3_constants(void)
 #endif
 }
 
+#if defined(_WIN32) && SQLITE_VERSION_NUMBER >= 3007006
+/*
+ * Point SQLite's "Sleep" system call at kernel32's Sleep.
+ *
+ * SQLite's Windows VFS sleeps through the "Sleep" entry of its system-call table (osSleep) while
+ * holding the connection's mutex: in winLock's lock retries, winRetryIoerr, winClose's retry, and
+ * winSleep, which the default busy handler behind Database#busy_timeout and the WAL read retry
+ * call. When SQLite is linked into this extension (the precompiled gems and the default build),
+ * the linker resolves that "Sleep" against the Ruby DLL, which exports a "Sleep" that forwards to
+ * rb_w32_Sleep (win32/mkexports.rb turns the "#define Sleep" in include/ruby/win32.h into an
+ * export, and mkmf links the Ruby DLL's import library ahead of kernel32). rb_w32_Sleep releases
+ * the GVL and checks for interrupts, so during SQLite's sleep another Ruby thread could enter the
+ * connection and block on its mutex with the GVL held, and a Thread#raise or Thread#kill could
+ * unwind through SQLite and leave the mutex locked. Either way the process deadlocks.
+ *
+ * kernel32's Sleep is looked up at runtime because a direct reference to Sleep in this extension
+ * would bind to the same Ruby-DLL import. The system-call table is shared by all of os_win.c's
+ * VFSes, so setting it on any of them sets it for all; each is tried in case some are compiled
+ * out. It is harmless when SQLite is a separate DLL (--enable-system-libraries, which SQLCipher
+ * builds imply), whose Sleep already comes from kernel32.
+ *
+ * xSetSystemCall is documented as a testing aid, and the set of calls it can override may change
+ * between SQLite releases. If no VFS accepts the override, this warns instead of failing silently.
+ * test/test_sqlite_sleep.rb guards the behavior on the Windows CI jobs.
+ *
+ * See https://github.com/sparklemotion/sqlite3-ruby/issues/766
+ */
+static void
+use_kernel32_sleep(void)
+{
+    static const char *const vfs_names[] = {
+        "win32", "win32-longpath", "win32-none", "win32-longpath-none"
+    };
+    HMODULE kernel32 = GetModuleHandleW(L"kernel32.dll");
+    sqlite3_syscall_ptr kernel32_sleep = NULL;
+    int installed = 0;
+    size_t i;
+
+    if (kernel32) {
+        kernel32_sleep = (sqlite3_syscall_ptr)(void (*)(void))GetProcAddress(kernel32, "Sleep");
+    }
+
+    if (kernel32_sleep) {
+        for (i = 0; i < sizeof(vfs_names) / sizeof(vfs_names[0]); i++) {
+            sqlite3_vfs *vfs = sqlite3_vfs_find(vfs_names[i]);
+            if (vfs && vfs->iVersion >= 3 && vfs->xSetSystemCall &&
+                    vfs->xSetSystemCall(vfs, "Sleep", kernel32_sleep) == SQLITE_OK) {
+                installed = 1;
+            }
+        }
+    }
+
+    if (!installed) {
+        rb_warn("sqlite3: could not point SQLite's Sleep at kernel32's Sleep; SQLite's lock waits "
+                "may release the GVL while holding the connection's mutex "
+                "(https://github.com/sparklemotion/sqlite3-ruby/issues/766)");
+    }
+}
+#endif
+
 RUBY_FUNC_EXPORTED
 void
 Init_sqlite3_native(void)
@@ -223,6 +283,9 @@ Init_sqlite3_native(void)
 #ifdef HAVE_SQLITE3_INITIALIZE
     sqlite3_initialize();
 #endif
+#if defined(_WIN32) && SQLITE_VERSION_NUMBER >= 3007006
+    use_kernel32_sleep();
+#endif
 
     init_sqlite3_exception();
     init_sqlite3_constants();

lib/sqlite3/database.rb

@@ -682,6 +682,12 @@ def readonly?
     # but only retries up to the indicated number of +milliseconds+.
     # This is an alternative to #busy_timeout, which holds the GVL
     # while SQLite sleeps and retries.
+    #
+    # SQLite runs the handler while holding this connection's mutex. Other
+    # threads can run during the wait only if they don't use this connection:
+    # a thread that calls into it meanwhile blocks on that mutex while holding
+    # the GVL, and the process hangs. If threads share a connection, use
+    # #busy_timeout, or give each thread its own connection.
     def busy_handler_timeout=(milliseconds)
       timeout_seconds = milliseconds.fdiv(1000)
 

test/test_sqlite_sleep.rb

@@ -0,0 +1,159 @@
+require "helper"
+require "rbconfig"
+require "tempfile"
+require "tmpdir"
+
+module SQLite3
+  # SQLite sleeps while it holds a connection's mutex: in the default busy handler that
+  # Database#busy_timeout installs, and on Windows also in os_win.c's lock and I/O retries. Those
+  # sleeps must not release the GVL or check for interrupts, as Database#busy_timeout documents.
+  # Otherwise another Ruby thread can enter the connection and block on SQLite's mutex while
+  # holding the GVL, or an interrupt can unwind through SQLite's frames and leave that mutex
+  # locked. On Windows, SQLite's Sleep used to resolve to the Ruby DLL's "Sleep" export, which
+  # forwards to rb_w32_Sleep and does both. See https://github.com/sparklemotion/sqlite3-ruby/issues/766
+  class TestSQLiteSleep < SQLite3::TestCase
+    SUBPROCESS_TIMEOUT = 30
+
+    def setup
+      @dir = Dir.mktmpdir("sqlite3-ruby-sleep")
+      @path = File.join(@dir, "test.db")
+      SQLite3::Database.new(@path) { |db| db.execute("create table t (x)") }
+      @holder = SQLite3::Database.new(@path)
+      @db = SQLite3::Database.new(@path)
+    end
+
+    def teardown
+      @ticker&.kill
+      @ticker&.join
+      @db&.close
+      @holder&.close
+      FileUtils.remove_entry(@dir) if @dir
+    end
+
+    def test_busy_timeout_wait_does_not_release_the_gvl
+      skip("only CRuby has a GVL") unless RUBY_ENGINE == "ruby"
+
+      @holder.execute("begin exclusive")
+      @db.busy_timeout = 300
+
+      ticks = 0
+      @ticker = Thread.new do
+        loop do
+          ticks += 1
+          sleep 0.005
+        end
+      end
+      sleep 0.05 until ticks > 0
+
+      before = ticks
+      assert_raises(SQLite3::BusyException) { @db.execute("select count(*) from t") }
+      during = ticks - before
+
+      # The wait takes about 300 ms. A wait that released the GVL would let the ticker run
+      # dozens of times; holding it allows at most a tick or two around the call's edges.
+      assert_operator(during, :<=, 3, "another thread ran #{during} times during the busy wait")
+    end
+
+    def test_thread_sharing_a_connection_during_a_busy_wait_does_not_deadlock
+      assert_completes_in_subprocess(<<~RUBY)
+        holder = SQLite3::Database.new(PATH)
+        holder.execute("begin exclusive")
+        shared = SQLite3::Database.new(PATH)
+        shared.busy_timeout = 500
+
+        waiter = Thread.new do
+          shared.execute("select count(*) from t")
+        rescue SQLite3::BusyException
+          nil
+        end
+        sleep 0.05 # let the waiter start its busy wait
+
+        # Use the same connection from this thread while the waiter may be in SQLite's sleep.
+        begin
+          shared.execute("select count(*) from t")
+        rescue SQLite3::BusyException
+          nil
+        end
+        waiter.join
+
+        holder.execute("rollback")
+        shared.execute("select count(*) from t")
+      RUBY
+    end
+
+    def test_interrupt_during_a_busy_wait_leaves_the_connection_usable
+      assert_completes_in_subprocess(<<~RUBY)
+        holder = SQLite3::Database.new(PATH)
+        holder.execute("begin exclusive")
+        shared = SQLite3::Database.new(PATH)
+        shared.busy_timeout = 500
+
+        waiter = Thread.new do
+          shared.execute("select count(*) from t")
+        rescue SQLite3::BusyException
+          nil
+        end
+        waiter.report_on_exception = false
+        sleep 0.05 # let the waiter start its busy wait
+
+        waiter.raise(RuntimeError, "interrupted")
+        begin
+          waiter.join
+        rescue RuntimeError
+          nil
+        end
+
+        holder.execute("rollback")
+        Thread.new { shared.execute("select count(*) from t") }.join
+      RUBY
+    end
+
+    private
+
+    # Runs +body+ in a child Ruby process with the same sqlite3 loaded and PATH set to the test
+    # database, and asserts that it finishes. A deadlock here stops every thread of the process,
+    # so it can only be detected from outside it. Uses spawn rather than fork so it also runs on
+    # Windows, where this bug lived.
+    def assert_completes_in_subprocess(body)
+      script = Tempfile.new(["sqlite3-sleep", ".rb"])
+      script.write(<<~RUBY)
+        require "sqlite3"
+        PATH = ARGV.fetch(0)
+        #{body}
+        print "ok"
+      RUBY
+      script.close
+
+      sqlite3_dir = File.dirname($LOADED_FEATURES.find { |f| f.end_with?("/sqlite3.rb") })
+      read, write = IO.pipe
+      pid = Process.spawn(RbConfig.ruby, "-I", sqlite3_dir, script.path, @path, out: write, err: write)
+      write.close
+
+      output = +""
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + SUBPROCESS_TIMEOUT
+      status = nil
+      loop do
+        if read.wait_readable(0.1)
+          chunk = read.read_nonblock(4096, exception: false)
+          output << chunk if chunk.is_a?(String)
+        end
+        _, status = Process.waitpid2(pid, Process::WNOHANG)
+        break if status
+        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline
+      end
+
+      unless status
+        Process.kill(:KILL, pid)
+        Process.waitpid(pid)
+        flunk("child process did not finish in #{SUBPROCESS_TIMEOUT}s (deadlocked on the connection?)")
+      end
+
+      output << read.read.to_s
+      assert_predicate(status, :success?, "child process failed: #{output}")
+      assert(output.end_with?("ok"), "child process did not finish its script: #{output}")
+    ensure
+      read&.close
+      script&.unlink
+    end
+  end
+end