Repository navigation
fix(indexer): close PageIndex's local SQLite connection so rollback doesn't hit WinError 32 #249
Description
Problem
index_long_document() in openkb/indexer.py (local PageIndex indexing for
long PDFs) never closes the local PageIndex SQLite connection it opens.
client = PageIndexClient(...) / col = client.collection() opens
.openkb/pageindex.db (WAL mode) and keeps it open for the lifetime of the
process — there is no close()/finally anywhere in the function.
If a later step in the same add (indexing itself, or the subsequent
compile_long_doc()) raises, the mutation rollback in openkb/mutation.py
tries to unlink()/rename pageindex.db/-wal/-shm as part of discarding
the failed add. On Windows this fails with WinError 32 ("the process cannot
access the file because it is being used by another process"), because the
SQLite connection opened earlier in the same process is still holding the
file open:
openkb.mutation WARNING: Rolled back interrupted add journal ....json.
...
openkb.mutation WARNING: Mutation rollback failed: [WinError 32] Der Prozess
kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess
verwendet wird: '...\.openkb\pageindex.db'
[ERROR] Rollback failed; mutation journal retained for recovery: ...
The rollback failure leaves a dirty mutation journal on disk and (per
add_coordinator.DirtyRollbackError) stops the rest of the batch instead of
continuing to the next file — a single failed long-document add can abort an
entire multi-file openkb add run.
Reproduction
openkb add <folder-with-a-long-pdf>on Windows.- Indexing (or the subsequent compile step) fails for any reason — e.g. the
credential-forwarding bug in [Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219, or any other transient error during
index_long_document()/compile_long_doc(). - The mutation rollback then hits the
WinError 32above and the batch stops.
Context
- Windows,
openkbinstalled fromVectifyAI/OpenKB(fork-based dev
install), Python 3.12. - The pinned
pageindexversion exposesclose()only on the low-level
SQLiteStorageits local backend holds internally (client._backend ._storage) — there is no publicclose()/context-manager API on
PageIndexClient/Collectionitself. - Related: [Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219 (PageIndex's own LLM calls not receiving the resolved
credential bundle) touches the same function and can be the trigger that
causes indexing to fail and hit this rollback bug — but the two are
independent problems (this one reproduces with any failure after PageIndex
has opened its local DB, not just an auth failure).
Vorschlag
Always close PageIndex's local storage connection(s) in a finally around
index_long_document()'s body, in both the success and the failure path, so
a subsequent mutation rollback can always remove pageindex.db on Windows —
best-effort (never raises), since cleanup must not mask the real indexing
error.
This issue was drafted with the assistance of an AI assistant.
Activity