Repository navigation
Windows: tree-sitter definitions pass fails on ALL files when repo_path contains non-ASCII (CJK / Cyrillic / Arabic / accented Latin / etc.) characters #636
Description
Bug Description
On Windows, when the repo_path passed to index_repository contains Chinese (CJK) characters in directory or file names, the definitions pass (tree-sitter AST parsing) fails on every file — defs=0, calls=0, imports=0, errors=N for all files. The resulting knowledge graph contains only File/Folder/Project nodes with no Function, Class, Method, or Variable nodes, making the index effectively useless for code intelligence.
This is more severe than issue #571 (project name truncation for CJK paths) — that issue is cosmetic, while this one prevents all code-level analysis entirely.
Steps to Reproduce
- On Windows, create or index a project whose path contains Chinese characters, e.g.:
F:/项目资产-ProjectAssets/智能体/智能体原型/ - Run:
codebase-memory-mcp cli index_repository '{"repo_path":"F:/项目资产-ProjectAssets/智能体/智能体原型"}' - Observe the pipeline log — the definitions pass shows:
pass=definitions defs=0 calls=0 imports=0 errors=41 get_architecturereturns only File/Folder nodes, zero Function/Class/Method nodes
Expected Behavior
Files at paths with Chinese characters should be parsed by tree-sitter identically to ASCII-only paths. A 41-file TypeScript/JS/CSS project should produce, e.g., defs=200+ with errors=0.
Actual Behavior (Verbose Pipeline Log)
pass=discover files=41
pass=structure nodes=60 edges=58 ← File/Folder tree works fine
pass=definitions defs=0 calls=0 imports=0 errors=41 ← ALL files fail
pass=calls total=0 resolved=0 unresolved=0 errors=41
pass=semantic inherits=0 decorates=0 implements=0 errors=41
Root Cause Verified
Identical file content, copied to an ASCII-only path, indexes correctly:
| Test | Path | defs | calls | errors |
|---|---|---|---|---|
| ASCII path | C:/tmp/core-test.js |
30 | 103 | 0 ✅ |
| Chinese path | F:/...智能体.../core.js |
0 | 0 | 41 ❌ |
The 417-line JS file is byte-identical. The only variable is whether the parent path contains CJK characters. Languages tested: JavaScript, TypeScript (.ts/.tsx), Python, CSS — all fail identically when the path has CJK characters.
Environment
- OS: Windows 10 (64-bit)
- codebase-memory-mcp version: v0.8.1
- Filesystem: NTFS (UTF-8)
- Shell tested: git-bash (MSYS2) — same result via direct Python subprocess call
Suspected Cause
Likely a Windows-specific encoding issue. The structure pass (directory enumeration via FindFirstFileW/FindNextFileW) works because it natively uses wide-char UTF-16 paths. But the definitions pass probably reads file paths as plain char* and passes them to tree-sitter via fopen or similar, which on Windows uses the ANSI codepage (e.g., GBK on zh-CN systems) rather than UTF-8. This causes tree-sitter to receive garbled paths or fail to open file contents for AST parsing.
On Linux/macOS, char* paths in UTF-8 work natively because the filesystem encoding is UTF-8 — this explains why similar issues (#571) were reported but the definitions-pass failure was not caught.
Activity
maxniu1 commentedon Jun 26, 2026
Update: This affects ALL non-English Windows users, not just CJK
I want to clarify the actual impact — this is not a Chinese-specific issue. It affects any language with non-ASCII characters on Windows:
Desktop,project项目智能体設定データ프로젝트ПроектНастройкиمشروعبياناتโครงการข้อมูลपरियोजनाडेटाÄnderungenétatfrançaisDự ándữ liệuΈργοΡυθμίσειςפרויקטנתוניםKey insight: The Windows
fopenAPI uses the system ANSI codepage (e.g., GBK on zh-CN, Shift-JIS on ja-JP, CP1251 on ru-RU) rather than UTF-8. So this will break for ALL users whose filenames contain characters outside ASCII-7 — which is the vast majority of Windows users worldwide.If a French Windows user has
C:\Users\élise\projets\état\config.json, it will also fail.The actual root cause is the lack of wide-character (UTF-16) file I/O on Windows, not CJK specifically. The structure pass works because
FindFirstFileWuses wide chars natively — butfopen/tree-sitter file reading uses the narrow-char API which depends on the active codepage.[-]Windows: tree-sitter definitions pass fails on all files when repo_path contains Chinese (CJK) characters[/-][+]Windows: tree-sitter definitions pass fails on ALL files when repo_path contains non-ASCII (CJK / Cyrillic / Arabic / accented Latin / etc.) characters[/+]maxniu1 commentedon Jun 26, 2026
Fix submitted as PR #637: #637
The fix adds
cbm_fopen()— a portable file-open wrapper that uses_wfopen()on Windows (wide-character, UTF-16) and delegates tofopen()on POSIX. Applied to all 3 pipeline files that read source files for tree-sitter parsing.Tested on a 41-file TypeScript/JS/Python project with CJK directory names. Before:
defs=0 errors=41. After:defs=215 errors=0.DeusData commentedon Jun 29, 2026
Thanks. I’ve triaged this under #394 and #592. This is more severe than project-name truncation because it kills code extraction; the fix needs wide-char/non-ASCII path handling through file discovery and tree-sitter reads, plus a Windows non-ASCII path regression.
test(windows): convert fixed reds to green guards, wire CI, rewrite d…
DeusData commentedon Jul 4, 2026
Fixed on main (non-ASCII Windows paths now use _wfopen via #703) and now covered by a green Windows regression guard, tests/windows/test_non_ascii_path.py (added in #639, passing on the Windows CI leg). Closing.
squall-cyber commentedon Jul 9, 2026
Still reproduces on v0.9.0 — the release whose notes claim end-to-end non-ASCII path support incl. "wide-argv + CreateProcessW, escaped/NUL-terminated worker command lines".
On Windows,
index_repository --repo-path "C:\test\雷達測試"(CJK directory, ASCII filenames, UTF-8 content) → the indexing worker crashes, exit 1:{"status":"error","outcome":"exit_nonzero","hint":"Indexing worker crashed on a file..."}. stderr echoes the path mojibaked:雷達測試→?琿?皜祈岫.Isolation (points at the supervisor→worker root-path argv handoff, not file I/O):
outcome=clean, 100 nodes / 371 edges).--args-file(UTF-8 JSON) makes the top-levelrepo_pathecho correctly (no mojibake), but the worker still crashes → the supervisor→worker handoff appears to re-pass the root path via (ANSI) argv and re-mojibakes it.outcome=clean, even a CJK-named file說明書.mdinside). So file enumeration (wide-char API) is fine — only the root-path argv handoff to the worker is still broken.This is worth flagging beyond a single repro. v0.9.0's notes explicitly list wide-argv / worker-command-line handling as fixed, and #639 added a Windows regression guard — yet the crash is back in the shipped release. That is the dangerous kind of regression: it is "marked fixed + CI green", so every release looks fixed while non-ASCII Windows users still cannot index their real directories, and the same class of bug keeps returning with each update. For anyone on a CJK path, "just update" doesn't help — the update itself still ships the crash.
The most likely gap:
tests/windows/test_non_ascii_path.pyprobably exercises the pipeline in-process and never actually spawns the real worker subprocess with a CJK root path — so it structurally cannot catch the argv handoff that breaks in production. A guard that shells out to the realindex_repositoryon a CJK directory and assertsoutcome=clean(end-to-end, real subprocess spawn, not in-process) would catch this whole class of argv-handoff regression that the current guard keeps missing.Environment:
codebase-memory-mcp 0.9.0, Windows 11 (build 26200) x64, standard build (GitHub release archive).DeusData commentedon Jul 9, 2026
Thanks for the follow-up. I am keeping this original issue closed so we do not split the regression across two trackers, but the fresh report is now open as #973, labelled high priority, and assigned to the 0.9.1-rc milestone. We will track the shipped v0.9.0 CJK repo_path regression there.