Repository navigation
Marshal libgit2 P/Invoke strings as UTF-8, not ANSI - #2110
Tyrie Vella (tyrielv) merged 1 commit into
Conversation
libgit2 treats every string it receives and returns as UTF-8. The LibGit2Repo P/Invoke declarations left CharSet unset, so a bare [DllImport] string parameter defaulted to CharSet.Ansi and encoded through the Windows ANSI code page. On a non-UTF-8 code page an unmappable character (for example a repository path under a non-English Windows user name, such as "C:\Users\...\café_日本語") was dropped to '?', so git_repository_open resolved the wrong path and the constructor threw InvalidDataException. Annotate every string parameter with [MarshalAs(UnmanagedType.LPUTF8Str)]. This covers the repository path (git_repository_open), the revspec (git_revparse_single), the object id string (git_oid_fromstr), and the config keys (git_config_get_string, git_config_get_bool, git_config_get_multivar_foreach). Read the borrowed git_error.message pointer as an IntPtr and copy it with Marshal.PtrToStringUTF8, matching the existing GitConfigEntry pattern, so the interop marshaller never frees libgit2's heap memory. The config value output path already used Marshal.PtrToStringUTF8, so it is unchanged. Add LibGit2NonAsciiPathTests, which opens a real repository at a non-ASCII path through git2.dll and reads a non-ASCII config value. Before the fix both cases fail with "No mapping for the Unicode character exists in the target multi-byte code page"; after the fix they pass. The fixture ignores itself when the active ANSI code page is already UTF-8 (CP65001), where the bug cannot reproduce. Document the UTF-8 marshalling contract on the LibGit2Repo class and in AGENTS.md so future libgit2 bindings do not regress. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
1701b75 to
b4e9194
Compare
Nathan Baird (ShiningMassXAcc)
left a comment
There was a problem hiding this comment.
Review — PR #2110: Marshal libgit2 P/Invoke strings as UTF-8, not ANSI
Recommendation: approve (one doc-comment correction, one optional follow-up).
This one is correct, well-scoped, and I reproduced both the bug and the fix end-to-end.
Verification performed
I built the repo's vcpkg libgit2 (libgit2[core,pcre,ssl,winhttp]:x64-windows-dynamic@1.9.4) and
ran a standalone harness declaring git_repository_open twice — once with the pre-fix implicit
CharSet.Ansi, once with [MarshalAs(UnmanagedType.LPUTF8Str)] — against a real repo at
...\MarshalProbe_café_日本語_☃_<rand>, on ACP 1252:
Active ANSI code page: 1252
=== git_repository_open, PRE-FIX (implicit CharSet.Ansi) ===
result code : -1
error : failed to resolve path '...MarshalProbe_caf?_???_?_...':
No mapping for the Unicode character exists in the target multi-byte code page.
=== git_repository_open, POST-FIX (LPUTF8Str) ===
result code : 0
OK: repository opened at the non-ASCII path.
=== git_config_get_string, non-ASCII value round-trip ===
value : café-日本語-☃
expected : café-日本語-☃
OK
That is byte-for-byte the failure in your PR description, so the diagnosis and the fix are both
confirmed against real libgit2 — not just reasoned about.
The git_error.message change is also a genuine fix, and a bigger one than the description claims:
pre-fix ([MarshalAs(LPStr)] string) : failed to resolve path '...café_日本語_☃...'
post-fix (IntPtr + PtrToStringUTF8) : failed to resolve path '...café_日本語_☃...'
Every libgit2 error message containing a non-ASCII path was mojibake in logs and in
InvalidDataException text. Good catch.
Also checked:
- Coverage is complete. Every
stringparameter underLibGit2Repo.Nativeis now annotated
(git_revparse_single,git_repository_open,git_oid_fromstr,git_config_get_string,
git_config_get_bool,git_config_get_multivar_foreach×2).GitConfigEntryalready used
IntPtr+ manual UTF-8, andGitConfigMultivarCallbacktakes onlyIntPtrs, so the reverse
P/Invoke needs nothing.LibGit2Repo.csis the only libgit2 binding site in the repo. net10.0-windows10.0.17763.0, soUnmanagedType.LPUTF8StrandMarshal.PtrToStringUTF8are both
fine.GetMultivarForeachis only called withregex: "", so no null-marshalling edge case.- Build: 0 warnings, 0 errors. Full unit suite 982 tests, 971 passed, 0 failed, 11 skipped.
GVFS.FunctionalTestscompiles clean (the 3CA2022warnings are pre-existing and unrelated). build.yamlruns onpull_requestand callsfunctional-tests.yaml, and the hosted
windows-2025runners are ACP 1252, so the new fixture will really execute rather than
self-ignore.
1. One correction: the GetLastError comment states the wrong reason
// git_error.message is a borrowed UTF-8 char* owned by libgit2. Keep it as an
// IntPtr and copy it with Marshal.PtrToStringUTF8 so the interop marshaller never
// frees libgit2's heap memory (same pattern as GitConfigEntry).Marshal.PtrToStructure<GitError> does not free string fields — only Marshal.DestroyStructure
does. The old [MarshalAs(UnmanagedType.LPStr)] public string Message; leaked nothing and
double-freed nothing; my probe read it repeatedly with no crash, it just produced mojibake. The real
defect was the encoding, exactly as shown above.
The free-the-wrong-allocator hazard is real for a [DllImport] out string, which is what the
AGENTS.md bullet and the existing GetConfigString comment describe — those are correct as
written. It's only this one comment that borrows the rationale where it doesn't apply. Suggest:
// git_error.message is a UTF-8 char* owned by libgit2. Marshalling it as a
// [MarshalAs(LPStr)] struct field decodes it through the ANSI code page and mangles
// any non-ASCII path in the message, so keep the raw pointer and decode it as UTF-8.2. Non-blocking
- Like
LibGit2ConfigTests, this fixture needs nothing from GVFS — no enlistment, no mount — but
living inGVFS.FunctionalTestsputs it behindGlobalSetup, which installsGVFS.Service.exe.
Locally I couldn't run it without a full GVFS install. Precedent is on your side so I won't hold
the PR, but a libgit2-only test assembly (or aGVFS.UnitTestsfixture gated ongit2.dllbeing
present) would make these much cheaper to run. Fine as a follow-up. - The
AGENTS.mdsection is genuinely useful — thanks for writing the contract down rather than
just patching the call sites.
f4728ba
into
microsoft:master
Problem
LibGit2Repo(GVFS.Common/Git/LibGit2Repo.cs) calls libgit2 (git2.dll) in-process for local Git object, tree, blob, and config lookups. libgit2 treats every string it receives and returns as UTF-8.The P/Invoke declarations left
CharSetunset, and there is no[module: DefaultCharSet]override, so a bare[DllImport]string parameter defaults toCharSet.Ansiand encodes through the Windows ANSI code page. On a non-UTF-8 code page, an unmappable character in a path — for example a repository cloned under a non-English Windows user name (C:\Users\…\café_日本語\…) — is dropped to?before libgit2 sees it.git_repository_openthen resolves the wrong path and theLibGit2Repoconstructor throwsInvalidDataException:LibGit2Repois production code reached from mount, prefetch, clone, hooks, and the status cache, so the blast radius is broad (though limited to non-ASCII inputs — ASCII bytes are identical in UTF-8 and ANSI).Fix
[MarshalAs(UnmanagedType.LPUTF8Str)]: the repository path (git_repository_open), the revspec (git_revparse_single), the object-id string (git_oid_fromstr), and the config keys (git_config_get_string,git_config_get_bool,git_config_get_multivar_foreach).git_error.messagepointer as anIntPtrand copy it withMarshal.PtrToStringUTF8, matching the existingGitConfigEntrypattern, so the interop marshaller never frees libgit2's heap memory.Marshal.PtrToStringUTF8, so it is unchanged.CharSet.Unicodeis deliberately not used — that marshals UTF-16, which libgit2 does not accept (the same bug in the other direction).Test
GVFS.FunctionalTests/Tests/LibGit2NonAsciiPathTests.cs(new, modelled onLibGit2ConfigTests) opens a real repository at a non-ASCII temp path throughgit2.dlland reads a non-ASCII config value. Verified with the dev functional-test harness:git_repository_openannotation reverted: both cases fail with the "No mapping for the Unicode character" error above — a genuine regression guard.The fixture ignores itself when the active ANSI code page is already UTF-8 (CP65001), where the bug cannot reproduce.
Docs
To keep future libgit2 bindings from regressing, the UTF-8 marshalling contract is documented on the
LibGit2Repoclass, on the nestedNativeclass, and in a newAGENTS.mdsection.