Detect file encoding from byte order mark

[dronov-dmitry]

Summary

Fixes opening files that are not UTF-8: the editor used the configured default encoding (UTF-8) for every file, so e.g. a UTF-16LE file was shown as garbage. Now the file encoding is detected from its byte order mark (BOM) and the file is opened (and saved) in the correct encoding.

How it works

• New utils/FileEncoding detects the BOM:
  • UTF-32 LE/BE, UTF-8, UTF-16 BE/LE (in order of signature length so the 4-byte UTF-32 LE signature is not misread as UTF-16 LE).
• On open (openNamedFileDirect, openNamedFileLegacyDirect, applyExternalDocument) the detected encoding is used and the BOM is stripped from the text.
• On save the document is written back in its original encoding with the BOM restored, so the file on disk stays unchanged in format.
• When no BOM is present, the configured default encoding is used as before.

Verification

• FileEncodingTest — 19 unit tests: BOM detection for all 5 encodings (incl. UTF-32 vs UTF-16 disambiguation and truncated BOMs), fromCharset() reconstruction, BOM stripping, BOM-only files, encode/decode round-trips with Cyrillic text
• New EditorFileEncodingTest — 5 instrumented tests: opens UTF-8+BOM, UTF-16LE+BOM, UTF-16BE+BOM, UTF-32LE+BOM and no-BOM files, checks the decoded text (no stray BOM char) and saves back byte-exact with the original BOM
• EditorRecoveryTest — UTF-16LE+BOM → edit → recreate/restore → save recovery round-trip
• RecoveryRepositoryTest — recovery metadata JSON round-trip covers encoding and hasBom
• ./gradlew test — OK
• ./gradlew compileDebugJavaWithJavac — OK
• ./gradlew lint — no errors
• ./gradlew :app:connectedDebugAndroidTest — 53/53 passed, 1 skipped (API-level gate), on Redmi 7 / Android 11
• UI tests now pre-grant storage permissions with GrantPermissionRule so they no longer fail behind the system permission dialog on fresh installs
• Verified against a real UTF-16LE (BOM) Cyrillic report file

[maxistar]

Thanks, this is a very valuable improvement and the BOM detection/round-trip implementation looks good. I found one data-integrity edge case: after restoring an edited UTF-16/UTF-32 recovery draft, documentEncoding is null, so the next save may rewrite the file using the configured default encoding and drop its BOM. The recovery metadata already has encoding and hasBom; could we persist both and reconstruct documentEncoding in restoreDraft()? A recovery test covering UTF-16LE+BOM → edit → recreate/restore → save would be ideal.

[dronov-dmitry]

Thanks for detailed description of the changes!

To ensure everything works reliably, it would be great to write unit tests covering all these edge cases you've listed. Also, please make sure to include tests for opening files in various character encodings to prevent any regressions there.

[dronov-dmitry]

Hi @maxistar — following up on your review and the test checklist above.

Your review (recovery draft encoding) — addressed in 84ad11c: encoding/hasBom are persisted in recovery metadata and documentEncoding is reconstructed in restoreDraft(), plus the requested UTF-16LE+BOM → edit → recreate/restore → save instrumented test (utf16leWithBomRoundTripsThroughRecoveryRestoreAndSave).

Tests added in 111c03a

Unit tests (FileEncodingTest, 10 → 19 cases, runs in CI via ./gradlew test):

• fromCharset() reconstruction for all 5 supported encodings + no-BOM / null / empty / unknown-charset cases
• truncated BOM detection (FF, EF BB, FE, 00 00 FE, ...)
• BOM-only file decodes to an empty string
• encode with null encoding falls back to the configured default; encode without BOM writes no signature
• full decode/encode round-trip for UTF-8, UTF-16LE/BE and UTF-32LE/BE with Cyrillic text

Opening files in various character encodings (EditorFileEncodingTest, 5 instrumented cases):

• UTF-8+BOM, UTF-16LE+BOM, UTF-16BE+BOM, UTF-32LE+BOM: open → correct text with no stray U+FEFF BOM character → edit → save → bytes keep the original BOM and encoding
• file without BOM: opened with the configured default encoding and saved without a BOM (regression guard)

Recovery metadata JSON round-trip now also asserts encoding and hasBom.

Also in 11be531: UI tests were failing on a fresh install because EditorActivity requests storage permissions in onCreate() and the system permission dialog covers the activity (NoActivityResumedException). Added GrantPermissionRule (the pattern already used in EditorActivityTest) to the affected classes — the full instrumented suite is now green: 53/53 passed (1 skipped by an API-level gate).

Verification: ./gradlew test ✅ · ./gradlew lint ✅ · ./gradlew :app:connectedDebugAndroidTest ✅ (Redmi 7, API 30)

Ready for another look.