Skip to content

Tool results and evidence can end on a lone surrogate half when clipped #207

Open

Description

@rahul05ranjan

What happened?

Mail and page text reaches tool results, recorded evidence and notifications through String.prototype.slice(0, limit) on the UTF-16 code units (for example the mail snippet at apps/server/src/engine/conversation.ts (snippet: body.slice(0, 240), read_mail_thread's body.slice(0, 12000)), the search/web/page clips in apps/server/src/engine/model.ts and the monitor/evidence clips in apps/server/src/engine/service.ts, plus browser.ts and jev/service.ts). When the cut index lands between the two halves of a surrogate pair (an astral character such as an emoji), the clipped value ends on a lone high surrogate:

  • it is not a character: UI renderers show U+FFFD in its place, and
  • consumers that encode the value to UTF-8 (for example any Python service that reads the same data) raise UnicodeEncodeError on the lone half.

Two places in the codebase already guard against exactly this: clipCalendarText in conversation.ts (added with the calendar tool) and sanitizeFileName in files.ts ("Slice code points, not UTF-16 units, so a surrogate pair is never split in half"), and #183 fixed the related case of a surrogate numeric character reference failing the mail list. The mail and page clips are the same class but do not use the guard.

Steps to reproduce

  1. Give a mail body or watched page an astral character (for example 👍, U+1F44D, the pair D83D DC4D) straddling index 240 (or 12000/30000/500 for the other surfaces) — or in code: "a".repeat(239) + "\u{1F44D}".repeat(3).
  2. const snippet = body.slice(0, 240) → the result ends with the lone half \uD83D: /[\uD800-\uDBFF]$/.test(snippet) is true.
  3. The same value passed through JSON.stringify keeps a \ud83d escape, and encoding it to UTF-8 outside JS replaces or rejects the lone half.

Expected: clip at the same limit, but drop the high half with its low half when the cut would split a pair, the way clipCalendarText already does for calendar text.

Commit or release

main @ a4944f4.

Platform

Any (server-side string handling).

Environment and safe logs

Pure string handling; no provider or platform dependency. Reproduced with Node: ("a".repeat(239) + "\u{1F44D}").slice(0, 240).charCodeAt(239).toString(16) → d83d.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions