Python: extract keywords from non-English text for topic selection

[Yufeng He (he-yufeng)]

Motivation and Context

_WORD_PATTERN in _harness/_memory.py was re.compile(r"[a-z0-9][a-z0-9_-]{1,}", flags=re.IGNORECASE) — ASCII-only. _extract_keywords runs it over the input messages, and _select_topics returns early when the keyword set is empty. So a message written in CJK, Cyrillic, or any other non-Latin script yields no keywords, and non-English users never get their memory topic files auto-loaded.

Description

Make the pattern Unicode-aware: re.compile(r"[^\W_][\w-]+"). [^\W_] is a Unicode letter or digit (excluding underscore) and [\w-] is a word char or hyphen, so this is the exact Unicode generalization of the old pattern — English tokenization is byte-for-byte unchanged, and CJK/Cyrillic/etc. text now produces keywords. Verified the invariant directly:

'hello world'        -> ['hello', 'world']      (unchanged)
'foo-bar_baz'        -> ['foo-bar_baz']          (unchanged)
'GPT-4 model'        -> ['GPT-4', 'model']       (unchanged)
'こんにちは 元気ですか'  -> ['こんにちは', '元気ですか']   (was [])
'привет мир друзья'  -> ['привет', 'мир', 'друзья'] (was [])
'a I x' / '!!! ???'  -> []                        (single chars / punctuation still skipped)

Related Issue

Fixes #6989

Contribution Checklist

• Tests pass locally (pytest test_harness_memory.py -k non_english, ruff format + check clean)
• Added a test proving non-English keyword extraction (and unchanged English extraction)
• Linked to an issue with a closing keyword
• Change is focused and touches no unrelated code

[Copilot]

Pull request overview

This PR fixes memory topic auto-loading for non-English user messages by making keyword extraction Unicode-aware in the Python harness memory subsystem.

Changes:

• Update _WORD_PATTERN to match 2+ character “word-like” tokens across Unicode scripts (CJK/Cyrillic/etc.), preventing empty keyword sets for non-Latin input.
• Add a regression test ensuring non-English keyword extraction works and that basic English extraction remains unchanged.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
python/packages/core/agent_framework/_harness/_memory.py	Replaces the ASCII-only keyword regex with a Unicode-aware pattern so non-English messages produce keywords for topic selection.
python/packages/core/tests/core/test_harness_memory.py	Adds a regression test covering CJK and Cyrillic keyword extraction plus a simple English invariance check.

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework/_harness
_memory.py	774	117	84%	78–79, 95, 102, 154, 167, 174–176, 199, 219, 228–232, 295, 301, 336, 417, 425, 427, 431, 434, 438, 533, 545, 563–564, 575–576, 587, 702, 719, 728, 743, 771, 774–776, 786, 808–811, 860, 870, 873, 875, 895, 905, 910, 913, 925, 993, 995, 997, 999, 1001, 1003, 1073, 1090, 1118, 1147, 1155–1157, 1159, 1213, 1239–1242, 1248, 1403, 1406, 1417–1418, 1422, 1425–1427, 1433, 1437, 1442, 1446, 1451, 1455, 1463–1464, 1515, 1518–1521, 1528–1529, 1557, 1561–1569, 1586, 1612–1613, 1616–1617, 1623, 1625, 1630, 1635, 1641
TOTAL	45595	4479	90%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
9415	34 💤	0 ❌	0 🔥	2m 28s ⏱️