Repository navigation
lab-ops-hub: container validator blanket-refuses CONTEXTJ-valid ZWNJ/ZWJ hosts (virama and Arabic-script joining contexts) the Worker validator accepts #531
Description
Finding
PR #530 added U+200C (ZWNJ) and U+200D (ZWJ) to the container validator's _IDNA_DISALLOWED set, so validate_crawl_url now refuses every host containing a zero-width joiner — including hosts where the joiner is in a valid joining context and IDNA/UTS-46 (with CheckJoiners, as the WHATWG URL parser applies it) accepts it.
ZWNJ/ZWJ are not DISALLOWED in IDNA2008; they are CONTEXTJ (RFC 5892 Appendix A.1/A.2):
- ZWJ (U+200D) is allowed when the preceding character has Canonical_Combining_Class 230 (a virama) — the standard Indic conjunct spelling, e.g. Devanagari क + ् + ZWJ + य.
- ZWNJ (U+200C) is allowed after a virama, or in a valid Arabic-script joining context — routine in ordinary Persian words, e.g. میخواهم.
PR #530's own regression test only pins the no-context case (exam<ZWJ>ple.com, comment: "no joining context here"), which both validators do refuse — but the implementation is a blanket ord(c) in _IDNA_DISALLOWED check with no context evaluation.
Evidence
- PR: lab-ops-hub: judge container crawl hosts on the full UTS-46 mapped form (#516, #526) #530 (merged, commit
43a89a81, mergea8a3bf7e) pkgs/lab-ops-hub/container/app.pyon current main:_IDNA_DISALLOWED(line ~55) contains0x200C, 0x200D, and the forbidden-character check refuses any host containing them, with no CONTEXTJ context rule.- Verified on current main (
a8a3bf7e) with the actualvalidate_crawl_urlfromcontainer/app.py:http://क्य.com/(U+0915 U+094D U+200D U+092F) → refused: "refusing to crawl host with a forbidden character"http://میخواهم.com/(with U+200C in joining context) → refused, same errorhttp://exam<ZWJ>ple.com/→ refused (correct — matches the Worker)
- Verified the Worker side with the WHATWG URL parser (Node
new URL, the same UTS-46 + CheckJoiners processing the Worker'svalidateCrawlUrlinpkgs/lab-ops-hub/src/lib.tsrelies on — it has no joiner check of its own):http://क्य.com/→ accepted, hostnamexn--11b6c9d222i.comhttp://میخواهم.com/→ accepted, hostnamexn--mgbn2ecje63gr19l.comhttp://exam<ZWJ>ple.com/andhttp://exam<ZWNJ>ple.com/→ rejected, matching the container
Why it matters
This is the same Worker/container parity class as #516/#526/#529, in the fail-closed direction: the Worker validator accepts these public IDN hosts and dispatches the crawl, and the container — the component that actually fetches — then refuses them, so legitimate crawls of Persian/Arabic-script and Indic domains whose names contain a contextual joiner fail at crawl time. ZWNJ in particular is not exotic: it appears in everyday Persian orthography, so a large class of real .ir / Persian-script hostnames is affected.
Single-repo: the validator pair (container/app.py + src/lib.ts) lives only in this repo (checked the opencode/openchamber/nix clones — no copies).
Acceptance criteria
-
validate_crawl_urlevaluates U+200C/U+200D under the RFC 5892 CONTEXTJ rules (A.1 for ZWNJ, A.2 for ZWJ) instead of blanket-refusing them: a joiner in a valid context (after a virama; valid Arabic-script joining context for ZWNJ) no longer triggers the forbidden-character refusal. - Joiners with no valid context (e.g.
exam<ZWJ>ple.com,exam<ZWNJ>ple.com, a label-leading joiner) are still refused, keeping PR lab-ops-hub: judge container crawl hosts on the full UTS-46 mapped form (#516, #526) #530's lab-ops-hub: container validator still allows IDNA-disallowed Unicode hosts (space separators, bidi/format controls, U+FFFD) the Worker validator refuses #516 regression tests green. - New tests in
pkgs/lab-ops-hub/test/test_container_validate.pypin both directions: at least one virama+ZWJ Indic host and one Arabic-script ZWNJ host accepted, plus the existing no-context refusals. - A differential probe against the Worker's WHATWG parsing over a host battery including the CONTEXTJ cases shows no container/Worker mismatch for joiner hosts.
Activity
dezren39 commentedon Oct 26, 2026
Intake review
pkgs/lab-ops-hub/container/app.py:72still lists0x200C, 0x200Din_IDNA_DISALLOWED, and the blanketord(c) in _IDNA_DISALLOWEDrefusals stand at :414 and :580 — no CONTEXTJ evaluation anywhere. Sibling of code lab-ops-hub: container crawler's 20s timeout is per-socket-read, not a deadline — a slow-drip target hangs the single-threaded crawler indefinitely #532 (same file, enriched last monitor run)._IDNA_DISALLOWEDtest in both call sites, add the issue's two positive hosts plus the existing negative cases as tests, and run the differential host battery from the acceptance criteria.— Fleet monitor intake enrichment, 2026-10-26 run; verified against current main this run (code
45b57e13, opencodec792e04fe0).dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up (triage)
pkgs/lab-ops-hub/container/app.pystill blanket-refuse ZWNJ/ZWJ hosts, and what does the Worker validator do with CONTEXTJ-valid ones?dezren39 commentedon Nov 1, 2026
🤖 Agent note · triage verdict — CONFIRMED (task 2115)
Evidence (
pkgs/lab-ops-hub/container/app.py, current main):_IDNA_DISALLOWED(lines 71-75) contains0x200Cand0x200Dunconditionally, andvalidate_crawl_urlrefuses any host with a member codepoint (the forbidden-character check at line ~414) — there is no joining-context logic anywhere in the function. The set's own comment (line 67) says "ZWNJ / ZWJ outside their joining context", but the code does not implement the qualifier; a later comment (line ~600) acknowledges the pair "is refused by the forbidden-character checks before this helper runs".test/test_container_validate.pylines 663-664 (\u200c/\u200d, commented "no joining context here"). No test accepts a CONTEXTJ-valid host — e.g. a Persian ZWNJ word (میخواهم) or a Devanagari virama+ZWJ conjunct — which the Worker side accepts via WHATWG/UTS-46CheckJoiners.Fix direction (pinned): take U+200C/U+200D out of the blanket
_IDNA_DISALLOWEDset and enforce RFC 5892 CONTEXTJ instead — ZWJ (A.1) only when the preceding character has Canonical_Combining_Class 230 (virama); ZWNJ (A.2) only after a virama or in a valid joining context (preceding character Joining_Type D/C, transparent characters aside) — or delegate the host check to a UTS-46 process with joiner checking enabled, matching the Worker. Tests: CONTEXTJ-valid Persian and Devanagari hosts accepted; the existing no-context refusals (test lines 663-664) keep refusing.Routing: follow-up fix enqueued as a
small_fixqueue task (deduperepo-health:v1:fix:developing-today/code:issue:531). This triage task completes astriaged.18 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (re-triage — CONFIRMED, defect resolved, follow-up live)
Verdict: CONFIRMED on current main
1e3763fa(verified fresh this shift).pkgs/lab-ops-hub/container/app.py:_IDNA_DISALLOWED(:72-73) still contains0x200C, 0x200D; the blanket checks stand at :415 and :669; and_judge_idna_label(:768) refuses any category-Cfcharacter — ZWNJ/ZWJ are Cf, so three sites refuse CONTEXTJ-valid hosts, not one. Tests still pin only the no-context refusals (test_container_validate.py:677-678). No PR for lab-ops-hub: container validator blanket-refuses CONTEXTJ-valid ZWNJ/ZWJ hosts (virama and Arabic-script joining contexts) the Worker validator accepts #531 exists.Task 4812's packet defect — resolved, with evidence (laptop probes, this shift):
idnadependency is needed. The service is stdlib-only by design (app.py header; the Dockerfile installs nothing), and stdliblabel.encode("idna")accepts both CONTEXTJ-valid labels from this issue (Devanagari U+0915 U+094D U+200D U+092F; Persian U+0645 U+06CC U+200C U+062E U+0648 U+0627 U+0647 U+0645). The blockers are only the validator's own three refusal sites.unicodedata.combining(U+094D) == 9(RFC 5892's symbolic class Virama). An implementation keyed on 230 would never match a virama and would fail its own acceptance tests. The corrected packet pins CCC 9.unicodedatadoes not expose: the ZWNJ Arabic-script rule (RFC 5892 A.1) needs a vendored, range-compressed Joining_Type table in app.py (source + Unicode version cited). That is now explicitly in the follow-up's scope.Routing: the corrected
small_fixpacket (deduperepo-health:v1:fix:developing-today/code:issue:531, staged athidden_files/packets/task6886-fix-531.jsonin the goal workspace) revived task 4812, verifiedclaimedin the queue at 2026-11-03T10:02Z — a lane is already working it with the corrected scope (stdlib-only CONTEXTJ at all three sites + vendored Joining_Type table + both-direction tests + the differential Worker battery).queue/claimed/work/assignedlabels are left on #531 mirroring task 4812's claim; this triage'sstatus/in-progressis removed.dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
bf419306-c4ef-470f-a553-beabf93239de)repo-health-code-pilot.dezren39 commentedon Nov 3, 2026
🤖 Agent note · blocked (packet defect)
bf419306-c4ef-470f-a553-beabf93239de)packet_defect; no source files changed.D or C, but RFC 5892 Appendix A.1 (with verified erratum 3312) requires(L or D), then zero or moreT, U+200C, zero or moreT, then(R or D).C(Join_Causing) is not a valid substitute; following the packet literally could accept contexts RFC/WHATWG reject, violating the security and differential-parity acceptance criteria./home/user/git/fleet/task-4812onqueue/4812-531-contextjis preserved.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 4812 (small_fix, blocked, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03