Repository navigation
fix(lab-ops-hub): keep the DNS root dot for percent-encoded host spellings (#647) - #657
Merged
Merged
Conversation
…returned URL for percent-encoded spellings (#647) Judging strips a single trailing root dot from the host, and the percent-decoded return path rebuilt the URL from that stripped host, so http://example.com%2e/ came back as http://example.com/ while the literal spelling http://example.com./ and the Worker validator both returned http://example.com./ — the same target recorded under two URLs. Track whether the mapped host ended in a root dot before the strip and re-attach it on the percent-decoded DNS return path. IP literals keep the canonical-address return (WHATWG serializes them without a root dot, as both validators already did). Verdicts are unchanged: localhost%2e and example.com%2e%2e stay refused. Tests: new Python battery (literal controls, %2e/%2E, subdomain, port, userinfo, ?# markers, percent-encoded U+3002, IP controls, refusals) and a TS pin of the Worker's forms for the same spellings.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing: 3
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #647
Problem
The container validator strips a single trailing DNS root dot from the host before judging (
app.py, #466), and its percent-decoded return path rebuilt the returned URL from that stripped host — so the dot was lost only for the percent-encoded spelling:http://example.com./http://example.com./http://example.com./http://example.com%2e/http://example.com/http://example.com./http://example.com%2e/pathhttp://example.com/pathhttp://example.com./pathThe validated URL is the crawl record and the fetched URL, so the same target was recorded under two different URLs depending on spelling, defeating cross-crawler dedup/comparison.
Solution
Track whether the (decoded, UTS-46-mapped) host ended in a root dot before the judging strip (
has_root_dot), and re-attach it to the host passed to_url_with_hoston the percent-decoded DNS return path. Judging still uses the stripped host, so verdicts are unchanged. IP literals keep the canonical-address return path — WHATWG serializes an IP host without a root dot, and both validators already agreed there (verified:8.8.8.8./,8.8.8.8%2e/,8%2e8%2e8%2e8%2e/all returnhttp://8.8.8.8/on both sides). A percent-encoded U+3002 (%E3%80%82), which maps to the root dot, is fixed by the same mechanism.Verification
a6639b82driving both real validators (Pythonvalidate_crawl_url, WorkervalidateCrawlUrlunder Bun): the issue's table reproduced exactly.?#markers, IP forms, refusals) byte-identical, plus a 150-spelling generated differential (6 hosts × 5 dot spellings × 5 tails) with zero diffs.test_returns_the_root_dot_for_percent_encoded_spellingsand a TS pin of the Worker's forms. Python 58/58, TS 39/39 pass. Ablation exact: withapp.pyreverted, only the new Python test fails.localhost%2eandexample.com%2e%2e/(empty label) stay refused on both sides.validate_crawl_url/validateCrawlUrlcopy exists in opencode, openchamber or dezren39/nix.Need help on this PR? Tag
@codesmith-botwith what you need.