fix(lab-ops-hub): return empty path, userinfo and ?/# markers WHATWG-normalized (#639) by dezren39 · Pull Request #648 · developing-today/code

dezren39 · 2026-10-23T06:01:59Z

Fixes #639

Problem

validate_crawl_url (container, pkgs/lab-ops-hub/container/app.py) returned three spellings without the WHATWG normalizations the Worker validator applies, so the two crawlers recorded different URLs for the same accepted target:

Input	Container returned	Worker returns
`http://example.com`	`http://example.com`	`http://example.com/`
`http://example.com:8080`	`http://example.com:8080`	`http://example.com:8080/`
`http://@example.com/`	`http://@example.com/`	`http://example.com/`
`http://example.com/path?`	`http://example.com/path`	`http://example.com/path?`
`http://example.com/path#`	`http://example.com/path`	`http://example.com/path#`

(The fourth arm from the issue, the empty port http://example.com:/path, was already fixed by #633's port canonicalization and is pinned by the new test.)

Solution

_normalize_validated_url supplies / for an empty path and drops an empty userinfo's @.
New _empty_query_fragment_markers helper: urlparse cannot represent an empty query or fragment (it parses .../path? and .../path identically and geturl() drops the marker), so the markers are computed from the spelling and re-attached as a suffix after each geturl() rebuild — in _normalize_validated_url, in _url_with_encoded_components, and in _url_with_host, which rebuilds percent-decoded-host and canonical-IP URLs before normalization and would otherwise drop the markers before the normalizer sees them (caught by the differential fuzz below). Decision on the markers, as the issue asked: preserve them, matching the Worker byte for byte; a ? inside the fragment stays data.
No verdict changes: which targets are accepted or refused is untouched, and blocked spellings in these forms are still refused by the ordinary checks.

Verification

New Python test test_returns_empty_components_normalized: the issue battery plus percent-decoded (%65xample.com) and IP-literal (8%2e8%2e8%2e8, 134744072) rebuilds, ?-in-fragment data controls, blocked controls, and a redirect-handler pin. New TS test pins the Worker reference forms.
Suites: Python 55/55, TS 38/38. Ablation exact: with only app.py reverted, only the new Python test fails.
Differential vs the real Worker validator under Bun over 43,200 generated spellings (12 hosts × userinfo × port × path × query/fragment tails × scheme): zero differences in this issue's scope. The only remaining differences are pre-existing and out of scope: the deliberate IDN host spelling the container keeps (lab-ops-hub: container validator blanket-refuses percent-encoded hosts the Worker validator decodes and accepts — public spellings (%65xample.com, 8%2e8%2e8%2e8) diverge, including as redirect targets #529/lab-ops-hub: container validator returns dot segments, host case and default ports un-normalized — fetcher requests a different path than the Worker #633; the Worker returns punycode) and the percent-encoded root dot (example.com%2e) tracked separately in lab-ops-hub: container validator drops the DNS root dot only for the percent-encoded spelling — returned URL differs from the Worker and from its own literal-dot form #647 — that issue's spellings are unchanged by this PR.
ruff format applied; diff confined to the validator helpers and the two test files.

Single-repo: the validator pair exists only in this repo (no copy in developing-today/opencode, developing-today/openchamber, or dezren39/nix).

^{Need help on this PR? Tag @codesmith-bot with what you need.}

…normalized (#639) validate_crawl_url returned http://example.com with an empty path, http://@example.com/ with an empty userinfo, and dropped empty ? / # markers, while the Worker validator returns the WHATWG form (http://example.com/, http://example.com/, markers kept), so the two crawlers recorded different URLs for the same target. - _normalize_validated_url now supplies "/" for an empty path and drops an empty userinfo's "@" (the empty port's colon was already dropped by the #633 port canonicalization). - New _empty_query_fragment_markers carries the empty ? / # markers urlparse cannot represent as a suffix re-attached after each geturl() rebuild — in _normalize_validated_url, in _url_with_encoded_components, and in _url_with_host (which rebuilds percent-decoded and canonical-IP URLs before normalization). Decision on the markers: preserve them, matching the Worker byte for byte. - No verdict changes: which targets are accepted or refused is untouched. Tests: new Python battery (empty path/port/userinfo, markers, percent-decoded and IP-literal rebuilds, redirect-handler pin, blocked controls) + TS pin of the Worker reference forms. Python 55/55, TS 38/38; ablation exact (only the new Python test fails with app.py reverted). Differential vs the Worker under Bun over 43,200 generated spellings (host x userinfo x port x path x query/fragment tails x scheme): zero differences in this issue's scope — the only remaining diffs are the deliberate IDN host spelling (#529/#633) and the percent-encoded root dot tracked separately in #647.

chatgpt-codex-connector · 2026-10-23T06:02:08Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T06:06:27.377107Z	`7bd77d0`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coderabbitai · 2026-10-23T06:02:34Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 915e6074-634c-4be6-a32c-1dc5338eb933

📥 Commits

Reviewing files that changed from the base of the PR and between e810ca8 and 7bd77d0.

📒 Files selected for processing: 3

pkgs/lab-ops-hub/container/app.py
pkgs/lab-ops-hub/test/lib.test.ts
pkgs/lab-ops-hub/test/test_container_validate.py

+-------------------------------------------------------------------------------------------------------------------+
| Work with a user to think like a user. It's the best way to gain insight into how the system will really be used. |
+-------------------------------------------------------------------------------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7bd77d0937

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-23T06:06:24Z

+        path=resolved_path,
        params="",
-    ).geturl()
+    ).geturl() + _empty_query_fragment_markers(url)


Insert the empty query marker before the fragment

When an accepted URL has an empty query followed by a non-empty fragment, such as http://example.com/path?#frag, geturl() first produces http://example.com/path#frag and this suffix append then returns http://example.com/path#frag?. The ? has consequently become fragment data instead of preserving the empty query, diverging from the Worker/WHATWG result http://example.com/path?#frag; the marker must be inserted before the fragment rather than appended to the whole serialization.

Useful? React with 👍 / 👎.

dezren39 requested a review from a team as a code owner October 23, 2026 06:02

github-actions Bot assigned dezren39 Oct 23, 2026

dezren39 merged commit 93065f0 into main Oct 23, 2026
4 of 5 checks passed

dezren39 deleted the sweep/639-empty-components branch October 23, 2026 06:03

chatgpt-codex-connector Bot reviewed Oct 23, 2026

View reviewed changes

This was referenced Oct 23, 2026

lab-ops-hub: container validator appends the empty-query marker after the fragment — path?#frag returns as path#frag?, differing from the Worker #654

Closed

fix(lab-ops-hub): keep the empty-query marker before the fragment (#654) #663

Merged

Navigation Menu

Repository navigation

fix(lab-ops-hub): return empty path, userinfo and ?/# markers WHATWG-normalized (#639) - #648

fix(lab-ops-hub): return empty path, userinfo and ?/# markers WHATWG-normalized (#639)#648
dezren39 merged 1 commit into
mainfrom
sweep/639-empty-components

dezren39 commented Oct 23, 2026 •

edited by blacksmith-sh Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 23, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dezren39 commented Oct 23, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Verification

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

coderabbitai Bot commented Oct 23, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dezren39 commented Oct 23, 2026 •

edited by blacksmith-sh Bot

Loading

chatgpt-codex-connector Bot commented Oct 23, 2026 •

edited

Loading