lab-ops-hub: Worker-fallback crawler follows redirects to non-public targets (no redirect validation, unlike the container) #460

Problem

The lab-ops-hub Worker-fallback crawler does not validate redirect targets, unlike the container crawler — so the #441 SSRF policy can be bypassed on the fallback path.

Evidence (verified on main 718d2674, 2026-10-21)

• pkgs/lab-ops-hub/src/index.ts:345 — the /api/crawl handler validates only the caller-supplied URL: const target = validateCrawlUrl(b.url).
• pkgs/lab-ops-hub/src/index.ts:159-161 — crawlFallback(url) then does a bare fetch with default redirect handling and no re-validation:

  const res = await fetch(url, { headers: { "user-agent": "LabOpsHub/0.1 (+personal crawler)" } });

  Per the Fetch spec, the default redirect mode is follow (up to 20 hops), so a public URL that replies 302 Location: http://169.254.169.254/… (or any private / loopback / link-local / localhost-style target that validateCrawlUrl in src/lib.ts:63 would refuse as a direct target) is fetched, and its body is extracted, returned in part, and archived to R2 (src/index.ts:358-382).
• The caller can also force this path directly with {"via":"worker"} (src/index.ts:353), so the gap does not depend on the container being unavailable.
• Contrast: the container crawler holds redirect targets to the same policy — pkgs/lab-ops-hub/container/app.py:47-50 (_ValidatingRedirectHandler.redirect_request calls validate_crawl_url(newurl)), and the comment at app.py:20-24 states "Redirect targets are held to the same policy."
• pkgs/lab-ops-hub/README.md:46-52 describes both crawlers as refusing non-public targets "including redirect targets in the container" — the Worker fallback is the uncovered half.
• Default-redirect behaviour reproduced locally this run: a spec-following HTTP client requesting a start URL that 302-redirects to a 127.0.0.1 target returned the private target's body (hits: ['/start', '/private']).

This is not the already-closed #441 (initial-URL validation + auth, both present) or #442 (budget).

Why it matters

The crawl endpoint is an authenticated fetch proxy by design; redirect-to-private is the standard SSRF bypass for exactly the initial-URL validation added in #441. The two crawler implementations were deliberately kept to one policy (defence in depth, per the app.py comment) — the fallback silently breaks that invariant.

Suggested fix / acceptance criteria

• crawlFallback follows redirects manually (redirect: "manual"), validating every Location target with validateCrawlUrl before fetching it, with a bounded hop count (e.g. 5); a redirect to a non-public or non-http(s) target fails the crawl with a clear error and nothing is archived to R2.
• Relative Location headers resolve against the current URL before validation.
• A regression test covers: public URL → redirect to http://127.0.0.1/… / http://169.254.169.254/… is refused on the worker path; a public → public redirect still succeeds. (The redirect-following logic may need to move into src/lib.ts alongside validateCrawlUrl so it is testable with plain Node like the existing test/lib.test.ts tests.)
• README crawl-target wording updated so it no longer implies redirect validation is container-only.

Single-repo: lab-ops-hub exists only in developing-today/code.