id: design: ID transport fallback for UDP-denied networks v1 #5671

Summary

ID servers should be reachable from networks where UDP is denied. Today ID's native transport assumes UDP (QUIC/iroh-style reachability). Many real networks — corporate egress, locked-down sandboxes, some cloud and CI environments, captive Wi-Fi — allow only TCP, and often only outbound HTTP/HTTPS through a proxy. This design expands #785 (the UDP-denied SSH fallback design) from "SSH still works" into a general transport-fallback feature set for ID itself: any ID traffic, not just SSH, should degrade gracefully to TCP/HTTP transports without giving up end-to-end encryption.

Relates: #785 (extends it — #785 is the SSH-specific case; this is the general ID transport), the serverless/rendezvous direction in #2046 (Cloudflare Workers edition — a Workers rendezvous/relay is one candidate component here).

Motivating case

Our own fleet sandbox: no direct UDP egress; outbound traffic goes through an HTTP CONNECT proxy only. We need, from inside that sandbox:

• Access to ID servers (data, queries, sync) with no UDP path available.
• SSH-class reachability through the same constrained egress — ideally locally terminated, so tools on the sandbox side see a local endpoint rather than a fragile chain of tunnels.
• The same pattern the fleet already uses for sandbox→laptop proxying, generalized so it is a property of ID, not a one-off fleet workaround.

If ID works from our sandbox, it works from most restricted networks.

Feature set (expanded)

1. Fallback ordering and transport negotiation

• Defined attempt order: direct UDP → relay-assisted UDP (hole-punch / relay) → TCP → HTTP/HTTPS proxy (CONNECT, and where necessary plain HTTP/WebSocket-style framing).
• Negotiation is automatic by default, with per-peer and per-network memory: remember which transport last worked for a peer/network and try it first next time, re-probing periodically.
• Explicit override surface: config and CLI flags to pin a transport, forbid a transport, or force proxy use (for testing and for networks where probing is noisy/slow).
• Failure classification: distinguish "UDP blocked" from "peer offline" from "proxy auth required" so error messages and retries are correct, and so fallback does not mask real outages.
• Connection migration: when a better transport becomes available (e.g. the client moves to an unrestricted network), upgrade in place where the protocol allows, without dropping sessions.

2. Broker / relay role

• A broker that proxies ID traffic over HTTP/TCP for clients that cannot use UDP: clients connect out to the broker over TCP/443 (directly or via their HTTP proxy); the broker forwards to the target ID server (or to a rendezvous point the server also connects out to, for servers themselves behind UDP-denied or NAT-only egress).
• Ideally native to ID itself: a serve/proxy role inside the id binary (id serve-adjacent mode) so any ID node can act as a broker for peers it is authorized to help — no separate daemon to deploy.
• Fleet-side surface (future): a fleet id subcommand that stands up/configures this broker role for fleet machines, mirroring the existing fleet sandbox→laptop proxy pattern.
• Broker is a dumb pipe by design: it routes opaque, end-to-end-encrypted traffic (see trust model). It must not need, hold, or be able to derive session/content keys.

3. Discovery and rendezvous

• How a UDP-denied client finds its server and a usable broker: rendezvous records (signed, verifiable) advertising a server's available transports, broker endpoints, and priority.
• Candidate rendezvous substrates to evaluate, with a recommendation: DNS-based records, a well-known HTTPS endpoint, and a Cloudflare Workers rendezvous/relay component (cheap, globally reachable over 443, no server to run) — including when Workers is the right shape vs. when a self-hosted id broker is required (sovereignty, cost, data-residency).
• Rendezvous must itself be reachable over plain HTTPS, or the fallback is circular.
• Stale/spoofed rendezvous handling: signature verification, TTLs, and fallback to last-known-good endpoints.

4. Relay trust model

• What the broker/relay can see: connection metadata (endpoints, timing, volume) — state this honestly and minimize it (no logging defaults, padding/batching options if cheap).
• What it cannot see: payload contents. ID payloads stay end-to-end encrypted between the true endpoints; the broker forwards ciphertext. SSH through the broker likewise stays end-to-end (broker never terminates SSH).
• Broker authorization: who may use a given broker (open, allowlist by identity/key, capability token), and who a broker will forward for — an open relay is an abuse magnet; default to authenticated.
• Compromised-broker analysis: worst case is metadata exposure + denial of service, never content exposure; design must keep it that way (no key escrow, no TLS termination of ID payloads at the broker).
• Local termination option: where the client side runs a local id proxy endpoint, applications (including SSH) connect to localhost and the encryption boundary is the local process — document exactly where plaintext exists in each topology.

5. Latency and performance budget

• Stated budgets per fallback tier (e.g. interactive SSH and ID queries must stay usable on the HTTP-proxy path; bulk sync may be slower but must complete).
• Overhead sources to measure: proxy CONNECT setup, TLS-in-TLS, framing, broker hop latency, head-of-line blocking on a single TCP stream (and whether multiple streams / multiplexing are needed).
• Keepalive and idle-timeout behavior for long-lived connections through proxies that kill idle TCP.
• Reconnection/resumption: dropped proxy connections resume sessions rather than restarting sync from zero.

6. What stays end-to-end encrypted (explicit list)

• ID data/sync payloads: E2E, broker-blind.
• ID queries and results (see the ID query design): E2E.
• SSH sessions tunneled through the fallback: E2E, broker is a pipe.
• Control/rendezvous traffic: authenticated and integrity-protected; confidentiality requirements stated per message type.
• Anything that is not E2E (if any) must be named in the design with justification — silence is not acceptable here.

Non-goals (v1)

• Defeating deliberate, policy-enforced egress restrictions the user is not authorized to bypass — this is for reaching your own ID servers from restrictive networks, with the broker as an authorized endpoint.
• A public, open relay network.
• Replacing the native UDP transport; fallback is additive.

Acceptance for this design

A follow-on reader can break this into the decomposition tickets below without re-deciding: fallback order, broker role shape (in-id vs Workers vs both), trust model, or the E2E boundary.

Decomposition sketch

1. Design: transport negotiation & fallback ordering (probing, per-peer memory, overrides, failure classification).
2. Design: id broker/serve-proxy role — protocol, framing over TCP/HTTP(S), multiplexing, keepalive/resumption.
3. Design: rendezvous & discovery for UDP-denied clients (signed transport advertisements; substrate selection).
4. Design: Cloudflare Workers rendezvous/relay component — when it is the right shape, limits, self-host parity.
5. Design: relay trust model & broker authorization (capabilities/allowlist, metadata minimization, compromised-broker analysis).
6. Design: HTTP CONNECT proxy traversal (proxy auth, corporate proxies, locally terminated endpoint for SSH-class traffic).
7. Design: latency budget & measurement plan per fallback tier (interactive vs bulk).
8. Design: fleet surface — future fleet id subcommand wiring the broker role into fleet sandbox→laptop patterns.

Open questions

• Broker in id itself, a Workers component, or both — and which ships first?
• What rendezvous substrate is authoritative when several disagree?
• How are brokers discovered by strangers vs. only by pre-configured peers (sovereignty vs. convenience)?
• Proxy-auth credentials: where stored, and how are they kept out of logs/config dumps?
• Is multiplexing over one TCP connection sufficient, or is head-of-line blocking disqualifying for mixed interactive+bulk traffic?
• Cost/ownership model for a hosted relay (who runs it, who pays, what are the abuse controls)?
• How does fallback interact with id: Phase 13 F13.3 SSH-over-iroh reachability (and iroh-over-SSH fallback) #785's SSH-specific design — supersede, wrap, or coexist?