[RFC] Roadmap to v1.9: xmux-style Stream Splitting, Temporal Jitter, and Resilience against Active Throttling #369

Activity

1. therealaleph commented on Apr 28, 2026

   

   therealaleph

   on Apr 28, 2026

   Owner

   More actions

   @w0l4i — substantive RFC, working through each piece. Ranking by impact/feasibility for the Iran-RST scenario:

   1. xmux stream splitting — accept, v1.9.x scope

   Agreed this is the highest-leverage feature for the "ISP RST-injects 30% of connections" scenario from #313. The current "one TCP stream = one deployment" model means a single bad deployment kills any session bound to it; splitting across 5–10 deployments means we degrade gracefully instead of failing hard.

   Implementation sketch:

   Client side (mhrv-rs Rust):

   • Each TCP session gets a logical stream ID + sequence numbers per byte range
   • Outbound bytes split into ~64 KB chunks; each chunk routed to a different deployment (round-robin with health-weighting)
   • Chunks tagged {stream_id, seq, offset, len} in the batch op
   • ACK + retransmit on chunk-level (not stream-level) — if deployment-3 RSTs, only the chunks routed there need retransmit, not the whole stream

   Server side (tunnel-node):

   • Receives chunks from any deployment carrying any stream ID
   • Reassembly buffer keyed by {client_id, stream_id}, ordered by seq
   • When contiguous prefix is available, write to the destination TCP connection
   • Out-of-order chunks buffered; drop window expires after (say) 30s

   Cost estimate: ~800–1500 LOC + non-trivial test infrastructure. Need:

   • Out-of-order packet arrival simulation tests
   • Late-write / duplicate suppression tests
   • Reassembly memory bounds tests (don't OOM if attacker sends adversarial seq numbers)
   • Health-weighted dispatcher tests
   • Existing tokio::test(start_paused = true) infra (fix(tunnel): per-deployment legacy fallback with auto-recovery #290) helps with timing tests

   Compatibility: breaking protocol change between client and tunnel-node. Need either:

   • Version negotiation in the auth handshake — client says "I speak xmux", server confirms
   • Both forms in parallel for one or two releases

   I'd want to ship this with a xmux: false config default for one release before flipping it on, so users can opt in + we can collect real-world failure modes before making it the default.

   Targeting v1.9.0 as the headline feature. Roughly 2–4 weeks of implementation + 1–2 weeks of testing + iteration before merge.

   2. Temporal jitter — accept, v1.8.x scope

   Agreed, packet inter-arrival timing is a real fingerprinting vector once length distribution is randomized (which v1.8.0 just did). Cheap to add:

   // In tunnel_client::mux_loop, before firing a batch:
   let jitter_ms = thread_rng().gen_range(0..=BATCH_TEMPORAL_JITTER_MS);
   tokio::time::sleep(Duration::from_millis(jitter_ms)).await;

   With BATCH_TEMPORAL_JITTER_MS = 50:

   • Average added latency: ~25 ms (sub-perceptible against Apps Script's ~250–500 ms baseline)
   • Distribution of inter-batch intervals widens from "tightly clustered around 8 ms BATCH_COALESCE_WINDOW" to "uniformly distributed 8–58 ms"
   • DPI fingerprints expecting a tight inter-arrival distribution miss

   Concern: if jitter is too high, throughput tanks because batches starve. 50 ms ceiling is conservative — even at maximum jitter, we add half of one Apps Script roundtrip, not multiple roundtrips.

   Concern 2: correlation. If all sessions in a single mhrv-rs process jitter together, the fingerprint stays. Each session needs independent jitter (which my sketch above does — thread_rng gives per-thread independence).

   Targeting v1.8.x point release alongside the Android UI batch. ~30 LOC, days not weeks.

   3. Resilience against active throttling

   This is more strategic than tactical. The core idea — "treat per-deployment health as a first-class signal in routing" — is what #319 (deployment auto-blacklist) already partially does. The roadmap continuation is:

   • Per-deployment RTT EMA in addition to the binary blacklist — slow deployments get less traffic share, not just timeout-failures = excluded
   • Per-deployment success rate window — recent error rate biases the round-robin
   • Adaptive concurrency — back off concurrent batch count to a deployment that's RTT-degrading even before it fully times out

   These are smaller individually than xmux but architecturally compatible — they all live in the next_script_id() / dispatcher layer.

   Targeting v1.8.x or v1.9.x depending on sequencing. If I do these before xmux, the dispatcher gets richer, then xmux plugs into the richer dispatcher cleanly. If I do xmux first, the dispatcher needs a partial rewrite. Inclined to do small dispatcher improvements first.

   Summary table

   Feature	Verdict	Target	Effort
   Temporal jitter (50 ms uniform)	Ship	v1.8.x point	days
   Per-deployment RTT EMA + adaptive routing	Ship	v1.8.x	week
   xmux stream splitting	Ship	v1.9.0	4–6 weeks
   Adaptive concurrency back-off	Ship	v1.9.0	piece of xmux

   Adding to roadmap explicitly. xmux gets its own tracking issue once I start design work.

   If you want to take temporal jitter as a small PR (~30 LOC + 1 test), happy to review same-day. xmux is too architecturally intertwined for a drive-by PR — better as a coordinated design discussion.

   Thanks for the careful threat model — the v1.8.0 random padding came from your previous RFC and the next round of hardening is going to come from this one.

   ---

   _{[reply via Anthropic Claude | reviewed by @therealaleph]}

   👍React with 👍1Reacted by W0L4I
2. therealaleph

   added

   enhancementNew feature or requestNew feature or request

   on Apr 28, 2026
3. 2bemoji commented on Apr 28, 2026

   

   2bemoji

   on Apr 28, 2026

   More actions

   This RFC identifies the most critical bottleneck in the current architecture: The fragility of the single-stream mapping. While v1.8’s padding addressed what the traffic looks like, these v1.9 proposals address how the traffic behaves and where it flows, which is the next frontier in bypassing state-level DPI.

   1. The Necessity of xmux (Stream Splitting) for Session Persistence
      The proposal for "True xmux" is a significant step toward Active Fault Tolerance.

   • Mitigating RST-Injection: In environments using Out-of-Band (OOB) TCP resets, the ISP doesn't need to block Google; they only need to identify one "suspicious" flow to kill the session. By distributing chunks across $N$ workers, you transform a binary "Success/Failure" state into a graceful degradation model.
   • Throughput Aggregation: Beyond security, xmux naturally bypasses per-flow rate-limiting. If an ISP caps a single TLS connection to 2Mbps, a pool of 5 workers theoretically restores a usable 10Mbps experience.

   2. Temporal Jitter vs. Packet Timing Attacks
      Static timing is the "silent killer" of encrypted tunnels. Advanced DPI (like the GFW or sophisticated corporate middleboxes) uses Inter-Arrival Time (IAT) histograms to identify tunnels even when the payload is fully encrypted.

   • Implementing a $\pm 25%$ jitter on the BATCH_COALESCE_WINDOW breaks the regularity that machine-learning classifiers look for.
   • By moving from a "Heartbeat" model to a "Stochastic" model, the traffic signature shifts from "Automated Proxy" to "Organic Web Traffic" (e.g., dynamic API calls or messy AJAX polling).

   3. Strategic "Light xmux" and State Machines
      The "Light xmux" approach for payloads $> 1200$ bytes is a pragmatic middle ground.

   • It avoids the overhead of fragmenting small ACK/handshake packets—which are sensitive to reordering—while effectively hiding the "Bulk Data" signature that triggers most active throttling.

   • The Hot/Cold State Machine is a clever solution to the Google Apps Script (GAS) concurrency limits. It ensures the 30-request quota is spent where it matters most: reducing the "Time to First Byte" during active usage.

     The transition from "Bypass" to "Invisibility" requires exactly this kind of shift toward Statistical Survivability. If one worker dies, the session must live. If the timing is analyzed, it must appear random. I fully support this roadmap and would be interested in seeing the performance benchmarks for the "Hot State" pre-fetching logic.

   ---

   Quick Question for the Devs: For the xmux reassembly layer, are we considering a sequence-number-based buffer at the tunnel-node to handle potential out-of-order delivery from the different Google workers?
   @w0l4i @therealaleph

   👍React with 👍1Reacted by W0L4I
4. therealaleph commented on Apr 28, 2026

   

   therealaleph

   on Apr 28, 2026

   Owner

   More actions

   @2bemoji — accurate framing. The "session persistence" angle on xmux is the right one — current architecture is single-stream-per-deployment, so any per-flow throttle or RST kills the entire session. Splitting across deployments at the byte-range level distributes the failure surface.

   For v1.9.0 design, the four sub-decisions to settle:

   1. Chunk size — too small (~4 KB) wastes Apps Script per-call overhead; too large (~256 KB) reduces failure-spread granularity. Sweet spot is probably ~64 KB chunks based on Apps Script's per-call latency floor (~250-500ms = chunk should take that long to be balanced).

   2. Reassembly buffer bound — adversarial sequence numbers could OOM the tunnel-node if not capped. Bound: per-stream-id, per-client-id, max 8 MB out-of-order buffer + reject anything that would push past it. Stream is then implicitly capped at 8 MB of in-flight unreordered.

   3. ACK granularity — chunk-level acks let the client retransmit only the failed chunks (which is the win), but require client-side retransmit logic. Stream-level acks are simpler but lose the "split across deployments" benefit on retransmit.

   4. Backwards-compat negotiation — at the auth handshake, client signals "I speak xmux v1" and tunnel-node responds "I speak xmux v1 / fallback to single-stream". Both binaries need the protocol-version exchange before the protocol switch happens. Ship in v1.8.x point release first (just the negotiation), then v1.9.0 turns on xmux when both sides have it.

   I'll start the design issue when I close out the v1.8.x batch (DNS, Android UI, googlevideo_ip, in-app updater). Tagging you + @w0l4i + @ipvsami as core reviewers — you've collectively contributed the most signal on the bottlenecks this is meant to fix.

   ETA on the design issue: ~3-4 weeks (after v1.8.x ships). Implementation: ~4-6 weeks after the design freezes. Total to v1.9.0 GA: ~8-12 weeks if everything goes smoothly.

   The "session persistence" framing is what I'll lead the design issue with — better than my earlier "high-survivability stream splitting" framing because it's user-facing.

   ---

   _{[reply via Anthropic Claude | reviewed by @therealaleph]}

   ❤️React with ❤️2Reacted by W0L4I and Sam Ashouri