perf: move streaming command processing off async workers

[Mohamed Mansour (mohamedmansour)]

Summary

• Move decoded start/resume state into the streaming continuation instead of borrowing and cloning its retained projection.
• Keep async ingestion limited to framing and record-size enforcement. Transfer owned record bytes through the existing capacity-one channel.
• Deserialize JSON, validate command order, and inject state defaults on the response's existing blocking renderer. No per-command worker or new public API is added.
• Preserve initial HTTP 502 command errors versus HTTP 500 rendering errors, precommit staging, cancellation, truncation, limits, and ordering. Add focused regressions and update the specification and ownership guidance.

*No streaming chunk-size, pool-retention, pool-sizing, watcher, or dependency changes are included.

Recorded before/after evidence

These measurements already isolated exactly the combined CLI ownership/worker-processing change, with the original transport and pool sizing in both variants. The split retains identical production code; its test-helper pool capacity also matches the measured original CHUNK_TARGET + 1024.

The recorded A/B baseline was f43db65067d48581ed7f99acc4b8d68f201c8bb9. This PR is based on newer main 2dab6fc6; the measurements were not rerun against that newer base. The intervening main changes did not modify this CLI module or the streaming writer. Ownership and parsing-placement effects were not independently ablated.

Method

Intel Core Ultra 9 285K; Linux x86_64 WSL2 6.6.114.1-microsoft-standard-WSL2; Rust/Cargo 1.98.0; release/thin-LTO. Three sequential before/after pairs pinned to CPUs 8 and 9. Each run/case used 40 warmups plus 180 latency responses: 540 measured responses per variant/case.

The harness measures actual private render() through to_bytes(), with prebuilt NDJSON delivered as 16 KiB in-memory backend chunks. It is not TCP or whole-server throughput. Start/resume retain a large rows subtree, and resume changes it. Components projection keeps output small to isolate processing.

State size per record	Two-record input bytes	Output bytes
Small	471	487
Approximately 64 KiB	131,255	487
Approximately 1 MiB	2,097,303	487

Every individual record stays below 2,000,000 bytes. All 487-byte bodies/checkpoints matched across cases, variants, warmups, and timed responses. Setup serialization, configuration, chunk clones, byte comparison, and CSV writing were outside the timing window.

Latency and CPU

All values are microseconds, median / P95, pooled across three runs. P50 uses the sample median; P95 uses sorted index floor((n - 1) * 0.95). CPU uses Linux process/current-thread CPU clocks.

State	Metric	Before	After
Small	Wall	84.056 / 180.574	76.818 / 131.396
Small	Process CPU	36.426 / 84.898	29.0185 / 60.674
Small	Event-thread CPU	18.4235 / 42.386	10.699 / 23.382
64 KiB	Wall	1468.1425 / 2108.629	996.589 / 1440.722
64 KiB	Process CPU	1404.395 / 1980.192	914.825 / 1319.581
64 KiB	Event-thread CPU	853.834 / 1275.773	41.839 / 81.282
1 MiB	Wall	39601.5925 / 48493.216	21582.768 / 26279.345
1 MiB	Process CPU	39481.3355 / 48401.554	21478.9435 / 26331.253
1 MiB	Event-thread CPU	11949.8735 / 17647.320	379.340 / 473.388

At 1 MiB, median wall latency falls approximately 45.5% and event-thread CPU approximately 96.8%. Small-response scheduling variance remains visible; these figures do not promise an equivalent server-wide RPS increase.

Separate cooperative ticker measurement

A continuously yielding task on the current thread records the maximum inter-tick gap per response. This phase is separate because spinner CPU must not be used as throughput evidence. It used 40 warmups plus 90 measured responses per run: 270 responses per variant/case.

Values are microseconds, median / P95 of the per-response maximum gap.

State	Before	After
Small	5.044 / 16.024	2.8065 / 5.679
64 KiB	412.1565 / 487.106	10.717 / 63.593
1 MiB	6147.572 / 8683.731	167.349 / 313.909

State	Wall median change: pairs 1 / 2 / 3	Maximum-gap median change: pairs 1 / 2 / 3
Small	-14.38% / -22.48% / -9.88%	-46.20% / -43.72% / -43.00%
64 KiB	-30.98% / -35.93% / -30.95%	-96.57% / -96.97% / -98.13%
1 MiB	-46.09% / -49.43% / -38.66%	-97.14% / -98.04% / -97.00%

Whole-process RSS high-water observations were before 54,372 / 54,136 / 55,336 KiB, after 50,736 / 61,040 / 82,960 KiB. They include setup, warmups, ticker activity, and allocator retention and establish neither a memory improvement nor regression. No allocation-count trace was collected.

Reproduction

cargo test --release -p microsoft-webui-cli --no-run

WEBUI_BENCH_ITERATIONS=180 WEBUI_BENCH_OUTPUT=before-1 \
  taskset -c 8,9 ./before-test measured_cli_render_pipeline --nocapture --test-threads=1
WEBUI_BENCH_ITERATIONS=180 WEBUI_BENCH_OUTPUT=after-1 \
  taskset -c 8,9 ./after-test measured_cli_render_pipeline --nocapture --test-threads=1

Repeat three sequential pairs, pooling raw CSV samples by case and phase. Use separate Cargo targets, or force the CLI rebuild in an isolated shared target: archived same-mtime sources can otherwise reuse the wrong executable.

Validation

The focused cargo test -p microsoft-webui-cli streaming_api selection and complete cargo xtask check passed on this independent branch. The gate included docs, workspace/WASM builds, examples, and benchmark smoke. The full gate completed in 83.7 seconds.

Regression coverage includes raw-byte ownership/backpressure, worker-side default preparation, initial error classification, and worker parse failure while the backend stalls. Public protocol version 2, the 2 MB record cap, and the 4 MB precommit cap are unchanged.

[Copilot]

🔵 Needs a closer look

The streaming execution-boundary changes warrant final human review.

Pull request overview

Moves streaming command parsing and state preparation to the blocking renderer while preserving backpressure, limits, ordering, and error classification.

Changes:

• Transfers raw records through the capacity-one channel.
• Adds focused streaming regressions.
• Updates ownership, performance, and architecture guidance.

File summaries

File	Summary
docs/guide/integrations/rust.md	Updates owned-state streaming guidance.
docs/guide/concepts/performance.md	Adds performance guidance for streaming state.
DESIGN.md	Documents the updated streaming architecture.
crates/webui-cli/src/commands/serve/streaming_api.rs	Moves decoding and state preparation to the renderer and adds regressions.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.