Skip to content

perf(moe): vectorize the hybrid expert ensure kernel - #907

Open
sword-k-w wants to merge 1 commit into
FlashML-org:mainfrom
sword-k-w:perf/moe-ensure-vectorize
Open

sword-k-w wants to merge 1 commit into
FlashML-org:mainfrom
sword-k-w:perf/moe-ensure-vectorize

Conversation

@sword-k-w

@sword-k-w sword-k-w commented Nov 14, 2026 •

Copy link
Copy Markdown

The hybrid decode guide (#859) calls out the per-layer CPU round trip in hybrid decode: "Every MoE layer makes a CPU round trip, about 6 us here (0.25 ms per step)." This PR reduces the GPU-side decision part: _ensure_experts_hybrid_kernel.

For Qwen3.6-35B-A3B (top_k = 8, MTP k = 3, four requests), one layer sees 4 x (3 + 1) x 8 = 128 routes. In the c4/5120/MTP trace, ensure takes 22.24 us/layer, 4.85% of the all-graph span; the old per-route scalar loop plus tl.debug_barrier() dominates. This PR cuts it to 5.07 us, 1.27%.

Part of #671. Related: #859, #778, #674, #491.

Changes

Three behavior-preserving changes:

  • Phase 3, route rewrite. Old: per-route load of expert_ids, slot load, tl.debug_barrier(), then stores. New: Phase 3 reuses the e_vec loaded in Phase 1, loads each lane's slot and weight, and performs the three stores; expert_ids is not read after the stores, so the per-route barrier goes away. BLOCK_A = next_power_of_2(max(1, num_active)); excess lanes are masked.
  • Phase 1, active mask. One vector load plus tl.histogram(e_vec, BLOCK_E, mask=a_mask) > 0 replaces the per-route scans; duplicate routes still count as active.
  • Phase 2, fetch selection. The fetch-priority score (rec or missing_rank) is computed only inside if num_fetch > 0; a no-fetch step skips the 256-wide recency load. The recency update is unchanged.

For the same cache state and routes, the LRU policy, victim selection, statistics and per-layer routes match an independent CPU reference; the only intended difference is kernel time.

Kernel performance

Local CUDA-graph microbenchmark of one call (script not in this PR; no model loaded): 40 layers, 256 experts, top_k = 8, 16 rows = 128 routes, synthetic power-law popularity, prefer-fetch; 3 fresh processes, each 4000 warmup + 300 replays, with the mutable cache state restored before each replay. Times include replay overhead and are for relative comparison; no L2 flush. main = 27740fec, RTX PRO 6000 (sm_120).

cache slots main this PR change (speedup)
2560 (25%) 30.6 us 14.0 us -54% (2.2x)
5120 (50%) 23.1 us 8.4 us -64% (2.8x)
7680 (75%) 24.1 us 8.5 us -65% (2.8x)
10240 (100%) 24.3 us 8.3 us -66% (2.9x)

10240 slots is one slot per (layer, expert), so the sweep is 25/50/75/100% capacity. The 2560 run has about 39 misses / 11 fetches per call; 5120+ has 3/1, so its fetch path is a larger share and the relative gain is smaller.

Same-load c4/5120/MTP trace (one 73-step + three 78-step requests; 307 per-request verify steps; 3120 ensure calls):

main this PR
ensure mean 22.24 us 5.07 us
share of all-graph span 4.85% 1.27%
ensure total 69.38 ms 15.81 ms

Single trace; the graph span also moved (1429.7 -> 1247.9 ms), but the controlled saving is the ensure total.

End-to-end decode performance

Official benchmarks/bench_decode_moe.py as-is (hybrid, greedy, 256 decode tokens), cache rate 25/50/75/100%, 2-5 fresh servers per side: no reproducible regression. Main and this PR produce the same output sha1 at every cache rate.
This is batch-size-1 (bs1) without MTP; it is a separate check from the c4/MTP numbers below.

c4/MTP spot check: four concurrent requests with the same AIME25 problem 0 prompt, MTP k = 3, greedy, ignore_eos, 256 decode tokens, thinking on; main = 27740fec. A fresh server is a new ft serve process with empty MoE and prefix
caches; its four warmup requests warm them before the measured batch. The six samples per side are independent server starts, not six servers running concurrently. Six fresh servers per side in two order-balanced passes: main -> PR with cache ascending (25/50/75/100%), then PR -> main with cache descending (100/75/50/25%), three servers per pass; each server runs four warmup and four measured c4 requests, released together from a client barrier.
Table values are medians over the six servers.

cache batch tok/s, main -> PR ms/verify-step, main -> PR
25% 436.8 -> 446.4 (+2.2%) 29.6 -> 29.9 (+1.0%)
50% 667.4 -> 672.9 (+0.8%) 19.5 -> 19.2 (-1.5%)
75% 777.3 -> 806.0 (+3.7%) 18.0 -> 17.2 (-4.0%)
100% 771.2 -> 806.3 (+4.6%) 18.1 -> 17.3 (-4.3%)

Higher tok/s and lower ms/verify-step are better. batch tok/s is total output-token intervals across the four requests
(4 x 255 = 1020) divided by the decode window. ms/verify-step is that window divided by the slowest request's verify steps, i.e. the maximum verify count among the four requests: the batch runs until the last request finishes. A verify step is one target forward over the current token plus k draft tokens (up to k + 1 = 4 committed tokens); higher MTP acceptance commits more tokens per step and reduces the number of steps (verify_steps_from_draft_n is draft_n / k). The table reports both because tok/s is user-facing but acceptance-confounded, while per-step time is the kernel-related cost. At 25%, tok/s is +2.2% but per-step is +1.0% slower; the six same-mode pairs are mixed, so that row is within noise.

Same-pass, same-output/verify pairs compare a main and a PR batch with the same multiset of output hashes and verify counts, which fixes the scheduler and acceptance mode. At 75-100% cache, those pairs are +2.7% to +5.2% tok/s and -2.6% to -4.9% per-step. At 50% there is one such pair (+4.4% / -4.2%); the table median over all six servers is +0.8% / -1.5%. At 25%, six pairs have median -0.2% / +0.4% and the two passes' medians differ by 5-10% on each side, so we claim neither a gain nor a regression there. An unpaired c4 median is not a strict A/B: admission order changes which requests share a verify step.

The c4 spot check is throughput-only; output hashes and verify counts differ across fresh servers even for the same build, so we do not compare c4 output identity. The controlled same-load trace in Kernel performance has identical measured text, verify counts and padding, and the reference test compares the full cache state and routes.

Both runs use the ft bench bw --cpu-threads 16 profile via FREETOKEN_BENCHBW_PATH. Official bs1, once per cache rate:

FREETOKEN_BENCHBW_PATH=<bw.json> \
taskset -c 52-68 python benchmarks/bench_decode_moe.py \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 --aime <aime25/test.jsonl> \
  --backend hybrid --greedy --decode 256 --cache-rate 0.25 --hybrid-fetch -1

c4/MTP server (5120 shown; sweep used 2560/5120/7680/10240):

FREETOKEN_BENCHBW_PATH=<bw.json> FREETOKEN_HYBRID_OVERLAP=1 \
FREETOKEN_HYBRID_FETCH_ROUND=prefer-fetch \
taskset -c 52-71 python -m freetoken.cli serve \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 --host 127.0.0.1 --port <port> \
  --moe-strategy hybrid --moe-cache-size 5120 --moe-cache-scope unified \
  --moe-cache-split equal --moe-hybrid-max-fetch -1 --moe-cpu-threads 16 \
  --num-pages 8192 --max-running-requests 4 --cuda-graph-max-bs 4 \
  --max-seq-len-override 8256 --memory-ratio 0.9 --kv-cache-dtype auto \
  --attention-backend auto --cache-type radix --speculative-algorithm mtp \
  --speculative-num-steps 3 --enable-timings-report

The client is a local harness (not part of this PR): four streamed /v1/chat/completions requests, AIME25 problem 0, temperature=0, top_p=1, top_k=-1, ignore_eos=true, max_tokens=256, thinking on.

Tests

  • test_hybrid_ensure_matches_lru_reference_wide_batch adds 36 cases (3 shapes x BY_RECENCY 2 x fetch_cap 3 x clamp 2), 54 reference cases total. Shapes: c4/MTP 16x8 = 128 routes, single-request MTP 4x8 = 32, and 13x5 with E=192.
  • Each case runs 16 calls from a cold cache to hits and compares against an independent CPU LRU reference: routes, slot_for_id, id_of_slot, usage, step, expert_recency, active_mask, evict_slots, src_indices, num_indices, num_missing_full; duplicate route ids included.
  • pytest tests/moe -q -m "not slow": 235 passed, 24 skipped, 23 deselected.
  • The same-load trace has identical measured text, verify counts and padding.

Tested on

  • RTX PRO 6000 Blackwell Server Edition 96 GB (sm_120), driver 590.44.01; Xeon Platinum 8470Q.
  • Both experiments use the same 16 CPU MoE workers on 52-67:
    • c4/MTP: taskset -c 52-71; 16 workers + 4 spare, spin_wait=True.
    • Official bs1: taskset -c 52-68; auto-sizing 17 cores -> 16 workers + 1 coordinator on 68, spin_wait=False.
  • torch 2.14.1, Triton 3.8.0, flashlib 0.4.0, flashinfer 0.6.18.post1.
  • Qwen3.6-35B-A3B-NVFP4 @ 1355db6...; main = 27740fec....
  • pytest tests -q -m "not slow": 2594 passed, 304 skipped, 4 failed, 32 deselected in 15m56s. The same four fail on main: the two fa cases on sm_120, the UVA pinned-pointer test, and test_swiglu_clamp.py::test_triton_matches_reference (CUDA invalid argument).

Not covered

  • ROCm and other GPU architectures; no ptxas register/spill audit.
  • A strict c4 A/B with a fixed scheduler batch; six fresh servers per side are a spot check.
  • The full model/checkpoint matrix: only Qwen3.6-35B-A3B-NVFP4; other models, expert counts/top_k, quant formats, TP/EP and MTP step counts were not run.
  • High-concurrency shapes above 512 routes (c16) were not benchmarked; BLOCK_A scales with the route count.

- vectorize the phase-3 route rewrite and remove the per-route barrier
- build the active mask with tl.histogram
- compute the fetch-priority score only when num_fetch > 0

Tests: pytest tests/moe -q -m "not slow" (235 passed, 24 skipped, 23 deselected)

Assisted-by: Claude Code
@github-actions github-actions Bot added the perf label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant