Repository navigation
Conversation
- vectorize the phase-3 route rewrite and remove the per-route barrier - build the active mask with tl.histogram - compute the fetch-priority score only when num_fetch > 0 Tests: pytest tests/moe -q -m "not slow" (235 passed, 24 skipped, 23 deselected) Assisted-by: Claude Code
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The hybrid decode guide (#859) calls out the per-layer CPU round trip in hybrid decode: "Every MoE layer makes a CPU round trip, about 6 us here (0.25 ms per step)." This PR reduces the GPU-side decision part:
_ensure_experts_hybrid_kernel.For Qwen3.6-35B-A3B (top_k = 8, MTP k = 3, four requests), one layer sees 4 x (3 + 1) x 8 = 128 routes. In the c4/5120/MTP trace, ensure takes 22.24 us/layer, 4.85% of the all-graph span; the old per-route scalar loop plus
tl.debug_barrier()dominates. This PR cuts it to 5.07 us, 1.27%.Part of #671. Related: #859, #778, #674, #491.
Changes
Three behavior-preserving changes:
expert_ids, slot load,tl.debug_barrier(), then stores. New: Phase 3 reuses thee_vecloaded in Phase 1, loads each lane's slot and weight, and performs the three stores;expert_idsis not read after the stores, so the per-route barrier goes away.BLOCK_A = next_power_of_2(max(1, num_active)); excess lanes are masked.tl.histogram(e_vec, BLOCK_E, mask=a_mask) > 0replaces the per-route scans; duplicate routes still count as active.recormissing_rank) is computed only insideif num_fetch > 0; a no-fetch step skips the 256-wide recency load. The recency update is unchanged.For the same cache state and routes, the LRU policy, victim selection, statistics and per-layer routes match an independent CPU reference; the only intended difference is kernel time.
Kernel performance
Local CUDA-graph microbenchmark of one call (script not in this PR; no model loaded): 40 layers, 256 experts, top_k = 8, 16 rows = 128 routes, synthetic power-law popularity, prefer-fetch; 3 fresh processes, each 4000 warmup + 300 replays, with the mutable cache state restored before each replay. Times include replay overhead and are for relative comparison; no L2 flush. main =
27740fec, RTX PRO 6000 (sm_120).10240 slots is one slot per (layer, expert), so the sweep is 25/50/75/100% capacity. The 2560 run has about 39 misses / 11 fetches per call; 5120+ has 3/1, so its fetch path is a larger share and the relative gain is smaller.
Same-load c4/5120/MTP trace (one 73-step + three 78-step requests; 307 per-request verify steps; 3120 ensure calls):
Single trace; the graph span also moved (1429.7 -> 1247.9 ms), but the controlled saving is the ensure total.
End-to-end decode performance
Official
benchmarks/bench_decode_moe.pyas-is (hybrid, greedy, 256 decode tokens), cache rate 25/50/75/100%, 2-5 fresh servers per side: no reproducible regression. Main and this PR produce the same output sha1 at every cache rate.This is batch-size-1 (bs1) without MTP; it is a separate check from the c4/MTP numbers below.
c4/MTP spot check: four concurrent requests with the same AIME25 problem 0 prompt, MTP k = 3, greedy,
ignore_eos, 256 decode tokens, thinking on; main =27740fec. A fresh server is a newft serveprocess with empty MoE and prefixcaches; its four warmup requests warm them before the measured batch. The six samples per side are independent server starts, not six servers running concurrently. Six fresh servers per side in two order-balanced passes: main -> PR with cache ascending (25/50/75/100%), then PR -> main with cache descending (100/75/50/25%), three servers per pass; each server runs four warmup and four measured c4 requests, released together from a client barrier.
Table values are medians over the six servers.
Higher
tok/sand lowerms/verify-stepare better.batch tok/sis total output-token intervals across the four requests(4 x 255 = 1020) divided by the decode window.
ms/verify-stepis that window divided by the slowest request's verify steps, i.e. the maximum verify count among the four requests: the batch runs until the last request finishes. A verify step is one target forward over the current token plus k draft tokens (up to k + 1 = 4 committed tokens); higher MTP acceptance commits more tokens per step and reduces the number of steps (verify_steps_from_draft_nisdraft_n / k). The table reports both becausetok/sis user-facing but acceptance-confounded, while per-step time is the kernel-related cost. At 25%,tok/sis +2.2% but per-step is +1.0% slower; the six same-mode pairs are mixed, so that row is within noise.Same-pass, same-output/verify pairs compare a main and a PR batch with the same multiset of output hashes and verify counts, which fixes the scheduler and acceptance mode. At 75-100% cache, those pairs are +2.7% to +5.2%
tok/sand -2.6% to -4.9% per-step. At 50% there is one such pair (+4.4% / -4.2%); the table median over all six servers is +0.8% / -1.5%. At 25%, six pairs have median -0.2% / +0.4% and the two passes' medians differ by 5-10% on each side, so we claim neither a gain nor a regression there. An unpaired c4 median is not a strict A/B: admission order changes which requests share a verify step.The c4 spot check is throughput-only; output hashes and verify counts differ across fresh servers even for the same build, so we do not compare c4 output identity. The controlled same-load trace in Kernel performance has identical measured text, verify counts and padding, and the reference test compares the full cache state and routes.
Both runs use the
ft bench bw --cpu-threads 16profile viaFREETOKEN_BENCHBW_PATH. Official bs1, once per cache rate:c4/MTP server (5120 shown; sweep used 2560/5120/7680/10240):
The client is a local harness (not part of this PR): four streamed
/v1/chat/completionsrequests, AIME25 problem 0,temperature=0,top_p=1,top_k=-1,ignore_eos=true,max_tokens=256, thinking on.Tests
test_hybrid_ensure_matches_lru_reference_wide_batchadds 36 cases (3 shapes xBY_RECENCY2 xfetch_cap3 xclamp2), 54 reference cases total. Shapes: c4/MTP 16x8 = 128 routes, single-request MTP 4x8 = 32, and 13x5 with E=192.slot_for_id,id_of_slot,usage,step,expert_recency,active_mask,evict_slots,src_indices,num_indices,num_missing_full; duplicate route ids included.pytest tests/moe -q -m "not slow": 235 passed, 24 skipped, 23 deselected.Tested on
taskset -c 52-71; 16 workers + 4 spare,spin_wait=True.taskset -c 52-68; auto-sizing 17 cores -> 16 workers + 1 coordinator on 68,spin_wait=False.1355db6...; main =27740fec....pytest tests -q -m "not slow": 2594 passed, 304 skipped, 4 failed, 32 deselected in 15m56s. The same four fail on main: the twofacases on sm_120, the UVA pinned-pointer test, andtest_swiglu_clamp.py::test_triton_matches_reference(CUDA invalid argument).Not covered
BLOCK_Ascales with the route count.