Repository navigation
Reduce SYCL-TLA Compiler Memory - #2193
Merged
Merged
Conversation
Collaborator
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines could not run because the pipeline triggers exclude this branch/path. |
luoyu-intel
requested review from
Zhenzhong1 and
a32543254
and
a lite review from Copilot
August 20, 2026 07:12
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restructures ARK’s SYCL-TLA build so concrete template instantiations are compiled in CMake-generated translation units rather than repeatedly through shared wrapper TUs, reducing peak compiler RSS under oneAPI 2026.0 while keeping Python-facing APIs and runtime routing intact.
Changes:
- Split Dense GEMM, S8 GEMM, MoE (gemm/decode/prefill), and SDPA kernel instantiations into generated
.cppfamilies and keep wrapper headers largely declaration-only. - Adjust ARK build/job scheduling (including
ARK_SYCL_TLA_JOBSvalidation) and add/ document static + empirical compiler-memory analysis tools (EN + CN). - Refine SDPA dispatch/instantiation axes (causal/mask/varlen variants) and add/organize helper headers for the new compilation model.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py | New tool to run compile commands serially under GNU time and record peak RSS + analysis metadata. |
| auto_round_extension/ark/tools/analyze_sycl_tla_templates.py | New static analyzer for reachable template declarations and dispatch call sites per TU. |
| auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py | Skip mixed BestLA SDPA tests when amx_bf16 CPU flag is unavailable. |
| auto_round_extension/ark/setup.py | Add cgroup-aware memory detection + affinity-aware CPU count; tighten ARK_SYCL_TLA_JOBS; change SYCL-TLA job sizing to ~3 GiB/job and improve build logging. |
| auto_round_extension/ark/README.md | Document build-memory controls and measurement workflow. |
| auto_round_extension/ark/README_CN.md | Add Chinese documentation counterpart including the new build-memory guidance. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp | Switch SYCL-TLA include to common declarations header to avoid pulling heavy implementations. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp | Make mask/varlen compile-time knobs for SDPA kernel families to reduce template graphs per TU. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp | Move toward common-declarations include and introduce fixed-tile helper entrypoints. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp | New declaration-only helper header for S8 fixed-tile routes + argument validation. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp | New macro helpers to define fixed-tile S8 route entrypoints in generated sources. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp | Remove header-only public MoE GEMM implementation (moved to generated sources). |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp | New header factoring INT4/INT2 → INT8 upcast kernels for DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp | New declaration-only helper header for direct S4 DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp | Add policy-dispatch wrapper for direct packed-S4 DPAS prefill. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp | New declaration-only helper header for INT8 DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp | New declaration-only helper header for per-tensor INT8 DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp | Add policy-dispatch wrappers for INT8 DPAS per-tensor/per-group prefill. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp | New declaration-only helper header for per-tensor FP8 DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp | New declaration-only helper header for per-group FP8 DPAS prefill routing. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp | Add policy-dispatch wrappers for FP8 DPAS per-tensor/per-group prefill. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp | Remove heavy header-only prefill routing logic; rely on common declarations + split implementations. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp | Remove heavy header-only decode routing logic; rely on common declarations + split implementations. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp | Add fixed-tile dense GEMM helper + move public entrypoint to generated sources. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp | New declaration-only helper header for dense GEMM fixed-tile routes. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp | Centralize lightweight declarations/parameter structs for MoE + GEMM + SDPA entrypoints. |
| auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp | Switch SYCL-TLA include to common declarations header. |
| auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in | New CMake-generated source template splitting MoE wrappers + dispatch + concrete instantiations. |
| auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake | New generator script producing Dense/S8/MoE SYCL-TLA implementation sources. |
| auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in | New CMake-generated source template for Dense + S8 GEMM dispatch + instantiation families. |
| auto_round_extension/ark/auto_round_kernel/sdpa.cpp | Update SDPA launcher selection to incorporate causal/mask/varlen variant routing. |
| auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in | Update SDPA instantiation template to match new compile-time variant parameters. |
| auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake | Expand SDPA generated TU matrix across causal/mask/varlen variants. |
| auto_round_extension/ark/auto_round_kernel/CMakeLists.txt | Wire SYCL-TLA generated sources into the module build. |
| auto_round_extension/ark/auto_round_kernel/ark.cpp | Reduce wrapper includes to declaration-only common header for SYCL-TLA paths. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Zhenzhong1
approved these changes
Aug 20, 2026
chensuyue
added a commit
that referenced
this pull request
Aug 22, 2026
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <suyue.chen@intel.com>
4 tasks done
yiliu30
added a commit
that referenced
this pull request
Sep 20, 2026
Brings in PR #2193 (Reduce SYCL-TLA Compiler Memory) as the prerequisite for extending the same instantiation-splitting to the sparse attention kernels. Co-Authored-By: Claude Code <noreply@anthropic.com>
yiliu30
added a commit
that referenced
this pull request
Sep 20, 2026
Merging main (PR #2193, "Reduce SYCL-TLA Compiler Memory") left the tree unbuildable: - ark.cpp failed with "use of undeclared identifier 'FlashAttnDtype'": the sparse binding code relied on sycl_tla_sdpa.hpp, which #2193 dropped from ark.cpp's includes to stop pulling heavy kernels into dispatch-only units. - Including sycl_tla_common.hpp from the heavy SDPA headers instead broke sdpa.cpp with "redefinition of default argument", because common's sdpa_impl(..., float* lse = nullptr) declaration collides with the definition in sdpa.cpp. FlashAttnDtype was defined identically in sycl_tla_sdpa_sparse.hpp, sycl_tla_sdpa.hpp and sycl_tla_wrapper.hpp. Move it to a new dependency-free flash_attn_dtype.hpp and include that from sycl_tla_common.hpp and the three headers, so dispatch-only translation units keep compiling without pulling in kernel headers. Co-Authored-By: Claude Code <noreply@anthropic.com>
yiliu30
added a commit
that referenced
this pull request
Sep 20, 2026
The sparse attention families (INT8 qks8 "sage" and native BF16/FP16 "sdpa") instantiated every causal/mask combination of every launcher inside a single translation unit. Measured peak compiler RSS was 18.3 GiB for sdpa_sparse.cpp and 17.9 GiB for sdpa_sparse_sdpa.cpp -- roughly 10x the worst dense SDPA source that PR #2193 ("Reduce SYCL-TLA Compiler Memory") brought under 2 GiB. Apply the same treatment here: - Promote causal state and mask presence from runtime options to template parameters on SparseSageConfig/SparseSDPAConfig and on the sparse launchers, so each instantiation compiles exactly one kernel. Cached KV was also a runtime branch; nothing in the sparse path sets it, so it becomes a compile time false with an explicit error if a caller asks for cached decode rather than silently taking a different kernel. - Generate one translation unit per (family, shape, dtype, causal, mask) via sdpa_generation.cmake, mirroring the dense generator, and select among the generated launchers at runtime the way sdpa.cpp does for dense SDPA. Measured with tools/measure_sycl_tla_compile_memory.py (serial, oneAPI 2026.0): sdpa_sparse.cpp 18759.3 -> 511.3 MiB sdpa_sparse_sdpa.cpp 18285.3 -> 508.5 MiB worst generated sparse TU 1887.6 MiB direct dispatches per TU 6 -> 1 The shared module also shrank from 22.8 MB to 20.8 MB. Kernel behaviour is unchanged: the sparse benchmark at seq 32768 reproduces pre-split timings to within 0.5%. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce SYCL-TLA Compiler Memory
Description
This PR reduces peak compiler memory for SYCL-TLA translation units under oneAPI 2026.0. The change moves concrete dense GEMM, S8 GEMM, MoE, and SDPA template instantiations out of shared wrapper translation units and into CMake-generated implementation sources.
The public Python-facing APIs remain unchanged. Runtime routing is preserved through lightweight declaration-only interfaces, while concrete template definitions are compiled only by the generated source families that need them.
Motivation
The previous include and instantiation structure caused large SYCL-TLA template graphs to be parsed and instantiated repeatedly. Representative translation units reached approximately 4 GiB peak RSS, making parallel builds unreliable on constrained build hosts.
The target is to keep representative per-translation-unit compiler memory below approximately 2 GiB while retaining the existing kernel routes and runtime behavior.
Main Changes
ark.cppand lightweight wrappers do not include the full implementation headers.ARK_MOE_PREFILL_DPAS_S4, with fallback to the existing routes when disabled or unsupported.ARK_SYCL_TLA_JOBSvalues.Validation
cmake --build auto_round_extension/ark/xbuild -j 2.setup.pyand both new analysis tools.git diff --checkpassed.Runtime Validation Limitation
The current validation host has no XPU device (
torch.xpu.is_available() == False, device count0). The focused MoE prefill accuracy suite therefore skipped all 36 cases. Direct packed-S4 numerical validation, including nibble ordering,cutlass::int4b_tinterpretation, DPAS results, and scale folding, still needs to be run on a BMG/XPU host with both settings:The existing fallback path remains available for unsupported shapes and for environments where the direct S4 route is disabled.
Related Issues
Not specified.
Checklist