Skip to content

Reduce SYCL-TLA Compiler Memory - #2193

Merged
luoyu-intel merged 8 commits into
mainfrom
fix/sycl-tla-memory-budget
Aug 20, 2026
Merged

luoyu-intel merged 8 commits into
mainfrom
fix/sycl-tla-memory-budget

Conversation

@luoyu-intel

Copy link
Copy Markdown
Contributor

Reduce SYCL-TLA Compiler Memory

Description

This PR reduces peak compiler memory for SYCL-TLA translation units under oneAPI 2026.0. The change moves concrete dense GEMM, S8 GEMM, MoE, and SDPA template instantiations out of shared wrapper translation units and into CMake-generated implementation sources.

The public Python-facing APIs remain unchanged. Runtime routing is preserved through lightweight declaration-only interfaces, while concrete template definitions are compiled only by the generated source families that need them.

Motivation

The previous include and instantiation structure caused large SYCL-TLA template graphs to be parsed and instantiated repeatedly. Representative translation units reached approximately 4 GiB peak RSS, making parallel builds unreliable on constrained build hosts.

The target is to keep representative per-translation-unit compiler memory below approximately 2 GiB while retaining the existing kernel routes and runtime behavior.

Main Changes

  • Generate dedicated SYCL-TLA implementation sources for:
    • Dense GEMM fixed-tile routes and dtype dispatch.
    • S8 GEMM normal and K-block routes.
    • MoE grouped GEMM, decode, and prefill routes.
    • Specialized INT8, FP8, low-bit, tensor, and dequantized MoE prefill paths.
  • Add declaration-only common and helper headers so ark.cpp and lightweight wrappers do not include the full implementation headers.
  • Split SDPA instantiations by dtype, head dimension, causal state, mask state, and varlen state.
  • Preserve runtime validation and dispatch behavior for dense GEMM, S8 GEMM, grouped GEMM, decode, prefill, and SDPA.
  • Add a direct packed symmetric S4 MoE prefill DPAS path, gated by ARK_MOE_PREFILL_DPAS_S4, with fallback to the existing routes when disabled or unsupported.
  • Change the default SYCL-TLA compiler-memory scheduling policy to approximately 3 GiB per compiler job.
  • Reject explicitly set empty, zero, negative, and non-numeric ARK_SYCL_TLA_JOBS values.
  • Add static template-analysis and serial compiler-memory measurement tools.
  • Document the build-memory controls and measurement workflow in English and Chinese.

Validation

  • Configured oneAPI XPU build completed successfully with cmake --build auto_round_extension/ark/xbuild -j 2.
  • Static analysis completed across 193 translation units.
  • The active compile database contains 72 generated SDPA translation units with unique causal, mask, and varlen variants.
  • Measured representative peak RSS:
    • Worst measured SDPA sources: approximately 1,941-1,950 MiB.
    • Direct packed-S4 source: approximately 1,516.6 MiB.
  • Python syntax checks passed for setup.py and both new analysis tools.
  • Both new analysis tools passed their command-line help checks.
  • The rebuilt shared library has no matching unresolved ARK API or generated-kernel symbols.
  • git diff --check passed.

Runtime Validation Limitation

The current validation host has no XPU device (torch.xpu.is_available() == False, device count 0). The focused MoE prefill accuracy suite therefore skipped all 36 cases. Direct packed-S4 numerical validation, including nibble ordering, cutlass::int4b_t interpretation, DPAS results, and scale folding, still needs to be run on a BMG/XPU host with both settings:

ARK_MOE_PREFILL_DPAS_S4=1
ARK_MOE_PREFILL_DPAS_S4=0

The existing fallback path remains available for unsupported shapes and for environments where the direct S4 route is disabled.

Related Issues

Not specified.

Checklist

  • Build and static validation completed locally.
  • Documentation updated, including the Chinese counterpart.
  • New compiler-analysis tools added and checked.
  • XPU runtime accuracy validation completed.
  • XPU direct-S4 performance comparison completed.
  • CUDA CI status checked. This PR targets the ARK SYCL-TLA/XPU backend.

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@a32543254 a32543254 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR restructures ARK’s SYCL-TLA build so concrete template instantiations are compiled in CMake-generated translation units rather than repeatedly through shared wrapper TUs, reducing peak compiler RSS under oneAPI 2026.0 while keeping Python-facing APIs and runtime routing intact.

Changes:

  • Split Dense GEMM, S8 GEMM, MoE (gemm/decode/prefill), and SDPA kernel instantiations into generated .cpp families and keep wrapper headers largely declaration-only.
  • Adjust ARK build/job scheduling (including ARK_SYCL_TLA_JOBS validation) and add/ document static + empirical compiler-memory analysis tools (EN + CN).
  • Refine SDPA dispatch/instantiation axes (causal/mask/varlen variants) and add/organize helper headers for the new compilation model.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py New tool to run compile commands serially under GNU time and record peak RSS + analysis metadata.
auto_round_extension/ark/tools/analyze_sycl_tla_templates.py New static analyzer for reachable template declarations and dispatch call sites per TU.
auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py Skip mixed BestLA SDPA tests when amx_bf16 CPU flag is unavailable.
auto_round_extension/ark/setup.py Add cgroup-aware memory detection + affinity-aware CPU count; tighten ARK_SYCL_TLA_JOBS; change SYCL-TLA job sizing to ~3 GiB/job and improve build logging.
auto_round_extension/ark/README.md Document build-memory controls and measurement workflow.
auto_round_extension/ark/README_CN.md Add Chinese documentation counterpart including the new build-memory guidance.
auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp Switch SYCL-TLA include to common declarations header to avoid pulling heavy implementations.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp Make mask/varlen compile-time knobs for SDPA kernel families to reduce template graphs per TU.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp Move toward common-declarations include and introduce fixed-tile helper entrypoints.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp New declaration-only helper header for S8 fixed-tile routes + argument validation.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp New macro helpers to define fixed-tile S8 route entrypoints in generated sources.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp Remove header-only public MoE GEMM implementation (moved to generated sources).
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp New header factoring INT4/INT2 → INT8 upcast kernels for DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp New declaration-only helper header for direct S4 DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp Add policy-dispatch wrapper for direct packed-S4 DPAS prefill.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp New declaration-only helper header for INT8 DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp New declaration-only helper header for per-tensor INT8 DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp Add policy-dispatch wrappers for INT8 DPAS per-tensor/per-group prefill.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp New declaration-only helper header for per-tensor FP8 DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp New declaration-only helper header for per-group FP8 DPAS prefill routing.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp Add policy-dispatch wrappers for FP8 DPAS per-tensor/per-group prefill.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp Remove heavy header-only prefill routing logic; rely on common declarations + split implementations.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp Remove heavy header-only decode routing logic; rely on common declarations + split implementations.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp Add fixed-tile dense GEMM helper + move public entrypoint to generated sources.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp New declaration-only helper header for dense GEMM fixed-tile routes.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp Centralize lightweight declarations/parameter structs for MoE + GEMM + SDPA entrypoints.
auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp Switch SYCL-TLA include to common declarations header.
auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in New CMake-generated source template splitting MoE wrappers + dispatch + concrete instantiations.
auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake New generator script producing Dense/S8/MoE SYCL-TLA implementation sources.
auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in New CMake-generated source template for Dense + S8 GEMM dispatch + instantiation families.
auto_round_extension/ark/auto_round_kernel/sdpa.cpp Update SDPA launcher selection to incorporate causal/mask/varlen variant routing.
auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in Update SDPA instantiation template to match new compile-time variant parameters.
auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake Expand SDPA generated TU matrix across causal/mask/varlen variants.
auto_round_extension/ark/auto_round_kernel/CMakeLists.txt Wire SYCL-TLA generated sources into the module build.
auto_round_extension/ark/auto_round_kernel/ark.cpp Reduce wrapper includes to declaration-only common header for SYCL-TLA paths.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@luoyu-intel
luoyu-intel merged commit ff2fd25 into main Aug 20, 2026
45 checks passed
@luoyu-intel
luoyu-intel deleted the fix/sycl-tla-memory-budget branch August 20, 2026 08:33
chensuyue added a commit that referenced this pull request Aug 22, 2026
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
yiliu30 added a commit that referenced this pull request Sep 20, 2026
Brings in PR #2193 (Reduce SYCL-TLA Compiler Memory) as the prerequisite for
extending the same instantiation-splitting to the sparse attention kernels.

Co-Authored-By: Claude Code <noreply@anthropic.com>
yiliu30 added a commit that referenced this pull request Sep 20, 2026
Merging main (PR #2193, "Reduce SYCL-TLA Compiler Memory") left the tree
unbuildable:

- ark.cpp failed with "use of undeclared identifier 'FlashAttnDtype'": the
  sparse binding code relied on sycl_tla_sdpa.hpp, which #2193 dropped from
  ark.cpp's includes to stop pulling heavy kernels into dispatch-only units.
- Including sycl_tla_common.hpp from the heavy SDPA headers instead broke
  sdpa.cpp with "redefinition of default argument", because common's
  sdpa_impl(..., float* lse = nullptr) declaration collides with the
  definition in sdpa.cpp.

FlashAttnDtype was defined identically in sycl_tla_sdpa_sparse.hpp,
sycl_tla_sdpa.hpp and sycl_tla_wrapper.hpp. Move it to a new dependency-free
flash_attn_dtype.hpp and include that from sycl_tla_common.hpp and the three
headers, so dispatch-only translation units keep compiling without pulling in
kernel headers.

Co-Authored-By: Claude Code <noreply@anthropic.com>
yiliu30 added a commit that referenced this pull request Sep 20, 2026
The sparse attention families (INT8 qks8 "sage" and native BF16/FP16 "sdpa")
instantiated every causal/mask combination of every launcher inside a single
translation unit. Measured peak compiler RSS was 18.3 GiB for sdpa_sparse.cpp
and 17.9 GiB for sdpa_sparse_sdpa.cpp -- roughly 10x the worst dense SDPA
source that PR #2193 ("Reduce SYCL-TLA Compiler Memory") brought under 2 GiB.

Apply the same treatment here:

- Promote causal state and mask presence from runtime options to template
  parameters on SparseSageConfig/SparseSDPAConfig and on the sparse launchers,
  so each instantiation compiles exactly one kernel. Cached KV was also a
  runtime branch; nothing in the sparse path sets it, so it becomes a compile
  time false with an explicit error if a caller asks for cached decode rather
  than silently taking a different kernel.
- Generate one translation unit per (family, shape, dtype, causal, mask) via
  sdpa_generation.cmake, mirroring the dense generator, and select among the
  generated launchers at runtime the way sdpa.cpp does for dense SDPA.

Measured with tools/measure_sycl_tla_compile_memory.py (serial, oneAPI 2026.0):

  sdpa_sparse.cpp                18759.3 -> 511.3 MiB
  sdpa_sparse_sdpa.cpp           18285.3 -> 508.5 MiB
  worst generated sparse TU                 1887.6 MiB
  direct dispatches per TU             6 -> 1

The shared module also shrank from 22.8 MB to 20.8 MB. Kernel behaviour is
unchanged: the sparse benchmark at seq 32768 reproduces pre-split timings to
within 0.5%.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants