Repository navigation
dpctl built with latest oneapi releases (2023.0.0, libsycl 6) fail to submit jobs to cpu #1023
Description
Activity
@fcharras It would be useful to know whether the compiler/tbb from oneAPI are activated (if so what is the output of
sycl-ls), and what version of dpctl are you using (0.13.0 from Intel channel, or 0.14.0dev1 from dppy/label/dev)?What is the output of
python -c "import dpctl; dpctl.lsplatform(verbosity=2)"?This build used to work only relying on python libraries such as
dpcpp-cpp-rtandmkl-dpcpp+ addingbin-llvmfolder from the oneapi basekit toPATH.Those packages are correctly bumped
root@3f55c48cd427:~/sklearn-numba-dpex# pip list |grep 2023 daal 2023.0.0 daal4py 2023.0.0 dpcpp-cpp-rt 2023.0.0 intel-cmplr-lib-rt 2023.0.0 intel-cmplr-lic-rt 2023.0.0 intel-fortran-rt 2023.0.0 intel-opencl-rt 2023.0.0 intel-openmp 2023.0.0 mkl 2023.0.0 mkl-dpcpp 2023.0.0 scikit-learn-intelex 2023.0.0 root@3f55c48cd427:~/sklearn-numba-dpex# pip list |grep 2022 pytz 2022.7I tried activating entirely the
oneapi basekitsource setvars.shbut then get the same results.
root@3f55c48cd427:~/sklearn-numba-dpex# sycl-ls [opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.15.12.0.01_081451] [opencl:cpu:1] Intel(R) OpenCL, 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 3.0 [2022.15.12.0.01_081451] [opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) Iris(R) Xe Graphics [0x9a49] 3.0 [22.38.24278] [opencl:acc:3] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.14.8.0.04_160000] [opencl:cpu:4] Intel(R) OpenCL, 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 3.0 [2022.14.8.0.04_160000] [ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics [0x9a49] 1.3 [1.3.24278]root@3f55c48cd427:~/sklearn-numba-dpex# python -c "import dpctl; dpctl.lsplatform(verbosity=2)" Platform 0 :: Name Intel(R) FPGA Emulation Platform for OpenCL(TM) Version OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3 Vendor Intel(R) Corporation Backend opencl Num Devices 1 # 0 Name Intel(R) FPGA Emulation Device Version 2022.15.12.0.01_081451 Filter string opencl:accelerator:0 Platform 1 :: Name Intel(R) OpenCL Version OpenCL 3.0 LINUX Vendor Intel(R) Corporation Backend opencl Num Devices 1 # 0 Name 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz Version 2022.15.12.0.01_081451 Filter string opencl:cpu:0 Platform 2 :: Name Intel(R) OpenCL HD Graphics Version OpenCL 3.0 Vendor Intel(R) Corporation Backend opencl Num Devices 1 # 0 Name Intel(R) Iris(R) Xe Graphics [0x9a49] Version 22.38.24278 Filter string opencl:gpu:0 Platform 3 :: Name Intel(R) FPGA Emulation Platform for OpenCL(TM) Version OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3 Vendor Intel(R) Corporation Backend opencl Num Devices 1 # 0 Name Intel(R) FPGA Emulation Device Version 2022.14.8.0.04_160000 Filter string opencl:accelerator:1 Platform 4 :: Name Intel(R) OpenCL Version OpenCL 3.0 LINUX Vendor Intel(R) Corporation Backend opencl Num Devices 1 # 0 Name 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz Version 2022.14.8.0.04_160000 Filter string opencl:cpu:1 Platform 5 :: Name Intel(R) Level-Zero Version 1.3 Vendor Intel(R) Corporation Backend ext_oneapi_level_zero Num Devices 1 # 0 Name Intel(R) Iris(R) Xe Graphics [0x9a49] Version 1.3.24278 Filter string level_zero:gpu:0oneapi basekit is also bumped accordingly
root@3f55c48cd427:/opt/intel/oneapi/compiler# ls 2023.0.0 latestThe version of
dpctlis0.14.0dev1, built from source. Maybe the build instructions changed recently ?I'll try a conda install to compare the results.
The issue is that DPC++ 2023 runtime is trying to submit a kernel to queue targeting CPU SYCL device built for DPC++ 2022 toolchain, the one with "2022.14.8.0.04_160000".
The DPC++ RT most likely picks it up from ICD files in
/etc/OpenCL/vendors. If my hunch is correct,cat /etc/OpenCL/vendors/intel64.icdwould show a path into/opt/intel/oneapi/compiler/2022.*.There are several solutions, other directly choose CPU device with driver version "2022.15.12.0.01_081451", or find a way to hide CPU/FPGA-emu drivers from 2022 installation (say uninstall older compiler).
sudo apt list --installed | grep oneapi-dpcpp-cpp | grep 2022
and then
sudo apt removethose listed.I don't think I have older compilers installed (since the docker image is rebuilt from scratch after all bumps), and the issue persists when specifically selecting the cpu with the good driver version.
Also, some kernels continues running fine. (see https://github.com/soda-inria/sklearn-numba-dpex/actions/runs/3733104423/jobs/6333449877 )
I'll try setting up a minimal reproducer to understand what specifically triggers kernel rejection. It seems to be related to
work_group_size: the test pipeline seems to hint that forcpukernels submitted with a group size equal tomax_work_group_sizeare rejected.Remaining errors encountered are detailed in IntelPython/numba-dpex#868 IntelPython/numba-dpex#867 and IntelPython/numba-dpex#865 with workarounds.
Unless you want to keep track of it here I think we can close this issue.
Thanks you @fcharras, closing
Getting errors such as
while it submits fine with
gpu, both withopenclandlevel_zerobackends 🤷