Skip to content

CUDA 10.1 on Windows 10 shows error: pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano #587

Open
Open
CUDA 10.1 on Windows 10 shows error: pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano#587

Description

@anupverma20
anupverma20
opened on May 14, 2019

OS: Windows 10
CUDA: 10.1

 Name                     Version                           Build  Channel
python                     3.5.6                            he025d50_0
libgpuarray                0.7.6                            vc14_0  [vc14]  mila-udem
pygpu                      0.7.6            	            py35h452e1ab_0
theano                     1.0.2            		    py35he980bc4_0
keras                      2.2.2                            0

Content of .theanorc.txt file:
[global]
device = cuda
floatX = float32

In System variables, the following are set:
CUDA_PATH
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

CUDA_PATH_V10_1
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

DEVICE
cuda

In the Path Variable of System variables, the following are set:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin

Error:

(gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend._BACKEND")
Using Theano backend.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 227, in <module>
    use(config.device)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 99, in init_dev
    **args)
  File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
  File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n'
theano

Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported?

Thanks!

Activity

  1. nouiz commented on May 15, 2019

    @nouiz
    nouiz
    on May 15, 2019 via email
    Member

  2. anupverma20 commented on May 15, 2019

    @anupverma20
    anupverma20
    on May 15, 2019
    Author

    It was fixed upstream, but no new release was done. So you must compile libgpuarray yourself.
    …
    On Tue, May 14, 2019 at 3:00 PM Anup Verma @.***> wrote: OS: Windows 10 CUDA: 10.1 Name Version Build Channel python 3.5.6 he025d50_0 libgpuarray 0.7.6 vc14_0 [vc14] mila-udem pygpu 0.7.6 py35h452e1ab_0 theano 1.0.2 py35he980bc4_0 keras 2.2.2 0 Content of .theanorc.txt file: [global] device = cuda floatX = float32 In System variables, the following are set: CUDA_PATH C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 CUDA_PATH_V10_1 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 GPUARRAY_TEST_DEVICE cuda In the Path Variable of System variables, the following are set: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin Error: (gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend.BACKEND") Using Theano backend. ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init.py", line 227, in use(config.device) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 214, in use init_dev(device, preallocate=preallocate) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 99, in init_dev **args) File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#587?email_source=notifications&email_token=AABMF65F4OFHAHWYO5QTOEDPVMD5VA5CNFSM4HM4YTW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTYBGHA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABMF64MUDG3BCDXM6LPBP3PVMD5VANCNFSM4HM4YTWQ .

    Thanks for the reply.
    I made libgpuarray.sln using cmake in Anaconda.:
    (gpuml) C:\Users\Anup\Desktop\lib\libgpuarray\build>cmake .. -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 16 2019" -A X64"

    I opened libgpuarray.sln in Visual Studio.
    When I select Build > gpuarray, I get the following error:

    1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>'python' is not recognized as an internal or external command,
1>operable program or batch file.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 9009.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

    I could not fix the python-related error
    I am new to cmake, Visual Studio and dll. Please help!

  3. NellTrico commented on May 16, 2019

    @NellTrico
    NellTrico
    on May 16, 2019

    Hello! I experience something similar.
    I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

    File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
    File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
    pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

    I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

    Could I please get some help with that?

  4. anupverma20 commented on May 16, 2019

    @anupverma20
    anupverma20
    on May 16, 2019
    Author

    I installed python, added its path in Environment Variables > Path. Now, that error is fixed.
    I also installed Mako, restarted the computer. But, I get the following error while building the sln file in Visual Studio:

    1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>Traceback (most recent call last):
1>  File "gen_types.py", line 3, in <module>
1>    from mako import exceptions
1>ModuleNotFoundError: No module named 'mako'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 1.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

    I was able to build gpuarray.dll in my office laptop. But, in the personal laptop, I am getting the above error. Trying to fix it now.

  5. anupverma20 commented on May 16, 2019

    @anupverma20
    anupverma20
    on May 16, 2019
    Author

    Hello! I experience something similar.
    I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

    File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
    File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
    pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

    I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

    Could I please get some help with that?

    As far as I have learnt from answers to similar issues on the internet, there are 2 options:

    1. The easier option is to install CUDA 9.
    2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
      Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

  6. NellTrico commented on May 20, 2019

    @NellTrico
    NellTrico
    on May 20, 2019

    Hello! I experience something similar.
    I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:
    File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
    File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
    pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'
    I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.
    Could I please get some help with that?

    As far as I have learnt from answers to similar issues on the internet, there are 2 options:

    1. The easier option is to install CUDA 9.

2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
   Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

    Thanks for your reply @anupverma20 !
    I tried both options without success. (Note that I don't use conda so I was already building libgpuarray myself)

    I am confused as to why the library insists on pointing to nvrtc64_102.dll, even after I uninstalled CUDA 10 and kept 9. I looked at the code in gpuarray_buffer_cuda.c and I am at loss to understand why and how I can even obtain major=10, minor=2... Nevertheless I tried changing major and minor in the source code to 9 and 1 respectively (having installed CUDA 9.1 with the corresponding nvrtc64_91.dll), but I was then faced with

    Could not load "nvrtc64_91.dll": %1 is not a valid Win32 application.\r\n'

    Similarly, when I still had CUDA 10.1, I tried renaming the dll to the expected "nvrtc64_102.dll", and had a similar error message.

    EDIT: It seems I get this error because I am using Python 36-32. I thought a compatible 32/64 bit version of Python would let me do it though? Am I wrong? If yes, do I have any option, like Running GPU acceleration with OpenCL for instance, which I saw in the tutorials, but have not given time yet, or is it limited for 64 bit architecture only?

  7. kumarnikhil936 commented on Aug 10, 2020

    @kumarnikhil936
    kumarnikhil936
    on Aug 10, 2020

    Any solutions so far?
    Still facing this issue.

  8. ThoenigAdrian commented on Oct 20, 2020

    @ThoenigAdrian
    ThoenigAdrian
    on Oct 20, 2020

    same issue here with cuda 11

  9. nouiz commented on Oct 20, 2020

    @nouiz
    nouiz
    on Oct 20, 2020
    Member

    This MR is the last one that update the supported CUDA version: https://github.com/Theano/libgpuarray/pull/580/files
    So the most recent CUDA version supported is 10.0

    This lib isn't maintained. But maybe you can find now to support newer version with that url.

  10. ThoenigAdrian commented on Oct 23, 2020

    @ThoenigAdrian
    ThoenigAdrian
    on Oct 23, 2020

    @nouiz
    Apparently there is an up to date fork : https://github.com/pymc-devs/Theano-PyMC . Haven't tried it though.
    Afaik it should be possible to install multiple CUDA versions on a machine. So one could install CUDA 9.0 and use Theano with it.

  11. nouiz commented on Oct 23, 2020

    @nouiz
    nouiz
    on Oct 23, 2020
    Member

    This is a Theano fork. I do not know if they forked libgpuarray too. Maybe you can ask them?
    I would recommend to try CUDA 10.0 first and not 9.0 as from the quick look at the code ,10.0 was supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions