You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Problems encountered during speculative decoding execution #888
An error occurred during the first execution:
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: gpu
Then I modified HFDecoderModel in hf_decoder_model.py to use cuda, and the following error occurred:
NotImplementedError: device "cuda" is not supported
On the third attempt, I changed it to use cpu and got the error:
ValueError: The following model_kwargs are not used by the model: ['use_accelerator']"
Is there any configuration or environment setting error on my part?
Activity
huangzl19 commentedon Sep 2, 2024
I encountered the same problem.