[Question]: Is A6000 supported? #23

Activity

1. yawzhe

   added

   questionFurther information is requestedFurther information is requested

   on Jul 8, 2024
2. Huiqiang Jiang (iofu728)

   assigned

   Huiqiang Jiang (iofu728)

   and

   Yucheng Li (liyucheng09)

   on Jul 8, 2024
3. Huiqiang Jiang (iofu728)

   added

   feature requestNew feature or requestNew feature or request

   on Jul 8, 2024
4. iofu728 commented on Jul 8, 2024

   

   Huiqiang Jiang (iofu728)

   on Jul 8, 2024

   Contributor

   More actions

   Hi yawzhe,

   Thank you for your support in MInference.

   After reviewing the code, it appears that our decoding stage relies on flash-attn. For the prefilling stage, the three ops used are based on a Triton-implemented version of dynamic sparse flash attention, which does not depend on the flash-attn library. We plan to support a version that does not rely on flash-attn in the future, although this version will have slightly higher latency compared to the one using flash-attn.
5. Huiqiang Jiang (iofu728)

   changed the title [-][Question]: A6000支持加速推理吗?[/-] [+][Question]: Is A6000 supported?[/+] on Jul 8, 2024
6. Huiqiang Jiang (iofu728)

   closed this as completedin #35on Jul 15, 2024
7. iofu728 commented on Jul 15, 2024

   

   Huiqiang Jiang (iofu728)

   on Jul 15, 2024

   Contributor

   More actions

   Please upgrade MInference to version 0.1.4.post3, which does not depend on flash_attn. You can do this by running the following command:

   pip install minference==0.1.4.post3