Skip to content

[Feature Request]: 请问MiniCPM Reranker何时支持vllm #258

Closed

Description

@IeohMingChan

Feature request / 功能建议

经过对比测试,MiniCPM Reranker模型的效果非常之好,然而响应速度较慢,请问何时可以支持vllm部署,万分感谢

Activity

  1. Kaguya-19 commented on Nov 7, 2024

    @Kaguya-19
    Collaborator

    您好,由于Reranker采用的是双向注意力,无kv cache机制,因此使用vllm部署并不会有较大的提升。您可以尝试转成onnx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions