Skip to content

Xenserver mount not working #2

Open

Description

@miesiu

when trying to mount the disk I get the following message, is there any solution?

MountVolume.MountDevice failed for volume "pvc-19c4c40c-6c7e-450a-a99a-277cc11479fd" : rpc error: code = Internal desc = Cannot find device path for volume e1ff8220-1e76-455f-bfef-ef6b5ba2e014: Failed to find device for the volumeID: "e1ff8220-1e76-455f-bfef-ef6b5ba2e014" within the alloted time

Warning FailedMount 49s (x8 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[example-volume], unattached volumes=[example-volume kube-api-access-g8mbx]: timed out waiting for the condition

Activity

  1. olivierlemasle commented on Jul 12, 2021

    @olivierlemasle
    Member

    Hi @miesiu, thank you for the report.

    Currently, this CSI driver works only with KVM and has not yet been tested with Xenserver.

    With KVM, I've found no direct link between CloudStack volume deviceId and the device id in the VM.

    That's why this CSI driver identifies the disk device by its device ID (as found in /dev/disk/by-id), which contains the "serial" of the volume, which is itself generated from the CloudStack volume ID. This is specific to KVM - cf https://github.com/apalia/cloudstack-csi-driver/blob/master/pkg/mount/mount.go#L129-L130

    However, CloudStack seems to define a direct link (in here):

    deviceId
    0 /dev/xvda
    1 /dev/xvdb
    2 /dev/xvdc
    4 /dev/xvde
    5 /dev/xvdf
    ... ...

    I'll try to get access to a CloudStack + XenServer environment, but could you please tell me if the "link" in the table above really works?

    In the following scenario:

    1. I create a VM, with no data volume (only a root volume)
    2. I create a volume A and attach it to the VM, with deviceId = 4
    3. THEN I create a volume B and attach it to the VM, with no deviceId specified

    Will I get:

    • /dev/xvda is the root volume?
    • /dev/xvde is volume A?
    • /dev/xvdb is volume B?
  2. miesiu commented on Jul 13, 2021

    @miesiu
    Author

    Hey @olivierlemasle

    Thank you for your response. I can confirm that the above scenario will work this way, remember deviceID = 3 is reserved for cd-rom. Let me know if I can help you somehow.

  3. olivierlemasle commented on Jul 13, 2021

    @olivierlemasle
    Member

    Thank you @miesiu. Then I'll try to use the deviceId to search the device with the mapping above. When it's ready, will you be ok to test if it works well in a CloudStack+Xenserver environment?

    I see that you started experimenting yourself. Did you get any success? I'll by happy to review a pull request if you want to contribute it.

  4. miesiu commented on Jul 14, 2021

    @miesiu
    Author

    @olivierlemasle, unfortunately without success, I have no experience in this field. I will run tests just let me know.

  5. joschi36 commented on Aug 3, 2021

    @joschi36

    Hi @miesiu @olivierlemasle we are just working on VMware support for the CSI and CS+VMware also doesn't provide /dev/disk/by-id.
    We noticed deviceId matches with our SCSI device number. Also, we are looking at chaininfo but this isn't always filled, at least with CS 4.14.1.0. We now try matching on both. My friend will publish a PR in a few days.
    If you (@miesiu) can verify deviceId matches with /dev/disk/by-id we maybe can also support Xen with the same PR.

  6. olivierlemasle commented on Aug 12, 2021

    @olivierlemasle
    Member

    Hi @joschi36,

    I'm also experimenting with VMware support, and I did not yet find how to match the devices with the CloudStack volumes.

    For example (with CloudStack 4.14 and VMware):

    • I create a VM with only a ROOT volume (20 GB)
    • I attach a 20 GB volume, with deviceId = 2
    • Then I attach a 10 GB volume, with deviceId = 1

    I specified the deviceId to test the case where the deviceIds are not increasing. As the CSI driver may attach multiple volumes to the same VM at "the same time", it is not possible to rely on this order...

    I have no chaininfo for my 2 data volumes.

    Then, when the volumes are recognized, lsblk gives:

    NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                      8:0    0   20G  0 disk 
    ├─sda1                   8:1    0    1G  0 part /boot
    └─sda2                   8:2    0   19G  0 part 
      └─fedora_fedora-root 253:0    0   15G  0 lvm  /
    sdb                      8:16   0   20G  0 disk 
    sdc                      8:32   0   10G  0 disk 
    sr0                     11:0    1 1024M  0 rom 
    

    It shows that the device names (sdb, sdc) follow the order of the volume attachments, not the device ids as known by CloudStack.

    And unlike with KVM, I do not find anything in /dev/disk/by-id (or any other /dev/disk/by-*) to link a device to its CloudStack volume ID, or to its CloudStack deviceId.

    ls -d /sys/block/sd*/device/scsi_device/* | awk -F '[/]' '{print $4,"- SCSI",$7}'
    sda - SCSI 0:0:1:0
    sdb - SCSI 2:0:0:0
    sdc - SCSI 2:0:1:0
    

    Also, the SCSI device numbers follow the order of the volume attachments, which do not match the device ids known by CloudStack.

    Do you have any idea?

  7. olivierlemasle commented on Aug 12, 2021

    @olivierlemasle
    Member
  8. schlapzz commented on Aug 12, 2021

    @schlapzz

    @olivierlemasle
    I encountered following behaviour with CloudStack and VMWare

    • CloudStack increases the device ID for every volume (or it takes the lowest unused device ID)
    • SCSI Device ID 3 is never assigned to a CloudStack volume (see here) This is because ID 3 is usually the CD-ROM device (of course doesn't make much sense in a cloud environmnet)
    • On VMWare Hosts SCSI DeviceID 7 is assigned to the Virtual SCSI Adapter and therefore can not used by a volume (see here) CloudStack however assigns this ID to the Volumes, which leads the they can not be found

    So for VMWare we're trying to match the deviceID by matching the path /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:<deviceID>:0

    There is maybe also a "bug" with the Unpublish Functions. According to the CSI Specification we need to undo everything we do in the Controller/Node Publish functions. This includes IMO also the cleaning of the SCSI Bus (counterpart to scan and add SCSI devices) For example:

    echo 1 > /sys/calss/scsi_device/0:10:0<deviceID>/delete
    

    Otherwise the controller is complaining about an already in use volume when you try to format the mounted volume.

  9. olivierlemasle commented on Aug 13, 2021

    @olivierlemasle
    Member

    @schlapzz Thank you for your answer.

    However, this works only because both CloudStack and VMware increment the deviceIds (as I noted above, this does not work when a deviceId is specified in the attachVolume request, which means that the CloudStack deviceId is not used).
    I feel it could create issues if multiple volumes were attached to the VM "simultaneously" (if possible).
    I'll make tests to verify if it is reliable; perhaps my concern is only theoretical...

    Also, thank you for your comment on the Unpublish functions; you're right.

  10. schlapzz commented on Mar 29, 2022

    @schlapzz

    @olivierlemasle
    I did a Pull Request to add support for VMWare. See #9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions