You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
when trying to mount the disk I get the following message, is there any solution?
MountVolume.MountDevice failed for volume "pvc-19c4c40c-6c7e-450a-a99a-277cc11479fd" : rpc error: code = Internal desc = Cannot find device path for volume e1ff8220-1e76-455f-bfef-ef6b5ba2e014: Failed to find device for the volumeID: "e1ff8220-1e76-455f-bfef-ef6b5ba2e014" within the alloted time
Warning FailedMount 49s (x8 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[example-volume], unattached volumes=[example-volume kube-api-access-g8mbx]: timed out waiting for the condition
Thank you for your response. I can confirm that the above scenario will work this way, remember deviceID = 3 is reserved for cd-rom. Let me know if I can help you somehow.
Thank you @miesiu. Then I'll try to use the deviceId to search the device with the mapping above. When it's ready, will you be ok to test if it works well in a CloudStack+Xenserver environment?
I see that you started experimenting yourself. Did you get any success? I'll by happy to review a pull request if you want to contribute it.
Hi @miesiu@olivierlemasle we are just working on VMware support for the CSI and CS+VMware also doesn't provide /dev/disk/by-id.
We noticed deviceId matches with our SCSI device number. Also, we are looking at chaininfo but this isn't always filled, at least with CS 4.14.1.0. We now try matching on both. My friend will publish a PR in a few days.
If you (@miesiu) can verify deviceId matches with /dev/disk/by-id we maybe can also support Xen with the same PR.
I'm also experimenting with VMware support, and I did not yet find how to match the devices with the CloudStack volumes.
For example (with CloudStack 4.14 and VMware):
I create a VM with only a ROOT volume (20 GB)
I attach a 20 GB volume, with deviceId = 2
Then I attach a 10 GB volume, with deviceId = 1
I specified the deviceId to test the case where the deviceIds are not increasing. As the CSI driver may attach multiple volumes to the same VM at "the same time", it is not possible to rely on this order...
I have no chaininfo for my 2 data volumes.
Then, when the volumes are recognized, lsblk gives:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
└─fedora_fedora-root 253:0 0 15G 0 lvm /
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
sr0 11:0 1 1024M 0 rom
It shows that the device names (sdb, sdc) follow the order of the volume attachments, not the device ids as known by CloudStack.
And unlike with KVM, I do not find anything in /dev/disk/by-id (or any other /dev/disk/by-*) to link a device to its CloudStack volume ID, or to its CloudStack deviceId.
@olivierlemasle
I encountered following behaviour with CloudStack and VMWare
CloudStack increases the device ID for every volume (or it takes the lowest unused device ID)
SCSI Device ID 3 is never assigned to a CloudStack volume (see here) This is because ID 3 is usually the CD-ROM device (of course doesn't make much sense in a cloud environmnet)
On VMWare Hosts SCSI DeviceID 7 is assigned to the Virtual SCSI Adapter and therefore can not used by a volume (see here) CloudStack however assigns this ID to the Volumes, which leads the they can not be found
So for VMWare we're trying to match the deviceID by matching the path /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:<deviceID>:0
There is maybe also a "bug" with the Unpublish Functions. According to the CSI Specification we need to undo everything we do in the Controller/Node Publish functions. This includes IMO also the cleaning of the SCSI Bus (counterpart to scan and add SCSI devices) For example:
However, this works only because both CloudStack and VMware increment the deviceIds (as I noted above, this does not work when a deviceId is specified in the attachVolume request, which means that the CloudStack deviceId is not used).
I feel it could create issues if multiple volumes were attached to the VM "simultaneously" (if possible).
I'll make tests to verify if it is reliable; perhaps my concern is only theoretical...
Also, thank you for your comment on the Unpublish functions; you're right.
Activity
olivierlemasle commentedon Jul 12, 2021
Hi @miesiu, thank you for the report.
Currently, this CSI driver works only with KVM and has not yet been tested with Xenserver.
With KVM, I've found no direct link between CloudStack volume
deviceIdand the device id in the VM.That's why this CSI driver identifies the disk device by its device ID (as found in
/dev/disk/by-id), which contains the "serial" of the volume, which is itself generated from the CloudStack volume ID. This is specific to KVM - cf https://github.com/apalia/cloudstack-csi-driver/blob/master/pkg/mount/mount.go#L129-L130However, CloudStack seems to define a direct link (in here):
I'll try to get access to a CloudStack + XenServer environment, but could you please tell me if the "link" in the table above really works?
In the following scenario:
Will I get:
miesiu commentedon Jul 13, 2021
Hey @olivierlemasle
Thank you for your response. I can confirm that the above scenario will work this way, remember deviceID = 3 is reserved for cd-rom. Let me know if I can help you somehow.
olivierlemasle commentedon Jul 13, 2021
Thank you @miesiu. Then I'll try to use the deviceId to search the device with the mapping above. When it's ready, will you be ok to test if it works well in a CloudStack+Xenserver environment?
I see that you started experimenting yourself. Did you get any success? I'll by happy to review a pull request if you want to contribute it.
miesiu commentedon Jul 14, 2021
@olivierlemasle, unfortunately without success, I have no experience in this field. I will run tests just let me know.
joschi36 commentedon Aug 3, 2021
Hi @miesiu @olivierlemasle we are just working on VMware support for the CSI and CS+VMware also doesn't provide
/dev/disk/by-id.We noticed
deviceIdmatches with our SCSI device number. Also, we are looking atchaininfobut this isn't always filled, at least with CS 4.14.1.0. We now try matching on both. My friend will publish a PR in a few days.If you (@miesiu) can verify
deviceIdmatches with/dev/disk/by-idwe maybe can also support Xen with the same PR.olivierlemasle commentedon Aug 12, 2021
Hi @joschi36,
I'm also experimenting with VMware support, and I did not yet find how to match the devices with the CloudStack volumes.
For example (with CloudStack 4.14 and VMware):
deviceId = 2deviceId = 1I specified the deviceId to test the case where the deviceIds are not increasing. As the CSI driver may attach multiple volumes to the same VM at "the same time", it is not possible to rely on this order...
I have no
chaininfofor my 2 data volumes.Then, when the volumes are recognized,
lsblkgives:It shows that the device names (
sdb,sdc) follow the order of the volume attachments, not the device ids as known by CloudStack.And unlike with KVM, I do not find anything in
/dev/disk/by-id(or any other/dev/disk/by-*) to link a device to its CloudStack volume ID, or to its CloudStack deviceId.Also, the SCSI device numbers follow the order of the volume attachments, which do not match the device ids known by CloudStack.
Do you have any idea?
olivierlemasle commentedon Aug 12, 2021
cc @schlapzz
schlapzz commentedon Aug 12, 2021
@olivierlemasle
I encountered following behaviour with CloudStack and VMWare
So for VMWare we're trying to match the deviceID by matching the path
/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:<deviceID>:0There is maybe also a "bug" with the Unpublish Functions. According to the CSI Specification we need to undo everything we do in the Controller/Node Publish functions. This includes IMO also the cleaning of the SCSI Bus (counterpart to scan and add SCSI devices) For example:
Otherwise the controller is complaining about an already in use volume when you try to format the mounted volume.
olivierlemasle commentedon Aug 13, 2021
@schlapzz Thank you for your answer.
However, this works only because both CloudStack and VMware increment the
deviceIds(as I noted above, this does not work when adeviceIdis specified in theattachVolumerequest, which means that the CloudStackdeviceIdis not used).I feel it could create issues if multiple volumes were attached to the VM "simultaneously" (if possible).
I'll make tests to verify if it is reliable; perhaps my concern is only theoretical...
Also, thank you for your comment on the Unpublish functions; you're right.
schlapzz commentedon Mar 29, 2022
@olivierlemasle
I did a Pull Request to add support for VMWare. See #9