Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: CNV v4.21.0
Affects Version/s: CNV v4.14.0
Component/s: Storage Platform
Labels:
None

Activity Type:
Quality / Stability / Reliability
Story Points:
2
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Epic Link:
cnv-storage-technical-debt-4.21
Component Fix Version(s):
CNV v4.99.0.rhel9-2358
Market:

Sprint:
CNV Storage 272, CNV Storage 273, CNV Storage 274, CNV Storage 275, CNV Storage 276, CNV Storage 277, CNV Storage 278, CNV Storage Sprint 279

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

PX Impact Score:

No indication why the clone is stuck when the 'volumeMode' and 'accessModes' are not supported by the target storage class

Description of problem:
Two issues:
1. No indication why the cloned PVC is Pending, when the 'volumeMode' and 'accessModes' are not supported by the target storage class.2. It attempts to use 'csi-clone' when cloning from HPP (that doesn't support snapshots) to OCS, and never switches to a network clone.

Version-Release number of selected component (if applicable):
4.14

How reproducible:
Always

Steps to Reproduce:

1. Create a source DV on HPP (ReadWriteOnce, Filesystem).

2. Create a target DV (to clone the source) on OCS, using 'pvc' api and only specifying 'accessModes - ReadWriteMany', but not mentioning the 'volumeMode' (can be done in the same namespace or in a different)

3. See that target DV is stuck in 'CSICloneInProgress'

$ oc get dv -A | grep dv
default                              dv-target                     CSICloneInProgress   N/A                   9m8s
openshift-virtualization-os-images   dv-source-hpp                 Succeeded            100.0%                9m41s

Actual results:

$ oc get dv dv-target -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata: 
  annotations: 
    cdi.kubevirt.io/cloneType: csi-clone
    cdi.kubevirt.io/storage.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjE2OTI2MDIyNjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWFwaXNlcnZlciIsIm5hbWUiOiJkdi1zb3VyY2UtaHBwIiwibmFtZXNwYWNlIjoib3BlbnNoaWZ0LXZpcnR1YWxpemF0aW9uLW9zLWltYWdlcyIsIm5iZiI6MTY5MjYwMTk2OSwib3BlcmF0aW9uIjoiQ2xvbmUiLCJwYXJhbXMiOnsidGFyZ2V0TmFtZSI6ImR2LXRhcmdldCIsInRhcmdldE5hbWVzcGFjZSI6ImRlZmF1bHQifSwicmVzb3VyY2UiOnsiZ3JvdXAiOiIiLCJyZXNvdXJjZSI6InBlcnNpc3RlbnR2b2x1bWVjbGFpbXMiLCJ2ZXJzaW9uIjoidjEifX0.dgw3d0iPqPFS6OA1ENZygJEinEHlQwNeOEonta57U3SAwRfjluIX163jB3H9HObB4tsqUuErlVuyNli0hGuUtIIFwfuFqpIzqK5bhnVyHTbWVj3qCCkosEg1pxhzGKzne1pEkvOinZc7xGgDOteCuyBlnSLE34zlq7UiP4B_5hsZd6V604-R5X5ePjvVoFoluNQY69-K2si1DlC5JzUigAXRp9hgdU-BP5HXFfxWY3zvLvFI6nthc4qGp65IzcGBaHqHk8SwvY4z79czoObwwX-1Sr6pW5tEaORC1dYHGhVqXQG0-koHBnX5hx70piIuTeBRZsPsERzwnL2jft3tPg
    cdi.kubevirt.io/storage.extended.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjIwMDc5NjE5NjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWRlcGxveW1lbnQiLCJuYW1lIjoiZHYtc291cmNlLWhwcCIsIm5hbWVzcGFjZSI6Im9wZW5zaGlmdC12aXJ0dWFsaXphdGlvbi1vcy1pbWFnZXMiLCJuYmYiOjE2OTI2MDE5NjksIm9wZXJhdGlvbiI6IkNsb25lIiwicGFyYW1zIjp7InRhcmdldE5hbWUiOiJkdi10YXJnZXQiLCJ0YXJnZXROYW1lc3BhY2UiOiJkZWZhdWx0IiwidWlkIjoiNzJkMTI3MWMtNjBmMi00N2M1LWFjMTMtOTczNGUzN2U2NGU4In0sInJlc291cmNlIjp7Imdyb3VwIjoiIiwicmVzb3VyY2UiOiJwZXJzaXN0ZW50dm9sdW1lY2xhaW1zIiwidmVyc2lvbiI6InYxIn19.WqmXCk9Etn0fn8P5ksKdTThtPnh08Ng0nUo3CReanDualLXSSi24vh1hZf1Vkq9bI9T9ZpkBnNbZfj-fthoqHIUH1LtPnFb2MxKVmKHzMxiL10BZWw6w0l7_2-do8NhPqoHzgX8Lp7PZNqCKMrUrSnl7tvK8mwO8zi3IIQY3vzyoFR_tDdjAJy1IMqUzvu54yxBLmSR54lxjFrYJ7VANwzLXtLRmmsS9VlNparTVphZGV7fXj8w_zDbE-rwH4TJgrT1AHJE6B57dJy_xGwl5LLTPYxS3Fmey38ekB6oIz4YQGS7TveLi602XElAjYsBjoTiKQTId_jaxyl1OVfmLZg
    cdi.kubevirt.io/storage.usePopulator: "true"
  creationTimestamp: "2023-08-21T07:12:49Z"
  finalizers: 
  - cdi.kubevirt.io/dataVolumeFinalizer
  generation: 1
  name: dv-target
  namespace: default
  resourceVersion: "14376859"
  uid: 72d1271c-60f2-47c5-ac13-9734e37e64e8
spec: 
  pvc: 
    accessModes: 
    - ReadWriteMany
    resources: 
      requests: 
        storage: 32Gi
    storageClassName: ocs-storagecluster-ceph-rbd
  source: 
    pvc: 
      name: dv-source-hpp
      namespace: openshift-virtualization-os-images
status: 
  claimName: dv-target
  conditions: 
  - lastHeartbeatTime: "2023-08-21T07:12:49Z"
    lastTransitionTime: "2023-08-21T07:12:49Z"
    message: PVC dv-target Pending
    reason: Pending
    status: "False"
    type: Bound
  - lastHeartbeatTime: "2023-08-21T07:12:52Z"
    lastTransitionTime: "2023-08-21T07:12:49Z"
    reason: TransferRunning
    status: "False"
    type: Ready
  - lastHeartbeatTime: "2023-08-21T07:12:50Z"
    lastTransitionTime: "2023-08-21T07:12:50Z"
    reason: Populator is running
    status: "True"
    type: Running
  phase: CSICloneInProgress
  progress: N/A

Target DV Events: 
Events: 
  Type     Reason              Age                From                             Message
  ----     ------              ----               ----                             -------
  Normal   CloneScheduled      14m (x2 over 14m)  datavolume-pvc-clone-controller  Cloning from openshift-virtualization-os-images/dv-source-hpp into default/dv-target scheduled
  Normal   Pending             14m                datavolume-pvc-clone-controller  PVC dv-target Pending
  Warning  Pending             14m                datavolume-pvc-clone-controller  Clone Pending
  Normal   CSICloneInProgress  14m                datavolume-pvc-clone-controller  CSI Volume clone in progress (for pvc openshift-virtualization-os-images/dv-source-hpp)

TARGET PVC got a default volumeMode: Filesystem (but our OCS storage can't be RWX + Filesystem)

$ oc get pvc dv-target -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata: 
  annotations: 
    cdi.kubevirt.io/clonePhase: CSIClone
    cdi.kubevirt.io/cloneType: csi-clone
    cdi.kubevirt.io/dataSourceNamespace: openshift-virtualization-os-images
    cdi.kubevirt.io/storage.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjE2OTI2MDIyNjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWFwaXNlcnZlciIsIm5hbWUiOiJkdi1zb3VyY2UtaHBwIiwibmFtZXNwYWNlIjoib3BlbnNoaWZ0LXZpcnR1YWxpemF0aW9uLW9zLWltYWdlcyIsIm5iZiI6MTY5MjYwMTk2OSwib3BlcmF0aW9uIjoiQ2xvbmUiLCJwYXJhbXMiOnsidGFyZ2V0TmFtZSI6ImR2LXRhcmdldCIsInRhcmdldE5hbWVzcGFjZSI6ImRlZmF1bHQifSwicmVzb3VyY2UiOnsiZ3JvdXAiOiIiLCJyZXNvdXJjZSI6InBlcnNpc3RlbnR2b2x1bWVjbGFpbXMiLCJ2ZXJzaW9uIjoidjEifX0.dgw3d0iPqPFS6OA1ENZygJEinEHlQwNeOEonta57U3SAwRfjluIX163jB3H9HObB4tsqUuErlVuyNli0hGuUtIIFwfuFqpIzqK5bhnVyHTbWVj3qCCkosEg1pxhzGKzne1pEkvOinZc7xGgDOteCuyBlnSLE34zlq7UiP4B_5hsZd6V604-R5X5ePjvVoFoluNQY69-K2si1DlC5JzUigAXRp9hgdU-BP5HXFfxWY3zvLvFI6nthc4qGp65IzcGBaHqHk8SwvY4z79czoObwwX-1Sr6pW5tEaORC1dYHGhVqXQG0-koHBnX5hx70piIuTeBRZsPsERzwnL2jft3tPg
    cdi.kubevirt.io/storage.condition.running: "true"
    cdi.kubevirt.io/storage.condition.running.message: ""
    cdi.kubevirt.io/storage.condition.running.reason: Populator is running
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.extended.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjIwMDc5NjE5NjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWRlcGxveW1lbnQiLCJuYW1lIjoiZHYtc291cmNlLWhwcCIsIm5hbWVzcGFjZSI6Im9wZW5zaGlmdC12aXJ0dWFsaXphdGlvbi1vcy1pbWFnZXMiLCJuYmYiOjE2OTI2MDE5NjksIm9wZXJhdGlvbiI6IkNsb25lIiwicGFyYW1zIjp7InRhcmdldE5hbWUiOiJkdi10YXJnZXQiLCJ0YXJnZXROYW1lc3BhY2UiOiJkZWZhdWx0IiwidWlkIjoiZTk1ZjU0N2YtMjkwOC00MThlLThjYTYtOTEzNTQ4NTlmMjI5In0sInJlc291cmNlIjp7Imdyb3VwIjoiIiwicmVzb3VyY2UiOiJwZXJzaXN0ZW50dm9sdW1lY2xhaW1zIiwidmVyc2lvbiI6InYxIn19.K0Yf9S9waiV_d8kQIDEa3zkYqsD6on6XTZCkbAFyoQSCUnojUsi3NjFM7j3X5F0XJxDqt1gQgRGV09Q1W8ehFk6vQIoasyWdBWhKFDCmd0XoENHXQjPGqzsvtcGAt9kvVR0ducEw6iiHaThebZxo0gN9K8ztAqURJdc5Tq1vuafJK99s6BajDmOwn4qD_9KF4CvtGiWZsC6aterQIYRjndgnI3qvAhbbEaw8kTZbjB91DMODM6ZaOHLFqusJLaH-O_K0IqKNqVYryaJ12wKPuUt2rwJEGADNYcPHFT-uzDM8izLV5P5ZHjmnvs0zUpxHnI8k_yjMdVTNW6rPSo6TCQ
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
  creationTimestamp: "2023-08-21T07:12:49Z"
  finalizers: 
  - kubernetes.io/pvc-protection
  - cdi.kubevirt.io/clonePopulator
  labels: 
    alerts.k8s.io/KubePersistentVolumeFillingUp: disabled
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
  name: dv-target
  namespace: default
  ownerReferences: 
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: dv-target
    uid: 72d1271c-60f2-47c5-ac13-9734e37e64e8
  resourceVersion: "14376854"
  uid: e95f547f-2908-418e-8ca6-91354859f229
spec: 
  accessModes: 
  - ReadWriteMany
  dataSource: 
    apiGroup: cdi.kubevirt.io
    kind: VolumeCloneSource
    name: volume-clone-source-72d1271c-60f2-47c5-ac13-9734e37e64e8
  dataSourceRef: 
    apiGroup: cdi.kubevirt.io
    kind: VolumeCloneSource
    name: volume-clone-source-72d1271c-60f2-47c5-ac13-9734e37e64e8
  resources: 
    requests: 
      storage: 32Gi
  storageClassName: ocs-storagecluster-ceph-rbd
  volumeMode: Filesystem
status: 
  phase: Pending

TMP-PVC -oyaml (also OCS with RWX + Filesystem, and it requests 'storage: 329Gi')

$ oc get pvc -n openshift-virtualization-os-images tmp-pvc-e95f547f-2908-418e-8ca6-91354859f229 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata: 
  annotations: 
    cdi.kubevirt.io/clonePhase: Pending
    cdi.kubevirt.io/cloneType: csi-clone
    cdi.kubevirt.io/dataSourceNamespace: openshift-virtualization-os-images
    cdi.kubevirt.io/storage.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjE2OTI2MDIyNjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWFwaXNlcnZlciIsIm5hbWUiOiJkdi1zb3VyY2UtaHBwIiwibmFtZXNwYWNlIjoib3BlbnNoaWZ0LXZpcnR1YWxpemF0aW9uLW9zLWltYWdlcyIsIm5iZiI6MTY5MjYwMTk2OSwib3BlcmF0aW9uIjoiQ2xvbmUiLCJwYXJhbXMiOnsidGFyZ2V0TmFtZSI6ImR2LXRhcmdldCIsInRhcmdldE5hbWVzcGFjZSI6ImRlZmF1bHQifSwicmVzb3VyY2UiOnsiZ3JvdXAiOiIiLCJyZXNvdXJjZSI6InBlcnNpc3RlbnR2b2x1bWVjbGFpbXMiLCJ2ZXJzaW9uIjoidjEifX0.dgw3d0iPqPFS6OA1ENZygJEinEHlQwNeOEonta57U3SAwRfjluIX163jB3H9HObB4tsqUuErlVuyNli0hGuUtIIFwfuFqpIzqK5bhnVyHTbWVj3qCCkosEg1pxhzGKzne1pEkvOinZc7xGgDOteCuyBlnSLE34zlq7UiP4B_5hsZd6V604-R5X5ePjvVoFoluNQY69-K2si1DlC5JzUigAXRp9hgdU-BP5HXFfxWY3zvLvFI6nthc4qGp65IzcGBaHqHk8SwvY4z79czoObwwX-1Sr6pW5tEaORC1dYHGhVqXQG0-koHBnX5hx70piIuTeBRZsPsERzwnL2jft3tPg
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Clone Pending
    cdi.kubevirt.io/storage.condition.running.reason: Pending
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.extended.clone.token: eyJhbGciOiJQUzI1NiJ9.eyJleHAiOjIwMDc5NjE5NjksImlhdCI6MTY5MjYwMTk2OSwiaXNzIjoiY2RpLWRlcGxveW1lbnQiLCJuYW1lIjoiZHYtc291cmNlLWhwcCIsIm5hbWVzcGFjZSI6Im9wZW5zaGlmdC12aXJ0dWFsaXphdGlvbi1vcy1pbWFnZXMiLCJuYmYiOjE2OTI2MDE5NjksIm9wZXJhdGlvbiI6IkNsb25lIiwicGFyYW1zIjp7InRhcmdldE5hbWUiOiJkdi10YXJnZXQiLCJ0YXJnZXROYW1lc3BhY2UiOiJkZWZhdWx0IiwidWlkIjoiZTk1ZjU0N2YtMjkwOC00MThlLThjYTYtOTEzNTQ4NTlmMjI5In0sInJlc291cmNlIjp7Imdyb3VwIjoiIiwicmVzb3VyY2UiOiJwZXJzaXN0ZW50dm9sdW1lY2xhaW1zIiwidmVyc2lvbiI6InYxIn19.K0Yf9S9waiV_d8kQIDEa3zkYqsD6on6XTZCkbAFyoQSCUnojUsi3NjFM7j3X5F0XJxDqt1gQgRGV09Q1W8ehFk6vQIoasyWdBWhKFDCmd0XoENHXQjPGqzsvtcGAt9kvVR0ducEw6iiHaThebZxo0gN9K8ztAqURJdc5Tq1vuafJK99s6BajDmOwn4qD_9KF4CvtGiWZsC6aterQIYRjndgnI3qvAhbbEaw8kTZbjB91DMODM6ZaOHLFqusJLaH-O_K0IqKNqVYryaJ12wKPuUt2rwJEGADNYcPHFT-uzDM8izLV5P5ZHjmnvs0zUpxHnI8k_yjMdVTNW6rPSo6TCQ
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.populator.kind: VolumeCloneSource
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
  creationTimestamp: "2023-08-21T07:12:50Z"
  finalizers: 
  - kubernetes.io/pvc-protection
  labels: 
    alerts.k8s.io/KubePersistentVolumeFillingUp: disabled
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
    cdi.kubevirt.io/OwnedByUID: e95f547f-2908-418e-8ca6-91354859f229
  name: tmp-pvc-e95f547f-2908-418e-8ca6-91354859f229
  namespace: openshift-virtualization-os-images
  resourceVersion: "14376799"
  uid: 015468c0-66fe-41e9-9b7a-38d4e5bb5a61
spec: 
  accessModes: 
  - ReadWriteMany
  dataSource: 
    apiGroup: null
    kind: PersistentVolumeClaim
    name: dv-source-hpp
  dataSourceRef: 
    apiGroup: null
    kind: PersistentVolumeClaim
    name: dv-source-hpp
  resources: 
    requests: 
      storage: 329Gi
  storageClassName: ocs-storagecluster-ceph-rbd
  volumeMode: Filesystem
status: 
  phase: Pending

No cloner pods in both namespaces:

$ oc get pods -n default
No resources found in default namespace.

$ oc get pods -n openshift-virtualization-os-images
No resources found in openshift-virtualization-os-images namespace.

Expected results:
1. Do not use 'Filesystem' as a default volume mode
2. Better explanation of why cloning is not progressing
3. 'csi-clone' should fall back to a network ('copy') clone

Additional info:
YAMLs to reproduce:

DV SOURCE

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata: 
  name: dv-source-hpp
  namespace: openshift-virtualization-os-images
  annotations: 
    cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
spec: 
  source: 
      http: 
         url: <cirros-0.4.0-x86_64-disk.qcow2>
  storage: 
    resources: 
      requests: 
        storage: 1Gi
    storageClassName: hostpath-csi-basic

DV TARGET

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata: 
  name: dv-target
spec: 
  source: 
    pvc: 
      namespace: "openshift-virtualization-os-images"
      name: "dv-source-hpp"
  pvc: 
    accessModes: 
    - ReadWriteMany
    resources: 
      requests: 
        storage: 32Gi
    storageClassName: ocs-storagecluster-ceph-rbd

links to

Followup PR

Assignee:: Danny Sanatar

Reporter:: Jenia Peimer

QA Contact:: Kate Shvaika

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2023/08/21 8:05 AM

Updated:: 2025/10/28 2:15 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates