Uploaded image for project: 'Data Foundation Bugs'
  1. Data Foundation Bugs
  2. DFBUGS-325

[2159791] [RDR] DR workloads remain stuck when deleted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • odf-4.18
    • odf-4.12
    • csi-driver
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • Hide
      .Disaster recovery workloads remain stuck when deleted

      When deleting a workload from a cluster, the corresponding pods may not terminate with events such as `FailedKillPod`. This may cause delay or failure in garbage collecting dependent DR resources such as the `PVC`, `VolumeReplication`, and `VolumeReplicationGroup`. It would also prevent a future deployment of the same workload to the cluster as the stale resources are not yet garbage collected.

      Workaround: Reboot the worker node on which the pod is currently running and stuck in a terminating state. This results in successful pod termination and subsequently related DR API resources are also garbage collected.
      Show
      .Disaster recovery workloads remain stuck when deleted When deleting a workload from a cluster, the corresponding pods may not terminate with events such as `FailedKillPod`. This may cause delay or failure in garbage collecting dependent DR resources such as the `PVC`, `VolumeReplication`, and `VolumeReplicationGroup`. It would also prevent a future deployment of the same workload to the cluster as the stale resources are not yet garbage collected. Workaround: Reboot the worker node on which the pod is currently running and stuck in a terminating state. This results in successful pod termination and subsequently related DR API resources are also garbage collected.
    • Known Issue
    • Ceph-CSI Team sprint #30, Ceph-CSI Team sprint #31, Ceph-CSI Team sprint #32, Ceph-CSI Team sprint #33, Ceph-CSI Team sprint #34, Ceph-CSI Team sprint 2024.1, Ceph-CSI Sprint 2024.2, Ceph-CSI Sprint 2024.3, Ceph-CSI Sprint 2024.4, Ceph-CSI Team sprint 2024.5, Ceph-CSI Team sprint 2024.6, Ceph-CSI Team sprint 2024.7, Ceph-CSI Team sprint 2024.8, Ceph-CSI Team sprint 2024.9
    • None

      Description of problem (please be detailed as possible and provide log
      snippests):

      Version of all relevant components (if applicable):
      ACM 2.7.0-DOWNSTREAM-2023-01-07-13-58-25
      ODF 4.12.0-158
      OCP 4.12.0-0.nightly-2023-01-08-142418
      ceph version 16.2.10-90.el8cp (821b516c325c19f31b81b943cd800c2190f1e685) pacific (stable)

      Does this issue impact your ability to continue to work with the product
      (please explain in detail what is the user impact)?

      Is there any workaround available to the best of your knowledge?

      Rate from 1 - 5 the complexity of the scenario you performed that caused this
      bug (1 - very simple, 5 - very complex)?

      Can this issue reproducible?

      Can this issue reproduce from the UI?

      If this is a regression, please provide more details to justify this:

      Steps to Reproduce:
      1. Created a RDR setup and deployed workloads from busybox 1 to 6 along with multi sub busybox on C1 having latency 175ms round trip
      C1 with 6W 3M (CoreOS) (Only 3 W nodes were selected for deploying ODF), C2 with 3M 3W (RHEL)
      2. Run the workloads for almost 10-11hrs until Ceph on C1 is ~55% full
      3. Cluster was healthy, did all pre-checks
      4. Delete busybox-workload-6, busybox-workload-2 and multi sub busybox from C1, observe

      Actual results: DR workloads remain stuck when deleted

      From C1-

      [amagrawa@amagrawa ~]$ busybox-6
      Now using project "busybox-workloads-6" on server "https://api.amagrawa-c1-9jan.qe.rh-ocs.com:6443".
      No resources found in busybox-workloads-6 namespace.

      [amagrawa@amagrawa ~]$ busybox-2
      Now using project "busybox-workloads-2" on server "https://api.amagrawa-c1-9jan.qe.rh-ocs.com:6443".
      buxyNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
      persistentvolumeclaim/busybox-pvc-25 Terminating pvc-0a7fd9fe-14d9-44a0-a0e7-fdc4ccd5c6ff 45Gi RWO ocs-storagecluster-ceph-rbd 12h

      NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE
      volumereplication.replication.storage.openshift.io/busybox-pvc-25 12h rbd-volumereplicationclass-539797778 busybox-pvc-25 primary Primary

      NAME DESIREDSTATE CURRENTSTATE
      volumereplicationgroup.ramendr.openshift.io/busybox-drpc primary Primary

      NAME READY STATUS RESTARTS AGE
      pod/busybox-25-7b84d6dd5d-w5xvm 1/1 Terminating 0 12h

      [amagrawa@amagrawa ~]$ multi
      Now using project "multi-busybox-workloads-1" on server "https://api.amagrawa-c1-9jan.qe.rh-ocs.com:6443".
      NAME READY STATUS RESTARTS AGE
      pod/busybox-1-8cb4bd97c-gfpb9 1/1 Terminating 0 12h
      pod/busybox-11-67bd665b5b-76qnq 1/1 Terminating 0 12h
      pod/busybox-17-dd479cdc6-vl5cr 1/1 Terminating 0 12h
      pod/busybox-18-844ffc74b-fnvhs 1/1 Terminating 0 12h
      pod/busybox-2-689fdcdf54-hqmnl 1/1 Terminating 0 12h
      pod/busybox-3-7867d488fd-kxcxt 1/1 Terminating 0 12h
      pod/busybox-5-7486849478-q4cm2 1/1 Terminating 0 12h
      pod/busybox-9-7b458df98b-pk44m 1/1 Terminating 0 12h

      NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
      persistentvolumeclaim/busybox-pvc-1 Terminating pvc-caa78916-67f4-4c64-97c2-f91a65833b55 94Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-11 Terminating pvc-b9e4669f-2c93-4434-b3a8-86365e7b163c 33Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-17 Terminating pvc-60325860-6284-4449-b107-cdb69d203276 58Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-18 Terminating pvc-ad71cce9-0647-4d48-a9b6-b303d9aa45b8 123Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-2 Terminating pvc-42c9eb74-40fa-4969-b00d-d60caca2fce7 44Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-3 Terminating pvc-25d5efd8-9f8d-474c-b51b-eeff1fbc1120 76Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-5 Terminating pvc-572010e5-33de-4949-ba24-0767d1527d7d 107Gi RWO ocs-storagecluster-ceph-rbd 12h
      persistentvolumeclaim/busybox-pvc-9 Terminating pvc-80902c29-c054-4e0f-bb3c-dc82d5973b96 111Gi RWO ocs-storagecluster-ceph-rbd 12h

      For multi-
      [amagrawa@amagrawa ~]$ oc describe pod busybox-1-8cb4bd97c-gfpb9
      Events:
      Type Reason Age From Message
      ---- ------ ---- ---- -------
      Warning FailedKillPod 41m (x2 over 105m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = failed to stop container 88735d22070e0e5c2f5529f10f35ec4c47b67af6e28f1b2b3eaf0fc2c3c25574: context deadline exceeded", failed to "KillPodSandbox" for "588fd1bf-1289-4950-a85d-438ad68dd058" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container for pod sandbox c311925bd2df9879be5f665e29783f1ec29dfbf614e6b41b44beb031857d79d5: failed to stop container 88735d22070e0e5c2f5529f10f35ec4c47b67af6e28f1b2b3eaf0fc2c3c25574: context deadline exceeded"]
      Normal Killing 32m (x22 over 127m) kubelet Stopping container busybox
      Warning FailedKillPod 28m (x6 over 123m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "588fd1bf-1289-4950-a85d-438ad68dd058" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container for pod sandbox c311925bd2df9879be5f665e29783f1ec29dfbf614e6b41b44beb031857d79d5: failed to stop container 88735d22070e0e5c2f5529f10f35ec4c47b67af6e28f1b2b3eaf0fc2c3c25574: context deadline exceeded"]
      Warning FailedKillPod 19m (x5 over 68m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = failed to stop container 88735d22070e0e5c2f5529f10f35ec4c47b67af6e28f1b2b3eaf0fc2c3c25574: context deadline exceeded", failed to "KillPodSandbox" for "588fd1bf-1289-4950-a85d-438ad68dd058" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
      Warning FailedKillPod 72s (x15 over 114m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "588fd1bf-1289-4950-a85d-438ad68dd058" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]

      [amagrawa@amagrawa ~]$ oc describe pod busybox-18-844ffc74b-fnvhs
      Events:
      Type Reason Age From Message
      ---- ------ ---- ---- -------
      Normal Killing 33m (x22 over 128m) kubelet Stopping container busybox
      Warning FailedKillPod 20m (x5 over 65m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = failed to stop container 2c806e073e8d128404acab388e18dfb09799778825687d7472797fc0d9b6f544: context deadline exceeded", failed to "KillPodSandbox" for "00f6f3f4-cc4b-4aaf-a15b-228972b931d2" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
      Warning FailedKillPod 6m37s (x17 over 123m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "00f6f3f4-cc4b-4aaf-a15b-228972b931d2" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
      Warning FailedKillPod 2m6s (x6 over 119m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "00f6f3f4-cc4b-4aaf-a15b-228972b931d2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container for pod sandbox bfca86be022f02d32305e26cb82cad1fb5193413c30b892da5fc8138a05480e4: failed to stop container 2c806e073e8d128404acab388e18dfb09799778825687d7472797fc0d9b6f544: context deadline exceeded"]

      [amagrawa@amagrawa ~]$ oc describe pod busybox-5-7486849478-q4cm2
      Events:
      Type Reason Age From Message
      ---- ------ ---- ---- -------
      Normal Killing 37m (x22 over 132m) kubelet Stopping container busybox
      Warning FailedKillPod 10m (x5 over 69m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = failed to stop container 5ef528ecfa31d45d365c3acd8736a246bbaf934a91043ef40bce8a29d0e06fd5: context deadline exceeded", failed to "KillPodSandbox" for "e66a0edf-466d-44e4-915e-f2638e26f6f8" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
      Warning FailedKillPod 6m11s (x17 over 128m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "e66a0edf-466d-44e4-915e-f2638e26f6f8" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
      Warning FailedKillPod 100s (x7 over 123m) kubelet error killing pod: [failed to "KillContainer" for "busybox" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "e66a0edf-466d-44e4-915e-f2638e26f6f8" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container for pod sandbox 00bb5be525fd355c250b77bec5c751ee85adc3b270d9960a4a09ec7a5949016f: failed to stop container 5ef528ecfa31d45d365c3acd8736a246bbaf934a91043ef40bce8a29d0e06fd5: context deadline exceeded"]

      From Hub-

      [amagrawa@amagrawa ~]$ oc get drpc -o wide -A
      NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
      busybox-workloads-1 busybox-drpc 12h amagrawa-c1-9jan Deployed Completed 2023-01-10T05:26:00Z 7.117676937s True
      busybox-workloads-2 busybox-drpc 12h amagrawa-c1-9jan Deployed Completed 2023-01-10T05:26:17Z 1.027336301s True
      busybox-workloads-3 busybox-drpc 12h amagrawa-c1-9jan Deployed Completed 2023-01-10T05:26:07Z 18.051119687s True
      busybox-workloads-4 busybox-drpc 12h amagrawa-c1-9jan Deployed Completed 2023-01-10T05:26:30Z 2.071596923s True
      busybox-workloads-5 busybox-drpc 12h amagrawa-c1-9jan Deployed Completed 2023-01-10T05:26:20Z 19.052064486s True

      [amagrawa@amagrawa ~]$ oc get drpc -o yaml -A
      apiVersion: v1
      items:

      • apiVersion: ramendr.openshift.io/v1alpha1
        kind: DRPlacementControl
        metadata:
        creationTimestamp: "2023-01-10T05:25:36Z"
        finalizers:
      • drpc.ramendr.openshift.io/finalizer
        generation: 2
        labels:
        app: busybox-sample
        cluster.open-cluster-management.io/backup: resource
        name: busybox-drpc
        namespace: busybox-workloads-1
        resourceVersion: "2249557"
        uid: ca9e0f91-e2b9-42c9-8aef-9eac621eb7e1
        spec:
        drPolicyRef:
        name: my-drpolicy-15
        placementRef:
        kind: PlacementRule
        name: busybox-placement
        namespace: busybox-workloads-1
        preferredCluster: amagrawa-c1-9jan
        pvcSelector:
        matchLabels:
        appname: busybox_app1
        status:
        actionDuration: 7.117676937s
        actionStartTime: "2023-01-10T05:26:00Z"
        conditions:
      • lastTransitionTime: "2023-01-10T05:25:37Z"
        message: Initial deployment completed
        observedGeneration: 2
        reason: Deployed
        status: "True"
        type: Available
      • lastTransitionTime: "2023-01-10T05:25:37Z"
        message: Ready
        observedGeneration: 2
        reason: Success
        status: "True"
        type: PeerReady
        lastGroupSyncTime: "2023-01-10T18:00:08Z"
        lastUpdateTime: "2023-01-10T18:03:46Z"
        phase: Deployed
        preferredDecision:
        clusterName: amagrawa-c1-9jan
        clusterNamespace: amagrawa-c1-9jan
        progression: Completed
        resourceConditions:
        conditions:
      • lastTransitionTime: "2023-01-10T05:25:58Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      • lastTransitionTime: "2023-01-10T05:25:39Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      • lastTransitionTime: "2023-01-10T05:25:37Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      • lastTransitionTime: "2023-01-10T05:25:47Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
        resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: busybox-drpc
        namespace: busybox-workloads-1
        protectedpvcs:
      • busybox-pvc-15
      • busybox-pvc-1
      • busybox-pvc-10
      • busybox-pvc-11
      • busybox-pvc-12
      • busybox-pvc-13
      • busybox-pvc-14
      • busybox-pvc-7
      • busybox-pvc-8
      • busybox-pvc-19
      • busybox-pvc-6
      • busybox-pvc-16
      • busybox-pvc-17
      • busybox-pvc-4
      • busybox-pvc-5
      • busybox-pvc-18
      • busybox-pvc-20
      • busybox-pvc-3
      • busybox-pvc-9
      • busybox-pvc-2
      • apiVersion: ramendr.openshift.io/v1alpha1
        kind: DRPlacementControl
        metadata:
        creationTimestamp: "2023-01-10T05:25:48Z"
        deletionGracePeriodSeconds: 0
        deletionTimestamp: "2023-01-10T15:58:39Z"
        finalizers:
      • drpc.ramendr.openshift.io/finalizer
        generation: 3
        labels:
        app: busybox-sample
        cluster.open-cluster-management.io/backup: resource
        name: busybox-drpc
        namespace: busybox-workloads-2
        resourceVersion: "2089228"
        uid: e16b57f6-1ff8-434c-8dcb-a07bba5a7b1a
        spec:
        drPolicyRef:
        name: my-drpolicy-15
        placementRef:
        kind: PlacementRule
        name: busybox-placement
        namespace: busybox-workloads-2
        preferredCluster: amagrawa-c1-9jan
        pvcSelector:
        matchLabels:
        appname: busybox_app2
        status:
        actionDuration: 1.027336301s
        actionStartTime: "2023-01-10T05:26:17Z"
        conditions:
      • lastTransitionTime: "2023-01-10T05:25:48Z"
        message: Initial deployment completed
        observedGeneration: 2
        reason: Deployed
        status: "True"
        type: Available
      • lastTransitionTime: "2023-01-10T05:25:48Z"
        message: Ready
        observedGeneration: 2
        reason: Success
        status: "True"
        type: PeerReady
        lastGroupSyncTime: "2023-01-10T06:45:13Z"
        lastUpdateTime: "2023-01-10T15:49:19Z"
        phase: Deployed
        preferredDecision:
        clusterName: amagrawa-c1-9jan
        clusterNamespace: amagrawa-c1-9jan
        progression: Completed
        resourceConditions:
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:13Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      • lastTransitionTime: "2023-01-10T05:25:50Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      • lastTransitionTime: "2023-01-10T05:25:49Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      • lastTransitionTime: "2023-01-10T05:26:00Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
        resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: busybox-drpc
        namespace: busybox-workloads-2
        protectedpvcs:
      • busybox-pvc-23
      • busybox-pvc-32
      • busybox-pvc-37
      • busybox-pvc-27
      • busybox-pvc-34
      • busybox-pvc-35
      • busybox-pvc-36
      • busybox-pvc-39
      • busybox-pvc-21
      • busybox-pvc-22
      • busybox-pvc-24
      • busybox-pvc-40
      • busybox-pvc-30
      • busybox-pvc-33
      • busybox-pvc-38
      • busybox-pvc-25
      • busybox-pvc-26
      • busybox-pvc-29
      • busybox-pvc-28
      • busybox-pvc-31
      • apiVersion: ramendr.openshift.io/v1alpha1
        kind: DRPlacementControl
        metadata:
        creationTimestamp: "2023-01-10T05:25:55Z"
        finalizers:
      • drpc.ramendr.openshift.io/finalizer
        generation: 2
        labels:
        app: busybox-sample
        cluster.open-cluster-management.io/backup: resource
        name: busybox-drpc
        namespace: busybox-workloads-3
        resourceVersion: "2254557"
        uid: 931946d3-bc6c-4b27-9926-c6ed5e304576
        spec:
        drPolicyRef:
        name: my-drpolicy-5
        placementRef:
        kind: PlacementRule
        name: busybox-placement
        namespace: busybox-workloads-3
        preferredCluster: amagrawa-c1-9jan
        pvcSelector:
        matchLabels:
        appname: busybox_app3
        status:
        actionDuration: 18.051119687s
        actionStartTime: "2023-01-10T05:26:07Z"
        conditions:
      • lastTransitionTime: "2023-01-10T05:25:55Z"
        message: Initial deployment completed
        observedGeneration: 2
        reason: Deployed
        status: "True"
        type: Available
      • lastTransitionTime: "2023-01-10T05:25:55Z"
        message: Ready
        observedGeneration: 2
        reason: Success
        status: "True"
        type: PeerReady
        lastGroupSyncTime: "2023-01-10T18:05:00Z"
        lastUpdateTime: "2023-01-10T18:07:35Z"
        phase: Deployed
        preferredDecision:
        clusterName: amagrawa-c1-9jan
        clusterNamespace: amagrawa-c1-9jan
        progression: Completed
        resourceConditions:
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:41Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      • lastTransitionTime: "2023-01-10T05:25:58Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      • lastTransitionTime: "2023-01-10T05:25:56Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      • lastTransitionTime: "2023-01-10T05:26:09Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
        resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: busybox-drpc
        namespace: busybox-workloads-3
        protectedpvcs:
      • busybox-pvc-54
      • busybox-pvc-55
      • busybox-pvc-57
      • busybox-pvc-60
      • busybox-pvc-44
      • busybox-pvc-48
      • busybox-pvc-58
      • busybox-pvc-46
      • busybox-pvc-47
      • busybox-pvc-43
      • busybox-pvc-45
      • busybox-pvc-49
      • busybox-pvc-52
      • busybox-pvc-53
      • busybox-pvc-41
      • busybox-pvc-42
      • busybox-pvc-56
      • busybox-pvc-59
      • busybox-pvc-50
      • busybox-pvc-51
      • apiVersion: ramendr.openshift.io/v1alpha1
        kind: DRPlacementControl
        metadata:
        creationTimestamp: "2023-01-10T05:26:02Z"
        finalizers:
      • drpc.ramendr.openshift.io/finalizer
        generation: 2
        labels:
        app: busybox-sample
        cluster.open-cluster-management.io/backup: resource
        name: busybox-drpc
        namespace: busybox-workloads-4
        resourceVersion: "2247337"
        uid: 17b30f88-380e-4af3-bf9a-93aebdcd19a1
        spec:
        drPolicyRef:
        name: my-drpolicy-30
        placementRef:
        kind: PlacementRule
        name: busybox-placement
        namespace: busybox-workloads-4
        preferredCluster: amagrawa-c1-9jan
        pvcSelector:
        matchLabels:
        appname: busybox_app4
        status:
        actionDuration: 2.071596923s
        actionStartTime: "2023-01-10T05:26:30Z"
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:02Z"
        message: Initial deployment completed
        observedGeneration: 2
        reason: Deployed
        status: "True"
        type: Available
      • lastTransitionTime: "2023-01-10T05:26:02Z"
        message: Ready
        observedGeneration: 2
        reason: Success
        status: "True"
        type: PeerReady
        lastGroupSyncTime: "2023-01-10T17:30:00Z"
        lastUpdateTime: "2023-01-10T18:02:08Z"
        phase: Deployed
        preferredDecision:
        clusterName: amagrawa-c1-9jan
        clusterNamespace: amagrawa-c1-9jan
        progression: Completed
        resourceConditions:
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:45Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      • lastTransitionTime: "2023-01-10T05:26:04Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      • lastTransitionTime: "2023-01-10T05:26:02Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      • lastTransitionTime: "2023-01-10T05:26:20Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
        resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: busybox-drpc
        namespace: busybox-workloads-4
        protectedpvcs:
      • busybox-pvc-71
      • busybox-pvc-74
      • busybox-pvc-77
      • busybox-pvc-62
      • busybox-pvc-65
      • busybox-pvc-70
      • busybox-pvc-76
      • busybox-pvc-61
      • busybox-pvc-63
      • busybox-pvc-64
      • busybox-pvc-78
      • busybox-pvc-79
      • busybox-pvc-66
      • busybox-pvc-73
      • busybox-pvc-75
      • busybox-pvc-72
      • busybox-pvc-80
      • busybox-pvc-67
      • busybox-pvc-68
      • busybox-pvc-69
      • apiVersion: ramendr.openshift.io/v1alpha1
        kind: DRPlacementControl
        metadata:
        creationTimestamp: "2023-01-10T05:26:09Z"
        finalizers:
      • drpc.ramendr.openshift.io/finalizer
        generation: 2
        labels:
        app: busybox-sample
        cluster.open-cluster-management.io/backup: resource
        name: busybox-drpc
        namespace: busybox-workloads-5
        resourceVersion: "2254120"
        uid: 9d316306-8c3e-4316-8dac-d737b481f0cc
        spec:
        drPolicyRef:
        name: my-drpolicy-60
        placementRef:
        kind: PlacementRule
        name: busybox-placement
        namespace: busybox-workloads-5
        preferredCluster: amagrawa-c1-9jan
        pvcSelector:
        matchLabels:
        appname: busybox_app5
        status:
        actionDuration: 19.052064486s
        actionStartTime: "2023-01-10T05:26:20Z"
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:09Z"
        message: Initial deployment completed
        observedGeneration: 2
        reason: Deployed
        status: "True"
        type: Available
      • lastTransitionTime: "2023-01-10T05:26:09Z"
        message: Ready
        observedGeneration: 2
        reason: Success
        status: "True"
        type: PeerReady
        lastGroupSyncTime: "2023-01-10T17:00:01Z"
        lastUpdateTime: "2023-01-10T18:07:15Z"
        phase: Deployed
        preferredDecision:
        clusterName: amagrawa-c1-9jan
        clusterNamespace: amagrawa-c1-9jan
        progression: Completed
        resourceConditions:
        conditions:
      • lastTransitionTime: "2023-01-10T05:26:54Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      • lastTransitionTime: "2023-01-10T05:26:11Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      • lastTransitionTime: "2023-01-10T05:26:10Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      • lastTransitionTime: "2023-01-10T05:26:34Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
        resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: busybox-drpc
        namespace: busybox-workloads-5
        protectedpvcs:
      • busybox-pvc-100
      • busybox-pvc-81
      • busybox-pvc-95
      • busybox-pvc-83
      • busybox-pvc-87
      • busybox-pvc-88
      • busybox-pvc-89
      • busybox-pvc-90
      • busybox-pvc-92
      • busybox-pvc-93
      • busybox-pvc-94
      • busybox-pvc-98
      • busybox-pvc-99
      • busybox-pvc-82
      • busybox-pvc-84
      • busybox-pvc-97
      • busybox-pvc-85
      • busybox-pvc-86
      • busybox-pvc-91
      • busybox-pvc-96
        kind: List
        metadata:
        resourceVersion: ""
        selfLink: ""

      Expected results: No resources should be left in those namespaces which are deleted

      Additional info: Did not perform any operation on the setup

              mrajanna@redhat.com Madhu R
              amagrawa@redhat.com Aman Agrawal
              Aman Agrawal, Raghavendra Talur
              Krishnaram Karthick Ramdoss Krishnaram Karthick Ramdoss
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: