Description of problem (please be detailed as possible and provide log
snippests):
Hi ,

I am trying to do Recovery to replace cluster with MDR of Discovered Apps by following mentioned steps.

[1]
https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.15/html/configuring_openshift_data_foundation_disaster_recovery_for_openshift_workloads/metro-dr-solution#recovering-to-a-replacement-cluster-with-mdr_manage-mdr

Followed below steps to disable DR of Discovered apps.
[2]
https://docs.google.com/document/d/1BoqbEqDBLCQZXp2qvd7Hw5mvg59njv1dqlrH6Hy7L58/edit#heading=h.1yx58g1ouy2

When I am trying to delete DRPC it get stuck in deleting state.Below is the drpc yaml output.

➜ hub oc get drpc imperative-1 -n openshift-dr-ops -oyaml
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
annotations:
drplacementcontrol.ramendr.openshift.io/app-namespace: openshift-dr-ops
drplacementcontrol.ramendr.openshift.io/last-app-deployment-cluster: asagare-sec
creationTimestamp: "2024-07-04T07:39:18Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2024-07-08T05:25:48Z"
finalizers:

• drpc.ramendr.openshift.io/finalizer
  generation: 6
  labels:
  cluster.open-cluster-management.io/backup: ramen
  name: imperative-1
  namespace: openshift-dr-ops
  ownerReferences:
• apiVersion: cluster.open-cluster-management.io/v1beta1
  blockOwnerDeletion: true
  controller: true
  kind: Placement
  name: imperative-1-placement-1
  uid: 0e05a1af-f89a-48b1-aa0f-28cb89b14344
  resourceVersion: "12464211"
  uid: 2f372e03-fb46-4ec5-96d2-e517f9f88d09
  spec:
  action: Failover
  drPolicyRef:
  apiVersion: ramendr.openshift.io/v1alpha1
  kind: DRPolicy
  name: odr-policy-mdr
  failoverCluster: asagare-sec
  kubeObjectProtection:
  captureInterval: 2m0s
  kubeObjectSelector:
  matchExpressions:
• key: appname
  operator: In
  values:
• busybox
  placementRef:
  apiVersion: cluster.open-cluster-management.io/v1beta1
  kind: Placement
  name: imperative-1-placement-1
  namespace: openshift-dr-ops
  preferredCluster: asagare-pri
  protectedNamespaces:
• busybox-discovered
  pvcSelector:
  matchExpressions:
• key: appname
  operator: In
  values:
• busybox
  status:
  actionStartTime: "2024-07-05T11:28:26Z"
  conditions:
• lastTransitionTime: "2024-07-05T11:28:27Z"
  message: Completed
  observedGeneration: 5
  reason: FailedOver
  status: "True"
  type: Available
• lastTransitionTime: "2024-07-05T11:28:26Z"
  message: cleaning secondaries
  observedGeneration: 5
  reason: Cleaning
  status: "False"
  type: PeerReady
• lastTransitionTime: "2024-07-05T11:29:57Z"
  message: VolumeReplicationGroup (openshift-dr-ops/imperative-1) on cluster asagare-sec
  is reporting errors (Cluster data of one or more PVs are unprotectedVRG Kube
  object protect errorunable to ListKeys in DeleteObjects from endpoint https://s3-openshift-storage.apps.asagare-pri.qe.rh-ocs.com
  bucket odrbucket-84427fcbc7ce keyPrefix openshift-dr-ops/imperative-1/kube-objects/1/velero/backups/)
  protecting workload resources, retrying till ClusterDataProtected condition
  is met
  observedGeneration: 5
  reason: Error
  status: "False"
  type: Protected
  lastKubeObjectProtectionTime: "2024-07-05T11:20:51Z"
  lastUpdateTime: "2024-07-07T19:49:50Z"
  observedGeneration: 6
  phase: Deleting
  preferredDecision:
  clusterName: asagare-pri
  clusterNamespace: asagare-pri
  progression: Deleting
  resourceConditions:
  conditions:
• lastTransitionTime: "2024-07-05T11:29:52Z"
  message: PVCs in the VolumeReplicationGroup are ready for use
  observedGeneration: 1
  reason: Ready
  status: "True"
  type: DataReady
• lastTransitionTime: "2024-07-05T11:29:52Z"
  message: VolumeReplicationGroup is replicating
  observedGeneration: 1
  reason: Replicating
  status: "False"
  type: DataProtected
• lastTransitionTime: "2024-07-05T11:29:16Z"
  message: Restored PVs and PVCs
  observedGeneration: 1
  reason: Restored
  status: "True"
  type: ClusterDataReady
• lastTransitionTime: "2024-07-05T11:29:52Z"
  message: Cluster data of one or more PVs are unprotectedVRG Kube object protect
  errorunable to ListKeys in DeleteObjects from endpoint https://s3-openshift-storage.apps.asagare-pri.qe.rh-ocs.com
  bucket odrbucket-84427fcbc7ce keyPrefix openshift-dr-ops/imperative-1/kube-objects/1/velero/backups/
  observedGeneration: 1
  reason: UploadError
  status: "False"
  type: ClusterDataProtected
  resourceMeta:
  generation: 1
  kind: VolumeReplicationGroup
  name: imperative-1
  namespace: openshift-dr-ops
  protectedpvcs:
• busybox-pvc
  resourceVersion: "7885562"

Version of all relevant components (if applicable):

OCP: 4.16.0-0.nightly-2024-06-27-091410
ODF: 4.16.0-134
ACM: 2.11.0-140
CEPH: 18.2.1-194.el9cp (04a992766839cd3207877e518a1238cdbac3787e) reef (stable)
OADP: 1.4.0
GitOps: 1.12.4

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?
yes

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. I have upgraded MDR setup from 4.15.4 to 4.16
2. Deployed and applied DRpolicy to Discovered apps on primary cluster 1
3. Powered off primary cluster.
4. Followed steps for replace cluster mentioned in doc[1]
5. Disabled DR for protected apps using doc[2]
6. Drpc deletion stuck in deleting state.

Actual results:
drpc deletion stuck in deleting state.

Expected results:
drpc should get deleted.

Additional info: