Observed LCA operator reporting error message "failed to remove backup label" when Prep stage failed followed by Abort operation/Setting back IBU CR to Idle stage. 

//
2024-04-25T19:53:32Z    INFO    controllers.ImageBasedUpgrade.BackupRestore    Backup deletion request has sent    {"backup": "acm-klusterlet"}
2024-04-25T19:53:32Z    ERROR    controllers.ImageBasedUpgrade.BackupRestore    failed to remove backup label    {"name": "klusterlet", "namespace": "", "resource": "clusterroles", "group": "rbac.authorization.k8s.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}
//

TALM bundle: 4.16.0-23
Seed SNO Cluster: OCP 4.16.0-0.nightly-2024-04-16-195622
Target SNO cluster before upgrade : OCP 4.14.21
Target SNO cluster after upgrade : OCP 4.16.0-0.nightly-2024-04-16-195622
Target Hub cluster: OCP 4.16.0-0.nightly-2024-04-16-195622
ACM: 2.10.2
GitOps: 1.12.0
LCA operator bundle container version: 4.16.0-36
LCA Operator version: 4.16.0-31
Recert image version: 4.16.0-8

Encountered 3 times, potential frequent occurrence issue.

[1] Make sure all policies reporting  compliant except site specific config policy (which has destination [new labels] extra manifests) on target sno cluster.
{{
[kni@registry.kni-qe-53 ~]$ oc get policy -A
NAMESPACE   NAME                                          REMEDIATION ACTION   COMPLIANCE STATE   AGE
helix57     ztp-common.common-config-policy               inform               Compliant          32h
helix57     ztp-common.common-lca-subscriptions-policy    inform               Compliant          32h
helix57     ztp-common.common-oadp-subscriptions-policy   inform               Compliant          32h
helix57     ztp-common.common-subscriptions-policy        inform               Compliant          32h
helix57     ztp-common.oadp-cm-common-policies                                 Compliant          3d23h
helix57     ztp-group.group-du-sno-config-policy          inform               Compliant          32h
helix57     ztp-group.group-ibu-finalize-policy           inform               NonCompliant       3d23h
helix57     ztp-group.group-ibu-prep-policy               inform               NonCompliant       3d23h
helix57     ztp-group.group-ibu-rollback-policy           inform               NonCompliant       3d23h
helix57     ztp-group.group-ibu-upgrade-policy            inform               NonCompliant       3d23h
helix57     ztp-site.helix57-config-policy                inform               NonCompliant       32h
helix57     ztp-site.helix57-oadp-config-policy           inform               Compliant          32h
[kni@registry.kni-qe-53 ~]$ 
}}
[3] Apply IBU Prep CGU via TALM from target hub cluster.
[4] Make sure IBU CR reporting Prep stage failed.
[5] Apply IBU finalize policy to set back IBU CR to Idle stage from target hub cluster. 

========================================================    
2024-04-25T19:53:31Z	INFO	controllers.ImageBasedUpgrade	Start reconciling IBU	{"name": {"name":"upgrade"}}2024-04-25T19:53:31Z	INFO	controllers.ImageBasedUpgrade	Loaded IBU	{"name": {"name":"upgrade"}, "version": "1049225", "desired stage": "Idle"}INFO[163160] Executing /usr/bin/env with args [-- rpm-ostree status --json] 2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade	Starting handleAbort2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade	Terminating precaching worker thread, will wait up to 30 seconds2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade	Cleaning up staterootINFO[163160] Executing /usr/bin/env with args [-- rpm-ostree status --json] 2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade	Cleaning up precache2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade	Cleaning up DeleteBackupRequest and Backup CRs2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade.BackupRestore	All DeleteBackupRequest CRs have been deleted2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade.BackupRestore	Backup deletion request has sent	{"backup": "acm-klusterlet"}2024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlet", "namespace": "", "resource": "clusterroles", "group": "rbac.authorization.k8s.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlet", "namespace": "open-cluster-management-agent", "resource": "deployments", "group": "apps", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "bootstrap-hub-kubeconfig", "namespace": "open-cluster-management-agent", "resource": "secrets", "group": "", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlet", "namespace": "open-cluster-management-agent", "resource": "serviceaccounts", "group": "", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "open-cluster-management:klusterlet-admin-aggregate-clusterrole", "namespace": "", "resource": "clusterroles", "group": "rbac.authorization.k8s.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlet", "namespace": "", "resource": "clusterrolebindings", "group": "rbac.authorization.k8s.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlet", "namespace": "", "resource": "klusterlets", "group": "operator.open-cluster-management.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "klusterlets.operator.open-cluster-management.io", "namespace": "", "resource": "customresourcedefinitions", "group": "apiextensions.k8s.io", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	ERROR	controllers.ImageBasedUpgrade.BackupRestore	failed to remove backup label	{"name": "open-cluster-management-image-pull-credentials", "namespace": "open-cluster-management-agent", "resource": "secrets", "group": "", "version": "v1", "error": "failed to patch object: the server rejected our request due to an error in our request"}github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).cleanupBackupLabels	/remote-source/app/internal/backuprestore/backup.go:232github.com/openshift-kni/lifecycle-agent/internal/backuprestore.(*BRHandler).CleanupBackups	/remote-source/app/internal/backuprestore/backup.go:392github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).cleanup	/remote-source/app/controllers/idle_handlers.go:162github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbort	/remote-source/app/controllers/idle_handlers.go:42github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleAbortOrFinalize	/remote-source/app/controllers/ibu_controller.go:307github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).handleStage	/remote-source/app/controllers/ibu_controller.go:289github.com/openshift-kni/lifecycle-agent/controllers.(*ImageBasedUpgradeReconciler).Reconcile	/remote-source/app/controllers/ibu_controller.go:216sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2272024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade.BackupRestore	Waiting for DeleteBackupRequest to be processed	{"deleteBackupRequest": "acm-klusterlet", "phase": "InProgress"}2024-04-25T19:53:32Z	INFO	controllers.ImageBasedUpgrade.BackupRestore	Waiting for DeleteBackupRequest to be processed	{"deleteBackupRequest": "acm-klusterlet", "phase": "InProgress"}2024-04-25T19:53:33Z	INFO	controllers.ImageBasedUpgrade.BackupRestore	All Backup CRs have been deleted successfully2024-04-25T19:53:33Z	INFO	controllers.ImageBasedUpgrade	Cleaning up IBU files2024-04-25T19:53:33Z	INFO	controllers.ImageBasedUpgrade	Finished handleAbort successfully2024-04-25T19:53:33Z	INFO	controllers.ImageBasedUpgrade	Finish reconciling IBU	{"name": {"name":"upgrade"}, "requeueRightAway": false}
========================================

=========================================
2024-04-24T11:28:07Z    INFO    controllers.ImageBasedUpgrade    Start reconciling IBU    {"name": {"name":"upgrade"}}
2024-04-24T11:28:07Z    INFO    controllers.ImageBasedUpgrade    Loaded IBU    {"name": {"name":"upgrade"}, "version": "687644", "desired stage": "Idle"}
INFO[46435] Executing /usr/bin/env with args [-- rpm-ostree status --json] 
2024-04-24T11:28:07Z    INFO    controllers.ImageBasedUpgrade    Starting handleAbort
2024-04-24T11:28:07Z    INFO    controllers.ImageBasedUpgrade    Terminating precaching worker thread, will wait up to 30 seconds
2024-04-24T11:28:07Z    INFO    controllers.ImageBasedUpgrade    Cleaning up stateroot
INFO[46436] Executing /usr/bin/env with args [-- rpm-ostree status --json] 
INFO[46436] Executing /usr/bin/env with args [-- rpm-ostree status --json] 
INFO[46436] Executing /usr/bin/env with args [-- ostree admin undeploy 1] 
INFO[46439] Executing /usr/bin/env with args [-- bash -c unshare -m /bin/sh -c "mount -o remount,rw /sysroot && rm -rf /ostree/deploy/rhcos_4.16.0_0.nightly_2024_04_16_195622"] 
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Cleaning up precache
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Cleaning up DeleteBackupRequest and Backup CRs
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade.BackupRestore    All DeleteBackupRequest CRs have been deleted
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade.BackupRestore    All Backup CRs have been deleted successfully
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Cleaning up IBU files
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Finished handleAbort successfully
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Finish reconciling IBU    {"name": {"name":"upgrade"}, "requeueRightAway": false}
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Start reconciling IBU    {"name": {"name":"upgrade"}}
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Loaded IBU    {"name": {"name":"upgrade"}, "version": "687673", "desired stage": "Idle"}
INFO[46439] Executing /usr/bin/env with args [-- rpm-ostree status --json] 
2024-04-24T11:28:11Z    INFO    controllers.ImageBasedUpgrade    Finish reconciling IBU    {"name": {"name":"upgrade"}, "requeueRightAway": false}
=================================

Valid Backup CR (acm-klusterlet) deleted and IBU CR set back to Idle stage successfully.    


//SUT logs snippet before the test//
[kni@registry.kni-qe-53 ~]$ oc get policy -A
NAMESPACE   NAME                                          REMEDIATION ACTION   COMPLIANCE STATE   AGE
helix57     ztp-common.common-config-policy               inform               Compliant          32h
helix57     ztp-common.common-lca-subscriptions-policy    inform               Compliant          32h
helix57     ztp-common.common-oadp-subscriptions-policy   inform               Compliant          32h
helix57     ztp-common.common-subscriptions-policy        inform               Compliant          32h
helix57     ztp-common.oadp-cm-common-policies                                 Compliant          4d
helix57     ztp-group.group-du-sno-config-policy          inform               Compliant          32h
helix57     ztp-group.group-ibu-finalize-policy           inform               NonCompliant       4d
helix57     ztp-group.group-ibu-prep-policy               inform               NonCompliant       4d
helix57     ztp-group.group-ibu-rollback-policy           inform               NonCompliant       4d
helix57     ztp-group.group-ibu-upgrade-policy            inform               NonCompliant       4d
helix57     ztp-site.helix57-config-policy                inform               NonCompliant       32h
helix57     ztp-site.helix57-oadp-config-policy           inform               Compliant          32h
[kni@registry.kni-qe-53 ~]$ oc get ibu
NAME      AGE   DESIRED STAGE   STATE    DETAILS
upgrade   45h   Prep            Failed   prep stage failed with error from prep worker: encountered error while running prep-stage worker goroutine: failed to validate IBU spec: failed to validate oadp configMap: Failed to cleanup stale Backups: all DeleteBackupRequests are processed, but backup localvolume is not deleted
[kni@registry.kni-qe-53 ~]$ oc -n openshift-adp  get backups
    NAME             AGE
acm-klusterlet   21h
[kni@registry.kni-qe-53 ~]$ oc -n openshift-adp  get backups acm-klusterlet -o yaml | grep -i clusterid
    config.openshift.io/clusterID: 6b0b914f-bb0a-4f27-914c-e3dce9a67e91
[kni@registry.kni-qe-53 ~]$ oc get clusterversions.config.openshift.io 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.21   True        False         6d3h    Cluster version is 4.14.21
[kni@registry.kni-qe-53 ~]$ oc get clusterversions.config.openshift.io -o yaml | grep -i clusterid
    clusterID: 6b0b914f-bb0a-4f27-914c-e3dce9a67e91
[kni@registry.kni-qe-53 ~]$ 
[kni@registry.kni-qe-53 ~]$ oc get no
NAME                              STATUS   ROLES                         AGE    VERSION
helix57.lab.eng.rdu2.redhat.com   Ready    control-plane,master,worker   6d4h   v1.27.11+749fe1d
[kni@registry.kni-qe-53 ~]$ 

//SUT logs snippet after the test//
[kni@registry.kni-qe-53 ~]$ oc -n openshift-adp  get backups acm-klusterlet -o yaml | grep -i clusterid
Error from server (NotFound): backups.velero.io "acm-klusterlet" not found
[kni@registry.kni-qe-53 ~]$ oc -n openshift-adp  get backups
No resources found in openshift-adp namespace.
[kni@registry.kni-qe-53 ~]$ oc get policies.policy.open-cluster-management.io -A
NAMESPACE   NAME                                          REMEDIATION ACTION   COMPLIANCE STATE   AGE
helix57     ztp-common.common-config-policy               inform               Compliant          32h
helix57     ztp-common.common-lca-subscriptions-policy    inform               Compliant          32h
helix57     ztp-common.common-oadp-subscriptions-policy   inform               Compliant          32h
helix57     ztp-common.common-subscriptions-policy        inform               Compliant          32h
helix57     ztp-common.oadp-cm-common-policies                                 Compliant          4d
helix57     ztp-group.group-du-sno-config-policy          inform               Compliant          32h
helix57     ztp-group.group-ibu-finalize-policy           inform               Compliant          4d
helix57     ztp-group.group-ibu-prep-policy               inform               NonCompliant       4d
helix57     ztp-group.group-ibu-rollback-policy           inform               NonCompliant       4d
helix57     ztp-group.group-ibu-upgrade-policy            inform               NonCompliant       4d
helix57     ztp-site.helix57-config-policy                inform               NonCompliant       32h
helix57     ztp-site.helix57-oadp-config-policy           inform               Compliant          32h
[kni@registry.kni-qe-53 ~]$ oc get ibu
NAME      AGE   DESIRED STAGE   STATE   DETAILS
upgrade   45h   Idle            Idle    Idle
[kni@registry.kni-qe-53 ~]$ oc get ibu upgrade -o yaml
apiVersion: lca.openshift.io/v1alpha1
kind: ImageBasedUpgrade
metadata:
  creationTimestamp: "2024-04-23T22:34:11Z"
  generation: 10
  name: upgrade
  resourceVersion: "1049241"
  uid: 93b8f038-1462-4de7-ab18-6b4f83734d70
spec:
  autoRollbackOnFailure: {}
  oadpContent:
  - name: oadp-cm
    namespace: openshift-adp
  seedImageRef:
    image: quay.io/rh_ee_pmohanra/ibu-seed-sno0:4.16.0-0.nightly-2024-04-16-195622
    pullSecretRef:
      name: seed-pull-secret
    version: 4.16.0-0.nightly-2024-04-16-195622
  stage: Idle
status:
  conditions:
  - lastTransitionTime: "2024-04-25T19:53:33Z"
    message: Idle
    observedGeneration: 10
    reason: Idle
    status: "True"
    type: Idle
  observedGeneration: 10
  validNextStages:
  - Prep
[kni@registry.kni-qe-53 ~]$