-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
False
-
None
-
False
-
-
-
-
Important
-
No
Description of problem:
ManagedCluster is deleted via GitOps. The CR on the hub takes ~5m to delete.
Reviewing the events seems to show some of the associated resources are re-created after the delete, then forceably deleted after 5 minutes:
# oc get events -A --sort-by=.lastTimestamp | grep -i managedcluster open-cluster-management-hub 5m26s Normal ClusterRoleDeleted deployment/cluster-manager-registration-controller Deleted ClusterRole.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17 open-cluster-management-hub 5m25s Normal RoleBindingDeleted deployment/cluster-manager-registration-controller Deleted RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17:registration -n cnfdf17 multicluster-engine 5m25s Warning ServiceAccountCreateFailed deployment/managedcluster-import-controller-v2 Failed to create ServiceAccount/cnfdf17-bootstrap-sa -n cnfdf17: serviceaccounts "cnfdf17-bootstrap-sa" already exists open-cluster-management-hub 5m25s Normal ClusterRoleBindingDeleted deployment/cluster-manager-registration-controller Deleted ClusterRoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17 multicluster-engine 5m24s Normal SecretCreated deployment/managedcluster-import-controller-v2 Created Secret/cnfdf17-import -n cnfdf17 because it was missing multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-managed-serviceaccount-deploy-0 is force deleted multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-work-manager-deploy-0 is force deleted multicluster-engine 37s Normal ManagedClusterAddonForceDeleted deployment/managedcluster-import-controller-v2 The managedClusterAddon cnfdf17/managed-serviceaccount is force deleted multicluster-engine 37s Normal ManagedClusterAddonForceDeleted deployment/managedcluster-import-controller-v2 The managedClusterAddon cnfdf17/config-policy-controller is force deleted multicluster-engine 37s Normal ManagedClusterAddonForceDeleted deployment/managedcluster-import-controller-v2 The managedClusterAddon cnfdf17/governance-policy-framework is force deleted multicluster-engine 37s Warning ClusterRoleCreateFailed deployment/managedcluster-import-controller-v2 Failed to create ClusterRole.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17: clusterroles.rbac.authorization.k8s.io "system:open-cluster-management:managedcluster:bootstrap:cnfdf17" already exists multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-governance-policy-framework-pre-delete is force deleted multicluster-engine 37s Warning ClusterRoleBindingCreateFailed deployment/managedcluster-import-controller-v2 Failed to create ClusterRoleBinding.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17: clusterrolebindings.rbac.authorization.k8s.io "system:open-cluster-management:managedcluster:bootstrap:cnfdf17" already exists open-cluster-management-hub 37s Normal ManagedClusterAvailableConditionUpdated deployment/cluster-manager-registration-controller update managed cluster "cnfdf17" available condition to unknown, due to its lease is not updated constantly multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/cnfdf17-klusterlet is force deleted open-cluster-management-hub 37s Normal ManagedClusterAddOnStatusUpdated deployment/cluster-manager-registration-controller update addon "governance-policy-framework" status to unknown on managed cluster "cnfdf17" multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-governance-policy-framework-deploy-0 is force deleted open-cluster-management-hub 37s Normal ManagedClusterAddOnStatusUpdated deployment/cluster-manager-registration-controller update addon "config-policy-controller" status to unknown on managed cluster "cnfdf17" multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/cnfdf17-klusterlet-crds is force deleted multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-config-policy-controller-deploy-0 is force deleted multicluster-engine 37s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/addon-config-policy-controller-pre-delete is force deleted multicluster-engine 36s Normal ServiceAccountCreated deployment/managedcluster-import-controller-v2 Created ServiceAccount/cnfdf17-bootstrap-sa -n cnfdf17 because it was missing open-cluster-management-hub 36s Normal ManagedClusterGC deployment/cluster-manager-registration-controller managed cluster cnfdf17 is deleting and the cluster rbac are deleted open-cluster-management-hub 36s Normal ManagedClusterAddOnStatusUpdated deployment/cluster-manager-registration-controller update addon "managed-serviceaccount" status to unknown on managed cluster "cnfdf17" multicluster-engine 36s Normal ClusterRoleCreated deployment/managedcluster-import-controller-v2 Created ClusterRole.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17 because it was missing multicluster-engine 36s Normal ClusterNamespaceDeletion deployment/managedcluster-import-controller-v2 cluster namespace cnfdf17 is deleted multicluster-engine 36s Normal ManagedClusterAddonForceDeleted deployment/managedcluster-import-controller-v2 The managedClusterAddon cnfdf17/managed-serviceaccount is force deleted multicluster-engine 36s Normal ManagedClusterNamespaceDeleted deployment/managedcluster-import-controller-v2 The managed cluster cnfdf17 namespace is deleted multicluster-engine 36s Normal ManagedClusterFinalizerRemoved deployment/managedcluster-import-controller-v2 The managed cluster cnfdf17 finalizer managedcluster-import-controller.open-cluster-management.io/manifestwork-cleanup is removed multicluster-engine 36s Normal ManifestWorksForceDeleted deployment/managedcluster-import-controller-v2 The manifest work cnfdf17/cnfdf17-observability is force deleted multicluster-engine 36s Normal ManagedClusterFinalizerRemoved deployment/managedcluster-import-controller-v2 The managed cluster cnfdf17 finalizer managedcluster-import-controller.open-cluster-management.io/cleanup is removed open-cluster-management-hub 36s Normal RoleBindingDeleted deployment/cluster-manager-registration-controller Deleted RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17:work -n cnfdf17 multicluster-engine 35s Normal ClusterRoleBindingCreated deployment/managedcluster-import-controller-v2 Created ClusterRoleBinding.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17 because it was missing
Version-Release number of selected component (if applicable):
MCE: multicluster-engine.v2.5.3 ACM: advanced-cluster-management.v2.10.3 LSO: local-storage-operator.v4.15.0-202405161507 TALM: topology-aware-lifecycle-manager.v4.15.0 GitOps: openshift-gitops-operator.v1.12.3
How reproducible:
Reproduceable 2 out of 2 attempts
Steps to Reproduce:
- Deploy SNO cluster (gitops) via assisted installer (ClusterDeployment, AgentClusterInstall, InfraEnv, bmh, ...).
- ACM has
- observability enabled
- Multiple policies bound to the deployed cluster
- Including creation of ManagedCluster and KlusterletAddonConfig
- Delete the ManagedCluster and assisted installer artifacts (including cluster namepace)
- Deletion of ManagedCluster stalls for 5m after which it is deleted and all other artifacts are successfully deleted.
Actual results:
5m delay in deleting resources
Expected results:
Deletion on the order of seconds.