Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-11962

ManagedCluster takes 5m to delete

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • Server Foundation
    • False
    • None
    • False
    • Important
    • No

      Description of problem:

      ManagedCluster is deleted via GitOps. The CR on the hub takes ~5m to delete. 

      Reviewing the events seems to show some of the associated resources are re-created after the delete, then forceably deleted after 5 minutes:

      # oc get events -A --sort-by=.lastTimestamp | grep -i managedcluster
      
      open-cluster-management-hub            5m26s       Normal    ClusterRoleDeleted                        deployment/cluster-manager-registration-controller   Deleted ClusterRole.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17
      open-cluster-management-hub            5m25s       Normal    RoleBindingDeleted                        deployment/cluster-manager-registration-controller   Deleted RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17:registration -n cnfdf17
      multicluster-engine                    5m25s       Warning   ServiceAccountCreateFailed                deployment/managedcluster-import-controller-v2       Failed to create ServiceAccount/cnfdf17-bootstrap-sa -n cnfdf17: serviceaccounts "cnfdf17-bootstrap-sa" already exists
      open-cluster-management-hub            5m25s       Normal    ClusterRoleBindingDeleted                 deployment/cluster-manager-registration-controller   Deleted ClusterRoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17
      multicluster-engine                    5m24s       Normal    SecretCreated                             deployment/managedcluster-import-controller-v2       Created Secret/cnfdf17-import -n cnfdf17 because it was missing
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-managed-serviceaccount-deploy-0 is force deleted
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-work-manager-deploy-0 is force deleted
      multicluster-engine                    37s         Normal    ManagedClusterAddonForceDeleted           deployment/managedcluster-import-controller-v2       The managedClusterAddon cnfdf17/managed-serviceaccount is force deleted
      multicluster-engine                    37s         Normal    ManagedClusterAddonForceDeleted           deployment/managedcluster-import-controller-v2       The managedClusterAddon cnfdf17/config-policy-controller is force deleted
      multicluster-engine                    37s         Normal    ManagedClusterAddonForceDeleted           deployment/managedcluster-import-controller-v2       The managedClusterAddon cnfdf17/governance-policy-framework is force deleted
      multicluster-engine                    37s         Warning   ClusterRoleCreateFailed                   deployment/managedcluster-import-controller-v2       Failed to create ClusterRole.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17: clusterroles.rbac.authorization.k8s.io "system:open-cluster-management:managedcluster:bootstrap:cnfdf17" already exists
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-governance-policy-framework-pre-delete is force deleted
      multicluster-engine                    37s         Warning   ClusterRoleBindingCreateFailed            deployment/managedcluster-import-controller-v2       Failed to create ClusterRoleBinding.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17: clusterrolebindings.rbac.authorization.k8s.io "system:open-cluster-management:managedcluster:bootstrap:cnfdf17" already exists
      open-cluster-management-hub            37s         Normal    ManagedClusterAvailableConditionUpdated   deployment/cluster-manager-registration-controller   update managed cluster "cnfdf17" available condition to unknown, due to its lease is not updated constantly
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/cnfdf17-klusterlet is force deleted
      open-cluster-management-hub            37s         Normal    ManagedClusterAddOnStatusUpdated          deployment/cluster-manager-registration-controller   update addon "governance-policy-framework" status to unknown on managed cluster "cnfdf17"
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-governance-policy-framework-deploy-0 is force deleted
      open-cluster-management-hub            37s         Normal    ManagedClusterAddOnStatusUpdated          deployment/cluster-manager-registration-controller   update addon "config-policy-controller" status to unknown on managed cluster "cnfdf17"
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/cnfdf17-klusterlet-crds is force deleted
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-config-policy-controller-deploy-0 is force deleted
      multicluster-engine                    37s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/addon-config-policy-controller-pre-delete is force deleted
      multicluster-engine                    36s         Normal    ServiceAccountCreated                     deployment/managedcluster-import-controller-v2       Created ServiceAccount/cnfdf17-bootstrap-sa -n cnfdf17 because it was missing
      open-cluster-management-hub            36s         Normal    ManagedClusterGC                          deployment/cluster-manager-registration-controller   managed cluster cnfdf17 is deleting and the cluster rbac are deleted
      open-cluster-management-hub            36s         Normal    ManagedClusterAddOnStatusUpdated          deployment/cluster-manager-registration-controller   update addon "managed-serviceaccount" status to unknown on managed cluster "cnfdf17"
      multicluster-engine                    36s         Normal    ClusterRoleCreated                        deployment/managedcluster-import-controller-v2       Created ClusterRole.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17 because it was missing
      multicluster-engine                    36s         Normal    ClusterNamespaceDeletion                  deployment/managedcluster-import-controller-v2       cluster namespace cnfdf17 is deleted
      multicluster-engine                    36s         Normal    ManagedClusterAddonForceDeleted           deployment/managedcluster-import-controller-v2       The managedClusterAddon cnfdf17/managed-serviceaccount is force deleted
      multicluster-engine                    36s         Normal    ManagedClusterNamespaceDeleted            deployment/managedcluster-import-controller-v2       The managed cluster cnfdf17 namespace is deleted
      multicluster-engine                    36s         Normal    ManagedClusterFinalizerRemoved            deployment/managedcluster-import-controller-v2       The managed cluster cnfdf17 finalizer managedcluster-import-controller.open-cluster-management.io/manifestwork-cleanup is removed
      multicluster-engine                    36s         Normal    ManifestWorksForceDeleted                 deployment/managedcluster-import-controller-v2       The manifest work cnfdf17/cnfdf17-observability is force deleted
      multicluster-engine                    36s         Normal    ManagedClusterFinalizerRemoved            deployment/managedcluster-import-controller-v2       The managed cluster cnfdf17 finalizer managedcluster-import-controller.open-cluster-management.io/cleanup is removed
      open-cluster-management-hub            36s         Normal    RoleBindingDeleted                        deployment/cluster-manager-registration-controller   Deleted RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managedcluster:cnfdf17:work -n cnfdf17
      multicluster-engine                    35s         Normal    ClusterRoleBindingCreated                 deployment/managedcluster-import-controller-v2       Created ClusterRoleBinding.rbac.authorization.k8s.io/system:open-cluster-management:managedcluster:bootstrap:cnfdf17 because it was missing
       

      Version-Release number of selected component (if applicable):

      MCE: multicluster-engine.v2.5.3
      ACM: advanced-cluster-management.v2.10.3
      LSO: local-storage-operator.v4.15.0-202405161507
      TALM: topology-aware-lifecycle-manager.v4.15.0
      GitOps: openshift-gitops-operator.v1.12.3

      How reproducible:

      Reproduceable 2 out of 2 attempts

      Steps to Reproduce:

      1. Deploy SNO cluster (gitops) via assisted installer (ClusterDeployment, AgentClusterInstall, InfraEnv, bmh, ...).
      2. ACM has
        1. observability enabled
        2. Multiple policies bound to the deployed cluster 
      3. Including creation of ManagedCluster and KlusterletAddonConfig
      4. Delete the ManagedCluster and assisted installer artifacts (including cluster namepace)
      5. Deletion of ManagedCluster stalls for 5m after which it is deleted and all other artifacts are successfully deleted.

      Actual results:

      5m delay in deleting resources

      Expected results:

      Deletion on the order of seconds.

      Additional info:

              zyin@redhat.com Zhiwei Yin
              rhn-support-imiller Ian Miller
              Hui Chen Hui Chen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: