Uploaded image for project: 'Data Foundation Bugs'
  1. Data Foundation Bugs
  2. DFBUGS-91

[2318506] Cannot delete the namespace after uninstalling ocs-client-operator and other dependency operators

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • odf-4.18
    • odf-4.17
    • ocs-operator
    • None
    • None

      Description of problem:
      After uninstalling ocs-client-operator and its dependent operators, the namespace 'openshift-storage-client' cannot be deleted. This is the namespace where the operator was installed. This is tested on client cluster in a provider-client setup.

      % oc get ns openshift-storage-client -o yaml
      apiVersion: v1
      kind: Namespace
      metadata:
      annotations:
      openshift.io/sa.scc.mcs: s0:c25,c20
      openshift.io/sa.scc.supplemental-groups: 1000640000/10000
      openshift.io/sa.scc.uid-range: 1000640000/10000
      creationTimestamp: "2024-10-14T05:04:29Z"
      deletionTimestamp: "2024-10-14T09:12:00Z"
      labels:
      kubernetes.io/metadata.name: openshift-storage-client
      pod-security.kubernetes.io/audit: restricted
      pod-security.kubernetes.io/audit-version: v1.24
      pod-security.kubernetes.io/enforce: restricted
      pod-security.kubernetes.io/enforce-version: v1.24
      pod-security.kubernetes.io/warn: restricted
      pod-security.kubernetes.io/warn-version: v1.24
      security.openshift.io/scc.podSecurityLabelSync: "true"
      name: openshift-storage-client
      resourceVersion: "87170"
      uid: 059a125c-da2c-4829-a0c1-f31ba9354a8f
      spec:
      finalizers:

      • kubernetes
        status:
        conditions:
      • lastTransitionTime: "2024-10-14T09:12:07Z"
        message: All resources successfully discovered
        reason: ResourcesDiscovered
        status: "False"
        type: NamespaceDeletionDiscoveryFailure
      • lastTransitionTime: "2024-10-14T09:12:07Z"
        message: All legacy kube types successfully parsed
        reason: ParsedGroupVersions
        status: "False"
        type: NamespaceDeletionGroupVersionParsingFailure
      • lastTransitionTime: "2024-10-14T09:12:07Z"
        message: 'Failed to delete all resource types, 1 remaining: admission webhook
        "admissionwebhook.noobaa.io" denied the request: Noobaa cleanup policy is not
        set, blocking Noobaa deletion'
        reason: ContentDeletionFailed
        status: "True"
        type: NamespaceDeletionContentFailure
      • lastTransitionTime: "2024-10-14T09:12:07Z"
        message: All content successfully removed
        reason: ContentRemoved
        status: "False"
        type: NamespaceContentRemaining
      • lastTransitionTime: "2024-10-14T09:12:07Z"
        message: All content-preserving finalizers finished
        reason: ContentHasNoFinalizers
        status: "False"
        type: NamespaceFinalizersRemaining
        phase: Terminating

      % oc get Noobaa
      NAME S3-ENDPOINTS STS-ENDPOINTS SYSLOG-ENDPOINTS IMAGE PHASE AGE
      noobaa-remote Rejected 77m

      % oc get Noobaa noobaa-remote -o yaml -n openshift-storage-client
      apiVersion: noobaa.io/v1alpha1
      kind: NooBaa
      metadata:
      annotations:
      remote-client-noobaa: "true"
      creationTimestamp: "2024-10-14T05:59:03Z"
      deletionGracePeriodSeconds: 0
      deletionTimestamp: "2024-10-14T07:12:26Z"
      finalizers:

      • noobaa.io/graceful_finalizer
        generation: 3
        labels:
        app: noobaa
        name: noobaa-remote
        namespace: openshift-storage-client
        ownerReferences:
      • apiVersion: ocs.openshift.io/v1alpha1
        kind: StorageClient
        name: storage-client
        uid: 6f40c83c-1fb9-4691-81d6-8a5ff6a6f0e6
        resourceVersion: "51471"
        uid: cf6140a5-44a8-4781-a9f4-fa70284fb061
        spec:
        autoscaler: {}
        bucketLogging: {}
        cleanupPolicy: {}
        joinSecret:
        name: noobaa-remote-join-secret
        namespace: openshift-storage-client
        loadBalancerSourceSubnets: {}
        security:
        kms: {}
        status:
        accounts:
        admin:
        secretRef: {}
        conditions:
      • lastHeartbeatTime: "2024-10-14T05:59:03Z"
        lastTransitionTime: "2024-10-14T05:59:03Z"
        message: Invalid system name "noobaa-remote" expected "noobaa"
        reason: InvalidSystemName
        status: Unknown
        type: Available
      • lastHeartbeatTime: "2024-10-14T05:59:03Z"
        lastTransitionTime: "2024-10-14T05:59:03Z"
        message: Invalid system name "noobaa-remote" expected "noobaa"
        reason: InvalidSystemName
        status: "False"
        type: Progressing
      • lastHeartbeatTime: "2024-10-14T05:59:03Z"
        lastTransitionTime: "2024-10-14T05:59:03Z"
        message: Invalid system name "noobaa-remote" expected "noobaa"
        reason: InvalidSystemName
        status: "True"
        type: Degraded
      • lastHeartbeatTime: "2024-10-14T05:59:03Z"
        lastTransitionTime: "2024-10-14T05:59:03Z"
        message: Invalid system name "noobaa-remote" expected "noobaa"
        reason: InvalidSystemName
        status: Unknown
        type: Upgradeable
        observedGeneration: 2
        phase: Rejected
        readme: "\n\tERROR: NooBaa operator cannot reconcile this system spec.\n\n\tCheck
        out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl
        -n openshift-storage-client describe noobaa\n\t\tkubectl -n openshift-storage-client
        get noobaa -o yaml\n\t\tkubectl -n openshift-storage-client get events --sort-by=metadata.creationTimestamp\n\n\tIn
        order to retry, edit the system spec and the operator is watching and will be
        notified.\n\n\tNooBaa Core Version: master-20240520\n\tNooBaa Operator Version:
        5.17.0\n"
        services:
        serviceMgmt: {}
        serviceS3: {}
        serviceSts: {}
        serviceSyslog: {}

      Subs and CSVs are deleted using CLI. There is no installed operators in UI.

      =================================================
      Version-Release number of selected component (if applicable):
      OCS client operator 4.17.0-120
      Openshift 4.17.0-rc.0

      ==================================================
      How reproducible:
      1/1

      Steps to Reproduce:

      1. Delete all the PVCs that use the storageclass from the client
      2. Delete the storageclaims that are manually created (if any for testing)
      3. Delete the storageclient
      4. Delete the configmap ocs-client-operator-config
      5. Delete the subscription for ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
      6. Delete the CSVs ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
      7. Delete the client namespace

      Adding output from the test.

      % oc get pv,pvc,volumesnapshot -A
      No resources found

      % oc get storageclient
      NAME PHASE CONSUMER
      storage-client Connected 122e3c08-067e-4ccf-b5bc-26d42ff8714d

      % oc delete storageclient storage-client
      storageclient.ocs.openshift.io ""storage-client"" deleted

      % oc get storageclient
      No resources found

      % oc get sc
      NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
      kubevirt-csi-infra-default (default) csi.kubevirt.io Delete Immediate false 120m

      Cephblockpoolradosnamespace and subvolumegroup are deleted from provider cluster

      Storageclient no longer listed in provider cluster UI

      After storageclient is off-boarded

      % oc get storageclient
      No resources found

      % oc delete configmap ocs-client-operator-config -n openshift-storage-client
      configmap "ocs-client-operator-config" deleted

      % oc get subs
      NAME PACKAGE SOURCE CHANNEL
      cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace cephcsi-operator ocs-catalogsource stable-4.17
      mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace mcg-operator ocs-catalogsource stable-4.17
      ocs-client-operator ocs-client-operator ocs-catalogsource stable-4.17
      odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace odf-csi-addons-operator ocs-catalogsource stable-4.17

      % oc delete subs ocs-client-operator -n openshift-storage-client
      subscription.operators.coreos.com ""ocs-client-operator"" deleted

      % oc delete subs odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
      subscription.operators.coreos.com ""odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

      % oc delete subs cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
      subscription.operators.coreos.com ""cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

      % oc delete subs mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
      subscription.operators.coreos.com ""mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

      % oc get subs
      No resources found in openshift-storage-client namespace.

      % oc get csv
      NAME DISPLAY VERSION REPLACES PHASE
      cephcsi-operator.v4.17.0-120.stable CephCSI operator 4.17.0-120.stable Succeeded
      mcg-operator.v4.17.0-120.stable NooBaa Operator 4.17.0-120.stable Succeeded
      ocs-client-operator.v4.17.0-120.stable OpenShift Data Foundation Client 4.17.0-120.stable Succeeded
      odf-csi-addons-operator.v4.17.0-120.stable CSI Addons 4.17.0-120.stable Succeeded

      % oc delete csv ocs-client-operator.v4.17.0-120.stable
      clusterserviceversion.operators.coreos.com ""ocs-client-operator.v4.17.0-120.stable"" deleted

      % oc delete csv odf-csi-addons-operator.v4.17.0-120.stable cephcsi-operator.v4.17.0-120.stable mcg-operator.v4.17.0-120.stable
      clusterserviceversion.operators.coreos.com ""odf-csi-addons-operator.v4.17.0-120.stable"" deleted
      clusterserviceversion.operators.coreos.com ""cephcsi-operator.v4.17.0-120.stable"" del

      % oc get csv
      No resources found in openshift-storage-client namespace.

      All operators are gone from installed Operators page in client cluster UI"

      Delete the namespace where the client operator was installed.

      % oc delete namespace openshift-storage-client
      namespace "openshift-storage-client" deleted
      ^C%

      ============================================

      Actual results:
      Namespace openshift-storage-client is not deleted.

      Expected results:
      Namespace openshift-storage-client should be deleted.

      Additional info:

              kmajumder@redhat.com Kaustav Majumder
              jijoy@redhat.com Jilju Joy
              Elad Ben Aharon Elad Ben Aharon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: