-
Bug
-
Resolution: Unresolved
-
Critical
-
odf-4.17
-
None
-
False
-
-
False
-
Committed
-
?
-
-
-
None
Description of problem:
After uninstalling ocs-client-operator and its dependent operators, the namespace 'openshift-storage-client' cannot be deleted. This is the namespace where the operator was installed. This is tested on client cluster in a provider-client setup.
% oc get ns openshift-storage-client -o yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
openshift.io/sa.scc.mcs: s0:c25,c20
openshift.io/sa.scc.supplemental-groups: 1000640000/10000
openshift.io/sa.scc.uid-range: 1000640000/10000
creationTimestamp: "2024-10-14T05:04:29Z"
deletionTimestamp: "2024-10-14T09:12:00Z"
labels:
kubernetes.io/metadata.name: openshift-storage-client
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.24
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.24
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.24
security.openshift.io/scc.podSecurityLabelSync: "true"
name: openshift-storage-client
resourceVersion: "87170"
uid: 059a125c-da2c-4829-a0c1-f31ba9354a8f
spec:
finalizers:
- kubernetes
status:
conditions: - lastTransitionTime: "2024-10-14T09:12:07Z"
message: All resources successfully discovered
reason: ResourcesDiscovered
status: "False"
type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2024-10-14T09:12:07Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2024-10-14T09:12:07Z"
message: 'Failed to delete all resource types, 1 remaining: admission webhook
"admissionwebhook.noobaa.io" denied the request: Noobaa cleanup policy is not
set, blocking Noobaa deletion'
reason: ContentDeletionFailed
status: "True"
type: NamespaceDeletionContentFailure - lastTransitionTime: "2024-10-14T09:12:07Z"
message: All content successfully removed
reason: ContentRemoved
status: "False"
type: NamespaceContentRemaining - lastTransitionTime: "2024-10-14T09:12:07Z"
message: All content-preserving finalizers finished
reason: ContentHasNoFinalizers
status: "False"
type: NamespaceFinalizersRemaining
phase: Terminating
% oc get Noobaa
NAME S3-ENDPOINTS STS-ENDPOINTS SYSLOG-ENDPOINTS IMAGE PHASE AGE
noobaa-remote Rejected 77m
% oc get Noobaa noobaa-remote -o yaml -n openshift-storage-client
apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
annotations:
remote-client-noobaa: "true"
creationTimestamp: "2024-10-14T05:59:03Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2024-10-14T07:12:26Z"
finalizers:
- noobaa.io/graceful_finalizer
generation: 3
labels:
app: noobaa
name: noobaa-remote
namespace: openshift-storage-client
ownerReferences: - apiVersion: ocs.openshift.io/v1alpha1
kind: StorageClient
name: storage-client
uid: 6f40c83c-1fb9-4691-81d6-8a5ff6a6f0e6
resourceVersion: "51471"
uid: cf6140a5-44a8-4781-a9f4-fa70284fb061
spec:
autoscaler: {}
bucketLogging: {}
cleanupPolicy: {}
joinSecret:
name: noobaa-remote-join-secret
namespace: openshift-storage-client
loadBalancerSourceSubnets: {}
security:
kms: {}
status:
accounts:
admin:
secretRef: {}
conditions: - lastHeartbeatTime: "2024-10-14T05:59:03Z"
lastTransitionTime: "2024-10-14T05:59:03Z"
message: Invalid system name "noobaa-remote" expected "noobaa"
reason: InvalidSystemName
status: Unknown
type: Available - lastHeartbeatTime: "2024-10-14T05:59:03Z"
lastTransitionTime: "2024-10-14T05:59:03Z"
message: Invalid system name "noobaa-remote" expected "noobaa"
reason: InvalidSystemName
status: "False"
type: Progressing - lastHeartbeatTime: "2024-10-14T05:59:03Z"
lastTransitionTime: "2024-10-14T05:59:03Z"
message: Invalid system name "noobaa-remote" expected "noobaa"
reason: InvalidSystemName
status: "True"
type: Degraded - lastHeartbeatTime: "2024-10-14T05:59:03Z"
lastTransitionTime: "2024-10-14T05:59:03Z"
message: Invalid system name "noobaa-remote" expected "noobaa"
reason: InvalidSystemName
status: Unknown
type: Upgradeable
observedGeneration: 2
phase: Rejected
readme: "\n\tERROR: NooBaa operator cannot reconcile this system spec.\n\n\tCheck
out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl
-n openshift-storage-client describe noobaa\n\t\tkubectl -n openshift-storage-client
get noobaa -o yaml\n\t\tkubectl -n openshift-storage-client get events --sort-by=metadata.creationTimestamp\n\n\tIn
order to retry, edit the system spec and the operator is watching and will be
notified.\n\n\tNooBaa Core Version: master-20240520\n\tNooBaa Operator Version:
5.17.0\n"
services:
serviceMgmt: {}
serviceS3: {}
serviceSts: {}
serviceSyslog: {}
Subs and CSVs are deleted using CLI. There is no installed operators in UI.
=================================================
Version-Release number of selected component (if applicable):
OCS client operator 4.17.0-120
Openshift 4.17.0-rc.0
==================================================
How reproducible:
1/1
Steps to Reproduce:
1. Delete all the PVCs that use the storageclass from the client
2. Delete the storageclaims that are manually created (if any for testing)
3. Delete the storageclient
4. Delete the configmap ocs-client-operator-config
5. Delete the subscription for ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
6. Delete the CSVs ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
7. Delete the client namespace
Adding output from the test.
% oc get pv,pvc,volumesnapshot -A
No resources found
% oc get storageclient
NAME PHASE CONSUMER
storage-client Connected 122e3c08-067e-4ccf-b5bc-26d42ff8714d
% oc delete storageclient storage-client
storageclient.ocs.openshift.io ""storage-client"" deleted
% oc get storageclient
No resources found
% oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
kubevirt-csi-infra-default (default) csi.kubevirt.io Delete Immediate false 120m
Cephblockpoolradosnamespace and subvolumegroup are deleted from provider cluster
Storageclient no longer listed in provider cluster UI
After storageclient is off-boarded
% oc get storageclient
No resources found
% oc delete configmap ocs-client-operator-config -n openshift-storage-client
configmap "ocs-client-operator-config" deleted
% oc get subs
NAME PACKAGE SOURCE CHANNEL
cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace cephcsi-operator ocs-catalogsource stable-4.17
mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace mcg-operator ocs-catalogsource stable-4.17
ocs-client-operator ocs-client-operator ocs-catalogsource stable-4.17
odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace odf-csi-addons-operator ocs-catalogsource stable-4.17
% oc delete subs ocs-client-operator -n openshift-storage-client
subscription.operators.coreos.com ""ocs-client-operator"" deleted
% oc delete subs odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted
% oc delete subs cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted
% oc delete subs mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted
% oc get subs
No resources found in openshift-storage-client namespace.
% oc get csv
NAME DISPLAY VERSION REPLACES PHASE
cephcsi-operator.v4.17.0-120.stable CephCSI operator 4.17.0-120.stable Succeeded
mcg-operator.v4.17.0-120.stable NooBaa Operator 4.17.0-120.stable Succeeded
ocs-client-operator.v4.17.0-120.stable OpenShift Data Foundation Client 4.17.0-120.stable Succeeded
odf-csi-addons-operator.v4.17.0-120.stable CSI Addons 4.17.0-120.stable Succeeded
% oc delete csv ocs-client-operator.v4.17.0-120.stable
clusterserviceversion.operators.coreos.com ""ocs-client-operator.v4.17.0-120.stable"" deleted
% oc delete csv odf-csi-addons-operator.v4.17.0-120.stable cephcsi-operator.v4.17.0-120.stable mcg-operator.v4.17.0-120.stable
clusterserviceversion.operators.coreos.com ""odf-csi-addons-operator.v4.17.0-120.stable"" deleted
clusterserviceversion.operators.coreos.com ""cephcsi-operator.v4.17.0-120.stable"" del
% oc get csv
No resources found in openshift-storage-client namespace.
All operators are gone from installed Operators page in client cluster UI"
Delete the namespace where the client operator was installed.
% oc delete namespace openshift-storage-client
namespace "openshift-storage-client" deleted
^C%
============================================
Actual results:
Namespace openshift-storage-client is not deleted.
Expected results:
Namespace openshift-storage-client should be deleted.
Additional info:
- relates to
-
DFBUGS-678 [2319352] [ODF-4.17.z] Cannot delete the namespace after uninstalling ocs-client-operator and other dependency operators
- ASSIGNED
- external trackers