Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI:

[RDR] After upgrading cluster internal build noobaa goes Connecting state

The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):

 Vmware-UPI

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):

 DR

The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

 Installed:- 4.18.0-48.stable

Upgrade:- 4.18.0-49.stable

Does this issue impact your ability to continue to work with the product?

 Yes

Is there any workaround available to the best of your knowledge?

Yes restart noobaa pods 

Can this issue be reproduced? If so, please provide the hit rate

Can this issue be reproduced from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:

1. Deploy DR cluster with 4.18.0-48.stable

2. Ugprade ODF to 4.18.0-49.stable

3. check noobaa status

The exact date and time when the issue was observed, including timezone details:

Actual results:

oc get noobaas.noobaa.io
NAME S3-ENDPOINTS STS-ENDPOINTS SYSLOG-ENDPOINTS IMAGE PHASE AGE
noobaa ["https://10.1.114.94:30657"] ["https://10.1.114.94:30191"] registry.redhat.io/odf4/mcg-core-rhel9@sha256:c0917568887a1c8c8f67bac4484b7811421695d4275a9e39d2b55d2038a1678c Connecting 28h
 

 $ oc describe noobaas
Name: noobaa
Namespace: openshift-storage
Labels: app=noobaa
Annotations: <none>
API Version: noobaa.io/v1alpha1
Kind: NooBaa
Metadata:
Creation Timestamp: 2024-11-11T05:42:07Z
Finalizers:
noobaa.io/graceful_finalizer
Generation: 1
Owner References:
API Version: ocs.openshift.io/v1
Block Owner Deletion: true
Controller: true
Kind: StorageCluster
Name: ocs-storagecluster
UID: 275d3e92-de56-4e62-a823-8994f0b41a8a
Resource Version: 4575887
UID: d2b0ef1c-312d-4a5a-ad88-67c8a0b3e1ab
Spec:
Affinity:
Node Affinity:
Required During Scheduling Ignored During Execution:
Node Selector Terms:
Match Expressions:
Key: cluster.ocs.openshift.io/openshift-storage
Operator: Exists
Autoscaler:
Autoscaler Type: hpav2
Prometheus Namespace: openshift-monitoring
Bucket Logging:
Cleanup Policy:
Core Resources:
Limits:
Cpu: 999m
Memory: 4Gi
Requests:
Cpu: 999m
Memory: 4Gi
Db Image: registry.redhat.io/rhel9/postgresql-15@sha256:24fb4e7914a6e1464d015be9e5582cc4b9da224137408bd429e7ea4f391aa198
Db Resources:
Limits:
Cpu: 500m
Memory: 4Gi
Requests:
Cpu: 500m
Memory: 4Gi
Db Storage Class: ocs-storagecluster-ceph-rbd
Db Type: postgres
Db Volume Resources:
Requests:
Storage: 50Gi
Endpoints:
Max Count: 2
Min Count: 1
Resources:
Limits:
Cpu: 999m
Memory: 2Gi
Requests:
Cpu: 999m
Memory: 2Gi
Image: registry.redhat.io/odf4/mcg-core-rhel9@sha256:c0917568887a1c8c8f67bac4484b7811421695d4275a9e39d2b55d2038a1678c
Labels:
Monitoring:
Load Balancer Source Subnets:
Pv Pool Default Storage Class: ocs-storagecluster-ceph-rbd
Security:
Kms:
Schedule: @weekly
Tolerations:
Effect: NoSchedule
Key: node.ocs.openshift.io/storage
Operator: Equal
Value: true
Status:
Accounts:
Admin:
Secret Ref:
Name: noobaa-admin
Namespace: openshift-storage
Actual Image: registry.redhat.io/odf4/mcg-core-rhel9@sha256:c0917568887a1c8c8f67bac4484b7811421695d4275a9e39d2b55d2038a1678c
Conditions:
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:34Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-2
Reason: TemporaryError
Status: False
Type: Available
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:34Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-2
Reason: TemporaryError
Status: True
Type: Progressing
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-11T05:42:07Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-2
Reason: TemporaryError
Status: False
Type: Degraded
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:34Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-2
Reason: TemporaryError
Status: False
Type: Upgradeable
Last Heartbeat Time: 2024-11-12T04:29:35Z
Last Transition Time: 2024-11-11T05:42:08Z
Status: k8s
Type: KMS-Type
Last Heartbeat Time: 2024-11-12T04:29:35Z
Last Transition Time: 2024-11-11T05:42:09Z
Status: Sync
Type: KMS-Status
Endpoints:
Ready Count: 1
Virtual Hosts:
s3.openshift-storage.svc
s3-openshift-storage.apps.akrai-c1.qe.rh-ocs.com
Observed Generation: 1
Phase: Connecting
Readme:

NooBaa operator is still working to reconcile this system.
Check out the system status.phase, status.conditions, and events with:

kubectl -n openshift-storage describe noobaa
kubectl -n openshift-storage get noobaa -o yaml
kubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp

You can wait for a specific condition with:

kubectl -n openshift-storage wait noobaa/noobaa --for condition=available --timeout -1s

NooBaa Core Version: master-20240520
NooBaa Operator Version: 5.18.0

Services:
Service Mgmt:
External DNS:
https://noobaa-mgmt-openshift-storage.apps.akrai-c1.qe.rh-ocs.com:443
Internal DNS:
https://noobaa-mgmt.openshift-storage.svc:443
Internal IP:
https://172.30.210.60:443
Node Ports:
https://10.1.114.94:0
Pod Ports:
https://10.128.3.123:8443
serviceS3:
External DNS:
https://s3-openshift-storage.apps.akrai-c1.qe.rh-ocs.com:443
Internal DNS:
https://s3.openshift-storage.svc:443
Internal IP:
https://172.30.206.94:443
Node Ports:
https://10.1.114.94:30657
Pod Ports:
https://10.128.3.125:6443
Service Sts:
External DNS:
https://sts-openshift-storage.apps.akrai-c1.qe.rh-ocs.com:443
Internal DNS:
https://sts.openshift-storage.svc:443
Internal IP:
https://172.30.157.52:443
Node Ports:
https://10.1.114.94:30191
Pod Ports:
https://10.128.3.125:7443
Service Syslog:
Events: <none>

Expected results:

 Noobaa should be in ready state

Logs collected and log location:

 http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/DFBUGS-832/

Additional info:

 $oc describe backingstores.noobaa.io
Name: noobaa-default-backing-store
Namespace: openshift-storage
Labels: app=noobaa
Annotations: rgw:
API Version: noobaa.io/v1alpha1
Kind: BackingStore
Metadata:
Creation Timestamp: 2024-11-11T05:43:33Z
Finalizers:
noobaa.io/finalizer
Generation: 1
Owner References:
API Version: noobaa.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: NooBaa
Name: noobaa
UID: d2b0ef1c-312d-4a5a-ad88-67c8a0b3e1ab
Resource Version: 4575891
UID: 9090c484-63aa-467e-aa32-0f3097040343
Spec:
s3Compatible:
Endpoint: https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443
Secret:
Name: rook-ceph-object-user-ocs-storagecluster-cephobjectstore-noobaa-ceph-objectstore-user
Namespace: openshift-storage
Signature Version: v4
Target Bucket: nb.1731303813911.apps.akrai-c1.qe.rh-ocs.com
Type: s3-compatible
Status:
Conditions:
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:35Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1
Reason: TemporaryError
Status: False
Type: Available
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:35Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1
Reason: TemporaryError
Status: True
Type: Progressing
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-11T05:43:35Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1
Reason: TemporaryError
Status: False
Type: Degraded
Last Heartbeat Time: 2024-11-12T05:56:24Z
Last Transition Time: 2024-11-12T04:29:35Z
Message: RPC: connection closed while request is pending wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1
Reason: TemporaryError
Status: False
Type: Upgradeable
Mode:
Mode Code: OPTIMAL
Time Stamp: 2024-11-11 05:43:35.87357944 +0000 UTC m=+534.293967674
Phase: Connecting
Events: <none>