Issue: Test-ns-onprem namespacestore in rejected state

Details:

1] Previously two namespace store was in Rejected Phase :

test-ns-onprem
test-ns-onprem2

2]Action plan to restart noobaa stack was performed after which now one namespacestore in rejected state now

+++

1. oc delete pods <noobaa-operator> -n openshift-storage
2. oc delete pods <noobaa-core> -n openshift-storage
3. oc delete pods <noobaa-endpoint> -n openshift-storage
4. oc delete pods <noobaa-db> -n openshift-storage

++

3] Files that are on onprem thay are actually present on prem2
4] The files generating the 404 are not sourced in test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6. However, bucket test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0 used the bucket class test-ns-onprem, which shows files from test-ns-onprem and test-ns-onprem2.
5] These files DO exist in test-ns-onprem2. test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6 is the bucket that is the target for test-ns-onprem, while test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0 is the bucket that uses bucket class test-bc-onprem and points to test-ns-onprem.

Logs

• test-ns-onprem s3-compatible test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6 Rejected 125d21h32m52s
• test-bc-gateway null {"type":"Multi","multi":{"readResources":["test-ns-onprem","test-ns-cloud"],"writeResource":"test-ns-onprem"}} null Rejected 125d20h54m39s

Events:

11h Normal nodeAssigned service/noobaa-mgmt announcing from node "dev-xp5wz-infra-zqr47" with protocol "layer2"
11h Normal nodeAssigned service/s3 announcing from node "dev-xp5wz-infra-hqbmw" with protocol "layer2"
11h Normal nodeAssigned service/sts announcing from node "dev-xp5wz-infra-hqbmw" with protocol "layer2"
time="2024-02-20T20:42:38Z" level=info msg="ReconcileNamespaceStores: [{Name:test-ns-onprem Mode:IO_ERRORS Undeletable:IN_USE EndpointType:S3_COMPATIBLE Endpoint:https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443 TargetBucket:test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6

10h Warning BackingStorePhaseRejected backingstore/noobaa-default-backing-store Backing store mode: ALL_NODES_OFFLINE
10h Warning RejectedBackingStore bucketclass/noobaa-default-bucket-class NooBaa BackingStore "noobaa-default-backing-store" is in rejected phase
11h Normal nodeAssigned service/noobaa-mgmt announcing from node "dev-xp5wz-infra-zqr47" with protocol "layer2"
11h Normal nodeAssigned service/s3 announcing from node "dev-xp5wz-infra-hqbmw" with protocol "layer2"
11h Normal nodeAssigned service/sts announcing from node "dev-xp5wz-infra-hqbmw" with protocol "layer2"
10h Warning RejectedNamespaceStore bucketclass/test-bc-gateway NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
10h Warning RejectedNamespaceStore bucketclass/test-bc-onprem NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
10h Normal NamespaceStorePhaseReady namespacestore/test-ns-onprem Namespace store mode: OPTIMAL
10h Warning NamespaceStorePhaseRejected namespacestore/test-ns-onprem Namespace store mode: IO_ERRORS

noobaa endpoint logs

• ./noobaa-operator-7f5bb8df89-lq5zz-noobaa-operator.log:time="2024-02-20T21:21:30Z" level=info msg="ReconcileNamespaceStores: nsr: {Name:test-ns-onprem Mode:IO_ERRORS Undeletable:IN_USE EndpointType:S3_COMPATIBLE Endpoint:https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443 TargetBucket:test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6 AccessMode:READ_WRITE Identity:Z1FIOGGGTKBGPURDEQLJ AuthMethod: CpCode:}

  " sys=openshift-storage/noobaa

$ cat events|grep test-bc-gateway
34s Warning RejectedNamespaceStore bucketclass/test-bc-gateway NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
$ cat events|grep test-ns-onprem
34s Warning RejectedNamespaceStore bucketclass/test-bc-gateway NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
34s Warning RejectedNamespaceStore bucketclass/test-bc-onprem NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
70s Normal NamespaceStorePhaseReady namespacestore/test-ns-onprem Namespace store mode: OPTIMAL
38s Warning NamespaceStorePhaseRejected namespacestore/test-ns-onprem Namespace store mode: IO_ERRORS

#--------------------#

1. • Namespace Stores -#
     #--------------------#

NAME TYPE TARGET-BUCKET PHASE AGE
test-ns-onprem s3-compatible test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6 Rejected 125d21h32m52s

#------------------#

1. • Bucket Classes -#
     #------------------#

test-bc-gateway null {"type":"Multi","multi":{"readResources":["test-ns-onprem","test-ns-cloud"],"writeResource":"test-ns-onprem"}} null Rejected 125d20h54m39s
test-bc-onprem null {"type":"Multi","multi":{"readResources":["test-ns-onprem","test-ns-onprem2"],"writeResource":"test-ns-onprem"}} null Rejected 125d21h27m57s

name: test-bc-gateway
namespace: openshift-storage
resourceVersion: "2308648286"
uid: 4b9dc303-7769-4416-9b72-77c1e9945398
spec:
namespacePolicy:
multi:
readResources:

• test-ns-onprem
• test-ns-cloud
  writeResource: test-ns-onprem
  type: Multi
  status:
  conditions:
• lastHeartbeatTime: "2023-10-18T00:26:51Z"
  lastTransitionTime: "2024-02-20T21:17:02Z"
  message: NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
  reason: RejectedNamespaceStore
  status: Unknown
  type: Available
• lastHeartbeatTime: "2023-10-18T00:26:51Z"
  lastTransitionTime: "2024-02-20T21:17:02Z"
  message: NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
  reason: RejectedNamespaceStore
  status: "False"
  type: Progressing

name: test-bc-onprem
namespace: openshift-storage
resourceVersion: "2308648281"
uid: 1023afb4-26bb-4dd8-928c-eb144cf1747c
spec:
namespacePolicy:
multi:
readResources:

• test-ns-onprem
• test-ns-onprem2
  writeResource: test-ns-onprem
  type: Multi
  status:
  conditions:
• lastHeartbeatTime: "2023-10-17T23:53:33Z"
  lastTransitionTime: "2024-02-20T21:17:02Z"
  message: NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
  reason: RejectedNamespaceStore
  status: Unknown
  type: Available
• lastHeartbeatTime: "2023-10-17T23:53:33Z"
  lastTransitionTime: "2024-02-20T21:17:02Z"
  message: NooBaa NamespaceStore "test-ns-onprem" is in rejected phase
  reason: RejectedNamespaceStore
  status: "False"
  type: Progressing

#-----------------#

1. • Bucket Claims -#
     #-----------------#
     NAMESPACE NAME BUCKET-NAME STORAGE-CLASS BUCKET-CLASS PHASE
     focus-mc-focus-face test-ns-obc-onprem-rgw test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6 ocs-storagecluster-ceph-rgw Bound
     pp remote-css-staged remote-css-staged-8c981263-4fef-496e-8872-6c2fbf9bf08a openshift-storage.noobaa.io pp-remote-css-staged Pending
     #13 (Associate) Make PublicPrivateHelps Resolution? 0
     (Target Date: 2/29/2024 8:55 PM SBT: 388 )

Created By: Oon Kwee Lim (2/29/2024 2:26 PM)
03744742/odf-mustgather210224/registry-redhat-io-odf4-ocs-must-gather-rhel8-sha256-025876630f383debb275abe8ab7899dd52d84afe106019878e432ba2f1cc90c0/namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims

test-ns-obc-onprem-rgw.yaml: bucketName: test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6

test-obc-onprem.yaml: bucketName: test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0

$ find . -name *yaml | xargs grep test-ns-obc-onprem-rgw
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: name: obc-focus-mc-focus-face-test-ns-obc-onprem-rgw
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: cephUser: obc-focus-mc-focus-face-test-ns-obc-onprem-rgw
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: bucketName: test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: name: obc-focus-mc-focus-face-test-ns-obc-onprem-rgw
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: cephUser: obc-focus-mc-focus-face-test-ns-obc-onprem-rgw
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-ns-obc-onprem-rgw.yaml: bucketName: test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-ns-obc-onprem-rgw.yaml: name: test-ns-obc-onprem-rgw
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-ns-obc-onprem-rgw.yaml: bucketName: test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-ns-obc-onprem-rgw.yaml: generateBucketName: test-ns-obc-onprem-rgw
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-ns-obc-onprem-rgw.yaml: objectBucketName: obc-focus-mc-focus-face-test-ns-obc-onprem-rgw
./noobaa/raw_output/NamespaceStoreList_crs.yaml: targetBucket: test-ns-obc-onprem-rgw-1bbe3dd9-2d01-44c9-a80e-b866856a1ba6

$ find . -name *yaml | xargs grep test-obc-onprem
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: name: obc-focus-mc-focus-face-test-obc-onprem
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: account: obc-account.test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0.652f1f30@noobaa.io
./cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: bucketName: test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: name: obc-focus-mc-focus-face-test-obc-onprem
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: account: obc-account.test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0.652f1f30@noobaa.io
./namespaces/all/cluster-scoped-resources/objectbucket.io/objectbuckets/obc-focus-mc-focus-face-test-obc-onprem.yaml: bucketName: test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-obc-onprem.yaml: name: test-obc-onprem
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-obc-onprem.yaml: bucketName: test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-obc-onprem.yaml: generateBucketName: test-obc-onprem
./namespaces/all/namespaces/focus-mc-focus-face/objectbucket.io/objectbucketclaims/test-obc-onprem.yaml: objectBucketName: obc-focus-mc-focus-face-test-obc-onprem
./noobaa/raw_output/NamespaceStoreList_crs.yaml: targetBucket: test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0

noobaa endpoint pod logs shows 'Error: InvalidMetadata: metadata keys are invalid'

^{[[32mApr-19 3:31:05.888}[[35m [Endpoint/13] ^[[31m[ERROR][[39m core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>SOAP requests must be made over an HTTPS connection.</Message><Resource>/test-obc-cloud-eff6c666-8c13-4b86-8246-36aa84e5eb94/testfile88.txt</Resource><RequestId>lv645dc8-5w4d0o-zn0</RequestId></Error> PUT /test-obc-cloud-eff6c666-8c13-4b86-8246-36aa84e5eb94/testfile88.txt

{"user-agent":"aws-sdk-nodejs/2.1127.0 linux/v16.20.2 promise","x-amz-copy-source":"/test-obc-onprem-b0b36a88-f77f-4909-b7df-0630f5a750e0/testfile88.txt","content-type":"application/octet-stream","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","content-length":"0","host":"s3.openshift-storage.svc.cluster.local","x-amz-date":"20240419T033104Z","authorization":"AWS4-HMAC-SHA256 Credential=L4VdfpSSGFVF8aJ60Qjo/20240419/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-date, Signature=a19534653346598c3a4a6277b575e85cff96bc1e360f6b09dda35579907b9a38","connection":"close"}

Error: InvalidMetadata: metadata keys are invalid

Version of all relevant components (if applicable):
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.12.22 True False 242d Cluster version is 4.12.22

ODF version 4.12 (internal)

mcg-operator.v4.12.11-rhodf NooBaa Operator 4.12.11-rhodf mcg-operator.v4.12.10-rhodf Succeeded
metallb-operator.4.12.0-202401291234 MetalLB Operator 4.12.0-202401291234 metallb-operator.4.12.0-202310311227 Succeeded
namespace-configuration-operator.v1.2.6 Namespace Configuration Operator 1.2.6 namespace-configuration-operator.v1.2.5 Succeeded
ocs-operator.v4.12.11-rhodf OpenShift Container Storage 4.12.11-rhodf ocs-operator.v4.12.10-rhodf Succeeded
odf-csi-addons-operator.v4.12.11-rhodf CSI Addons 4.12.11-rhodf odf-csi-addons-operator.v4.12.10-rhodf Succeeded
odf-operator.v4.12.11-rhodf OpenShift Data Foundation 4.12.11-rhodf odf-operator.v4.12.10-rhodf Succeeded

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

No

Is there any workaround available to the best of your knowledge?

Restarting noobaa stack helped for one namespacestore to be in Ready state

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

3

Can this issue reproducible?

Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:

NA

Steps to Reproduce:
NA

Actual results:

Namespacestore test-ns-onprem should be in Rejected Phase

Expected results:

Namespacestore test-ns-onprem should be in Ready Phase

Additional info: