Description of problem (please be detailed as possible and provide log
snippests):

After upgrading from ocs-operator.v4.16.1-rhodf to ocs-registry:4.17.0-78, image "rhceph-7-rhel9" is not upgraded

Version of all relevant components (if applicable):
ocs-registry:4.17.0-78

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
Not aware

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
2/2

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
yes

Steps to Reproduce:
1. install odf ocs-operator.v4.16.1-rhodf ( GA'ed )
2. upgrade to 4.17.0-78
3. check all images are upgraded

Actual results:

2024-08-20 12:36:16 03:06:15 - MainThread - ocs_ci.ocs.resources.pod - WARNING - Images:

{'registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3'}

weren't upgraded in: rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2!

Expected results:

image should be upgraded to rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580

Additional info:

from csv, image should be rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580

$ oc get csv ocs-operator.v4.17.0-78.stable -o yaml | grep -i rhceph
value: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580

• image: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580

but from pod image is "rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3"

$ oc get pod rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2 -o yaml | grep -i rhceph
value: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad

> rook-ceph-operator-7694fff5fc-665cm.log logs

2024-08-20T10:05:56.916+0000 7f92efd7e800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1
2024-08-20T10:05:56.916+0000 7f92efd7e800 0 failed to load current period: (5) Input/output error. : exit status 1
2024-08-20 10:05:57.407564 I | ceph-spec: parsing mon endpoints: a=172.30.49.190:3300,b=172.30.102.23:3300,c=172.30.164.34:3300
2024-08-20 10:05:57.407642 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found
2024-08-20 10:05:57.407800 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found
2024-08-20 10:05:57.489822 E | ceph-object-store-user-controller: failed to reconcile CephObjectStoreUser "openshift-storage/noobaa-ceph-objectstore-user". failed to initialized rgw admin ops client api: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "ocs-storagecluster-cephobjectstore": failed to create s3 user. Unable to initialize site config.2024-08-20T10:05:57.473+0000 7f1ef172f800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1

job: https://url.corp.redhat.com/e23d2b4
must gather: https://url.corp.redhat.com/0a9f93d