-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
odf-4.18
-
None
Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI:
[RDR] Upgrade from 4.17 to 4.18 is stuck
The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):
VMware UPI
The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):
DR
The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):
OCP version:- 4.18.0-0.nightly-2024-11-15-113437
ODF version:- 4.17.0-126
CEPH version:- ceph version 18.2.1-229.el9cp (ef652b206f2487adfc86613646a4cac946f6b4e0) reef (stable)
ACM version:- 2.12.0
SUBMARINER version:- v0.19.0
VOLSYNC version:- volsync-product.v0.11.0
OADP version:- 1.4.1
VOLSYNC method:- destinationCopyMethod: Direct
Does this issue impact your ability to continue to work with the product?
Yes
Is there any workaround available to the best of your knowledge?
Can this issue be reproduced? If so, please provide the hit rate
Yes
Can this issue be reproduced from the UI?
If this is a regression, please provide more details to justify this:
Steps to Reproduce:
1. Deploy 4..17 RDR cluster
2. Upgrade ODF to 4.18 from 4.17
3. check csv status
The exact date and time when the issue was observed, including timezone details:
Actual results:
oc get csv NAME DISPLAY VERSION REPLACES PHASE cephcsi-operator.v4.17.0-rhodf CephCSI operator 4.17.0-rhodf Succeeded mcg-operator.v4.17.0-rhodf NooBaa Operator 4.17.0-rhodf Succeeded ocs-client-operator.v4.17.0-rhodf OpenShift Data Foundation Client 4.17.0-rhodf Succeeded ocs-operator.v4.17.0-rhodf OpenShift Container Storage 4.17.0-rhodf Succeeded odf-csi-addons-operator.v4.17.0-rhodf CSI Addons 4.17.0-rhodf Succeeded odf-operator.v4.17.0-rhodf OpenShift Data Foundation 4.17.0-rhodf Replacing odf-operator.v4.18.0-64.stable OpenShift Data Foundation 4.18.0-64.stable odf-operator.v4.17.0-rhodf Installing odf-prometheus-operator.v4.17.0-rhodf Prometheus Operator 4.17.0-rhodf Succeeded odr-cluster-operator.v4.18.0-64.stable Openshift DR Cluster Operator 4.18.0-64.stable odr-cluster-operator.v4.17.0-rhodf Succeeded openshift-gitops-operator.v1.14.2 Red Hat OpenShift GitOps 1.14.2 openshift-gitops-operator.v1.14.1 Succeeded recipe.v4.17.0-rhodf Recipe 4.17.0-rhodf Succeeded rook-ceph-operator.v4.17.0-rhodf Rook-Ceph 4.17.0-rhodf Succeeded volsync-product.v0.11.0 VolSync 0.11.0 Succeeded
Snippet from odf-operator pod logs
2024-11-21T01:31:58Z ERROR Reconciler error {"controller": "storagesystem", "controllerGroup": "odf.openshift.io", "controllerKind": "StorageSystem", "StorageSystem": {"name":"ocs-storagecluster-storagesystem","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "ocs-storagecluster-storagesystem", "reconcileID": "ebec4b7a-c47d-428d-bebd-ae847a8ccda9", "error": "InstallPlan not found for CSV odf-dependencies.v4.18.0-64.stable; InstallPlan not found for CSV ocs-operator.v4.18.0-64.stable; InstallPlan not found for CSV rook-ceph-operator.v4.18.0-64.stable; InstallPlan not found for CSV mcg-operator.v4.18.0-64.stable; InstallPlan not found for CSV odf-prometheus-operator.v4.18.0-64.stable; InstallPlan not found for CSV recipe.v4.18.0-64.stable; InstallPlan not found for CSV ocs-client-operator.v4.18.0-64.stable; InstallPlan not found for CSV odf-csi-addons-operator.v4.18.0-64.stable; InstallPlan not found for CSV cephcsi-operator.v4.18.0-64.stable", "errorCauses": [{"error": "InstallPlan not found for CSV odf-dependencies.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV ocs-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV rook-ceph-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV mcg-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV odf-prometheus-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV recipe.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV ocs-client-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV odf-csi-addons-operator.v4.18.0-64.stable"}, {"error": "InstallPlan not found for CSV cephcsi-operator.v4.18.0-64.stable"}]}sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:2242024-11-21T01:32:04Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV mcg-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:04Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:14Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV ocs-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:14Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:24Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV ocs-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:24Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:34Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV ocs-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:34Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:37Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV mcg-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:37Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:44Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV rook-ceph-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:44Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:32:54Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV ocs-operator.v4.18.0-64.stable is not found"}2024-11-21T01:32:54Z INFO controller-runtime.healthz healthz check failed {"statuses": [{}]}2024-11-21T01:33:04Z DEBUG controller-runtime.healthz healthz check failed {"checker": "readyz", "error": "CSV ocs-operator.v4.18.0-64.stable is not found"}
Snippet from csv
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 17m operator-lifecycle-manager requirements not yet checked Normal InstallWaiting 17m operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" waiting for 1 outdated replica(s) to be terminated Warning InstallCheckFailed 12m operator-lifecycle-manager install timeout Normal NeedsReinstall 12m (x2 over 12m) operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability. Normal AllRequirementsMet 12m (x4 over 17m) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 12m (x4 over 17m) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 12m (x2 over 13m) operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability. Warning InstallCheckFailed 7m3s operator-lifecycle-manager install timeout Normal NeedsReinstall 7m2s (x2 over 7m2s) operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability. Normal AllRequirementsMet 7m2s (x2 over 7m2s) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 7m1s (x2 over 7m1s) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 7m1s (x2 over 7m1s) operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability. Warning InstallCheckFailed 3m27s (x2 over 3m27s) operator-lifecycle-manager install failed: deployment odf-operator-controller-manager not ready before timeout: deployment "odf-operator-controller-manager" exceeded its progress deadline
Expected results:
upgrade should work fine
Logs collected and log location:
Additional info:
