-
Bug
-
Resolution: Done
-
Major
-
Logging 5.4.0
-
False
-
False
-
NEW
-
VERIFIED
-
Before this update, defining a toleration with no key and the exists operator caused the operator to be unable to complete an upgrade. With this update, this toleration no longer blocks the upgrade from completing.
-
-
Logging (LogExp) - Sprint 214, Logging (LogExp) - Sprint 215
Issue: Elasticsearch cluster upgrade stuck.
Findings:
We can see pod elasticsearch-cdm-tnwarhmo-1 is on the latest image and elasticsearch-cdm-tnwarhmo-2 and elasticsearch-cdm-tnwarhmo-3 are on the old image.
~~~
nodes:
- deploymentName: elasticsearch-cdm-tnwarhmo-1
upgradeStatus:
scheduledUpgrade: "True"
underUpgrade: "True"
upgradePhase: preparationComplete
- deploymentName: elasticsearch-cdm-tnwarhmo-2
upgradeStatus:
scheduledUpgrade: "True"
- deploymentName: elasticsearch-cdm-tnwarhmo-3
upgradeStatus:
scheduledUpgrade: "True"
~~~
Verified the latest image by installing the test logging cluster.
Elasticsearch operator logs.
~~~
{"_ts":"2022-01-103048.820353447Z","_level":"0","_component":"elasticsearch-operator","_message":"unable to update node","_error":\{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-tnwarhmo-1"},"cluster":"elasticsearch","namespace":"openshift-logging"}
~~~
Pod 'elasticsearch-cdm-tnwarhmo-1' is already restarted and is up and running. I am wondering why the operator is stuck at 'timed out waiting for node to rollout'.
I tried to replicate this issue in the test cluster but the upgrade was completed successfully.
~~~
[demeestg@pinocp01 ~]$ oc get sub,ip,csv -n openshift-logging
NAME PACKAGE SOURCE CHANNEL
subscription.operators.coreos.com/cluster-logging cluster-logging redhat-operators stable-5.3NAME CSV APPROVAL APPROVED
installplan.operators.coreos.com/install-hlbhw cluster-logging.5.3.0-55 Automatic true
installplan.operators.coreos.com/install-m5bvh cluster-logging.5.3.2-20 Automatic true
installplan.operators.coreos.com/install-r6wfq cluster-logging.5.3.1-12 Automatic trueNAME DISPLAY VERSION REPLACES PHASE
clusterserviceversion.operators.coreos.com/cluster-logging.5.3.2-20 Red Hat OpenShift Logging 5.3.2-20 cluster-logging.5.3.1-12 Succeeded
clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.3.2-20 OpenShift Elasticsearch Operator 5.3.2-20 elasticsearch-operator.5.3.1-12 Succeeded
clusterserviceversion.operators.coreos.com/redhat-openshift-pipelines.v1.5.2 Red Hat OpenShift Pipelines 1.5.2 redhat-openshift-pipelines.v1.4.1 Succeeded
~~~
Please let me know in case of any other information is required.
- clones
-
LOG-2126 [Logging 5.3] Elasticsearch cluster upgrade stuck
- Closed
- links to