-
Task
-
Resolution: Done
-
Major
-
Logging 5.1
-
None
-
devex docs #204 Jul 8-Jul 29
-
5
-
Documentation (Ref Guide, User Guide, etc.), Release Notes
-
Undefined
Description of problem:
Deploy logging, don't enable storage for ES, after all ES pods running, add storage to ES, then the ES cluster upgrade status becomes `scheduledUpgrade: "True"`, and `shardAllocationEnabled` is set to `primaries`:
nodes:
- deploymentName: elasticsearch-cdm-ns8u94oy-1
upgradeStatus:
scheduledUpgrade: "True"
underUpgrade: "True"
upgradePhase: preparationComplete
- deploymentName: elasticsearch-cdm-ns8u94oy-2
upgradeStatus:
scheduledUpgrade: "True"
- deploymentName: elasticsearch-cdm-ns8u94oy-3
upgradeStatus:
scheduledUpgrade: "True"
pods:
client:
failed: []
notReady: []
ready:
- elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
- elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
- elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
data:
failed: []
notReady: []
ready:
- elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
- elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
- elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
master:
failed: []
notReady: []
ready:
- elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
- elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
- elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
shardAllocationEnabled: primaries
EO logs:
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:57:37.671156772Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:58:09.855651975Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:58:40.714884796Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Unregistering future events","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:59:05.448910336Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:59:11.619126665Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Flushing nodes","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","objectKey":{"Namespace":"openshift-logging","Name":"elasticsearch"},"operator_version":"4.7.0","ts":"2021-05-07T00:59:12.619891446Z"}
{"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Registering future events","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","objectKey":{"Namespace":"openshift-logging","Name":"elasticsearch"},"operator_version":"4.7.0","ts":"2021-05-07T01:01:40.435653703Z"}
Version-Release number of selected component (if applicable):
elasticsearch-operator.5.1.0-20
How reproducible:
Always
Steps to Reproduce:
condition 1:
1. deploy CLO and EO
2. create cl/instance with
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 60h
infra:
maxAge: 3h
audit:
maxAge: 1d
elasticsearch:
nodeCount: 3
redundancyPolicy: "SingleRedundancy"
resources:
requests:
memory: "2Gi"
storage: {}
3. wait until ES cluster status becomes green, update cl/instance, add storage to ES:
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 60h
infra:
maxAge: 3h
audit:
maxAge: 1d
elasticsearch:
nodeCount: 3
redundancyPolicy: "SingleRedundancy"
resources:
requests:
memory: "2Gi"
storage:
storageClassName: "gp2"
size: "20Gi"
4. check ES status in es/elasticsearch
condition 2:
1. deploy CLO and EO
2. create cl/instance with
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 60h
infra:
maxAge: 3h
audit:
maxAge: 1d
elasticsearch:
nodeCount: 3
redundancyPolicy: "SingleRedundancy"
resources:
requests:
memory: "2Gi"
storage:
storageClassName: "gp2"
size: "20Gi"
3. wait until ES cluster status becomes green, update cl/instance, remove storage :
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 60h
infra:
maxAge: 3h
audit:
maxAge: 1d
elasticsearch:
nodeCount: 3
redundancyPolicy: "SingleRedundancy"
resources:
requests:
memory: "2Gi"
storage: {}
4. check ES status in es/elasticsearch
Actual results:
No ES pod restarts, but the ES cluster `upgradeStatus` is changed to `scheduledUpgrade: "True"`
No message `Changing the storage structure for a custom resource is not supported` when follow the steps in condition 1.
Expected results:
ES cluster should not in upgrade status, and in es/elasticsearch, there should have message like:
conditions:
- lastTransitionTime: "2021-05-07T01:05:13Z"
message: Changing the storage structure for a custom resource is not supported
reason: StorageStructureChangeIgnored
status: "True"
type: StorageStructureChangeIgnored
Additional info:
- documents
-
LOG-1351 [Logging 5.1]EO shouldn't try to upgrade ES cluster after adding/removing storage.
-
- Closed
-
- links to