Uploaded image for project: 'Docs for Red Hat Developers'
  1. Docs for Red Hat Developers
  2. RHDEVDOCS-3018

Document: [Logging 5.1]EO shouldn't try to upgrade ES cluster after adding/removing storage.

    XMLWordPrintable

Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Done
    • Logging 5.1
    • Logging 5.1
    • Logging
    • None

    Description

      Description of problem:

      Deploy logging, don't enable storage for ES, after all ES pods running, add storage to ES, then the ES cluster upgrade status becomes `scheduledUpgrade: "True"`, and `shardAllocationEnabled` is set to `primaries`: 

          nodes:
          - deploymentName: elasticsearch-cdm-ns8u94oy-1
            upgradeStatus:
              scheduledUpgrade: "True"
              underUpgrade: "True"
              upgradePhase: preparationComplete
          - deploymentName: elasticsearch-cdm-ns8u94oy-2
            upgradeStatus:
              scheduledUpgrade: "True"
          - deploymentName: elasticsearch-cdm-ns8u94oy-3
            upgradeStatus:
              scheduledUpgrade: "True"
          pods:
            client:
              failed: []
              notReady: []
              ready:
              - elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
              - elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
              - elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
            data:
              failed: []
              notReady: []
              ready:
              - elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
              - elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
              - elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
            master:
              failed: []
              notReady: []
              ready:
              - elasticsearch-cdm-ns8u94oy-3-5d86576b94-5kx69
              - elasticsearch-cdm-ns8u94oy-1-64f97c8746-hr4mq
              - elasticsearch-cdm-ns8u94oy-2-698d6d5c5b-zlk4n
          shardAllocationEnabled: primaries
      

      EO logs:

      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:57:37.671156772Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:58:09.855651975Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:58:40.714884796Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Unregistering future events","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:59:05.448910336Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"unable to update node","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","operator_version":"4.7.0","ts":"2021-05-07T00:59:11.619126665Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Flushing nodes","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","objectKey":{"Namespace":"openshift-logging","Name":"elasticsearch"},"operator_version":"4.7.0","ts":"2021-05-07T00:59:12.619891446Z"}
      {"cluster":"elasticsearch","component":"elasticsearch-operator","error":{"msg":"timed out waiting for node to rollout","node":"elasticsearch-cdm-ns8u94oy-1"},"go_arch":"amd64","go_os":"linux","go_version":"go1.15.7","level":"0","message":"Registering future events","name":{"Namespace":"openshift-logging","Name":"kibana"},"namespace":"openshift-logging","node":"elasticsearch-cdm-ns8u94oy-1","objectKey":{"Namespace":"openshift-logging","Name":"elasticsearch"},"operator_version":"4.7.0","ts":"2021-05-07T01:01:40.435653703Z"}
      

      Version-Release number of selected component (if applicable):

      elasticsearch-operator.5.1.0-20
      How reproducible:

      Always
      Steps to Reproduce:

      condition 1:

      1. deploy CLO and EO
      2. create cl/instance with

        logStore:
          type: "elasticsearch"
          retentionPolicy: 
            application:
              maxAge: 60h 
            infra:
              maxAge: 3h
            audit:
              maxAge: 1d
          elasticsearch:
            nodeCount: 3
            redundancyPolicy: "SingleRedundancy"
            resources:
              requests:
                memory: "2Gi"
            storage: {}

      3.  wait until ES cluster status becomes green, update cl/instance, add storage to ES:

        logStore:
          type: "elasticsearch"
          retentionPolicy: 
            application:
              maxAge: 60h 
            infra:
              maxAge: 3h
            audit:
              maxAge: 1d
          elasticsearch:
            nodeCount: 3
            redundancyPolicy: "SingleRedundancy"
            resources:
              requests:
                memory: "2Gi"
            storage:
              storageClassName: "gp2"
              size: "20Gi"
      

      4. check ES status in es/elasticsearch

      condition 2:

      1. deploy CLO and EO
      2. create cl/instance with

        logStore:
          type: "elasticsearch"
          retentionPolicy: 
            application:
              maxAge: 60h 
            infra:
              maxAge: 3h
            audit:
              maxAge: 1d
          elasticsearch:
            nodeCount: 3
            redundancyPolicy: "SingleRedundancy"
            resources:
              requests:
                memory: "2Gi"
            storage: 
              storageClassName: "gp2" 
              size: "20Gi"

      3.  wait until ES cluster status becomes green, update cl/instance, remove storage :

        logStore:
          type: "elasticsearch"
          retentionPolicy: 
            application:
              maxAge: 60h 
            infra:
              maxAge: 3h
            audit:
              maxAge: 1d
          elasticsearch:
            nodeCount: 3
            redundancyPolicy: "SingleRedundancy"
            resources:
              requests:
                memory: "2Gi"
            storage: {}
      

      4. check ES status in es/elasticsearch

       

      Actual results:

      No ES pod restarts, but the ES cluster `upgradeStatus` is changed to `scheduledUpgrade: "True"`

      No message `Changing the storage structure for a custom resource is not supported` when follow the steps in condition 1.

      Expected results:

      ES cluster should not in upgrade status, and in es/elasticsearch, there should have message like:

          conditions:
          - lastTransitionTime: "2021-05-07T01:05:13Z"
            message: Changing the storage structure for a custom resource is not supported
            reason: StorageStructureChangeIgnored
            status: "True"
            type: StorageStructureChangeIgnored
      

      Additional info:

      Attachments

        Activity

          People

            rdlugyhe Rolfe Dlugy-Hegwer
            rdlugyhe Rolfe Dlugy-Hegwer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: