Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2075

The elasticsearch-im-xxx job failed when trying to start index management process for a non-existent(empty-named) index [openshift-logging-5.0]

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Normal
    • Logging 5.0.11
    • Logging 5.0
    • Log Storage
    • None
    • False
    • False
    • NEW
    • NEW
    • Logging (LogExp) - Sprint 211

    Description

      Copy of BZ#1985826 for visibility:

      * OCP: 4.7.24
      * cluster-logging version 5.1.0-75
      * OpenShift Elasticsearch Operator 5.1.0-96
      
      2021-08-22T14:15:16.936670135Z Checking if infra-000225 is the write index for infra-write
      2021-08-22T14:15:16.988613680Z Done!
      2021-08-22T14:15:16.990845795Z ========================
      2021-08-22T14:15:16.990845795Z Index management rollover process starting for {"error":{"root_cause":[{"type":"security_exception","reason":"_opendistro_security_dls_query
      2021-08-22T14:15:16.990845795Z 
      2021-08-22T14:15:16.999455040Z Received an empty response from elasticsearch -- server may not be readyChecking the ES instances for messages caused by the request:
      
      % for pod in $(oc get pods -n openshift-logging|awk '/-cdm-/ {print $1}'); do echo $pod; oc logs pod/$pod -n openshift-logging -c elasticsearch |grep "2021-08-22T14:15:1";  done
      elasticsearch-cdm-u27cy1z5-1-75ffb9c455-k9b7p
      elasticsearch-cdm-u27cy1z5-2-6d4668f58-9s9dr
      elasticsearch-cdm-u27cy1z5-3-745687cf85-fdf9q
      2021-08-22T14:15:11.787176744Z [2021-08-22T14:15:11,669][WARN ][r.suppressed             ] [elasticsearch-cdm-u27cy1z5-3] path: /audit-write/_rollover, params: {pretty=, index=audit-write}
      2021-08-22T14:15:11.787176744Z org.elasticsearch.transport.RemoteTransportException: [elasticsearch-cdm-u27cy1z5-2][10.131.2.18:9300][indices:admin/rollover]
      2021-08-22T14:15:11.787176744Z Caused by: org.elasticsearch.ElasticsearchSecurityException: _opendistro_security_dls_query does not match (SG 900D)
      2021-08-22T14:15:11.787176744Z     at com.amazon.opendistroforelasticsearch.security.privileges.DlsFlsEvaluator.evaluate(DlsFlsEvaluator.java:128) ~[?:?]
      2021-08-22T14:15:11.787176744Z     at com.amazon.opendistroforelasticsearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:443) ~[?:?]
      2021-08-22T14:15:11.787176744Z     at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply0(OpenDistroSecurityFilter.java:287) ~[?:?]
      ...
      2021-08-22T14:15:11.787410641Z [2021-08-22T14:15:11,669][WARN ][r.suppressed             ] [elasticsearch-cdm-u27cy1z5-3] path: /_cat/aliases/infra-write, params: {h=alias, alias=infra-write}
      2021-08-22T14:15:11.787410641Z org.elasticsearch.transport.RemoteTransportException: [elasticsearch-cdm-u27cy1z5-2][10.131.2.18:9300][indices:admin/aliases/get]
      2021-08-22T14:15:11.787410641Z Caused by: org.elasticsearch.ElasticsearchSecurityException: _opendistro_security_dls_query does not match (SG 900D)
      2021-08-22T14:15:11.787410641Z     at com.amazon.opendistroforelasticsearch.security.privileges.DlsFlsEvaluator.evaluate(DlsFlsEvaluator.java:128) ~[?:?]
      2021-08-22T14:15:11.787410641Z     at com.amazon.opendistroforelasticsearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:443) ~[?:?]
      
      

      Attachments

        Issue Links

          Activity

            People

              gvanloo Gerard Vanloo (Inactive)
              gvanloo Gerard Vanloo (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: