Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1748

EO pod is not able to contact with ES svc and the IM cronjobs are failing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • Logging 5.0, Logging 5.2
    • Log Storage
    • None
    • False
    • False
    • NEW
    • NEW
    • Logging (LogExp) - Sprint 207, Logging (LogExp) - Sprint 208

    Description

      CLO and EO in latest "stable" (cluster-logging.5.2.0-57).

      The EO pod is not able to contact with the ES svc, but doing a curl, it's network reachable

      ~~~
      2021-09-14T09:01:44.555786100Z

      {"_ts":"2021-09-14T09:01:44.555606401Z","_level":"0","_component":"elasticsearch-operator","_message":"failed sending payload using bearer token","method":"GET","url":"_template/common.*,ocp-gen-*"}

      2021-09-14T09:01:44.636683272Z

      {"_ts":"2021-09-14T09:01:44.636538375Z","_level":"0","_component":"elasticsearch-operator","_message":"failed sending payload using mTLS PKI","method":"GET","url":"_template/common.*,ocp-gen-*"}

      2021-09-14T09:01:50.626170581Z

      {"_ts":"2021-09-14T09:01:50.626056237Z","_level":"0","_component":"elasticsearch-operator","_message":"failed sending payload using bearer token","method":"GET","url":"_template"}

      2021-09-14T09:01:50.685397005Z

      {"_ts":"2021-09-14T09:01:50.685214906Z","_level":"0","_component":"elasticsearch-operator","_message":"failed sending payload using mTLS PKI","method":"GET","url":"_template"}

      2021-09-14T09:01:50.685819631Z {"_ts":"2021-09-14T09:01:50.685747613Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to list existing templates in order to reconcile stale ones","_error":{"cluster":"elasticsearch","msg":"failed to get list of index templates","namespace":"openshift-logging","response_body":

      {"results":"Cannot authenticate user because admin user is not permitted to login via HTTP"}

      ,"response_error":null,"response_status":403}}
      ~~~

      The IM cronjobs are failing with error:

      ~~~
      $ oc logs elasticsearch-im-app-1631540700-g4tnv
      /cases/02973788/0140-must-gather-02973788-2021-09-13.tgz/must-gather-02973788-2021-09-13/quay-io-openshift-origin-cluster-logging-operator-sha256-69e49a1f5e36575e84225c50f89257aed1d60ccaac22288d1997de7faa3d18ae/namespaces/openshift-logging/pods/elasticsearch-im-app-1631540700-g4tnv/indexmanagement/indexmanagement/logs/current.log
      2021-09-13T13:45:13.506897742Z Index management delete process starting
      2021-09-13T13:45:13.674470205Z Error while attemping to determine the active write alias: {'error': {'root_cause': [

      {'type': 'security_exception', 'reason': 'no permissions for [indices:admin/aliases/get] and User [name=system:serviceaccount:openshift-logging:elasticsearch, roles=[admin_reader], requestedTenant=null]'}

      ], 'type': 'security_exception', 'reason': 'no permissions for [indices:admin/aliases/get] and User [name=system:serviceaccount:openshift-logging:elasticsearch, roles=[admin_reader], requestedTenant=null]'}, 'status': 403}
      2021-09-13T13:45:13.680466418Z Index management rollover process starting
      2021-09-13T13:45:14.503441129Z Error while attemping to determine the active write alias: {'error': {'root_cause': [

      {'type': 'security_exception', 'reason': 'no permissions for [indices:admin/aliases/get] and User [name=system:serviceaccount:openshift-logging:elasticsearch, roles=[admin_reader], requestedTenant=null]'}

      ], 'type': 'security_exception', 'reason': 'no permissions for [indices:admin/aliases/get] and User [name=system:serviceaccount:openshift-logging:elasticsearch, roles=[admin_reader], requestedTenant=null]'}, 'status': 403}
      ~~~

      A cluster-logging must-gather will be attached.

      Attachments

        Issue Links

          Activity

            People

              ikarpukh Igor Karpukhin (Inactive)
              rhn-support-ocasalsa Oscar Casal Sanchez
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: