Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4949

Cronjob elasticsearch-im-<type> fails every time when <type> logs are not captured


    • False
    • None
    • False
    • NEW
    • NEW
    • Before this update, the elasticsearch-im-<type>-* pods failed when no <type> logs were collected, where <type> is audit, infra, or app. After this update, no pods are failing when <type> logs are not collected.
    • Bug Fix
    • Log Storage - Sprint 248, Log Storage - Sprint 249, Log Storage - Sprint 250, Log Storage - Sprint 251, Log Storage - Sprint 252
    • Important

      Description of problem:

      Version-Release number of selected component (if applicable):

      OpenShift Elasticsearch Operator 5.8.1

      How reproducible:


      Steps to Reproduce:

      1. Install latest version of Logging 5.8 and Elasticsearch 5.8.1. It captures infra and application loga by default.
      2.  Check logs of elasticsearch-im-audit-xxxx-xxxx pod triggered by its related cronjob.


      $ oc logs elasticsearch-im-audit-28414935-bt95h
      Index management delete process starting for Received
      Received an empty response from elasticsearch -- server may not be ready 

      Rest of the job completes fine:

      elasticsearch-im-app-28414935-pw55v            0/1     Completed   0          2m24s
      elasticsearch-im-audit-28414935-bt95h          0/1     Error       0          2m24s
      elasticsearch-im-infra-28414935-lq5m9          0/1     Completed   0          2m24s 


      Actual results:

      The job elasticsearch-im-audit fails every time when audit logs are not being captured elasticsearch.

      Expected results:

      All the jobs should complete fine without any problems when elasticsearch is healthy.

      Additional info:

      Ran tests on 5.8.0 version as well but the jobs works fine there. The issue starts from 5.8.1 version of OpenShift Elasticsearch Operator.

      In previous versions, there used be to be a blank index "audit-0000xx" even when audit logs are not being captured. However, this index is missing from 5.8.1 version.

      When this index is created manually, the job starts to complete just fine.

      $ oc rsh elasticsearch-cdm-ticteo7u-1-86d5cb957-qjbp2
      Defaulted container "elasticsearch" out of: elasticsearch, proxy
      sh-5.1$ es_util --query=audit-000001 -XPUT
      sh-5.1$ exit
      ---- After some time ----
      $ oc get pods | grep -i elasticsearch-im
      elasticsearch-im-app-28414935-hwqp8             0/1     Completed   0          5m4s
      elasticsearch-im-audit-28414935-l7qm2           0/1     Completed   0          5m4s
      elasticsearch-im-infra-28414935-zk4p5           0/1     Completed   0          5m4s


            btaani@redhat.com Bayan Taani
            rhn-support-dgautam Dhruv Gautam
            Qiaoling Tang Qiaoling Tang
            0 Vote for this issue
            6 Start watching this issue