Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1273

The index management job status is always `Completed` even when there has an error in the job log.

    XMLWordPrintable

Details

    • Logging (LogExp) - Sprint 204
    • Passed
    • NEW
    • VERIFIED
    • Hide
      Before this update, if an index management cronjob terminated with an error, it did not report the error exit code: instead, its job status was "complete." This update resolves the issue by reporting the error exit codes of index management cronjobs that terminate with errors.
      Show
      Before this update, if an index management cronjob terminated with an error, it did not report the error exit code: instead, its job status was "complete." This update resolves the issue by reporting the error exit codes of index management cronjobs that terminate with errors.

    Description

      Description of problem:

      The index management scripts fail, but the im job status is `Completed`.

      $ oc get pod
      NAME                                            READY   STATUS      RESTARTS   AGE
      cluster-logging-operator-7b96786d97-72kjl       1/1     Running     0          24m
      elasticsearch-cdm-zi2vxyc8-1-785cb8d59-8rjql    2/2     Running     0          22m
      elasticsearch-cdm-zi2vxyc8-2-7dbf55b468-26z2r   2/2     Running     0          22m
      elasticsearch-cdm-zi2vxyc8-3-7475c78549-67qdp   2/2     Running     0          22m
      elasticsearch-im-app-26980020-k2wxc             0/1     Completed   0          98s
      elasticsearch-im-audit-26980020-sjlsz           0/1     Completed   0          98s
      elasticsearch-im-infra-26980020-xgl2w           0/1     Completed   0          98s
      $ oc logs elasticsearch-im-infra-26980020-xgl2w
      Index management delete process startingNo indices to delete
      Index management rollover process starting
      Current write index for infra-write: infra-000001
      Checking results from _rollover call
      Calculating next write index based on current write index...
      Next write index for infra-write: infra-000002
      Checking if infra-000002 exists
      {
        "error" : {
          "root_cause" : [
            {
              "type" : "cluster_block_exception",
              "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
            }
          ],
          "type" : "cluster_block_exception",
          "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
        },
        "status" : 403
      }

      Version-Release number of selected component (if applicable):

      elasticsearch-operator.5.0.2-18

      How reproducible:

      Always

      Steps to Reproduce:
      1.  deploy CLO and EO
      2. set storage size to 1Gi when create clusterlogging/instance
      3. wait until the ES disk storage usage > 95%, monitor the status of index management jobs

      Actual results:

      Expected results:

      The index management job should fail when the scripts fail.

       

      Attachments

        Activity

          People

            ptsiraki@redhat.com Periklis Tsirakidis
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: