Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3063

Index management cronjobs are rolling over indices when parsing JSON is enabled and not expected

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Before this update, each rollover job created empty indices when the `ClusterLogForwarder` custom resource had JSON parsing defined. With this update, new indices are not empty
    • High
    • Hide

      1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator
      2) Create ClusterLogging Custom Resource (CR) instance.
      3) Create ClusterLogForwarder (CLF) instance enabling parsing json logs.

       

      We did different tests, and also we reproduced the issue that a client has.

       

      Test 1: 

      ClusterLogForwarder instance:

      spec:
  pipelines:
  - inputRefs:
    - application
    name: elasticsearch-application
    outputRefs:
    - default
    parse: json  

      Initial indices:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6 -- es_util --query=_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   app-000001   fUYMsPkISHa4g_zAAiiVaw   1   0         23            0     94.5kb         94.5kb
green  open   infra-000001 Dz8Y5zUIRNeJ2c2TufPvhA   1   0          0            0       261b           261b
green  open   .security    rSD6nIALS4medtfKUWFxzw   1   0          6            0     33.2kb         33.2kb
green  open   audit-000001 uN6_cLhaR8euOIW6P8C9IQ   1   0          0            0       261b           261b
green  open   .kibana_1    Fle3m71ER_SzqG35JTZbAQ   1   0          0            0       261b           261b

      After a while:

      $ oc get pods -n openshift-logging
NAME                                           READY   STATUS      RESTARTS   AGE
cluster-logging-operator-5c86f6bf9-sd95n       1/1     Running     0          4d
collector-4drgt                                2/2     Running     0          98m
collector-4t25l                                2/2     Running     0          96m
collector-chsz5                                2/2     Running     0          97m
collector-lrn9h                                2/2     Running     0          96m
collector-s8lht                                2/2     Running     0          97m
collector-t6p6q                                2/2     Running     0          96m
elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6   2/2     Running     0          98m
elasticsearch-im-app-27627225-jjnh4            0/1     Completed   0          8m15s
elasticsearch-im-audit-27627225-gzdzb          0/1     Completed   0          8m16s
elasticsearch-im-infra-27627225-bztt5          0/1     Completed   0          8m16s
kibana-6bf9fbc84d-d6h4v                        2/2     Running     0          98m
kibana-6bf9fbc84d-sx578                        2/2     Running     0          98m

      New empty indices are created:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6 -- es_util --query=_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   app-000002   uSIUbUomQBWqCenrLqBPcw   1   0          0            0       261b           261b
green  open   infra-000001 Dz8Y5zUIRNeJ2c2TufPvhA   1   0          0            0       261b           261b
green  open   infra-000002 Zn5epx0mSLCDSkJbtgtpuQ   1   0          0            0       261b           261b
green  open   audit-000002 JWuwEJ8AQ2KTaJgVDVBlxQ   1   0          0            0       261b           261b
green  open   audit-000001 uN6_cLhaR8euOIW6P8C9IQ   1   0          0            0       261b           261b
green  open   .kibana_1    Fle3m71ER_SzqG35JTZbAQ   1   0          0            0       261b           261b
green  open   app-000001   fUYMsPkISHa4g_zAAiiVaw   1   0         23            0     94.5kb         94.5kb
green  open   .security    rSD6nIALS4medtfKUWFxzw   1   0          6            0     33.2kb         33.2kb

       

       

      Test 2:

      ClusterLogForwarder instance (deleting "parse: json" and adding "outputDefaults":

      spec:
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.logFormat
      structuredTypeName: json
  pipelines:
  - inputRefs:
    - application
    name: elasticsearch-application
    outputRefs:
    - default

      Same behavior.

      Initial indices:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2 -- es_util --query=_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   app-000001   5oA5aoOzQuuTZod6AyP6cw   1   0        119            0    215.7kb        215.7kb
green  open   .security    PP8U8rGTTV-H4IJUsltsmg   1   0          6            0     33.2kb         33.2kb
green  open   infra-000001 mdiyyrYCQUmCuUWF4f_MCA   1   0          0            0       261b           261b
green  open   .kibana_1    m1LJ3pWFQxeW56bhdCCzGQ   1   0          0            0       261b           261b
green  open   audit-000001 TENxomASTPWZT3-UCbhsTg   1   0          0            0       261b           261b

      After a while:

      $ oc get pods -n openshift-logging
NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-5c86f6bf9-sd95n        1/1     Running     0          3d22h
collector-26fng                                 2/2     Running     0          71m
collector-cxvlm                                 2/2     Running     0          71m
collector-jwr2g                                 2/2     Running     0          70m
collector-lsdwg                                 2/2     Running     0          70m
collector-nww96                                 2/2     Running     0          71m
collector-zbjfg                                 2/2     Running     0          70m
elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2   2/2     Running     0          72m
elasticsearch-im-app-27627120-vdhwt             0/1     Completed   0          9m8s
elasticsearch-im-audit-27627120-nx6zv           0/1     Completed   0          9m8s
elasticsearch-im-infra-27627120-ftnfb           0/1     Completed   0          9m8s
kibana-648778968-92cf5                          2/2     Running     0          72m
kibana-648778968-vb64j                          2/2     Running     0          72m

      New empty indices are created:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2 -- es_util --query=_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   app-000001   5oA5aoOzQuuTZod6AyP6cw   1   0        119            0    216.3kb        216.3kb
green  open   .security    PP8U8rGTTV-H4IJUsltsmg   1   0          6            0     33.2kb         33.2kb
green  open   infra-000001 mdiyyrYCQUmCuUWF4f_MCA   1   0          0            0       261b           261b
green  open   .kibana_1    m1LJ3pWFQxeW56bhdCCzGQ   1   0          0            0       261b           261b
green  open   app-000002   r1qc1skYSPGlP-noNiPjqA   1   0          0            0       261b           261b
green  open   audit-000002 EbUGAFv5TMKqRhu2GY8C-Q   1   0          0            0       261b           261b
green  open   infra-000002 JBvjHyTkSse0C9alXEsmgA   1   0          0            0       261b           261b
green  open   audit-000001 TENxomASTPWZT3-UCbhsTg   1   0          0            0       261b           261b

      If you wait more time, indices continue to be created.

       

      Reproduction of the client's case:

      1. Creating a CLF instance with this configuration:

      spec:
    outputDefaults:
        elasticsearch:
            structuredTypeKey: kubernetes.namespace_name
            structuredTypeName: mo-app-index

      2. After that, we deployed an application in "adri" namespace sending json logs and we can see that a new index "app-adri-XXX" is created in ES:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-ba8fs4vh-1-554c975987-d9g86 -- es_util --query=_cat/indices?v
health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   infra-000008    QckoCZuvSpKL8_B5zLtDCw   1   0    1813534            0      1.2gb          1.2gb
green  open   infra-000007    w6cbUtORScuq2Y69q-ba7Q   1   0    1853346            0      1.2gb          1.2gb
green  open   app-000003      r126x_fvRuSwEHg7Un20vg   1   0          0            0       261b           261b
green  open   app-000005      5ltz4g1mS7ip7xq_BuB5Rw   1   0          0            0       261b           261b
green  open   app-adri-000001 RU-dyoXFRB64lGsJdwbozg   1   0         37            0    144.3kb        144.3kb

      3. Once we can see this index, we changed the CLF instance configuration to:

      spec:
    outputDefaults:
        elasticsearch:
            structuredTypeKey: kubernetes.labels.logFormat
                        structuredTypeName: json

      4. After applying the new changes, a new index is created, and another empty new index  for "app-adri-XXX".

      oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-ba8fs4vh-1-554c975987-d9g86 -- es_util --query=_cat/indices?v
health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
...
green  open   app-adri-000001 RU-dyoXFRB64lGsJdwbozg   1   0         37            0    144.7kb        144.7kb
green  open   app-json-000001 2PmXBCcjTreQe5S4wqSd8Q   1   0         21            0    100.6kb        100.6kb
green  open   app-adri-000002 EWTCtRzFS0Kklw6Qm701kg   1   0          0            0       261b           261b
...

       

      Show
      1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator 2) Create ClusterLogging Custom Resource (CR) instance. 3) Create ClusterLogForwarder (CLF) instance enabling parsing json logs.   We did different tests, and also we reproduced the issue that a client has.   Test 1:  ClusterLogForwarder instance: spec:   pipelines:   - inputRefs:     - application     name: elasticsearch-application     outputRefs:     - default   parse: json   Initial indices: $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6 -- es_util --query=_cat/indices?v health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   app-000001   fUYMsPkISHa4g_zAAiiVaw   1   0         23            0     94.5kb         94.5kb green  open   infra-000001 Dz8Y5zUIRNeJ2c2TufPvhA   1   0          0            0       261b           261b green  open   .security    rSD6nIALS4medtfKUWFxzw   1   0          6            0     33.2kb         33.2kb green  open   audit-000001 uN6_cLhaR8euOIW6P8C9IQ   1   0          0            0       261b           261b green  open   .kibana_1    Fle3m71ER_SzqG35JTZbAQ   1   0          0            0       261b           261b After a while: $ oc get pods -n openshift-logging NAME                                           READY   STATUS      RESTARTS   AGE cluster-logging-operator-5c86f6bf9-sd95n       1/1     Running     0          4d collector-4drgt                                2/2     Running     0          98m collector-4t25l                                2/2     Running     0          96m collector-chsz5                                2/2     Running     0          97m collector-lrn9h                                2/2     Running     0          96m collector-s8lht                                2/2     Running     0          97m collector-t6p6q                                2/2     Running     0          96m elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6   2/2     Running     0          98m elasticsearch-im-app-27627225-jjnh4            0/1     Completed   0          8m15s elasticsearch-im-audit-27627225-gzdzb          0/1     Completed   0          8m16s elasticsearch-im-infra-27627225-bztt5          0/1     Completed   0          8m16s kibana-6bf9fbc84d-d6h4v                        2/2     Running     0          98m kibana-6bf9fbc84d-sx578                        2/2     Running     0          98m New empty indices are created: $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-7ybko9li-1-c8565bf79-49kx6 -- es_util --query=_cat/indices?v health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   app-000002   uSIUbUomQBWqCenrLqBPcw   1   0          0            0       261b           261b green  open   infra-000001 Dz8Y5zUIRNeJ2c2TufPvhA   1   0          0            0       261b           261b green  open   infra-000002 Zn5epx0mSLCDSkJbtgtpuQ   1   0          0            0       261b           261b green  open   audit-000002 JWuwEJ8AQ2KTaJgVDVBlxQ   1   0          0            0       261b           261b green  open   audit-000001 uN6_cLhaR8euOIW6P8C9IQ   1   0          0            0       261b           261b green  open   .kibana_1    Fle3m71ER_SzqG35JTZbAQ   1   0          0            0       261b           261b green  open   app-000001   fUYMsPkISHa4g_zAAiiVaw   1   0         23            0     94.5kb         94.5kb green  open   .security    rSD6nIALS4medtfKUWFxzw   1   0          6            0     33.2kb         33.2kb     Test 2: ClusterLogForwarder instance (deleting "parse: json" and adding "outputDefaults": spec:   outputDefaults:     elasticsearch:       structuredTypeKey: kubernetes.labels.logFormat       structuredTypeName: json   pipelines:   - inputRefs:     - application     name: elasticsearch-application     outputRefs:   - default Same behavior. Initial indices: $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2 -- es_util --query=_cat/indices?v health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   app-000001   5oA5aoOzQuuTZod6AyP6cw   1   0        119            0    215.7kb        215.7kb green  open   .security    PP8U8rGTTV-H4IJUsltsmg   1   0          6            0     33.2kb         33.2kb green  open   infra-000001 mdiyyrYCQUmCuUWF4f_MCA   1   0          0            0       261b           261b green  open   .kibana_1    m1LJ3pWFQxeW56bhdCCzGQ   1   0          0            0       261b           261b green  open   audit-000001 TENxomASTPWZT3-UCbhsTg   1   0          0            0       261b           261b After a while: $ oc get pods -n openshift-logging NAME                                            READY   STATUS      RESTARTS   AGE cluster-logging-operator-5c86f6bf9-sd95n        1/1     Running     0          3d22h collector-26fng                                 2/2     Running     0          71m collector-cxvlm                                 2/2     Running     0          71m collector-jwr2g                                 2/2     Running     0          70m collector-lsdwg                                 2/2     Running     0          70m collector-nww96                                 2/2     Running     0          71m collector-zbjfg                                 2/2     Running     0          70m elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2   2/2     Running     0          72m elasticsearch-im-app-27627120-vdhwt             0/1     Completed   0          9m8s elasticsearch-im-audit-27627120-nx6zv           0/1     Completed   0          9m8s elasticsearch-im-infra-27627120-ftnfb           0/1     Completed   0          9m8s kibana-648778968-92cf5                          2/2     Running     0          72m kibana-648778968-vb64j                          2/2     Running     0          72m New empty indices are created: $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-yezhl2vc-1-5c5cb65765-kg4k2 -- es_util --query=_cat/indices?v health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   app-000001   5oA5aoOzQuuTZod6AyP6cw   1   0        119            0    216.3kb        216.3kb green  open   .security    PP8U8rGTTV-H4IJUsltsmg   1   0          6            0     33.2kb         33.2kb green  open   infra-000001 mdiyyrYCQUmCuUWF4f_MCA   1   0          0            0       261b           261b green  open   .kibana_1    m1LJ3pWFQxeW56bhdCCzGQ   1   0          0            0       261b           261b green  open   app-000002   r1qc1skYSPGlP-noNiPjqA   1   0          0            0       261b           261b green  open   audit-000002 EbUGAFv5TMKqRhu2GY8C-Q   1   0          0            0       261b           261b green  open   infra-000002 JBvjHyTkSse0C9alXEsmgA   1   0          0            0       261b           261b green  open   audit-000001 TENxomASTPWZT3-UCbhsTg   1   0          0            0       261b           261b If you wait more time, indices continue to be created.   Reproduction of the client's case: 1. Creating a CLF instance with this configuration: spec:     outputDefaults:         elasticsearch:             structuredTypeKey: kubernetes.namespace_name             structuredTypeName: mo-app-index 2. After that, we deployed an application in "adri" namespace sending json logs and we can see that a new index "app-adri-XXX" is created in ES: $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-ba8fs4vh-1-554c975987-d9g86 -- es_util --query=_cat/indices?v health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   infra-000008    QckoCZuvSpKL8_B5zLtDCw   1   0    1813534            0      1.2gb          1.2gb green  open   infra-000007    w6cbUtORScuq2Y69q-ba7Q   1   0    1853346            0      1.2gb          1.2gb green  open   app-000003      r126x_fvRuSwEHg7Un20vg   1   0          0            0       261b           261b green  open   app-000005      5ltz4g1mS7ip7xq_BuB5Rw   1   0          0            0       261b           261b green  open   app-adri-000001 RU-dyoXFRB64lGsJdwbozg   1   0         37            0    144.3kb        144.3kb 3. Once we can see this index, we changed the CLF instance configuration to: spec:     outputDefaults:         elasticsearch:             structuredTypeKey: kubernetes.labels.logFormat                       structuredTypeName: json 4. After applying the new changes, a new index is created, and another empty new index  for "app-adri-XXX". oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-ba8fs4vh-1-554c975987-d9g86 -- es_util --query=_cat/indices?v health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size ... green  open   app-adri-000001 RU-dyoXFRB64lGsJdwbozg   1   0         37            0    144.7kb        144.7kb green  open   app-json-000001 2PmXBCcjTreQe5S4wqSd8Q   1   0         21            0    100.6kb        100.6kb green  open   app-adri-000002 EWTCtRzFS0Kklw6Qm701kg   1   0          0            0       261b           261b ...  
    • Log Storage - Sprint 222, Log Storage - Sprint 223, Log Storage - Sprint 224

    Description

      -OCP version 4.10
      -RHOL VERSION 5.4( cluster-logging.5.4.2 and elasticsearch-operator.5.4.2)

       

      After enabling parsing JSON logs functionality in ClusterLogForwarder (CLF) instance, new empty indices (app, infra, and audit) are created without fulfilling the following characteristics:

      -The index is older than the rollover.maxAge value in the Elasticsearch CR.
      -The index size is greater than 40 GB × the number of primary shards.
      -The index doc count is greater than 40960 KB × the number of primary shards.

       

      These new indices created are empty but they are created without complying with the characteristics that the cronjobs have to do the rollover. After a time we found a lot of empty indexes.

      This could give performance problems since the indices use metadata.

      Reproducing the environment with RHOL 5.3 I am not able to see the current issue.

       

       

       

       

       

       

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. LOG-2823 Index management cronjobs are rolling over indices when parsing JSON is enabled and not expected

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link [KCS] Unexpected rolling over indices in RHOCP 4

          GitHub openshift/cluster-logging-operator#1691: [release-5.4] LOG-3063: Detect correct prometheus bind address

          Activity

            People

              btaani@redhat.com Bayan Taani
              acandelp Adrian Candel
              Qiaoling Tang Qiaoling Tang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: