Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2022

resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.1]

XMLWordPrintable

    • False
    • False
    • NEW
    • VERIFIED
    • Hide
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed.

      If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
      Show
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed. If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
    • Logging (LogExp) - Sprint 211, Logging (LogExp) - Sprint 212

      Logs from the eventrouter started to fail to get sent to elasticsearch.

      2021-11-22 18:30:28 +0000 [warn]: send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.event.metadata.resourceVersion] of type [integer] in document with id 'a080cf7b-0c09-4743-855d-a2e1f647bd53''" location=nil tag="kubernetes.var.log.containers.eventrouter-554c8f49f4-dlbzv_openshift-logging_kube-eventrouter-4a23b8779f6ede533e872b944940f8ef441d800270456b26ba984a59cf5aaf91.log" time=2021-11-22 18:10:21.537752890 +0000

      Looking at the processed event, you can see the resourceVersion is 2159599094 which is greater than a 32 bit int can hold.

      {"verb":"ADDED","event":{"metadata":{"name":"comms-api-25-build.16ba408e20da9db6","namespace":"tools","selfLink":"/api/v1/namespaces/tools/events/comms-api-25-build.16ba408e20da9db6","uid":"00b80141-b38b-4595-8ec3-95b766435583","resourceVersion":"2159599094","creationTimestamp":"2021-11-23T18:33:13Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-23T18:33:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:fieldPath":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"tools","name":"comms-api-25-build","uid":"49fb6671-0022-4a23-8f08-8d4ba29c46ee","apiVersion":"v1","resourceVersion":"2159598320","fieldPath":"spec.initContainers{manage-dockerfile}"},"reason":"Pulle...

              gvanloo Gerard Vanloo (Inactive)
              rhn-support-mrobson Matt Robson
              Qiaoling Tang Qiaoling Tang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: