Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2089

resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.0]

XMLWordPrintable

    • False
    • False
    • NEW
    • NEW
    • Hide
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed.

      If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
      Show
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed. If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
    • Logging (LogExp) - Sprint 212

      Logs from the eventrouter started to fail to get sent to elasticsearch.

      2021-11-22 18:30:28 +0000 [warn]: send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.event.metadata.resourceVersion] of type [integer] in document with id 'a080cf7b-0c09-4743-855d-a2e1f647bd53''" location=nil tag="kubernetes.var.log.containers.eventrouter-554c8f49f4-dlbzv_openshift-logging_kube-eventrouter-4a23b8779f6ede533e872b944940f8ef441d800270456b26ba984a59cf5aaf91.log" time=2021-11-22 18:10:21.537752890 +0000

      Looking at the processed event, you can see the resourceVersion is 2159599094 which is greater than a 32 bit int can hold.

      {"verb":"ADDED","event":{"metadata":{"name":"comms-api-25-build.16ba408e20da9db6","namespace":"tools","selfLink":"/api/v1/namespaces/tools/events/comms-api-25-build.16ba408e20da9db6","uid":"00b80141-b38b-4595-8ec3-95b766435583","resourceVersion":"2159599094","creationTimestamp":"2021-11-23T18:33:13Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-23T18:33:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:fieldPath":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"tools","name":"comms-api-25-build","uid":"49fb6671-0022-4a23-8f08-8d4ba29c46ee","apiVersion":"v1","resourceVersion":"2159598320","fieldPath":"spec.initContainers{manage-dockerfile}"},"reason":"Pulle...

              gvanloo Gerard Vanloo (Inactive)
              gvanloo Gerard Vanloo (Inactive)
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: