Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2088

resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.3]

XMLWordPrintable

    • False
    • False
    • NEW
    • NEW
    • Hide
      Before this update, Elasticsearch rejected logs from the eventrouter due to a parsing error. This update changes the data model to resolve the parsing error. However, as a result, existing indices will cause an warning/error within Kibana. The field ("kubernetes.event.metadata.resourceVersion") will cause an error until the removal or reindexing of existing indices.
           
      If this field is not in use in Kibana, then you can ignore the error message. For those with a delete policy, that policy will eventually remove old indices. When that occurs, the message will no longer generate. Those without a delete policy in place will need to manually reindex in order to remove the message.
      Show
      Before this update, Elasticsearch rejected logs from the eventrouter due to a parsing error. This update changes the data model to resolve the parsing error. However, as a result, existing indices will cause an warning/error within Kibana. The field ("kubernetes.event.metadata.resourceVersion") will cause an error until the removal or reindexing of existing indices.       If this field is not in use in Kibana, then you can ignore the error message. For those with a delete policy, that policy will eventually remove old indices. When that occurs, the message will no longer generate. Those without a delete policy in place will need to manually reindex in order to remove the message.
    • Logging (LogExp) - Sprint 212

      Logs from the eventrouter started to fail to get sent to elasticsearch.

      2021-11-22 18:30:28 +0000 [warn]: send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.event.metadata.resourceVersion] of type [integer] in document with id 'a080cf7b-0c09-4743-855d-a2e1f647bd53''" location=nil tag="kubernetes.var.log.containers.eventrouter-554c8f49f4-dlbzv_openshift-logging_kube-eventrouter-4a23b8779f6ede533e872b944940f8ef441d800270456b26ba984a59cf5aaf91.log" time=2021-11-22 18:10:21.537752890 +0000

      Looking at the processed event, you can see the resourceVersion is 2159599094 which is greater than a 32 bit int can hold.

      {"verb":"ADDED","event":{"metadata":{"name":"comms-api-25-build.16ba408e20da9db6","namespace":"tools","selfLink":"/api/v1/namespaces/tools/events/comms-api-25-build.16ba408e20da9db6","uid":"00b80141-b38b-4595-8ec3-95b766435583","resourceVersion":"2159599094","creationTimestamp":"2021-11-23T18:33:13Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-23T18:33:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:fieldPath":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"tools","name":"comms-api-25-build","uid":"49fb6671-0022-4a23-8f08-8d4ba29c46ee","apiVersion":"v1","resourceVersion":"2159598320","fieldPath":"spec.initContainers{manage-dockerfile}"},"reason":"Pulle...

              gvanloo Gerard Vanloo (Inactive)
              gvanloo Gerard Vanloo (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: