Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2088

resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.3]

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Hide
      Before this update, Elasticsearch rejected logs from the eventrouter due to a parsing error. This update changes the data model to resolve the parsing error. However, as a result, existing indices will cause an warning/error within Kibana. The field ("kubernetes.event.metadata.resourceVersion") will cause an error until the removal or reindexing of existing indices.
           
      If this field is not in use in Kibana, then you can ignore the error message. For those with a delete policy, that policy will eventually remove old indices. When that occurs, the message will no longer generate. Those without a delete policy in place will need to manually reindex in order to remove the message.
      Show
      Before this update, Elasticsearch rejected logs from the eventrouter due to a parsing error. This update changes the data model to resolve the parsing error. However, as a result, existing indices will cause an warning/error within Kibana. The field ("kubernetes.event.metadata.resourceVersion") will cause an error until the removal or reindexing of existing indices.       If this field is not in use in Kibana, then you can ignore the error message. For those with a delete policy, that policy will eventually remove old indices. When that occurs, the message will no longer generate. Those without a delete policy in place will need to manually reindex in order to remove the message.
    • Logging (LogExp) - Sprint 212

    Description

      Logs from the eventrouter started to fail to get sent to elasticsearch.

      2021-11-22 18:30:28 +0000 [warn]: send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.event.metadata.resourceVersion] of type [integer] in document with id 'a080cf7b-0c09-4743-855d-a2e1f647bd53''" location=nil tag="kubernetes.var.log.containers.eventrouter-554c8f49f4-dlbzv_openshift-logging_kube-eventrouter-4a23b8779f6ede533e872b944940f8ef441d800270456b26ba984a59cf5aaf91.log" time=2021-11-22 18:10:21.537752890 +0000

      Looking at the processed event, you can see the resourceVersion is 2159599094 which is greater than a 32 bit int can hold.

      {"verb":"ADDED","event":{"metadata":{"name":"comms-api-25-build.16ba408e20da9db6","namespace":"tools","selfLink":"/api/v1/namespaces/tools/events/comms-api-25-build.16ba408e20da9db6","uid":"00b80141-b38b-4595-8ec3-95b766435583","resourceVersion":"2159599094","creationTimestamp":"2021-11-23T18:33:13Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-23T18:33:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:fieldPath":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"tools","name":"comms-api-25-build","uid":"49fb6671-0022-4a23-8f08-8d4ba29c46ee","apiVersion":"v1","resourceVersion":"2159598320","fieldPath":"spec.initContainers{manage-dockerfile}"},"reason":"Pulle...

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. LOG-2022 resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.1]

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub openshift/origin-aggregated-logging#2224: Log 2022: Updating resourceVersion attribute to a keyword

          Activity

            People

              gvanloo Gerard Vanloo (Inactive)
              gvanloo Gerard Vanloo (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: