Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2089

resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.0]

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Hide
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed.

      If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
      Show
      The fix provided requires that the data model be changed. However, as a result, existing indices will cause an warning/error within Kibana. This field ("kubernetes.event.metadata.resourceVersion") will be broken until the existing indices are removed or reindexed. If this field is not in use in Kibana, then the message can be ignored. For those with a delete policy, the old indices will eventually be removed. When that occurs, the message will be removed. For those without that policy, the cluster will need to be reindexed in order to remove the message.
    • Logging (LogExp) - Sprint 212

    Description

      Logs from the eventrouter started to fail to get sent to elasticsearch.

      2021-11-22 18:30:28 +0000 [warn]: send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.event.metadata.resourceVersion] of type [integer] in document with id 'a080cf7b-0c09-4743-855d-a2e1f647bd53''" location=nil tag="kubernetes.var.log.containers.eventrouter-554c8f49f4-dlbzv_openshift-logging_kube-eventrouter-4a23b8779f6ede533e872b944940f8ef441d800270456b26ba984a59cf5aaf91.log" time=2021-11-22 18:10:21.537752890 +0000

      Looking at the processed event, you can see the resourceVersion is 2159599094 which is greater than a 32 bit int can hold.

      {"verb":"ADDED","event":{"metadata":{"name":"comms-api-25-build.16ba408e20da9db6","namespace":"tools","selfLink":"/api/v1/namespaces/tools/events/comms-api-25-build.16ba408e20da9db6","uid":"00b80141-b38b-4595-8ec3-95b766435583","resourceVersion":"2159599094","creationTimestamp":"2021-11-23T18:33:13Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-23T18:33:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:fieldPath":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"tools","name":"comms-api-25-build","uid":"49fb6671-0022-4a23-8f08-8d4ba29c46ee","apiVersion":"v1","resourceVersion":"2159598320","fieldPath":"spec.initContainers{manage-dockerfile}"},"reason":"Pulle...

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. LOG-2022 resourceVersion is overflowing type Integer causing ES rejection [openshift-logging 5.1]

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub openshift/origin-aggregated-logging#2225: Updating viaq data models

          Activity

            People

              gvanloo Gerard Vanloo (Inactive)
              gvanloo Gerard Vanloo (Inactive)
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: